Obstacle avoidance advancing method and device of unmanned aerial vehicle, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an obstacle avoidance advancing method and device of an unmanned aerial vehicle, electronic equipment and a storage medium. The method comprises the following steps: determining target terminal point information of the unmanned aerial vehicle, and determining a current advancing front image of the unmanned aerial vehicle advancing at the current moment; based on the target end point information and the current advancing front image, determining advancing control parameter information of the unmanned aerial vehicle at the next moment through an obstacle avoidance model, and guiding the obstacle avoidance model to screen out image information having preset correlation with the target end point information from the current advancing front image through the target end point information, the prediction module is used for predicting advancing control parameters of the unmanned aerial vehicle for obstacle avoidance advancing towards a target end point position at the next moment; and controlling the unmanned aerial vehicle to perform obstacle avoidance advancing at the next moment based on the advancing control parameter information. According to the scheme provided by the invention, the obstacle-avoiding advancing can be quickly carried out under the scene of multi-scene and multi-size obstacles, so that efficient autonomous navigation can be realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of path planning technology, and in particular to an obstacle avoidance method, device, electronic device and storage medium for an unmanned aerial vehicle. Background Art

[0002] Drones are being used more and more widely in various fields, especially in logistics and distribution, agricultural plant protection, surveying and exploration, emergency rescue, etc. In the drone technology system, navigation and obstacle avoidance technology is the key core technology to ensure the safe and efficient operation of drones.

[0003] At present, the obstacle avoidance capability of UAVs in complex flight environments faces severe challenges. Traditional binocular parallax models and obstacle avoidance algorithms have obvious defects in dealing with large-scale and complex obstacles, and focus on local path planning and short-range obstacle avoidance. It is difficult to respond promptly and accurately to large and long-distance obstacles, which seriously threatens the flight safety of UAVs. In addition, the traditional UAV obstacle avoidance algorithm consists of multiple modules such as perception, path planning, and trajectory optimization. This complex structure not only increases the computational burden of the system, but also leads to poor coordination efficiency of each module. Especially in practical applications, its performance cannot meet the growing demand for UAV application scenarios, such as low-altitude logistics distribution in complex urban environments, and precise obstacle avoidance in emergency rescue scenarios. Therefore, it is urgent to develop a new type of UAV obstacle avoidance solution to break through the bottleneck of traditional technology and improve the obstacle avoidance capability and autonomous flight level of UAVs in complex environments. Summary of the invention

[0004] The embodiments of the present invention provide an obstacle avoidance method, device, electronic device and storage medium for a drone, so as to realize rapid obstacle avoidance in scenes with obstacles of multiple scenes and multiple sizes, thereby enabling efficient autonomous navigation.

[0005] In a first aspect, an embodiment of the present invention provides an obstacle avoidance method for a drone, the method comprising:

[0006] Determine the target destination information of the drone, and determine the current forward image of the drone when it is moving at the current moment;

[0007] Based on the target endpoint information and the current image ahead, the obstacle avoidance model is used to determine the travel control parameter information of the drone at the next moment. The target endpoint information guides the obstacle avoidance model to screen out image information that has a preset correlation with the target endpoint information from the current image ahead, so as to predict the travel control parameters for the drone to avoid obstacles when traveling toward the target endpoint position at the next moment. The obstacle avoidance model is a model that enables the drone to avoid obstacles obtained through reinforcement learning training. The target endpoint information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the drone's obstacle avoidance at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the drone's travel direction that needs to be paid attention to when predicting the drone's obstacle avoidance at the current moment. Each current image block is generated by dividing the current image ahead.

[0008] Based on the travel control parameter information, the drone is controlled to avoid obstacles at the next moment.

[0009] In a second aspect, an embodiment of the present invention further provides an obstacle avoidance device for a drone, the device comprising:

[0010] The first determination module is used to determine the target end point information of the drone and determine the current forward image of the drone when it is moving at the current moment;

[0011] a second determination module, for determining the travel control parameter information of the UAV at the next moment through the obstacle avoidance model based on the target endpoint information and the current image ahead, the target endpoint information guiding the obstacle avoidance model to screen out image information having a preset correlation with the target endpoint information from the current image ahead, for predicting the travel control parameters for the UAV to avoid obstacles when traveling toward the target endpoint position at the next moment, the obstacle avoidance model is a model obtained through reinforcement learning training that enables the UAV to avoid obstacles, the target endpoint information is used to determine the first vector information and the second vector information, the first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the obstacle avoidance of the UAV at the current moment, the second vector information is used to guide the obstacle avoidance model to identify the direction of the UAV that needs to be paid attention to when predicting the obstacle avoidance of the UAV at the current moment, and each current image block is generated by dividing the current image ahead;

[0012] A control module is used to control the UAV to avoid obstacles at the next moment based on the travel control parameter information.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the obstacle avoidance method for a drone described in any one of the above embodiments.

[0017] In a fourth aspect, a computer-readable medium is further provided in an embodiment of the present invention, wherein the computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement the obstacle avoidance method for a drone as described in any one of the above embodiments when executed.

[0018] In the technical solution of the embodiment of the present invention, the target end point information clarifies the mission direction of the drone and provides target guidance for subsequent path planning. The current image ahead will provide real-time feedback on the environmental conditions of the drone. The target end point information guides the obstacle avoidance model to select key image information closely related to the obstacle avoidance movement of the drone toward the target end point based on the target end point information. The target end point information runs through the entire obstacle avoidance movement process of the drone so that the obstacle avoidance model can always give priority to image information that is valuable for obstacle avoidance movement toward the end point, and then accurately predict the movement control parameters to ensure that the drone can avoid obstacles and move toward the target in a complex environment. At the same time, the current image ahead provides rich environmental information, and no matter the size of the obstacle or the scene it is in, it can be captured by the image. When the obstacle avoidance model performs obstacle avoidance movement based on the image information ahead, it will not be restricted by the scene and size differences of the obstacle, so that the drone can autonomously adapt to various flight conditions without human intervention, and continuously adjust the flight speed, direction and other movement control parameters to flexibly make obstacle avoidance decisions.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1It is a flow chart of an obstacle avoidance method for a drone provided by an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a scenario in which an unmanned aerial vehicle performs obstacle avoidance flight during an obstacle avoidance operation provided by an embodiment of the present invention;

[0023] Figure 3 It is a flow chart of another obstacle avoidance method for a drone provided by an embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of the algorithm structure of an obstacle avoidance model used in the obstacle avoidance process of a drone provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of an obstacle avoidance traveling device for a UAV provided by an embodiment of the present invention;

[0026] Figure 6 It is a structural schematic diagram of an electronic device for implementing an obstacle avoidance traveling method for a drone provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0028] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0033] Figure 1 A flow chart of an obstacle avoidance method for a drone provided in an embodiment of the present invention is applicable to situations where a drone performs obstacle avoidance in a limited space, especially situations where a drone performs obstacle avoidance when performing inspection and shooting in areas such as substations. The obstacle avoidance method for the drone can be executed by an obstacle avoidance device for the drone, which can be implemented in the form of software and / or hardware and is generally integrated on any electronic device with network communication function, which can be a mobile terminal, a PC or a server, etc.

[0034] like Figure 1 As shown, the obstacle avoidance method of the drone according to the embodiment of the present invention may include the following process:

[0035] S110: Determine the target destination information of the drone, and determine the current forward image of the drone when it is moving at the current moment.

[0036] The drone in this solution can be configured to be used in the target inspection area to realize drone obstacle avoidance during drone inspection. The target inspection area can be a substation area, etc. The target destination information is used to indicate the target destination position that the drone is going to reach. And / or, the target destination information can refer to the destination-related information that the drone needs to reach, including location coordinates, destination environmental characteristics, etc. The target destination information used when the drone is traveling can be determined by using GPS positioning, map matching, preset instructions, etc. The target destination information can be expressed in the form of location coordinates and / or image screen features at the destination position.

[0037] The current forward image can be acquired in real time through the image acquisition device carried by the drone, and the presented environment picture of the drone in the current direction of travel and in front of the drone covers the image picture information such as the obstacle distribution information, terrain features, and lighting conditions in the current direction of travel and in front of the drone. Among them, the important task of the drone in the process of moving is to avoid obstacles and reach the target location during the process of moving forward. Therefore, by giving priority to the front of the drone and acquiring the image of the front of the drone, it is possible to respond to the obstacles in front in a timely manner and make reasonable obstacle avoidance actions.

[0038] The obstacle distribution information ahead of the drone may include various objects ahead of the drone that may hinder the flight of the drone, such as buildings, trees, wires, etc. Specifically, the shape, size, position and relative distance of the obstacles covered in the ahead image can be identified through image analysis. The environmental features ahead of the drone may include the topography and lighting conditions ahead of the drone, such as whether it is a bright sunny day or a dim cloudy day. The lighting conditions will affect the visual effect of the image and the accuracy of the subsequent image recognition algorithm. The ahead image is obtained in real time during the flight of the drone, which can reflect the latest status of the drone's surrounding environment at the current moment, so that the drone can adjust its flight strategy in time according to the latest environmental information.

[0039] As an optional but non-limiting implementation scheme, determining the current forward image of the UAV when it is moving at the current moment includes the following steps:

[0040] During the flight of the drone, the fisheye camera on the drone is used to capture images in the direction in front of the drone at the current moment, so as to obtain the current forward image of the drone when it is moving at the current moment.

[0041] The image ahead is usually captured by an image acquisition device installed at the front of the drone. Image acquisition devices include ordinary cameras, fisheye cameras, etc. The image acquisition device is pre-set at a specific position and angle of the drone to ensure that the scene directly in front of the drone's direction of travel can be clearly captured. The drone will trigger the camera to shoot at a preset frequency or under specific conditions. For example, it will take an image at a certain interval, or start shooting immediately when it detects that there may be an obstacle ahead.

[0042] Optionally, fisheye cameras have unique optical properties, and their lenses are designed to provide a very wide viewing angle, usually up to 180° or even more than 180°. This ultra-wide-angle field of view is essential for drones to fully perceive the environment ahead during flight. Compared to the narrow viewing angle range of ordinary cameras, fisheye cameras can capture a wider range of scenes at one time, reducing visual blind spots, allowing drones to obtain richer information about the environment ahead in advance, including distant obstacles, terrain changes, and other factors that may affect the flight path.

[0043] During the flight of a drone, environmental information is constantly changing. Only real-time images can accurately reflect the environmental conditions in front of the drone's current location, which is crucial for the drone to make timely decisions. In other words, only based on the drone's latest environmental information in front of it can the drone accurately judge whether there are obstacles in front, as well as the real-time location and status of the obstacles, so as to accurately plan the flight path at the next moment and ensure flight safety.

[0044] Optionally, during the flight of the drone, the fisheye camera can be triggered to shoot in front of the drone according to a preset time interval, flight distance or specific event (such as detecting a specific target, reaching a specific location, etc.) to obtain an image of the drone in front of the drone at the current moment.

[0045] S120. Based on the target endpoint information and the current image ahead, the obstacle avoidance model is used to determine the travel control parameter information of the UAV at the next moment. The target endpoint information guides the obstacle avoidance model to filter out image information that has a preset correlation with the target endpoint information from the current image ahead, and is used to predict the travel control parameters of the UAV for obstacle avoidance travel toward the target endpoint position at the next moment. The obstacle avoidance model is a model that enables the UAV to avoid obstacles obtained through reinforcement learning training. The target endpoint information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the obstacle avoidance travel of the UAV at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the UAV travel direction that needs to be paid attention to when predicting the obstacle avoidance travel of the UAV at the current moment. Each current image block is generated by dividing the current image ahead.

[0046] Optionally, the obstacle avoidance model is a model obtained through reinforcement learning training that enables the drone to avoid obstacles.

[0047] See also Figure 2The obstacle avoidance model in this scheme can be an end-to-end model trained by reinforcement learning (RL), which can directly map the input target endpoint information and the current front image of the drone as input, and use the travel control parameter information of the next moment after the current moment of the drone as output. During the training of the obstacle avoidance model, the drone continuously tries different flight strategies in the simulation environment, optimizes its own decision-making based on the reward mechanism (such as positive rewards for successful obstacle avoidance and approaching the target, and negative rewards for colliding with obstacles), and gradually learns to safely fly around obstacles in a complex environment. In other words, reinforcement learning focuses on learning how to take actions through trial and error mechanisms and interaction with the environment to achieve the goal of maximizing long-term cumulative rewards. The obstacle avoidance model trained by reinforcement learning will be exposed to a large number of samples of different scenes and obstacles during the training process, learn obstacle avoidance strategies in various situations, and give the obstacle avoidance model a strong generalization ability, so that it can cope with new scenes and obstacles of different sizes that have never been seen before, and ensure that the model continuously optimizes decisions in different environments and tasks according to the actual situation of obstacles based on real-time images and target endpoint information.

[0048] Optionally, the obstacle avoidance model is an end-to-end model obtained by reinforcement learning training that enables the drone to avoid obstacles. The end-to-end model trained by reinforcement learning can directly map the raw data input by the sensor to the obstacle avoidance decision output, skipping the complex intermediate processing links in the traditional method, greatly reducing the processing time, and enabling the drone to respond quickly to obstacles in a very short time. The travel control parameter information may include the travel speed and yaw angle of the drone. The travel speed of the drone may include the linear velocity in the horizontal direction and the linear velocity in the vertical direction and the acceleration for controlling the linear velocity change rate of the drone. The travel control parameter information may also include the throttle parameter of the drone, which can be used to control the output power of the drone motor, thereby adjusting the lift and flight speed of the drone. In different flight stages and obstacle avoidance operations, the throttle parameters need to be reasonably adjusted. For example, in the take-off and ascent stages, the throttle needs to be increased to provide sufficient lift; if the flight altitude or speed needs to be changed quickly during the obstacle avoidance process, the throttle also needs to be adjusted accordingly.

[0049] See also Figure 2, the current image ahead of the drone contains a lot of information, but not all of it is related to obstacle avoidance and traveling to the target destination. The target destination information provides a clear screening direction for the obstacle avoidance model, allowing the obstacle avoidance model to prioritize image content related to the target destination to be reached. The target destination information provides a target guide for the drone's global path planning, which is not only used to deal with the obstacle avoidance problem of the drone at the current moment, but also guides the flight path of the drone as a whole. During the entire flight process, the obstacle avoidance model will continue to adjust the drone's direction and path based on the target destination information and real-time acquisition of the image ahead, ensuring that the drone always moves towards the target destination while avoiding all obstacles along the way, achieving safe and efficient flight from the starting position to the target destination.

[0050] See also Figure 2 , the target endpoint information can run through the entire travel process of the drone to achieve the guidance of the obstacle avoidance model's drone obstacle avoidance travel prediction, so that the obstacle avoidance model can accurately select the image content with preset correlation with the target endpoint information from the image information obtained in front of the drone. The preset correlation can be a pre-set standard or rule to measure the degree of correlation between the image information in the image in front of the drone and the target endpoint information. Specifically, the preset correlation is pre-set based on the characteristics of the drone's flight mission, the specific attributes of the target endpoint (such as the coordinate information of the target endpoint position, the environmental characteristics of the target endpoint position, etc.), and the empirical rules summarized by past flight data. The screened image content contains key visual clues closely related to the target endpoint, such as clear path markings leading to the target endpoint position, unique environmental characteristics of the target endpoint position, etc. These key clues play a vital role in predicting the drone's travel parameters at the next moment, and can guide the obstacle avoidance model to comprehensively consider the current environmental conditions and goal orientation, and predict the travel control parameter information that the drone should adopt at the next moment.

[0051] Each current image block is generated by dividing the current image ahead. The first vector information can reflect the correlation between each current image block divided in the current image ahead and the target endpoint information. Considering the obstacles, traversable areas and information close to the target endpoint contained in each current image block, a weight value is assigned to each image block. These weight values ​​together constitute the first vector information. The vector generated from the local image block level by combining the first vector information with the target endpoint position of the drone can well reflect the importance of different image blocks in obstacle avoidance and travel prediction, thereby guiding the obstacle avoidance model to identify and pay attention to reference image blocks that are important for obstacle avoidance and travel to the target.

[0052] Among them, the importance of the reference image block when used for the prediction of the obstacle avoidance of the UAV is greater than the importance of other image blocks in each current image block except the reference image block. Optionally, the reference performance of the reference image block when used for the prediction of the obstacle avoidance of the UAV is greater than the reference performance of other image blocks in each current image block except the reference image block when used for the prediction of the obstacle avoidance of the UAV, and the reference performance can be described by but not limited to at least one of the following information: the efficiency of guiding the UAV to the position indicated by the target terminal information, the accuracy of the prediction of guiding the UAV to the position indicated by the target terminal information, and the size of the role played in guiding the UAV to the position indicated by the target terminal information.

[0053] When predicting the obstacle avoidance of the drone, the first vector information guides the obstacle avoidance model to focus on the detailed information included in the current image block with a higher weight value when predicting the obstacle avoidance of the drone at the current moment to achieve the obstacle avoidance prediction. For example, if there is an image block that contains a traversable passage close to the target destination and there are no obvious obstacles, the weight of this image block in the first vector will be higher, and the obstacle avoidance model will focus on analyzing the detailed information of the reference image block to determine whether the drone can pass through the area, thereby planning a more reasonable obstacle avoidance path to ensure that the drone moves towards the target destination while avoiding obstacles.

[0054] The second vector information can reflect the direction vector from the drone to the target end position, and combined with the current forward image, such as the overall obstacle distribution, terrain characteristics, etc., to guide the obstacle avoidance model to adjust and optimize the direction of the drone. The second vector information can guide the obstacle avoidance direction of the drone from a global level, and assist the obstacle avoidance model to determine the direction of the drone that needs to be focused on.

[0055] When the UAV makes obstacle avoidance predictions, the first vector information guides the obstacle avoidance model to focus on environmental information in the same or similar direction as the vector when making obstacle avoidance predictions at the current moment. For example, when the second vector indicates that the target end point is in the northeast direction of the current position of the UAV, the obstacle avoidance model will focus on analyzing the obstacles in the northeast direction, the airspace passability, etc. If there are obstacles in this direction, the obstacle avoidance model will find a suitable detour direction based on the distance, size and other information of the obstacle, combined with the guidance of the second vector, to ensure that the UAV always moves in the general direction of the target end point, and avoids deviation from the target end point position due to excessive obstacle avoidance.

[0056] Optionally, for the first vector information, the obstacle avoidance model analyzes the correlation between each image block and the target endpoint position from the perspective of local image blocks. If there is an image block located in the direction close to the target endpoint position and contains clues related to the target key position, the image block is given a higher weight in the first vector to generate the first vector information. For the second vector information, the obstacle avoidance model considers it from a global perspective. According to the relative orientation relationship between the drone and the target endpoint at the current moment, it determines a vector representing the importance of the drone's direction of travel at the current moment, and guides the obstacle avoidance model to give priority to the direction of travel of the drone that needs attention.

[0057] By adopting the above method, the obstacle avoidance model can provide decision-making guidance for the movement of the drone from both local and global levels. The first vector helps the model to prioritize those key reference image blocks, and then extract effective features from the key reference image blocks for obstacle avoidance prediction, while the second vector can guide the obstacle avoidance model to obtain image information from the correct direction, so that the drone can always move towards the target destination while avoiding obstacles, ensuring the rationality of the global path planning.

[0058] S130: Control the UAV to avoid obstacles at the next moment based on the travel control parameter information.

[0059] See also Figure 2 , the flight control system of the UAV receives the travel control parameter information of the UAV at the next moment, which is obtained by the obstacle avoidance model based on the target endpoint information and the current front image. The travel control parameter information is usually transmitted in the form of digital signals, including multiple key control indicators, such as travel speed, yaw angle, etc. The flight control system of the UAV will parse these received parameters and convert them into a command format that the flight control system can understand and execute. For example, the numerical value representing the flight speed is parsed into the motor speed control command, and the yaw angle parameter is converted into the deflection angle command of the servo.

[0060] In an optional example, the drone is controlled to avoid obstacles at the next moment based on the travel control parameter information, including: the power of the drone mainly comes from the motor, and the motor speed directly determines the lift and flight speed of the drone. If the travel control parameters require an increase in the flight speed, the flight control system will send a signal to the motor controller to increase the input voltage or current of the motor, thereby increasing the motor speed, so that the drone can obtain greater lift and forward momentum. On the contrary, if the speed needs to be reduced, the motor speed will be reduced accordingly. For multi-rotor drones, the speed coordination of different motors can also achieve the steering and attitude adjustment of the drone. For example, when it is necessary to turn left, the flight control system will appropriately reduce the speed of the left motor and increase the speed of the right motor, so that the drone generates a torque to the left to achieve a turning action.

[0061] In an optional example, the drone is controlled to avoid obstacles at the next moment based on the travel control parameter information, including: the flight control system controls the deflection angle of the drone according to the yaw angle in the travel control parameter. For example, when it is necessary to climb upward, the flight control system controls the elevator to deflect upward, so that the drone's nose is tilted up, the flight attitude is changed, and the climbing action is achieved; when it is necessary to turn right, the aileron and rudder are controlled to work together to make the drone tilt to the right and turn.

[0062] During the entire obstacle avoidance process, the UAV's flight control system will continuously monitor the travel control parameters to be adopted by the UAV at different times in real time, and control the UAV in a timely manner according to the travel control parameters. At the same time, the actual flight state is constantly compared with the expected flight state set by the travel control parameters. If a deviation is found between the actual flight state and the expected flight state, such as the actual flight speed is lower than the set speed, the flight direction deviates from the predetermined direction, etc., the flight control system will adjust the power system and flight attitude in a timely manner according to the deviation, and dynamically correct the travel control parameters to ensure that the UAV always flies safely and stably according to the predetermined obstacle avoidance plan.

[0063] According to the technical solution of the embodiment of the present invention, the target endpoint information clarifies the mission direction of the UAV and provides target guidance for subsequent path planning. The current forward image will provide real-time feedback on the environmental conditions of the UAV. The target endpoint information guides the obstacle avoidance model to screen out key image information closely related to the obstacle avoidance movement of the UAV toward the target endpoint based on the target endpoint information. The target endpoint information runs through the entire obstacle avoidance movement process of the UAV so that the obstacle avoidance model can always give priority to image information that is valuable for obstacle avoidance movement toward the endpoint, and then accurately predict the movement control parameters to ensure that the UAV can avoid obstacles and move toward the target in a complex environment. At the same time, the current image ahead provides rich environmental information. No matter the size of the obstacle or the scene it is in, it can be captured by the image. The obstacle avoidance model will not be restricted by the scene and size differences of the obstacle when avoiding obstacles based on the image information ahead. The end-to-end obstacle avoidance model obtained through reinforcement learning training will be exposed to a large number of samples of different scenes and obstacles during the training process, and learn obstacle avoidance strategies in various situations, giving the obstacle avoidance model a strong generalization ability, so that it can cope with new scenes and obstacles of different sizes that have never been seen before, and ensure that the model continuously optimizes decisions in different environments and tasks according to the actual situation of the obstacle based on real-time images and target endpoint information. The UAV can autonomously adapt to various flight conditions without human intervention, and continuously adjust flight speed, direction and other travel control parameters to flexibly make obstacle avoidance decisions.

[0064] Figure 3A flow chart of another obstacle avoidance method for an unmanned aerial vehicle provided in an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the travel control parameter information of the unmanned aerial vehicle at the next moment through an obstacle avoidance model based on the target endpoint information and the current front image in the aforementioned embodiment on the basis of the technical solution of the aforementioned embodiment. This embodiment can be combined with various optional schemes in one or more of the aforementioned embodiments.

[0065] like Figure 3 As shown, the obstacle avoidance method of the drone according to the embodiment of the present invention may include the following process:

[0066] S310: Determine the target destination information of the drone, and determine the current forward image of the drone when it is moving at the current moment.

[0067] S320. Based on the current image ahead, position coding information corresponding to each of the multiple current image blocks is determined through an obstacle avoidance model. The multiple current image blocks are generated by dividing the current image ahead, and the position coding information corresponding to each current image block is used to represent the position information of the current image block in the current image ahead.

[0068] Among them, the obstacle avoidance model is an end-to-end model obtained through reinforcement learning training that enables the drone to avoid obstacles.

[0069] The current image ahead is divided into small blocks according to certain rules (such as evenly dividing into square or rectangular areas of fixed size) to obtain multiple current image blocks corresponding to the current image ahead, each of which contains local image information in the current image ahead.

[0070] At the same time, after obtaining multiple current image blocks corresponding to the current forward image, each current image block can be subjected to position coding embedding processing, and the position coding information corresponding to each current image block represents the information of the position of the current image block in the current forward image. The position coding information corresponding to the current image block can be digitally encoded based on the row and column coordinates of the current image block in the current forward image, or it can be a vector representation generated by conversion through a specific algorithm, so that the obstacle avoidance model can perceive the spatial position of each current image block, thereby better understanding the overall structure and local relationship of the image.

[0071] As an optional but non-limiting implementation scheme, based on the current forward image, the position coding information corresponding to each of the multiple current image blocks is determined by the obstacle avoidance model, including the following steps:

[0072] The current image ahead is divided into a plurality of current image blocks of the same size, and position information of each current image block in the current image ahead is given by embedding position coding.

[0073] The current image ahead contains rich environmental information, but directly processing the entire image ahead requires high computing resources and model complexity. For this reason, the image ahead can be divided into multiple image blocks, which can decompose the complex image ahead information into multiple local small blocks, so that the obstacle avoidance model can process them separately, improving computing efficiency and feature extraction accuracy. Optionally, when dividing the current image ahead into blocks, a uniform division method can be used, that is, the image is cut into square or rectangular areas of uniform size according to fixed row and column spacing. This division method allows each image block to have a clear coordinate representation in the image.

[0074] Since the image block itself does not contain the position information of the image block in the image ahead, it is difficult for the obstacle avoidance model to understand the spatial position relationship between these image blocks. The position information of the image block is converted into a numerical representation through position encoding, so that the obstacle avoidance model can perceive the position of the image block in the overall image, so as to better understand the structure of the image and the distribution of objects. For example, position encoding methods include sine-cosine position encoding and encoding based on relative distance. By embedding the position of each current image block, the obstacle avoidance model can represent different images as special vectors similar to word vectors when performing obstacle avoidance. By mining the relationship between different images, the entire operating scene of the drone can be perceived.

[0075] In an optional example, the position coding information corresponding to each of the multiple current image blocks is determined by the obstacle avoidance model, including: constructing a patch by equally dividing the window image block, dividing the current image ahead into current image blocks of the same size, and then embedding the position information into each current image block, and embedding the position coding to express the position information of the current image block in the current image ahead, thus completing the position embedding. Then, using the characteristics of the obstacle avoidance model, different current image blocks are represented as special vectors similar to word vectors, and the entire scene is perceived by mining the connections between different current image blocks.

[0076] S330. Determine the first vector information and the second vector information based on the target endpoint information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when performing the drone obstacle avoidance movement prediction at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the drone travel direction that needs to be paid attention to when performing the drone obstacle avoidance movement prediction at the current moment.

[0077] Among them, the target endpoint information guides the obstacle avoidance model to filter out image information that has a preset correlation with the target endpoint information from the current front image, which is used to predict the travel control parameters of the drone for obstacle avoidance travel towards the target endpoint position at the next moment.

[0078] As an optional but non-limiting implementation scheme, determining the first vector information based on the target endpoint information includes the following steps B11-B13:

[0079] Step B11: Determine the current travel position information of the UAV at the current moment.

[0080] Optionally, the latitude and longitude coordinates of the drone on the earth are obtained through the GPS positioning system to obtain the current travel position information of the drone at the current moment. At the same time, the inertial navigation system (INS) carried by the drone can also be used to measure the acceleration and angular velocity of the drone using an accelerometer and a gyroscope, and the travel position information and attitude changes of the drone at the current moment can be calculated through integral calculation. In addition, the visual positioning technology can be combined to match the current forward image taken by the drone at the current moment with the pre-built map to determine the current travel position information of the drone at the current moment.

[0081] Step B12: Based on the target endpoint information and the current travel position information, determine the weight vector of each current image block. The weight vector of the current image block is used to indicate the importance of the information contained in the current image block in guiding the obstacle avoidance model to perform drone obstacle avoidance travel prediction.

[0082] Step B13: Determine first vector information based on the weight vectors of each current image block.

[0083] The weight vector of the current image block can be a vector that assigns different weight values ​​to each current image block in combination with the current moving position of the drone and the target destination information. The weight vector of the current image block can be based on the local angle of the current moving front image, guide some image blocks in the current moving front image, and select reference image blocks containing obstacle details, terrain features, etc. that are valuable for obstacle avoidance prediction according to the weight size, and assist the obstacle avoidance model to accurately identify the key local image information in each current image block, so that the key reference image blocks can be used first to make more accurate obstacle avoidance decisions, such as planning a reasonable avoidance path, adjusting the flight altitude and speed, etc.

[0084] The current image ahead is divided into multiple current image blocks. The higher the weight corresponding to the weight vector of each current image block, the more important the information contained in the current image block is for the obstacle avoidance model to predict obstacle avoidance. This enables the obstacle avoidance model to select key features from a large amount of local image information to predict obstacle avoidance and ignore irrelevant background information, thereby processing image data more efficiently. During the flight of the drone, the weight vector of the current image block can guide the obstacle avoidance model to prioritize the detailed analysis of the local area within the current field of view to predict obstacle avoidance.

[0085] The environment is changing dynamically, and the local situations faced by the drone are different at different times. The image block weight vector can reflect these changes in real time. By adjusting the weights of each image block, the obstacle avoidance model can quickly adapt to the new local environment, and focus more on the image information included in the reference image block that needs attention when making obstacle avoidance predictions. For example, when a drone enters a narrow street from an open area, the weight vector will be redistributed to focus more on the buildings and possible obstacles on both sides of the street, ensuring the drone's safe flight in a complex dynamic environment.

[0086] The weight vectors of each current image block are integrated, such as by weighted summation, splicing, etc., to obtain the first vector information. This first vector information comprehensively reflects the importance distribution of all current image blocks in the obstacle avoidance model's obstacle avoidance travel prediction process, guides the obstacle avoidance model to focus on image blocks with higher weights, extracts key information from them for obstacle avoidance decisions, and realizes that for the navigation task of the drone, the weight of each image block in the image ahead is different, and the image blocks representing the direction of the target end point should have a higher position weight, while the image blocks irrelevant to the drone navigation task should have a lower position weight.

[0087] As an optional but non-limiting implementation scheme, determining the second vector information based on the target endpoint information includes the following steps B21-B23:

[0088] Step B21: Determine the current travel position information of the UAV at the current moment.

[0089] Step B22: Based on the target endpoint information and the current traveling position information, determine the latent vector of the target endpoint information at the current moment, the latent vector of the target endpoint information at the current moment reflects the relative position of the target endpoint information with respect to the current traveling position information, and the latent vector of the target endpoint information at the current moment is used to indicate the role played by the current traveling front image in guiding the obstacle avoidance model to perform drone obstacle avoidance travel prediction.

[0090] Step B23: determine the latent vector of the target endpoint information at the current moment as the second vector information.

[0091] The latent vector of the target endpoint information at the current moment is the relative position information formed by combining the current travel position of the UAV at the current moment and the target endpoint position. The latent vector of the target endpoint information at the current moment determines the direction of the UAV travel that the obstacle avoidance model needs to pay attention to when making obstacle avoidance travel predictions from the perspective of the global overall image, and guides the obstacle avoidance model to preferentially obtain the image information ahead of the direction of travel of the UAV, so as to plan a rough path towards the target endpoint position, while ensuring that the UAV will not deviate from the overall travel direction during the local obstacle avoidance process, thereby achieving effective coordination between the global and local.

[0092] The latent vector of the target endpoint information at the current moment can provide a macroscopic direction guide for the UAV's movement after the current moment, and assist the obstacle avoidance model to plan the approximate path from the current travel position to the target endpoint position. The latent vector of the target endpoint information at the current moment can guide the UAV to move in the direction of the target endpoint position during the movement after the current moment, avoiding getting lost in the local area. The latent vector of the target endpoint information at the current moment determines the direction of travel that the UAV should pay attention to when acquiring the image ahead of it after the current moment. According to the target endpoint information, the obstacle avoidance model can determine which direction of travel the image ahead of it is most critical for obstacle avoidance and travel decision-making, thereby guiding the priority processing of obstacles in that direction of travel, ensuring that the UAV always moves towards the target. The latent vector of the target endpoint information at the current moment can pass the global goal to the local decision-making process, so that the local obstacle avoidance action will not deviate from the overall direction of travel.

[0093] The obstacle avoidance model maps the target endpoint information and the current travel position information to a low-dimensional vector space through a specific algorithm (such as the fully connected layer and attention mechanism in the neural network), and generates a latent vector of the target endpoint information at the current moment. This latent vector contains the relationship between the target endpoint and the current position, as well as the importance of the target endpoint information to the current obstacle avoidance decision.

[0094] The latent vector of the target endpoint information at the current moment is used to indicate the role of the current forward image in guiding the obstacle avoidance model to predict the drone's obstacle avoidance. For example, if the latent vector indicates that the target endpoint is directly in front of the current position and the distance is relatively close, then the information in the forward area of ​​the current forward image is more critical to the obstacle avoidance decision. The obstacle avoidance model will focus on analyzing the forward image information based on this vector and plan the obstacle avoidance path towards the target endpoint.

[0095] S340, based on the position coding information, the first vector information and the second vector information corresponding to each of the multiple current image blocks, determine the travel control parameter information of the drone at the next moment through the obstacle avoidance model.

[0096] Specifically, the obstacle avoidance model takes the position coding information, first vector information and second vector information corresponding to each of the multiple current image blocks as input. Combining the position coding information and the first vector information, the obstacle avoidance model performs weighted processing on each image block to highlight the reference image block features that need to be focused on. Then, the processed image block feature vector is fused with the second vector information, comprehensively considering the local key image blocks and the global travel direction. The obstacle avoidance model extracts and calculates features through the internal neural network layer, and uses the parameters and algorithms obtained through training to predict the travel control parameters required by the drone at the next moment to avoid obstacles and move to the target terminal position, such as flight speed, yaw angle, etc.

[0097] By adopting the above scheme, multi-source information is integrated to achieve accurate decision-making. Specifically, by fusing position coding, first vector and second vector information, the model fully considers the position of image blocks, local key information and global travel direction, and can more comprehensively and accurately analyze the current environment and target requirements, thereby predicting reasonable travel control parameters to ensure that the UAV can avoid obstacles safely and efficiently and move towards the target destination.

[0098] As an optional but non-limiting implementation, see Figure 4 The obstacle avoidance model is configured with a first multilayer perceptron for determining first vector information based on current traveling position information and a second multilayer perceptron for determining second vector information based on current traveling position information. The current traveling position information is the traveling position information of the UAV at the current moment.

[0099] As an optional but non-limiting implementation scheme, the obstacle avoidance model is configured with a first multi-layer perceptron for obtaining the weight vector of each current image block based on the current travel position information and the target endpoint information, and a second multi-layer perceptron for obtaining the latent vector of the target endpoint information at the current moment.

[0100] See also Figure 4 The multi-layer perceptron (MLP) configured in the obstacle avoidance model is a feedforward neural network consisting of an input layer, a hidden layer, and an output layer. Neurons transmit information through weighted connections. Information is processed from the input layer to the hidden layer in sequence, and finally the result is generated in the output layer. During the training process of the obstacle avoidance model, the connection weights between neurons are continuously adjusted through the back-propagation algorithm, so that the output of the obstacle avoidance model is as close to the expected result as possible.

[0101] See also Figure 4In the obstacle avoidance model of the present application, the first multi-layer perceptron receives the current travel position information and the target destination information as input, and these two information are encoded into a vector form suitable for network input. For example, the current travel position information can be converted into a numerical vector by means of GPS coordinates, etc., and the target destination information is also expressed in numerical forms such as coordinates and distances. The input layer passes this information to the hidden layer, and the neurons in the hidden layer perform weighted summation on the input and perform nonlinear transformation through activation functions (such as ReLU, Sigmoid, etc.). After being processed by multiple hidden layers, data features are continuously extracted and abstracted. Finally, the output layer outputs a weight vector for each current image block. Each element in this weight vector represents the importance of the corresponding current image block in guiding the obstacle avoidance model to predict the obstacle avoidance of the drone.

[0102] See also Figure 4 In the obstacle avoidance model of the present application, the second multi-layer perceptron receives the current travel position information and the target end point information as input. Similar to the first multi-layer perceptron, this information is first converted into a suitable vector form and input into the network. Inside the network of the second multi-layer perceptron, after layer-by-layer processing of the hidden layer, the relative position, distance, direction and other relationships between the current travel position and the target end point are deeply extracted and abstracted. Finally, the output layer outputs a latent vector, which reflects the key information such as the relative position of the target end point information relative to the current travel position information. For example, the latent vector can encode information such as the target end point is in the northeast direction of the current position and is several meters away. The latent vector is used as the second vector information to indicate the role of the current front image in guiding the obstacle avoidance model to predict the obstacle avoidance of the drone. For example, if the latent vector indicates that the target end point is near the left front of the current position, then in the current front image, the image information of the left front area is more critical for the obstacle avoidance model to plan the path to the left front, and the obstacle avoidance model will focus on and analyze the image content of the area based on this latent vector.

[0103] As an optional but non-limiting implementation scheme, based on the position coding information, the first vector information and the second vector information corresponding to each of the multiple current image blocks, the travel control parameter information of the UAV at the next moment is determined by the obstacle avoidance model, including the following steps C1-C3:

[0104] Step C1, based on the position coding information corresponding to each current image block in the position coding information corresponding to each of the multiple current image blocks and the weight vector of each current image block, determine the update result of the position coding information of each current image block.

[0105] Step C2: input the updated result of the position coding information of each current image block and the latent vector of the target end point information at the current moment into the reference encoder configured in the obstacle avoidance model, where the reference encoder is an encoder based on the architecture of the attention mechanism.

[0106] Step C3: input the output result of the reference encoder into the third multi-layer perceptron configured in the obstacle avoidance model, and output the travel control parameter information that the drone needs to load and use at the next moment through the third multi-layer perceptron.

[0107] The position coding information corresponding to the current image block is a digital representation of the position of the current image block in the current image ahead, which enables the obstacle avoidance model to perceive the spatial position of the image block and assists the obstacle avoidance model in understanding the image structure and the relationship between the local and the global. The update result of the position coding information of the current image block is obtained by performing specific operations on the position coding information of the current image block and the weight vector, which comprehensively considers the position of the image block and the importance of its information.

[0108] In an alternative example, see Figure 4 For each current image block, first obtain its corresponding position coding information and weight vector. The position coding information can be a vector generated by sine cosine function and other methods, representing the position of the image block in the image. The weight vector is generated by the first multi-layer perceptron according to the current travel position information and the target end point information. Then, multiply the position coding information and the weight vector, or use other fusion methods (such as weighted summation) to obtain the updated result of the position coding information corresponding to the current image block.

[0109] Alternatively, see Figure 4 The reference encoder can be an encoder based on the attention mechanism architecture, for example, the reference encoder is an encoder based on the Tranformer architecture. The third multi-layer perceptron is a neural network component in the obstacle avoidance model, which receives the output result of the reference encoder, further processes and maps the output result of the reference encoder, and outputs the control parameter information of the drone at the next moment.

[0110] In an alternative example, see Figure 4, the updated results of the position encoding information of all current image blocks are combined into a matrix or vector sequence, which is used as the input of the reference encoder together with the latent vector of the target endpoint information at the current moment. The reference encoder is based on the attention mechanism. When processing the input information, it calculates the degree of association between each position encoding information update result and the latent vector (i.e., the attention weight). For the position encoding information update results that are closely related to the target endpoint information, a higher attention weight will be assigned, so that these key information will be more prominent in the encoding process. For example, the attention weight is obtained by calculating the dot product, using scaled dot product attention, and then the weighted summation and other operations are performed on the position encoding information update results to complete the encoding process.

[0111] In an alternative example, see Figure 4 The output of the reference encoder contains the key feature information processed by the attention mechanism, and this feature information is input into the input layer of the third multi-layer perceptron. The third multi-layer perceptron further performs nonlinear transformation and feature extraction on the input features through multiple hidden layers. The neurons in each hidden layer perform weighted summation on the input and process it through the activation function. After complex processing by multiple hidden layers, the output layer outputs the travel control parameter information required by the drone at the next moment based on the trained parameters and model structure.

[0112] For example, see Figure 4 , divide the current image ahead into multiple current image blocks, input the current image blocks into the linear projection of flattened patches (Linear Projetion of Flattened Patches) to obtain the position encoding information of each current image block; then, input the target endpoint information into the first multi-layer perceptron MLP and the second multi-layer perceptron MLP, the first multi-layer perceptron MLP outputs a weight vector for each image block, and the second multi-layer perceptron MLP outputs the latent vector of the target endpoint information at the current moment. Multiply the position encoding corresponding to each image block by the corresponding weight vector. Then input the position encoding result after multiplying the weight vector and the latent vector of the target endpoint information at the current moment into the Tranformer Encoder to obtain the output result, and input the output result into the third multi-layer perceptron MLP to obtain the speed and yaw angle of the drone. The drone follows the speed and yaw angle heading and arrives at the destination position.

[0113] By fusing the position encoding information of the image blocks and the importance of the position encoding information in obstacle avoidance prediction, subsequent processing can pay more attention to the position features of important image blocks, providing more accurate position-related information for accurate obstacle avoidance decisions. The reference encoder of the attention mechanism can automatically focus on the key image block information related to the target endpoint, effectively extract the most valuable features for obstacle avoidance decisions, enhance the model's ability to process important information, and improve the pertinence and effectiveness of information processing; and the third multi-layer perceptron uses its powerful nonlinear mapping ability to convert the feature information output by the reference encoder into specific travel control parameters, realizing the conversion from environmental information and target information to actual control instructions for the drone.

[0114] On the basis of the above embodiment, when training the obstacle avoidance model, this solution uses a simulation scene that has been accurately reconstructed in 3D, and performs reinforcement learning training on the obstacle avoidance model in the simulation scene to reduce the gap of sim2real, and in order to improve the generalization of the obstacle avoidance model, the starting point and end point of the drone are random. In the end, the drone achieved a success rate of more than 90% in the simulation scene, and can be migrated to the real scene with a probability of more than 80% to reach the end point.

[0115] S350: Control the UAV to avoid obstacles at the next moment based on the travel control parameter information.

[0116] The technical solution of the embodiment of the present invention, in addition to achieving the technical effects of the aforementioned embodiment, can also divide the current front image into multiple image blocks and determine the position coding information, so that the obstacle avoidance model can carefully perceive the position of each part of the image in the global context; the first vector information guides the model to focus on the key image blocks based on the target endpoint information from the local image block level, and the second vector information determines the direction of travel from a global perspective, so that the drone can avoid obstacles while keeping facing the target, and the two work together to ensure that the decision takes into account both local safety and global goals; furthermore, by integrating the position coding information, the first vector information and the second vector information, the obstacle avoidance model can comprehensively analyze the current environment and target requirements, accurately predict the travel control parameters, and face complex and changeable flight environments, such as different scenes and obstacles. The obstacle avoidance model can flexibly adjust the decision, improve the drone's ability to cope with complex environments, and ensure flight safety, and reduce unnecessary calculations by processing images in blocks and screening targeted information, and avoid invalid processing of a large amount of irrelevant information. Under limited computing resources, the obstacle avoidance model can quickly focus on key image information content, efficiently complete obstacle avoidance decisions, and enable the drone to respond to environmental changes in real time.

[0117] Figure 5A schematic structural diagram of an obstacle avoidance device for a UAV provided in an embodiment of the present invention. The embodiment of the present invention is applicable to situations where a UAV performs obstacle avoidance in a limited space, especially situations where a UAV performs obstacle avoidance when performing inspection and photography in areas such as substations. The obstacle avoidance device for the UAV can be implemented in the form of software and / or hardware, and is generally integrated on any electronic device with network communication function, which can be a mobile terminal, a PC or a server, etc.

[0118] like Figure 5 As shown, the obstacle avoidance device of the drone according to the embodiment of the present invention may include the following:

[0119] The first determination module 510 is used to determine the target destination information of the drone and determine the current forward image of the drone when it is traveling at the current moment;

[0120] The second determination module 520 is used to determine the travel control parameter information of the UAV at the next moment through the obstacle avoidance model based on the target endpoint information and the current forward image. The target endpoint information guides the obstacle avoidance model to screen out image information with a preset correlation with the target endpoint information from the current forward image, and is used to predict the travel control parameters of the UAV to avoid obstacles when traveling toward the target endpoint position at the next moment. The obstacle avoidance model is a model obtained by reinforcement learning training that enables the UAV to avoid obstacles. The target endpoint information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the obstacle avoidance of the UAV at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the direction of the UAV that needs to be paid attention to when predicting the obstacle avoidance of the UAV at the current moment. Each current image block is generated by dividing the current forward image.

[0121] The control module 530 is used to control the UAV to avoid obstacles at the next moment based on the travel control parameter information.

[0122] Based on the above embodiment, optionally, determining the current front image of the drone when it is moving at the current moment includes:

[0123] During the flight of the drone, the fisheye camera on the drone is used to capture images in the direction in front of the drone at the current moment, so as to obtain the current forward image of the drone when it is moving at the current moment.

[0124] On the basis of the above embodiment, optionally, based on the target endpoint information and the current front image, the travel control parameter information of the drone at the next moment is determined by an obstacle avoidance model, including:

[0125] Based on the current image ahead, determine position coding information corresponding to each of a plurality of current image blocks through an obstacle avoidance model, wherein the plurality of current image blocks are generated by dividing the current image ahead, and the position coding information corresponding to each of the current image blocks is used to represent position information of the current image block in the current image ahead;

[0126] Determine first vector information and second vector information based on the target endpoint information, wherein the first vector information is used to guide the obstacle avoidance model to identify reference image blocks that need to be paid attention to in each current image block when performing obstacle avoidance travel prediction for the drone at the current moment, and the second vector information is used to guide the obstacle avoidance model to identify the direction of travel of the drone that needs to be paid attention to when performing obstacle avoidance travel prediction for the drone at the current moment;

[0127] Based on the position coding information corresponding to each of the multiple current image blocks, the first vector information and the second vector information, the travel control parameter information of the drone at the next moment is determined through the obstacle avoidance model.

[0128] On the basis of the above embodiment, optionally, based on the current forward image, the position coding information corresponding to each of the multiple current image blocks is determined by the obstacle avoidance model, including:

[0129] The current moving ahead image is divided into a plurality of current image blocks of the same size, and position information of each current image block in the current moving ahead image is given by embedding position coding.

[0130] Based on the above embodiment, optionally, determining the first vector information based on the target endpoint information includes:

[0131] Determine the current travel position information of the UAV at the current moment;

[0132] Based on the target endpoint information and the current travel position information, determine a weight vector for each current image block, wherein the weight vector for the current image block is used to indicate the importance of the information contained in the current image block in guiding the obstacle avoidance model to perform obstacle avoidance travel prediction for the drone;

[0133] The first vector information is determined based on the weight vectors of each of the current image blocks.

[0134] Based on the above embodiment, optionally, determining the second vector information based on the target endpoint information includes:

[0135] Determine the current travel position information of the UAV at the current moment;

[0136] Based on the target endpoint information and the current travel position information, determine the latent vector of the target endpoint information at the current moment, the latent vector of the target endpoint information at the current moment reflects the relative position of the target endpoint information relative to the current travel position information, and the latent vector of the target endpoint information at the current moment is used to indicate the role played by the current forward image in guiding the obstacle avoidance model to perform obstacle avoidance travel prediction for the UAV;

[0137] The latent vector of the target endpoint information at the current moment is determined as the second vector information.

[0138] Based on the above embodiment, optionally, the obstacle avoidance model is configured with a first multilayer perceptron for obtaining the weight vector of each current image block based on the current traveling position information and the target endpoint information and a second multilayer perceptron for obtaining the latent vector of the target endpoint information at the current moment.

[0139] On the basis of the above embodiment, optionally, based on the position coding information corresponding to each of the multiple current image blocks, the first vector information and the second vector information, the travel control parameter information of the drone at the next moment is determined by the obstacle avoidance model, including:

[0140] Determine an update result of the position coding information of each current image block based on the position coding information corresponding to each current image block in the position coding information corresponding to each of the multiple current image blocks and a weight vector of each current image block;

[0141] Inputting the updated result of the position encoding information of each current image block and the latent vector of the target endpoint information at the current moment into a reference encoder configured in the obstacle avoidance model, wherein the reference encoder is an encoder based on an attention mechanism architecture;

[0142] The output result of the reference encoder is input into the third multi-layer perceptron configured in the obstacle avoidance model, and the third multi-layer perceptron is used to output the travel control parameter information that the drone needs to load and use at the next moment.

[0143] According to the technical solution of the embodiment of the present invention, the target endpoint information clarifies the mission direction of the UAV and provides target guidance for subsequent path planning. The current forward image will provide real-time feedback on the environmental conditions of the UAV. The target endpoint information guides the obstacle avoidance model to screen out key image information closely related to the obstacle avoidance movement of the UAV toward the target endpoint based on the target endpoint information. The target endpoint information runs through the entire obstacle avoidance movement process of the UAV so that the obstacle avoidance model can always give priority to image information that is valuable for obstacle avoidance movement toward the endpoint, and then accurately predict the movement control parameters to ensure that the UAV can avoid obstacles and move toward the target in a complex environment. At the same time, the current image ahead provides rich environmental information. No matter the size of the obstacle or the scene it is in, it can be captured by the image. The obstacle avoidance model will not be restricted by the scene and size differences of the obstacle when avoiding obstacles based on the image information ahead. The end-to-end obstacle avoidance model obtained through reinforcement learning training will be exposed to a large number of samples of different scenes and obstacles during the training process, and learn obstacle avoidance strategies in various situations, giving the obstacle avoidance model a strong generalization ability, so that it can cope with new scenes and obstacles of different sizes that have never been seen before, and ensure that the model continuously optimizes decisions in different environments and tasks according to the actual situation of the obstacle based on real-time images and target endpoint information. The UAV can autonomously adapt to various flight conditions without human intervention, and continuously adjust flight speed, direction and other travel control parameters to flexibly make obstacle avoidance decisions.

[0144] The obstacle avoidance device for a drone provided in an embodiment of the present invention can execute the obstacle avoidance method for a drone provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the obstacle avoidance method for a drone.

[0145] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0146] Figure 6 A schematic diagram of the structure of an electronic device for implementing an obstacle avoidance method for a drone provided by an embodiment of the present invention. Figure 6 , which shows an electronic device (eg Figure 6 The terminal device in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0147] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0148] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data.

[0149] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the obstacle avoidance method of the drone shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the obstacle avoidance method of the drone of the embodiment of the present invention are executed.

[0150] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0151] The electronic device provided in the embodiment of the present invention and the obstacle avoidance method for a drone provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0152] An embodiment of the present invention provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the obstacle avoidance method for a drone provided in the above embodiment is implemented.

[0153] It should be noted that the computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0154] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0155] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0156] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines the target endpoint information of the drone, and determines the current front image of the drone when it is moving at the current moment; based on the target endpoint information and the current front image, determines the travel control parameter information of the drone at the next moment through an obstacle avoidance model, the target endpoint information guides the obstacle avoidance model to filter out image information that has a preset correlation with the target endpoint information from the current front image, which is used to predict the travel control parameters of the drone for obstacle avoidance travel toward the target endpoint position at the next moment; based on the travel control parameter information, controls the drone to avoid obstacles at the next moment.

[0157] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The units involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a unit does not limit the unit itself in some cases.

[0160] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0161] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

[0163] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0164] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for avoiding obstacles in a drone, characterized in that: The method comprises: Determine the target destination information of the drone, and determine the current forward image of the drone when it is moving at the current moment; Based on the target endpoint information and the current image ahead, the obstacle avoidance model is used to determine the travel control parameter information of the drone at the next moment. The target endpoint information guides the obstacle avoidance model to screen out image information that has a preset correlation with the target endpoint information from the current image ahead, so as to predict the travel control parameters for the drone to avoid obstacles when traveling toward the target endpoint position at the next moment. The obstacle avoidance model is a model that enables the drone to avoid obstacles obtained through reinforcement learning training. The target endpoint information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the drone's obstacle avoidance at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the drone's travel direction that needs to be paid attention to when predicting the drone's obstacle avoidance at the current moment. Each current image block is generated by dividing the current image ahead. Based on the travel control parameter information, the drone is controlled to avoid obstacles at the next moment.

2. The method according to claim 1, characterized in that Based on the target endpoint information and the current forward image, the travel control parameter information of the UAV at the next moment is determined by the obstacle avoidance model, including: Based on the current image ahead, determine position coding information corresponding to each of a plurality of current image blocks through an obstacle avoidance model, wherein the plurality of current image blocks are generated by dividing the current image ahead, and the position coding information corresponding to each of the current image blocks is used to represent position information of the current image block in the current image ahead; Determine first vector information and second vector information based on the target endpoint information; Based on the position coding information corresponding to each of the multiple current image blocks, the first vector information and the second vector information, the travel control parameter information of the drone at the next moment is determined through the obstacle avoidance model.

3. The method according to claim 2, characterized in that Based on the current forward image, the obstacle avoidance model is used to determine the position coding information corresponding to each of the multiple current image blocks, including: The current moving ahead image is divided into a plurality of current image blocks of the same size, and position information of each current image block in the current moving ahead image is given by embedding position coding.

4. The method according to claim 2, characterized in that: Determining first vector information based on the target endpoint information includes: Determine the current travel position information of the UAV at the current moment; Based on the target endpoint information and the current travel position information, determine a weight vector for each current image block, wherein the weight vector for the current image block is used to indicate the importance of the information contained in the current image block in guiding the obstacle avoidance model to perform obstacle avoidance travel prediction for the drone; The first vector information is determined based on the weight vectors of each of the current image blocks.

5. The method according to claim 2, characterized in that: Determining second vector information based on the target endpoint information includes: Determine the current travel position information of the UAV at the current moment; Based on the target endpoint information and the current travel position information, determine the latent vector of the target endpoint information at the current moment, the latent vector of the target endpoint information at the current moment reflects the relative position of the target endpoint information relative to the current travel position information, and the latent vector of the target endpoint information at the current moment is used to indicate the role played by the current forward image in guiding the obstacle avoidance model to perform obstacle avoidance travel prediction for the UAV; The latent vector of the target endpoint information at the current moment is determined as the second vector information.

6. The method according to claim 2, characterized in that The obstacle avoidance model is configured with a first multilayer perceptron for determining first vector information based on current travel position information and a second multilayer perceptron for determining second vector information based on current travel position information, wherein the current travel position information is the travel position information of the drone at the current moment.

7. The method according to claim 2, characterized in that Based on the position coding information corresponding to each of the multiple current image blocks, the first vector information and the second vector information, the travel control parameter information of the UAV at the next moment is determined by the obstacle avoidance model, including: Determine an update result of the position coding information of each current image block based on the position coding information corresponding to each current image block in the position coding information corresponding to each of the multiple current image blocks and a weight vector of each current image block; Inputting the updated result of the position encoding information of each current image block and the latent vector of the target endpoint information at the current moment into a reference encoder configured in the obstacle avoidance model, wherein the reference encoder is an encoder based on an attention mechanism architecture; The output result of the reference encoder is input into the third multi-layer perceptron configured in the obstacle avoidance model, and the third multi-layer perceptron is used to output the travel control parameter information that the drone needs to load and use at the next moment.

8. An obstacle avoidance device for a drone, characterized in that: The device comprises: The first determination module is used to determine the target end point information of the drone and determine the current forward image of the drone when it is moving at the current moment; a second determination module, for determining the travel control parameter information of the UAV at the next moment through the obstacle avoidance model based on the target endpoint information and the current image ahead, the target endpoint information guiding the obstacle avoidance model to screen out image information having a preset correlation with the target endpoint information from the current image ahead, for predicting the travel control parameters for the UAV to avoid obstacles when traveling toward the target endpoint position at the next moment, the obstacle avoidance model is a model obtained through reinforcement learning training that enables the UAV to avoid obstacles, the target endpoint information is used to determine the first vector information and the second vector information, the first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the obstacle avoidance of the UAV at the current moment, the second vector information is used to guide the obstacle avoidance model to identify the direction of the UAV that needs to be paid attention to when predicting the obstacle avoidance of the UAV at the current moment, and each current image block is generated by dividing the current image ahead; A control module is used to control the UAV to avoid obstacles at the next moment based on the travel control parameter information.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the obstacle avoidance method for the drone described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the obstacle avoidance method of the drone described in any one of claims 1-7 when executed by a computer processor.

Images