Obstacle Avoidance Traveling Method, Device, Electronic Device and Storage Medium of Unmanned Aerial Vehicle

Abstract

An embodiment of the present invention provides an obstacle avoidance traveling method, device, electronic device, and storage medium for an unmanned aerial vehicle. The method includes: determining target end point information of the unmanned aerial vehicle and determining a current forward traveling image of the unmanned aerial vehicle when traveling at the current moment; based on the target end point information and the current forward traveling image, determining traveling control parameter information of the unmanned aerial vehicle at the next moment through an obstacle avoidance model, where the target end point information guides the obstacle avoidance model to screen out image information having a preset correlation with the target end point information from the current forward traveling image for predicting the traveling control parameters for the unmanned aerial vehicle to perform obstacle avoidance traveling towards the target end point position at the next moment; and controlling the unmanned aerial vehicle to perform obstacle avoidance traveling at the next moment based on the traveling control parameter information. The solution of the present application can achieve rapid obstacle avoidance traveling in scenarios with multi-scene and multi-size obstacles, thereby enabling efficient autonomous navigation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of path planning, and in particular, to an obstacle avoidance traveling method, device, electronic device, and storage medium for an unmanned aerial vehicle (UAV). Background Art

[0002] UAVs are increasingly widely used in various fields, especially in logistics distribution, agricultural plant protection, mapping exploration, emergency rescue, etc. In the UAV technology system, navigation and obstacle avoidance technologies are the key core technologies to ensure the safe and efficient operation of UAVs.

[0003] At present, the obstacle avoidance ability of UAVs faces severe challenges in the face of complex flight environments. The traditional binocular parallax model and obstacle avoidance algorithms have obvious defects in dealing with large-scale and complex obstacles, and focus on local path planning and short-distance obstacle avoidance, making it difficult to respond promptly and accurately to large-scale and long-distance obstacles, seriously threatening the flight safety of UAVs. In addition, the traditional UAV obstacle avoidance algorithm consists of multiple modules such as perception, path planning, and trajectory optimization. This complex structure not only increases the system calculation burden but also results in poor cooperation efficiency among modules. Especially in practical applications, its performance cannot meet the growing requirements of UAV application scenarios, such as low-altitude logistics distribution in urban complex environments and precise obstacle avoidance in emergency rescue scenarios. Therefore, there is an urgent need to develop a new UAV obstacle avoidance solution to break through the bottleneck of traditional technologies and improve the obstacle avoidance ability and autonomous flight level of UAVs in complex environments. Summary of the Invention

[0004] The embodiments of the present invention provide an obstacle avoidance traveling method, device, electronic device, and storage medium for an unmanned aerial vehicle, so as to achieve rapid obstacle avoidance traveling in scenarios with multi-scene and multi-size obstacles, thereby enabling efficient autonomous navigation.

[0005] In a first aspect, the embodiments of the present invention provide an obstacle avoidance traveling method for an unmanned aerial vehicle, the method including:

[0006] Determine the target end point information of the unmanned aerial vehicle, and determine the current forward image when the unmanned aerial vehicle is traveling at the current moment;

[0007] Based on the target end point information and the current forward image, determine the travel control parameter information of the UAV at the next moment through an obstacle avoidance model. The target end point information guides the obstacle avoidance model to screen out image information with a preset correlation with the target end point information from the current forward image, which is used to predict the travel control parameters for the UAV to avoid obstacles and travel towards the target end point position at the next moment. The obstacle avoidance model is a model obtained through reinforcement learning that enables the UAV to avoid obstacles and travel. The target end point information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be concerned in each current image block when predicting the UAV's obstacle avoidance travel at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the UAV travel direction that needs to be concerned when predicting the UAV's obstacle avoidance travel at the current moment. The current image blocks are generated by dividing the current forward image;

[0008] Control the UAV to avoid obstacles and travel at the next moment based on the travel control parameter information.

[0009] In a second aspect, an embodiment of the present invention further provides an obstacle avoidance travel device for a UAV, and the device includes:

[0010] A first determination module, configured to determine the target end point information of the UAV and determine the current forward image when the UAV travels at the current moment;

[0011] A second determination module, configured to determine the travel control parameter information of the UAV at the next moment through an obstacle avoidance model based on the target end point information and the current forward image. The target end point information guides the obstacle avoidance model to screen out image information with a preset correlation with the target end point information from the current forward image, which is used to predict the travel control parameters for the UAV to avoid obstacles and travel towards the target end point position at the next moment. The obstacle avoidance model is a model obtained through reinforcement learning that enables the UAV to avoid obstacles and travel. The target end point information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be concerned in each current image block when predicting the UAV's obstacle avoidance travel at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the UAV travel direction that needs to be concerned when predicting the UAV's obstacle avoidance travel at the current moment. The current image blocks are generated by dividing the current forward image;

[0012] A control module, configured to control the UAV to avoid obstacles and travel at the next moment based on the travel control parameter information.

[0013] In a third aspect, an electronic device is further provided in an embodiment of the present invention. The electronic device includes:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the obstacle avoidance traveling method of the unmanned aerial vehicle according to any one of the above embodiments.

[0017] In a fourth aspect, a computer-readable medium is further provided in an embodiment of the present invention. The computer-readable medium stores computer instructions for enabling a processor to implement the obstacle avoidance traveling method of the unmanned aerial vehicle according to any one of the above embodiments when executed.

[0018] In the technical solution of the embodiment of the present invention, the target end point information clarifies the task direction of the unmanned aerial vehicle, provides a target orientation for subsequent path planning, and the image in front of the current travel will timely feedback the environmental conditions of the unmanned aerial vehicle. The target end point information guides the obstacle avoidance model to screen out the key image information closely related to the obstacle avoidance traveling of the unmanned aerial vehicle towards the target end point from the target end point information. By passing through the entire obstacle avoidance traveling process of the unmanned aerial vehicle with the target end point information, the obstacle avoidance model can always give priority to focusing on the image information valuable for the obstacle avoidance traveling towards the end point, thereby accurately predicting the traveling control parameters and ensuring that the unmanned aerial vehicle can avoid obstacles and move towards the target in a complex environment. At the same time, the image in front of the current travel provides rich environmental information. Whether the obstacle is large or small and in what kind of scene, it can be captured by the image. When the obstacle avoidance model performs obstacle avoidance traveling based on the image information in front of the travel, it will not be restricted by the scene and size differences of the obstacles, enabling the unmanned aerial vehicle to autonomously adapt to various flight conditions without manual intervention and continuously adjust traveling control parameters such as flight speed and direction to flexibly make obstacle avoidance decisions.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In combination with the drawings and referring to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present invention will become more obvious. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the original components and elements are not necessarily drawn to scale.

[0021] Figure 1It is a schematic flowchart of an obstacle avoidance traveling method for a drone provided by an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of a scene where a drone performs obstacle bypass flight during obstacle avoidance traveling provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic flowchart of another obstacle avoidance traveling method for a drone provided by an embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of the algorithm structure of an obstacle avoidance model used during the obstacle avoidance traveling of a drone provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of the structure of an obstacle avoidance traveling device for a drone provided by an embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of the structure of an electronic device for implementing the obstacle avoidance traveling method of a drone provided by an embodiment of the present invention. Detailed Embodiments

[0027] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not used to limit the protection scope of the present invention.

[0028] It should be understood that the various steps recited in the method embodiments of the present invention can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0029] As used herein, the term "including" and its variations are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependent relationships.

[0031] It should be noted that the modifiers "one" and "more than one" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless clearly specified otherwise in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] Figure 1 As shown in the figure, it is a schematic flowchart of an obstacle avoidance traveling method for a drone provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation where the drone performs obstacle avoidance traveling in a limited space area, especially the situation where the drone performs obstacle avoidance traveling during inspection and shooting in areas such as substations. The obstacle avoidance traveling method of the drone can be executed by an obstacle avoidance traveling device of the drone. The obstacle avoidance traveling device of the drone can be implemented in the form of software and / or hardware and is generally integrated on any electronic device with network communication functions. The electronic device can be a mobile terminal, a PC terminal, a server, etc.

[0034] As Figure 1 shown, the obstacle avoidance traveling method of the drone in the embodiment of the present invention may include the following processes:

[0035] S110. Determine the target end point information of the drone and determine the current forward image when the drone is traveling at the current moment.

[0036] The drone in this solution can be configured to be applied to the process of drone inspection in a target inspection area to achieve drone obstacle avoidance. Among them, the target inspection area can be a substation area, etc. The target end point information is used to indicate the target end point position that the drone needs to travel to. And / or, the target end point information can refer to the information related to the end point that the drone needs to reach, including position coordinates, end point environment characteristics, etc. The target end point information adopted when the drone travels can be determined by using methods such as GPS positioning, map matching, preset instructions, etc. The target end point information can be represented in the form of position coordinates and / or image frame features at the end point position.

[0037] The current forward image can be obtained by real-time acquisition using an image acquisition device mounted on the drone. It presents the environmental picture in the forward direction of the drone at the current moment and in front of the drone's path, covering image information such as the distribution of obstacles, topographical features, and lighting conditions in the forward direction of the drone at the current moment and in front of the drone's path. Among them, an important task during the drone's flight is to avoid obstacles and reach the target location during forward movement. Therefore, by focusing on the forward direction of the drone first, obtaining the current forward image of the drone, and effectively responding to possible obstacles ahead in a timely manner, reasonable obstacle avoidance actions can be taken.

[0038] The distribution information of obstacles in front of the drone can include various objects that may impede the drone's flight in the forward direction of the drone, such as buildings, trees, wires, etc. Specifically, the shape, size, position, and relative distance from the drone of the obstacles covered in the forward image can be identified through image analysis. The environmental characteristics in front of the drone can include the topographical features and lighting conditions in the forward direction of the drone. For example, whether it is a bright sunny day or a dim cloudy day, the lighting conditions will affect the visual effect of the image and the accuracy of subsequent image recognition algorithms. The forward image is obtained in real-time during the drone's flight and can reflect the latest state of the environment around the drone at the current moment, enabling the drone to adjust its flight strategy in a timely manner according to the latest environmental information.

[0039] As an optional but non-limiting implementation solution, determining the current forward image of the drone during its current movement includes the following steps:

[0040] During the drone's flight, use the fisheye camera on the drone to take an image towards the forward direction of the drone at the current moment to obtain the current forward image of the drone during its current movement.

[0041] The forward image is usually taken by an image acquisition device installed at the front end of the drone. The image acquisition device includes ordinary cameras, fisheye cameras, etc. The image acquisition device is pre-set at a specific position and angle on the drone to ensure that the scene directly in front of the drone's flight direction can be clearly captured. The drone will trigger the camera to take pictures at a preset frequency or under specific conditions. For example, take an image at regular intervals, or immediately start taking pictures when it detects that there may be obstacles ahead.

[0042] Optionally, the fisheye camera has unique optical properties. The design of its lens can provide a very wide viewing angle, usually reaching a field of view of 180° or even exceeding 180°. This ultra-wide-angle view is crucial for the drone to comprehensively perceive the front environment during flight. Compared with the narrow viewing angle range of ordinary cameras, the fisheye camera can capture a larger range of scenes at one time, reducing the visual blind area, enabling the drone to obtain richer information about the front environment in advance, including distant obstacles, terrain changes, and other factors that may affect the flight path.

[0043] During the flight of the drone, the environmental information is constantly changing. Only the real-time captured images can accurately reflect the front environmental conditions of the current position of the drone, which is crucial for the drone to make decisions in a timely manner. That is to say, only based on the latest environmental information in front of the drone's current movement, the drone can accurately determine whether there are obstacles ahead, as well as the real-time position and state of the obstacles, so as to precisely plan the flight path for the next moment and ensure flight safety.

[0044] Optionally, during the flight of the drone, the fisheye camera can be triggered to face the front of the drone's movement for shooting according to a preset time interval, flight distance, or specific events (such as detecting a specific target, reaching a specific position, etc.) to obtain an image of the front of the drone's movement at the current moment.

[0045] S120. Based on the target end point information and the current front movement image, determine the movement control parameter information of the drone for the next moment through the obstacle avoidance model. The target end point information guides the obstacle avoidance model to screen out the image information with a preset correlation with the target end point information from the current front movement image, which is used to predict the movement control parameters for the drone to avoid obstacles and move towards the target end point position at the next moment. The obstacle avoidance model is a model obtained through reinforcement learning that enables the drone to avoid obstacles and move forward. The target end point information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be concerned in each current image block when predicting the drone's obstacle avoidance movement at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the drone's movement direction that needs to be concerned when predicting the drone's obstacle avoidance movement at the current moment. Each current image block is generated by dividing the current front movement image.

[0046] Optionally, the obstacle avoidance model is a model obtained through reinforcement learning that enables the drone to avoid obstacles and move forward.

[0047] See Figure 2, the obstacle avoidance model in this solution can be an end-to-end model trained through Reinforcement Learning (RL for short). It can directly map the input target end point information and the current forward image of the drone as inputs, and use the travel control parameter information of the next moment after the current moment of the drone as the output. During the training process of the obstacle avoidance model, the drone continuously tries different flight strategies in the simulation environment, optimizes its own decisions based on the reward mechanism (such as getting positive rewards for successful obstacle avoidance and approaching the target, and negative rewards for hitting obstacles), and gradually learns to fly around obstacles safely in a complex environment. That is to say, through reinforcement learning, it focuses on learning how to take actions to achieve the goal of maximizing long-term cumulative rewards through a trial-and-error mechanism and interacting with the environment. The obstacle avoidance model trained through reinforcement learning will come into contact with a large number of samples of different scenarios and obstacles during the training process, learn obstacle avoidance strategies in various situations, endow the obstacle avoidance model with strong generalization ability, enable it to cope with new scenarios and obstacles of different sizes that have never been seen before, and ensure that the model continuously optimizes decisions in different environments and tasks according to the real-time image, target end point information, and the actual situation of the obstacles.

[0048] Optionally, the obstacle avoidance model is an end-to-end model trained through reinforcement learning that enables the drone to perform obstacle avoidance travel. The end-to-end model trained through reinforcement can directly map the raw data input by the sensor to the obstacle avoidance decision output, skipping the complex intermediate processing links in the traditional method, greatly reducing the processing time, and enabling the drone to respond quickly to obstacles in an extremely short time. The travel control parameter information can include the drone's travel speed and yaw angle. The drone's travel speed can include the linear speed for controlling the horizontal direction, the linear speed for controlling the vertical direction, and the acceleration for controlling the rate of change of the drone's linear speed. The travel control parameter information can also include the throttle parameter of the drone. The throttle parameter can be used to control the output power of the drone's motor, and thus adjust the lift and flight speed of the drone. In different flight stages and obstacle avoidance operations, it is necessary to reasonably adjust the throttle parameter. For example, in the takeoff and ascent stages, it is necessary to increase the throttle to provide sufficient lift; during obstacle avoidance, if it is necessary to quickly change the flight height or speed, the throttle also needs to be adjusted accordingly.

[0049] See Figure 2, the current forward image obtained by the drone contains a large amount of information, but not all of it is relevant to obstacle avoidance and moving towards the target end point. The target end point information provides a clear screening direction for the obstacle avoidance model, enabling the obstacle avoidance model to prioritize the image content related to the target end point to be reached. The target end point information provides a target orientation for the global path planning of the drone, not only for dealing with the obstacle avoidance problem of the drone at the current moment, but also for guiding the flight path of the drone as a whole. During the entire flight process, the obstacle avoidance model will continuously adjust the moving direction and path of the drone according to the target end point information and the real-time obtained forward image, ensuring that the drone avoids all obstacles along the way while always moving towards the target end point, achieving a safe and efficient flight from the starting position to the target end point.

[0050] See Figure 2 , the target end point information can play a guiding role in predicting the obstacle avoidance movement of the drone by the obstacle avoidance model throughout the entire movement process of the drone, enabling the obstacle avoidance model to accurately screen out the image content that has a preset correlation with the target end point information from the image information obtained in front of the drone's movement. The preset correlation can be a standard or rule preset to measure the degree of association between the image information in the forward image and the target end point information. Specifically, the preset correlation is preset based on the characteristics of the drone's flight mission, the specific attributes of the target end point (such as the coordinate information of the target end point position, the environmental characteristics of the target end point position, etc.), and the empirical rules summarized from past flight data. The screened image content contains key visual clues closely related to the target end point, such as clear path signs leading to the target end point position, unique environmental characteristics of the target end point position, etc. And these key clues are crucial for predicting the movement parameters of the drone at the next moment, and can guide the obstacle avoidance model to comprehensively consider the current environmental conditions and target orientation, and predict the movement control parameter information that the drone should adopt at the next moment.

[0051] Each current image block is generated by dividing the current forward image. The first vector information can reflect the correlation between each current image block divided in the current forward image and the target end point information. Considering the obstacles, passable areas, and proximity to the target end point information contained in each current image block, a weight value is assigned to each image block, and these weight values together constitute the first vector information. The vector generated from the local image block level by combining the first vector information with the target end point position of the drone can well reflect the importance of different image blocks in predicting obstacle avoidance movement, thus guiding the obstacle avoidance model to be able to identify and focus on the reference image blocks that are important for obstacle avoidance and moving towards the target.

[0052] Among them, the importance of the reference image block when used for UAV obstacle avoidance travel prediction is greater than that of other image blocks except the reference image block among each current image block. Optionally, the reference performance of the reference image block when used for UAV obstacle avoidance travel prediction is greater than the reference performance of other image blocks except the reference image block among each current image block when used for UAV obstacle avoidance travel prediction. The reference performance can be described by using at least one of the following information, but not limited to: the efficiency when guiding the UAV to travel to the position indicated by the target end information, the accuracy of travel prediction when guiding the UAV to travel to the position indicated by the target end information, and the magnitude of the role played when guiding the UAV to travel to the position indicated by the target end information.

[0053] When performing UAV obstacle avoidance travel prediction, the first vector information guides the obstacle avoidance model to focus on the detailed information included in the current image block with a higher weight value first when performing UAV obstacle avoidance travel prediction at the current moment to achieve obstacle avoidance travel prediction. For example, if there is an image block that contains a passable travel channel in the direction close to the target end and there are no obvious obstacles, the weight of this image block in the first vector will be relatively high. The obstacle avoidance model will focus on analyzing the detailed information of the reference image block to determine whether the UAV can pass through this area, so as to plan a more reasonable obstacle avoidance path and ensure that the UAV moves towards the target end while avoiding obstacles.

[0054] The second vector information can reflect the direction vector from the UAV to the target end position, and combine with the current forward travel image, such as the overall obstacle distribution, terrain characteristics, etc., so as to guide the obstacle avoidance model to adjust and optimize the UAV travel direction. The second vector information can guide the UAV obstacle avoidance travel direction from a global level and assist the obstacle avoidance model to determine the UAV travel direction that needs to be focused on.

[0055] When the UAV performs obstacle avoidance travel prediction, the first vector information guides the obstacle avoidance model to focus on the environmental information in the direction consistent with or close to the direction pointed by the vector when performing UAV obstacle avoidance travel prediction at the current moment. For example, when the second vector indicates that the target end is in the northeast direction of the current UAV position, the obstacle avoidance model will focus on analyzing the obstacle situation and airspace passability in the northeast direction. If there are obstacles in this direction, the obstacle avoidance model will, according to information such as the distance and size of the obstacles, combine with the guidance of the second vector to find a suitable detour direction to ensure that the UAV always moves in the general direction of the target end and avoids deviating from the target end position due to excessive obstacle avoidance.

[0056] Optionally, for the first vector information, from the perspective of local image patches, the obstacle avoidance model analyzes the correlation between each image patch and the target end position. If there is an image patch located in the direction close to the target end position and containing clues related to the target end position, a relatively high weight is assigned to the image patch in the first vector, thereby generating the first vector information. For the second vector information, the obstacle avoidance model considers from a global perspective and determines a vector representing the importance of the drone's traveling direction at the current moment according to the relative orientation relationship between the drone and the target end position at the current moment, guiding the obstacle avoidance model to prioritize the attention to the drone's traveling direction that needs to be concerned about.

[0057] By adopting the above method, decision-making guidance for the drone's traveling can be provided for the obstacle avoidance model from both local and global levels. The first vector helps the model prioritize the attention to those key reference image patches, and then extract effective features from the key reference image patches for obstacle avoidance prediction. The second vector can guide the obstacle avoidance model to obtain image information from the correct direction, enabling the drone to always move towards the target end while avoiding obstacles, ensuring the rationality of the global path planning.

[0058] S130. Control the drone to perform obstacle avoidance traveling at the next moment based on the traveling control parameter information.

[0059] See Figure 2 , the flight control system of the drone receives the traveling control parameter information of the drone at the next moment obtained by the obstacle avoidance model according to the target end information and the image in front of the current traveling. The traveling control parameter information is usually transmitted in the form of digital signals and contains multiple key control indicators, such as traveling speed, yaw angle, etc. The flight control system of the drone will analyze these received parameters and convert them into an instruction format that the flight control system can understand and execute. For example, the numerical value representing the flight speed is parsed into a motor speed control instruction, and the yaw angle parameter is converted into a deflection angle instruction of the servo, etc.

[0060] In an optional example, controlling the drone to perform obstacle avoidance traveling at the next moment based on the traveling control parameter information includes: The power of the drone mainly comes from the motor, and the motor speed directly determines the lift and flight speed of the drone. If the traveling control parameter requires an increase in the flight speed, the flight control system will send a signal to the motor controller to increase the input voltage or current of the motor, thereby increasing the motor speed and enabling the drone to obtain greater lift and forward power. On the contrary, if the speed needs to be reduced, the motor speed will be correspondingly reduced. For a multi-rotor drone, the speed coordination of different motors can also achieve the turning and attitude adjustment of the drone. For example, when a left turn is required, the flight control system will appropriately reduce the speed of the left motor and increase the speed of the right motor at the same time, causing the drone to generate a left torque to achieve the turning action.

[0061] In an optional example, controlling the UAV to perform obstacle avoidance movement at the next moment based on the movement control parameter information includes: the flight control system controls the deflection angle of the UAV according to the yaw angle in the movement control parameters. For example, when climbing upward is required, the flight control system controls the elevator to deflect upward, causing the nose of the UAV to pitch up, changing the flight attitude, and achieving the climbing action; when turning right is required, the aileron and rudder work together to make the UAV tilt to the right and turn.

[0062] During the entire obstacle avoidance movement process, the flight control system of the UAV continuously and real-time monitors the movement control parameters to be adopted by the UAV at different moments, and timely controls the UAV according to the movement control parameters. At the same time, it continuously compares the actual flight state with the desired flight state set by the movement control parameters. If it is found that there is a deviation between the actual flight state and the desired flight state, such as the actual flight speed is lower than the set speed, the flight direction deviates from the predetermined direction, etc., the flight control system will timely adjust the power system and flight attitude according to the deviation situation, and dynamically correct the movement control parameters to ensure that the UAV always flies safely and stably according to the predetermined obstacle avoidance movement plan.

[0063] The technical solution of the embodiment of the present invention, the target end point information clarifies the task direction of the UAV, provides a target orientation for subsequent path planning, and the current forward image during movement will real-time feedback the environmental conditions of the UAV. The target end point information guides the obstacle avoidance model to screen out the key image information closely related to the UAV's obstacle avoidance movement towards the target end point from the target end point information. Through the target end point information running through the entire obstacle avoidance movement process of the UAV, the obstacle avoidance model can always give priority to focusing on the image information valuable for obstacle avoidance movement towards the end point, thereby accurately predicting the movement control parameters, ensuring that the UAV can avoid obstacles and move towards the target in a complex environment. At the same time, the current forward image during movement provides rich environmental information. Whether the size of the obstacle is large or small, or in what kind of scene, it can be captured by the image. When the obstacle avoidance model performs obstacle avoidance movement based on the forward image information during movement, it will not be restricted by the scene and size differences of the obstacles. And the end-to-end obstacle avoidance model obtained through reinforcement learning training will come into contact with a large number of samples of different scenes and obstacles during the training process, learn the obstacle avoidance strategies in various situations, endow the obstacle avoidance model with strong generalization ability, enabling it to cope with new scenes and obstacles of different sizes that have never been seen before, ensuring that the model continuously optimizes decisions according to the real-time image and target end point information according to the actual situation of the obstacles in different environments and tasks. The UAV can autonomously adapt to various flight conditions without manual intervention, continuously adjust movement control parameters such as flight speed and direction, and flexibly make obstacle avoidance decisions.

[0064] Figure 3A schematic flowchart of another obstacle avoidance traveling method for an unmanned aerial vehicle provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the traveling control parameter information of the unmanned aerial vehicle at the next moment through an obstacle avoidance model based on the target end point information and the current forward traveling image in the foregoing embodiment. This embodiment can be combined with each optional solution in one or more of the above embodiments.

[0065] As Figure 3 shown, the obstacle avoidance traveling method for an unmanned aerial vehicle according to an embodiment of the present invention may include the following processes:

[0066] S310. Determine the target end point information of the unmanned aerial vehicle, and determine the current forward traveling image when the unmanned aerial vehicle is traveling at the current moment.

[0067] S320. Based on the current forward traveling image, determine the position encoding information corresponding to each of a plurality of current image blocks through an obstacle avoidance model. The plurality of current image blocks are generated by dividing the current forward traveling image. The position encoding information corresponding to each current image block is used to represent the position information of the current image block in the current forward traveling image.

[0068] Among them, the obstacle avoidance model is an end-to-end model obtained by reinforcement learning and capable of enabling the unmanned aerial vehicle to perform obstacle avoidance traveling.

[0069] The current forward traveling image is segmented into small image blocks according to a certain rule (such as evenly dividing it into square or rectangular regions of a fixed size) to obtain a plurality of current image blocks corresponding to the current forward traveling image. Each current image block contains local image information in the current forward traveling image.

[0070] At the same time, when obtaining a plurality of current image blocks corresponding to the current forward traveling image, position encoding embedding processing can be performed on each current image block, and the position information of the current image block in the current forward traveling image is characterized by the position encoding information corresponding to each current image block. The position encoding information corresponding to the current image block can be numerically encoded based on the row and column coordinates of the current image block in the current forward traveling image, or can be a vector representation generated through a specific algorithm, enabling the obstacle avoidance model to perceive the spatial position of each current image block, so as to better understand the overall structure and local relationship of the image.

[0071] As an optional but non-limiting implementation solution, determining the position encoding information corresponding to each of a plurality of current image blocks through an obstacle avoidance model based on the current forward traveling image includes the following steps:

[0072] Divide the image in front of the current movement into multiple current image patches of the same size, and assign position information of each current image patch in the image in front of the current movement by embedding position encoding.

[0073] The image in front of the current movement contains rich environmental information. However, directly processing the entire image in front of the movement requires high computing resources and model complexity. Therefore, the image in front of the movement can be divided into multiple image patches, which can decompose the complex information of the image in front of the movement into multiple local small images, facilitating the obstacle avoidance model to process separately and improving the computing efficiency and the accuracy of feature extraction. Optionally, when dividing the image in front of the current movement into image patches, a uniform division method can be adopted, that is, according to fixed row and column spacings, the image is cut into square or rectangular regions of the same size. This division method enables each image patch to have a clear coordinate representation of its position in the image.

[0074] Since the image patches themselves do not contain the position information of the image patches in the image in front of the movement, it is difficult for the obstacle avoidance model to understand the spatial position relationship between these image patches. Through position encoding, the position information of the image patches is converted into a numerical representation, enabling the obstacle avoidance model to perceive the position of the image patches in the overall image, thereby better understanding the structure of the image and the distribution of objects. For example, the position encoding methods include sine-cosine position encoding, encoding based on relative distance, etc. By performing position embedding on each current image patch, different images can be represented as special vectors similar to word vectors when the obstacle avoidance model performs obstacle avoidance movement. By exploring the connections between different images, the entire operating scenario where the UAV is located can be perceived.

[0075] In an optional example, determine the respective position encoding information of multiple current image patches through the obstacle avoidance model, including: adopting the idea of equally dividing window picture patches to construct patches, dividing the image in front of the current movement into current image patches of the same size, and then embedding position information into each current image patch. The position encoding is embedded to express the position information of the current image patch in the image in front of the current movement, thus completing the position embedding. Then, utilizing the characteristics of the obstacle avoidance model, different current image patches are represented as special vectors similar to word vectors, and the entire scene is perceived by exploring the connections between different current image patches.

[0076] S330. Determine the first vector information and the second vector information based on the target end point information. The first vector information is used to guide the obstacle avoidance model to identify the reference image patches that need to be concerned in each current image patch when predicting the UAV obstacle avoidance movement at the current moment, and the second vector information is used to guide the obstacle avoidance model to identify the UAV movement direction that needs to be concerned when predicting the UAV obstacle avoidance movement at the current moment.

[0077] Among them, the target end - point information guides the obstacle - avoidance model to screen out image information with a preset correlation with the target end - point information from the current forward - moving image, which is used to predict the travel control parameters for the UAV to avoid obstacles and move towards the target end - point position at the next moment.

[0078] As an optional but non - limiting implementation solution, determining the first vector information based on the target end - point information includes the following steps B11 - B13:

[0079] Step B11: Determine the current travel position information of the UAV when traveling at the current moment.

[0080] Optionally, the GPS positioning system is used to obtain the longitude and latitude coordinates of the UAV on the earth, so as to obtain the current travel position information of the UAV when traveling at the current moment. At the same time, the inertial navigation system (INS) carried on the UAV can also be used. The accelerometer and gyroscope are used to measure the acceleration and angular velocity of the UAV, and the travel position information and attitude change of the UAV at the current moment are calculated through integral operation. In addition, visual positioning technology can be combined. Based on the current forward - moving image captured by the UAV and the pre - constructed map, the current travel position information of the UAV when traveling at the current moment is determined.

[0081] Step B12: Based on the target end - point information and the current travel position information, determine the weight vector of each current image block. The weight vector of the current image block is used to indicate the importance degree when the information contained in the current image block is used to guide the obstacle - avoidance model to predict the UAV's obstacle - avoidance travel.

[0082] Step B13: Determine the first vector information based on the weight vectors of each current image block.

[0083] The weight vector of the current image block can be a vector that assigns different weight values to each current image block by combining the current travel position of the UAV and the target end - point information. The weight vector of the current image block can start from the local angle of the current forward - moving image, guide some image blocks in the current forward - moving image, and according to the weight size, screen out reference image blocks containing obstacle details, terrain features, etc. that are valuable for obstacle - avoidance travel prediction, assist the obstacle - avoidance model to accurately identify the key local image information in each current image block, so that making more accurate obstacle - avoidance decisions by preferentially using the key reference image blocks, such as planning a reasonable avoidance path, adjusting the flight height and speed, etc.

[0084] The image in front of the current movement is divided into multiple current image blocks. The higher the weight corresponding to the weight vector of each current image block, the more important the information contained in the current image block is for the obstacle avoidance model to predict obstacle avoidance movement. This enables the obstacle avoidance model to screen out key features from a large amount of local image information for obstacle avoidance movement prediction, ignoring those unimportant background information, and thus processing image data more efficiently. During the flight of the drone, the weight vector of the current image block can guide the obstacle avoidance model to give priority to a detailed analysis of the local area within the current field of view for obstacle avoidance movement prediction.

[0085] The environment is dynamically changing, and the local situations faced by the drone at different times are also different. The image block weight vector can reflect these changes in real time. By adjusting the weights of each image block, the obstacle avoidance model can quickly adapt to the new local environment and focus more attention on the image information included in the reference image blocks that need to be concerned when making obstacle avoidance movement predictions. For example, when the drone enters a narrow street from an open area, the weight vector will be redistributed to focus more attention on the buildings on both sides of the street and possible obstacles, ensuring the safe flight of the drone in a complex dynamic environment.

[0086] Integrate the weight vectors of each current image block, such as by weighted summation, splicing, etc., to obtain the first vector information. This first vector information comprehensively reflects the importance distribution of all current image blocks in the process of the obstacle avoidance model making obstacle avoidance movement predictions, guiding the obstacle avoidance model to focus on the image blocks with higher weights, extracting key information from them for obstacle avoidance decision-making, and realizing that for the navigation task of the drone, the weights of each image block in the image in front of the movement are different. The image blocks adjacent to the direction of the target end point should have higher position weights, while the image blocks irrelevant to the drone navigation task should have lower position weights.

[0087] As an optional but non-limiting implementation solution, determine the second vector information based on the target end point information, including the following steps B21 - B23:

[0088] Step B21: Determine the current movement position information of the drone when it is moving at the current moment.

[0089] Step B22: Based on the target end point information and the current movement position information, determine the hidden vector of the target end point information at the current moment. The hidden vector of the target end point information at the current moment reflects the relative position of the target end point information with respect to the current movement position information, and the hidden vector of the target end point information at the current moment is used to indicate the magnitude of the role played by the image in front of the current movement when it is used to guide the obstacle avoidance model to perform drone obstacle avoidance movement prediction.

[0090] Step B23: Determine the hidden vector of the target end point information at the current moment as the second vector information.

[0091] The hidden vector of the target end point information at the current moment is the relative position information formed by combining the current traveling position of the UAV at the current moment with the target end point position. From the perspective of the global overall image, the hidden vector of the target end point information at the current moment determines the traveling direction of the UAV that the obstacle avoidance model needs to focus on when making obstacle avoidance traveling predictions, guiding the obstacle avoidance model to preferentially obtain the forward image information in the traveling direction of the UAV, thereby planning a general path towards the target end point position. At the same time, it ensures that during the local obstacle avoidance process, the UAV will not deviate from the overall traveling direction, achieving effective coordination between the global and local aspects.

[0092] The hidden vector of the target end point information at the current moment can provide a macroscopic direction guidance for the traveling of the UAV after the current moment, assisting the obstacle avoidance model to plan a general path from the current traveling position to the target end point position. The hidden vector of the target end point information at the current moment can guide the UAV to move forward in the direction of the target end point during the traveling process after the current moment, avoiding getting lost in the local area. The hidden vector of the target end point information at the current moment determines the traveling direction that the UAV should focus on when obtaining the forward image after the current moment. Based on the target end point information, the obstacle avoidance model can judge which forward image obtained from which traveling direction is the most critical for obstacle avoidance and traveling decisions, thereby guiding the priority processing of obstacles in that traveling direction to ensure that the UAV always moves towards the target. The hidden vector of the target end point information at the current moment can transmit the global target to the local decision-making process, so that the local obstacle avoidance actions will not deviate from the overall traveling direction.

[0093] Based on the target end point information and the current traveling position information, the obstacle avoidance model maps the relative position, distance, direction, etc. information of the target end point relative to the current traveling position to a low-dimensional vector space through specific algorithms (such as fully connected layers, attention mechanisms in neural networks), generating the hidden vector of the target end point information at the current moment. This hidden vector contains the relationship information between the target end point and the current position, as well as the importance information of the target end point information for the current obstacle avoidance decision.

[0094] The hidden vector of the target end point information at the current moment is used to indicate the magnitude of the role played by the current forward image in guiding the obstacle avoidance model to make UAV obstacle avoidance traveling predictions. For example, if the hidden vector indicates that the target end point is directly in front of the current position and the distance is relatively close, then the information in the directly forward area of the current forward image is more critical for the obstacle avoidance decision. The obstacle avoidance model will analyze the forward image information in the directly forward area based on this vector and plan an obstacle avoidance path towards the target end point.

[0095] S340. Based on the position encoding information, the first vector information, and the second vector information corresponding to each of the multiple current image patches, determine the travel control parameter information of the drone at the next moment through the obstacle avoidance model.

[0096] Specifically, the obstacle avoidance model takes the position encoding information, the first vector information, and the second vector information corresponding to each of the multiple current image patches as inputs. Combining the position encoding information and the first vector information, the obstacle avoidance model performs weighted processing on each image patch to highlight the features of the reference image patches that need to be focused on. Then, the processed image patch feature vectors are fused with the second vector information, comprehensively considering the local key image patches and the global travel direction. The obstacle avoidance model performs feature extraction and calculation through the internal neural network layers, and uses the trained parameters and algorithms to predict the travel control parameters required for the drone to avoid obstacles and travel towards the target end position at the next moment, such as flight speed, yaw angle, etc.

[0097] With the above solution, multi-source information is integrated to achieve precise decision-making. Specifically, by fusing the position encoding, the first vector, and the second vector information, the model fully considers the position of the image patches, the local key information, and the global travel direction, and can analyze the current environment and target requirements more comprehensively and accurately, so as to predict reasonable travel control parameters and ensure the safe and efficient obstacle avoidance of the drone and its travel towards the target end.

[0098] As an optional but non-limiting implementation solution, refer to Figure 4 , in the obstacle avoidance model, there is a first multi-layer perceptron for determining the first vector information based on the current travel position information and a second multi-layer perceptron for determining the second vector information based on the current travel position information, where the current travel position information is the travel position information of the drone when traveling at the current moment.

[0099] As an optional but non-limiting implementation solution, in the obstacle avoidance model, there is a first multi-layer perceptron for obtaining the weight vector of each current image patch based on the current travel position information and the target end information, and a second multi-layer perceptron for obtaining the hidden vector of the target end information at the current moment.

[0100] Refer to Figure 4 , the multi-layer perceptron (MLP) configured in the obstacle avoidance model is a feedforward neural network, which consists of an input layer, a hidden layer, and an output layer. Information is transmitted between neurons through connections with weights. The information is processed sequentially through the hidden layer from the input layer, and finally a result is generated at the output layer. During the training process of the obstacle avoidance model, the connection weights between neurons are continuously adjusted through the backpropagation algorithm to make the output of the obstacle avoidance model as close as possible to the expected result.

[0101] Refer to Figure 4, in the obstacle avoidance model of the solution of this application, the first multi-layer perceptron receives the current moving position information and the target end point information as inputs, and these two pieces of information are encoded into a vector form suitable for network input. For example, the current moving position information can be converted into a numerical vector through methods such as GPS coordinates, and the target end point information is also represented in numerical forms such as coordinates and distances. The input layer passes this information to the hidden layer, and the neurons in the hidden layer perform weighted summation on the inputs and perform non-linear transformation through activation functions (such as ReLU, Sigmoid, etc.). After being processed by multiple hidden layers, data features are continuously extracted and abstracted. Finally, the output layer outputs a weight vector for each current image block. Each element in this weight vector represents the importance degree of the corresponding current image block when guiding the obstacle avoidance model to predict the obstacle avoidance movement of the drone.

[0102] See Figure 4 , in the obstacle avoidance model of the solution of this application, the second multi-layer perceptron receives the current moving position information and the target end point information as inputs. Similar to the first multi-layer perceptron, these pieces of information are first converted into a suitable vector form and input into the network. Inside the network of the second multi-layer perceptron, through the successive processing of the hidden layer, in-depth feature extraction and abstraction are performed on the relative position, distance, direction, etc. between the current moving position and the target end point. Finally, the output layer outputs a hidden vector, and this hidden vector reflects key information such as the relative position of the target end point information relative to the current moving position information. For example, the hidden vector can encode information such as the target end point being in the northeast direction of the current position and several meters away. This hidden vector serves as the second vector information and is used to indicate the magnitude of the role played by the current forward image when guiding the obstacle avoidance model to predict the obstacle avoidance movement of the drone. For instance, if the hidden vector indicates that the target end point is in a relatively close position in the front left of the current position, then in the current forward image, the image information in the front left region is more crucial for the obstacle avoidance model to plan a path to move forward in the front left direction, and the obstacle avoidance model will focus on and analyze the image content in this region based on this hidden vector.

[0103] As an optional but non-limiting implementation solution, based on the position encoding information, the first vector information, and the second vector information corresponding to each of the multiple current image blocks, the obstacle avoidance model determines the movement control parameter information of the drone at the next moment, including the following steps C1 - C3:

[0104] Step C1: Based on the position encoding information corresponding to each current image block among the position encoding information corresponding to each of the multiple current image blocks and the weight vector of each current image block, determine the updated result of the position encoding information of each current image block.

[0105] Step C2: Input the updated position encoding information of each current image block and the hidden vector of the target end point information at the current moment into the reference encoder configured in the obstacle avoidance model. The reference encoder is an encoder based on the attention mechanism architecture.

[0106] Step C3: Input the output result of the reference encoder into the third multi-layer perceptron configured in the obstacle avoidance model, and output the travel control parameter information that the drone needs to load and use at the next moment through the third multi-layer perceptron.

[0107] The position encoding information corresponding to the current image block is a digital representation of the position of the current image block in the current forward-moving image, which can enable the obstacle avoidance model to perceive the spatial position of the image block and assist the obstacle avoidance model in understanding the image structure and the relationship between the local and the global. The updated result of the position encoding information of the current image block is obtained by performing a specific operation on the position encoding information of the current image block and the weight vector, taking into account both the position of the image block and the importance of its information.

[0108] In an optional example, see Figure 4 , for each current image block, first obtain its corresponding position encoding information and weight vector. The position encoding information can be a vector generated by means such as sine and cosine functions, representing the position of the image block in the image, and the weight vector is generated by the first multi-layer perceptron according to the current travel position information and the target end point information. Then, perform a multiplication operation on the position encoding information and the weight vector, or obtain the updated result of the position encoding information corresponding to the current image block through other fusion methods (such as weighted summation).

[0109] Optionally, see Figure 4 , the reference encoder can be an encoder based on the attention mechanism architecture. For example, the reference encoder is an encoder based on the Transformer architecture. The third multi-layer perceptron is a neural network component in the obstacle avoidance model, which receives the output result of the reference encoder and outputs the travel control parameter information of the drone at the next moment by further processing and mapping the output result of the reference encoder.

[0110] In an optional example, see Figure 4, combine the updated position encoding information results of all current image patches into a matrix or a sequence of vectors, and use them together with the hidden vector of the target end point information at the current moment as the input of the reference encoder. Based on the attention mechanism, when processing the input information, the reference encoder calculates the degree of association (i.e., attention weights) between the updated result of each position encoding information and the hidden vector. For the updated results of position encoding information that are closely associated with the target end point information, higher attention weights will be assigned, so that these key information will be more prominent during the encoding process. For example, the attention weights are obtained by calculating the dot product, using scaled dot product attention, etc., and then operations such as weighted summation are performed on the updated results of the position encoding information to complete the encoding process.

[0111] In an optional example, refer to Figure 4 , the output result of the reference encoder contains the key feature information processed by the attention mechanism. Input these feature information into the input layer of the third multi-layer perceptron. The third multi-layer perceptron further performs non-linear transformation and feature extraction on the input features through multiple hidden layers. Neurons in each hidden layer perform weighted summation on the input and process it through an activation function. After complex processing through multiple hidden layers, the output layer outputs the travel control parameter information required by the drone at the next moment according to the trained parameters and the model structure.

[0112] Exemplarily, refer to Figure 4 , divide the current image in front of the travel into multiple current image patches, input the current image patches into the linear projection of flattened patches to obtain the position encoding information of each current image patch; furthermore, input the target end point information into the first multi-layer perceptron MLP and the second multi-layer perceptron MLP. The output of the first multi-layer perceptron MLP is the weight vector for each image patch, and the output of the second multi-layer perceptron MLP is the hidden vector of the target end point information at the current moment. Multiply the position encoding corresponding to each image patch by the corresponding weight vector. Then input the position encoding result after multiplying by the weight vector and the hidden vector of the target end point information at the current moment into the Transformer Encoder to obtain the output result, and input the output result into the third multi-layer perceptron MLP to obtain the speed and yaw angle of the drone. The drone heads towards and reaches the end position according to the speed and yaw angle.

[0113] By fusing the position encoding information of the image patches and the importance information of the position encoding information in obstacle avoidance prediction, subsequent processing can pay more attention to the position features of important image patches, providing more accurate position-related information for accurate obstacle avoidance decisions. Moreover, the reference encoder of the attention mechanism can automatically focus on the key image patch information related to the target end point, effectively extracting the most valuable features for obstacle avoidance decisions, enhancing the model's ability to process important information, and improving the pertinence and effectiveness of information processing; in addition, the third multi-layer perceptron uses its powerful non-linear mapping ability to convert the feature information output by the reference encoder into specific travel control parameters, realizing the conversion from environmental information and target information to the actual control commands of the UAV.

[0114] Based on the above embodiments, when training the obstacle avoidance model, this solution uses a simulated scenario with accurate three-dimensional reconstruction, and conducts reinforcement learning training on the obstacle avoidance model in the simulated scenario with three-dimensional reconstruction to reduce the sim2real gap. Moreover, in order to improve the generalization of the obstacle avoidance model, the starting point and end point positions of the UAV are both random. Finally, the UAV achieves a success rate of more than 90% in the simulated scenario and can be transferred to the real scenario to reach the end point with a probability of more than 80%.

[0115] S350. Control the UAV to perform obstacle avoidance travel at the next moment based on the travel control parameter information.

[0116] The technical solution of the embodiment of the present invention, in addition to achieving the technical effects of the foregoing embodiments, can also divide the image in front of the current travel into multiple image patches and determine the position encoding information in this embodiment, enabling the obstacle avoidance model to carefully perceive the position of each part of the image in the global context; the first vector information guides the model to focus on the key image patches based on the target end point information from the local image patch level, and the second vector information determines the travel direction from a global perspective, enabling the UAV to maintain its orientation towards the target while avoiding obstacles. The two work together to ensure that the decision-making takes into account both local safety and global goals; furthermore, by integrating the position encoding information, the first vector information, and the second vector information, the obstacle avoidance model can comprehensively analyze the current environment and target requirements, accurately predict the travel control parameters. In the face of complex and changing flight environments, such as different scenarios and obstacle situations, the obstacle avoidance model can flexibly adjust its decisions, improving the UAV's ability to cope with complex environments, ensuring flight safety, and reducing unnecessary computational workload through block processing of images and targeted information screening, avoiding ineffective processing of a large amount of irrelevant information. With limited computing resources, the obstacle avoidance model can quickly focus on the key image information content, efficiently complete obstacle avoidance decisions, and enable the UAV to respond to environmental changes in real time.

[0117] Figure 5The figure is a schematic structural diagram of an obstacle avoidance and progress device for a drone provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation where the drone performs obstacle avoidance and progress in a limited space area, especially the situation where the drone performs obstacle avoidance and progress during inspection and shooting in areas such as substations. The obstacle avoidance and progress device of the drone can be implemented in the form of software and / or hardware, and is generally integrated on any electronic device with network communication functions. The electronic device can be a mobile terminal, a PC terminal, a server, etc.

[0118] As Figure 5 shown, the obstacle avoidance and progress device of the drone in the embodiment of the present invention may include the following:

[0119] A first determination module 510, configured to determine the target end point information of the drone, and determine the current forward image when the drone is moving at the current moment;

[0120] A second determination module 520, configured to determine the movement control parameter information of the drone at the next moment through an obstacle avoidance model based on the target end point information and the current forward image. The target end point information guides the obstacle avoidance model to screen out image information having a preset correlation with the target end point information from the current forward image, for predicting the movement control parameters for the drone to perform obstacle avoidance and progress towards the target end point position at the next moment. The obstacle avoidance model is a model obtained through reinforcement learning that enables the drone to perform obstacle avoidance and progress. The target end point information is used to determine a first vector information and a second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be concerned in each current image block when predicting the obstacle avoidance and progress of the drone at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the drone movement direction that needs to be concerned when predicting the obstacle avoidance and progress of the drone at the current moment. Each current image block is generated by dividing the current forward image;

[0121] A control module 530, configured to control the drone to perform obstacle avoidance and progress at the next moment based on the movement control parameter information.

[0122] Based on the above embodiment, optionally, determining the current forward image when the drone is moving at the current moment includes:

[0123] During the flight of the drone, use the fisheye camera on the drone to capture an image towards the forward direction of the drone at the current moment, and obtain the current forward image when the drone is moving at the current moment.

[0124] Based on the above embodiment, optionally, determining the movement control parameter information of the drone at the next moment through an obstacle avoidance model based on the target end point information and the current forward image includes:

[0125] Based on the current forward image, determine the position encoding information corresponding to each of multiple current image blocks through an obstacle avoidance model. The multiple current image blocks are generated by dividing the current forward image, and the position encoding information corresponding to each current image block is used to represent the position information of the current image block in the current forward image;

[0126] Based on the target end point information, determine a first vector information and a second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be concerned in each of the current image blocks when predicting the obstacle avoidance movement of the drone at the current moment, and the second vector information is used to guide the obstacle avoidance model to identify the drone movement direction that needs to be concerned when predicting the obstacle avoidance movement of the drone at the current moment;

[0127] Based on the position encoding information corresponding to each of the multiple current image blocks, the first vector information, and the second vector information, determine the movement control parameter information of the drone at the next moment through the obstacle avoidance model.

[0128] Based on the above embodiments, optionally, determining the position encoding information corresponding to each of multiple current image blocks through an obstacle avoidance model based on the current forward image includes:

[0129] Divide the current forward image into multiple current image blocks of the same size, and assign the position information of each current image block in the current forward image by means of embedding position encoding.

[0130] Based on the above embodiments, optionally, determining the first vector information based on the target end point information includes:

[0131] Determine the current movement position information of the drone when moving at the current moment;

[0132] Based on the target end point information and the current movement position information, determine the weight vector of each current image block. The weight vector of the current image block is used to indicate the importance degree when the information contained in the current image block is used to guide the obstacle avoidance model to predict the obstacle avoidance movement of the drone;

[0133] Determine the first vector information based on the weight vectors of each of the current image blocks.

[0134] Based on the above embodiments, optionally, determining the second vector information based on the target end point information includes:

[0135] Determine the current movement position information of the drone when moving at the current moment;

[0136] Based on the target end - point information and the current moving position information, determine the hidden vector of the target end - point information at the current moment. The hidden vector of the target end - point information at the current moment reflects the relative position of the target end - point information with respect to the current moving position information. The hidden vector of the target end - point information at the current moment is used to indicate the magnitude of the role played by the image in front of the current movement when it is used to guide the obstacle - avoidance model for predicting the obstacle - avoidance movement of the UAV.

[0137] Determine the hidden vector of the target end - point information at the current moment as the second vector information.

[0138] Based on the above - mentioned embodiments, optionally, in the obstacle - avoidance model, a first multi - layer perceptron for obtaining the weight vector of each current image patch based on the current moving position information and the target end - point information, and a second multi - layer perceptron for obtaining the hidden vector of the target end - point information at the current moment are configured.

[0139] Based on the above - mentioned embodiments, optionally, based on the position - encoding information corresponding to each of the multiple current image patches, the first vector information, and the second vector information, determine the movement control parameter information of the UAV at the next moment through the obstacle - avoidance model, including:

[0140] Based on the position - encoding information corresponding to each current image patch in the position - encoding information corresponding to each of the multiple current image patches and the weight vector of each current image patch, determine the updated result of the position - encoding information of each current image patch;

[0141] Input the updated results of the position - encoding information of each current image patch and the hidden vector of the target end - point information at the current moment into the reference encoder configured in the obstacle - avoidance model. The reference encoder is an encoder based on the attention mechanism architecture;

[0142] Input the output result of the reference encoder into the third multi - layer perceptron configured in the obstacle - avoidance model, and output the movement control parameter information that the UAV needs to load and use at the next moment through the third multi - layer perceptron.

[0143] In the technical solution of the embodiment of the present invention, the target end point information clarifies the task direction of the unmanned aerial vehicle (UAV), providing a target orientation for subsequent path planning. The current forward image in the traveling direction will real-time feedback the environmental conditions of the UAV. The target end point information guides the obstacle avoidance model to screen out the key image information closely related to the UAV's obstacle avoidance movement towards the target end point from the target end point information. By passing through the entire obstacle avoidance movement process of the UAV with the target end point information, the obstacle avoidance model can always prioritize the image information valuable for obstacle avoidance movement towards the end point, thereby accurately predicting the movement control parameters and ensuring that the UAV can avoid obstacles and move towards the target in a complex environment. At the same time, the current forward image in the traveling direction provides rich environmental information. Whether the size of the obstacle is large or small and in whatever scene, it can be captured by the image. When the obstacle avoidance model performs obstacle avoidance movement based on the forward image information in the traveling direction, it will not be restricted by the scene and size differences of the obstacles. Moreover, for the end-to-end obstacle avoidance model obtained through reinforcement learning training, during the training process, it will come into contact with a large number of samples of different scenes and obstacles, learn obstacle avoidance strategies in various situations, endow the obstacle avoidance model with strong generalization ability, enabling it to cope with new scenes and obstacles of different sizes that it has never seen before, ensuring that the model continuously optimizes decisions according to the real-time image and target end point information based on the actual situation of the obstacles in different environments and tasks. The UAV can autonomously adapt to various flight conditions without manual intervention, continuously adjust movement control parameters such as flight speed and direction, and flexibly make obstacle avoidance decisions.

[0144] The obstacle avoidance movement device of the UAV provided by the embodiment of the present invention can execute the obstacle avoidance movement method of the UAV provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the obstacle avoidance movement method of the UAV.

[0145] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiment of the present invention.

[0146] Figure 6 It is a schematic structural diagram of an electronic device for implementing the obstacle avoidance movement method of a UAV provided by an embodiment of the present invention. Referring below to Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing the embodiment of the present invention (such as Figure 6 the terminal device or server in). The terminal device in the embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6The electronic device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

[0147] As Figure 6 shown, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0148] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device 600 to communicate with other devices wirelessly or wiredly to exchange data.

[0149] Specifically, according to the embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present invention include a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the obstacle avoidance traveling method of the drone shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above functions defined in the obstacle avoidance traveling method of the drone in the embodiments of the present invention are executed.

[0150] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and do not limit the scope of these messages or information.

[0151] The electronic device provided by the embodiments of the present invention and the obstacle avoidance traveling method of the drone provided by the above embodiments belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0152] An embodiment of the present invention provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the obstacle avoidance traveling method of the drone provided in the above embodiment is implemented.

[0153] It should be noted that the computer-readable medium in the present invention may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or combined with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0154] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0155] The above computer-readable medium may be included in the above electronic device; or it may exist separately and not be assembled into the electronic device.

[0156] The above computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: determine the target end point information of the unmanned aerial vehicle (UAV), and determine the current forward image when the UAV is traveling at the current moment; based on the target end point information and the current forward image, determine the travel control parameter information of the UAV at the next moment through an obstacle avoidance model, where the target end point information guides the obstacle avoidance model to screen out image information having a preset correlation with the target end point information from the current forward image for predicting the travel control parameters for the UAV to avoid obstacles and travel towards the target end point position at the next moment; and control the UAV to avoid obstacles and travel at the next moment based on the travel control parameter information.

[0157] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0159] The units involved in the embodiments of the present invention can be implemented in software or in hardware. In some cases, the name of a unit does not constitute a limitation on the unit itself.

[0160] The functions described above herein can be performed, at least in part, by one or more hardware logic components. By way of example, and without limitation, the types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGA), Application Specific Integrated Circuits (ASIC), Application Specific Standard Products (ASSP), System on a Chip (SOC), Complex Programmable Logic Devices (CPLD), and the like.

[0161] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] The above description is only a preferred embodiment of the present invention and an illustration of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present invention.

[0163] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Certain features described in the context of separate embodiments can also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0164] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A method for avoiding obstacles in a drone, characterized in that: The method comprises: Determine the target destination information of the drone, and determine the current forward image of the drone when it is moving at the current moment; Based on the target endpoint information and the current image ahead, the obstacle avoidance model is used to determine the travel control parameter information of the drone at the next moment. The target endpoint information guides the obstacle avoidance model to screen out image information that has a preset correlation with the target endpoint information from the current image ahead, so as to predict the travel control parameters for the drone to avoid obstacles when traveling toward the target endpoint position at the next moment. The obstacle avoidance model is a model that enables the drone to avoid obstacles obtained through reinforcement learning training. The target endpoint information is used to determine the first vector information and the second vector information. The first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the drone's obstacle avoidance at the current moment. The second vector information is used to guide the obstacle avoidance model to identify the drone's travel direction that needs to be paid attention to when predicting the drone's obstacle avoidance at the current moment. Each current image block is generated by dividing the current image ahead. Based on the travel control parameter information, the drone is controlled to avoid obstacles at the next moment.

2. The method according to claim 1, characterized in that Based on the target endpoint information and the current forward image, the travel control parameter information of the UAV at the next moment is determined by the obstacle avoidance model, including: Based on the current image ahead, determine position coding information corresponding to each of a plurality of current image blocks through an obstacle avoidance model, wherein the plurality of current image blocks are generated by dividing the current image ahead, and the position coding information corresponding to each of the current image blocks is used to represent position information of the current image block in the current image ahead; Determine first vector information and second vector information based on the target endpoint information; Based on the position coding information corresponding to each of the multiple current image blocks, the first vector information and the second vector information, the travel control parameter information of the drone at the next moment is determined through the obstacle avoidance model.

3. The method according to claim 2, characterized in that Based on the current forward image, the obstacle avoidance model is used to determine the position coding information corresponding to each of the multiple current image blocks, including: The current moving ahead image is divided into a plurality of current image blocks of the same size, and position information of each current image block in the current moving ahead image is given by embedding position coding.

4. The method according to claim 2, characterized in that: Determining first vector information based on the target endpoint information includes: Determine the current travel position information of the UAV at the current moment; Based on the target endpoint information and the current travel position information, determine a weight vector for each current image block, wherein the weight vector for the current image block is used to indicate the importance of the information contained in the current image block in guiding the obstacle avoidance model to perform obstacle avoidance travel prediction for the drone; The first vector information is determined based on the weight vectors of each of the current image blocks.

5. The method according to claim 2, characterized in that: Determining second vector information based on the target endpoint information includes: Determine the current travel position information of the UAV at the current moment; Based on the target endpoint information and the current travel position information, determine the latent vector of the target endpoint information at the current moment, the latent vector of the target endpoint information at the current moment reflects the relative position of the target endpoint information relative to the current travel position information, and the latent vector of the target endpoint information at the current moment is used to indicate the role played by the current forward image in guiding the obstacle avoidance model to perform obstacle avoidance travel prediction for the UAV; The latent vector of the target endpoint information at the current moment is determined as the second vector information.

6. The method according to claim 2, characterized in that The obstacle avoidance model is configured with a first multilayer perceptron for determining first vector information based on current travel position information and a second multilayer perceptron for determining second vector information based on current travel position information, wherein the current travel position information is the travel position information of the drone at the current moment.

7. The method according to claim 2, characterized in that Based on the position coding information corresponding to each of the multiple current image blocks, the first vector information and the second vector information, the traveling control parameter information of the UAV at the next moment is determined by the obstacle avoidance model, including: Determine an update result of the position coding information of each current image block based on the position coding information corresponding to each current image block in the position coding information corresponding to each of the multiple current image blocks and a weight vector of each current image block; Inputting the updated result of the position encoding information of each current image block and the latent vector of the target endpoint information at the current moment into a reference encoder configured in the obstacle avoidance model, wherein the reference encoder is an encoder based on an attention mechanism architecture; The output result of the reference encoder is input into the third multi-layer perceptron configured in the obstacle avoidance model, and the third multi-layer perceptron is used to output the travel control parameter information that the drone needs to load and use at the next moment.

8. An obstacle avoidance device for a drone, characterized in that: The device comprises: The first determination module is used to determine the target end point information of the drone and determine the current forward image of the drone when it is moving at the current moment; a second determination module, for determining the travel control parameter information of the UAV at the next moment through the obstacle avoidance model based on the target endpoint information and the current image ahead, the target endpoint information guiding the obstacle avoidance model to screen out image information having a preset correlation with the target endpoint information from the current image ahead, for predicting the travel control parameters for the UAV to avoid obstacles when traveling toward the target endpoint position at the next moment, the obstacle avoidance model is a model obtained through reinforcement learning training that enables the UAV to avoid obstacles, the target endpoint information is used to determine the first vector information and the second vector information, the first vector information is used to guide the obstacle avoidance model to identify the reference image blocks that need to be paid attention to in each current image block when predicting the obstacle avoidance of the UAV at the current moment, the second vector information is used to guide the obstacle avoidance model to identify the direction of the UAV that needs to be paid attention to when predicting the obstacle avoidance of the UAV at the current moment, and each current image block is generated by dividing the current image ahead; A control module is used to control the UAV to avoid obstacles at the next moment based on the travel control parameter information.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the obstacle avoidance method for the drone described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the obstacle avoidance method of the drone described in any one of claims 1-7 when executed by a computer processor.

Images