Unmanned aerial vehicle high-altitude autonomous return flight obstacle avoidance method and system applied to mobile airport

Abstract

The invention provides an unmanned aerial vehicle high-altitude autonomous homeward voyage obstacle avoidance method and system applied to a mobile airport, and aims to solve the problem that obstacles possibly exist in a complex and changeable homeward voyage route of the mobile airport, whether the obstacles exist in front or not can be remotely judged in advance, and if the obstacles exist, an obstacle avoidance strategy needs to be autonomously selected. The obstacle avoidance strategy of the unmanned aerial vehicle during high-altitude autonomous return flight is realized, and the safety of the unmanned aerial vehicle is ensured. The high-altitude autonomous return flight obstacle avoidance of the unmanned aerial vehicle refers to that the unmanned aerial vehicle needs to return to a new coordinate position after the mobile airport moves to a new return flight point during mobile operation of the mobile airport. In the course of returning of the unmanned aerial vehicle, safe returning can be comprehensively judged according to information such as the field environment, an obstacle avoidance strategy is given, and high-altitude autonomous returning of the unmanned aerial vehicle is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of obstacle avoidance for unmanned aerial vehicles (UAVs), and in particular relates to a method and system for high-altitude autonomous return and obstacle avoidance for UAVs applied to mobile airports. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Drone inspections are characterized by high efficiency, high precision, and high safety. Drone airport automated inspections have become a relatively common automated inspection method. Traditional drone airports are fixed airports that need to be installed in fixed locations and can only inspect areas within the inspection radius. Although this gets rid of the problem that inspectors need to master drone operation skills, it also increases the limitations of the inspection area to a certain extent. Therefore, mobile airport products have been derived. Mobile airports have realized the mobility of airports and got rid of the limitations of traditional fixed airport inspection areas. At the same time, the mobile airport's off-site take-off and landing function (taking off from point A, the mobile airport moves to point B and waits for the drone to land at point B) has also greatly improved the inspection efficiency of mobile airports. However, there are still some problems when returning and landing in a different location.

[0004] The return path of the drone will be different from the path it flies to the inspection site as the mobile airport moves. This means that there is a possibility that the drone will crash due to obstacles blocking the return path. If the drone simply follows the navigation without avoiding obstacles, it is very likely to crash when the altitude is not enough or there is a problem with human operation.

[0005] For the high-altitude return site environment of drones used in mobile airports, the existing solutions still have certain problems in this application scenario. The existing obstacle avoidance solutions are generally applied to a certain fixed site environment, and there is no universal solution suitable for all scenarios.

[0006] The following is an example. The Chinese invention patent for a method and system for visual obstacle avoidance of unmanned aerial vehicles (CN116540776A) is equipped with external devices. This practice will reduce the endurance of the unmanned aerial vehicle, especially for small unmanned aerial vehicles used in mobile airports. Its own load-bearing capacity is weak, and the additional external devices have a greater impact on the endurance. The Chinese invention patent for a heterogeneous unmanned aerial vehicle visual obstacle avoidance system and method based on NeRF neural network (CN115826628B) is equipped with a laser radar at the same time as a camera, which is not suitable for small unmanned aerial vehicles. In addition, the main obstacle avoidance solution of this system method relies on laser radar to solve the problem of weak visual obstacle avoidance ability caused by unclear images under direct sunlight, and is not suitable for the high-altitude return scene of unmanned aerial vehicles at mobile airports. The Chinese invention patent for a method for visual obstacle avoidance of unmanned aerial vehicles based on point cloud space binarization (CN108764080B) requires the three-dimensional laser point cloud in the scene to be scanned in advance, and then compared with the image of the drone binocular camera, which is not suitable for changing environments. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a method and system for autonomous high-altitude return and obstacle avoidance of a UAV applied to a mobile airport. For a mobile airport, there may be obstacles on the complex and changeable return route. It can judge in advance from a long distance whether there are obstacles ahead. If there are obstacles, it is necessary to autonomously select an obstacle avoidance strategy, thereby realizing the obstacle avoidance strategy of the UAV during autonomous high-altitude return and ensuring the safety of the UAV.

[0008] The high-altitude autonomous return and obstacle avoidance of the UAV of the present invention means that during the mobile operation of the mobile airport, after the mobile airport moves to a new return point, the UAV needs to return to the new coordinate position. During the return of the UAV, it can be comprehensively judged whether it can return safely based on the on-site environment and other information, and an obstacle avoidance strategy can be given to realize the high-altitude autonomous return of the UAV.

[0009] According to some embodiments, the present invention adopts the following technical solutions:

[0010] A method for autonomously returning to flight at high altitude and avoiding obstacles for a drone comprises the following steps:

[0011] Obtain images taken by drones;

[0012] Set the captured image into rectangles and divide it into grids;

[0013] Process the image to obtain the grayscale value of each segmented area;

[0014] Preliminarily determine the location and size of obstacles based on the grayscale value and control the flight altitude of the drone;

[0015] When the grayscale value of the segmented area corresponding to the front of the UAV changes, the optimal path planning is determined according to the Bayesian decision, and the optimal obstacle avoidance path of the UAV is adjusted.

[0016] As an optional implementation, the drone is provided with a camera, and the camera has a built-in visual module for acquiring the shot video and capturing it as an image.

[0017] As an optional implementation, the process of rectangular setting and grid segmentation of the captured image includes correcting the image and segmenting the image into n*m multi-grid areas according to a preset grid size, where n and m are positive integers.

[0018] As an optional implementation, the location and size of the obstacle are preliminarily determined based on the grayscale value, and the process of controlling the flight altitude of the drone includes:

[0019] If the grayscale value of the segmented area at the bottom is greater than the grayscale value of the segmented area at the top, and the difference exceeds the set value, it is considered that there is an obstacle at the bottom and the flight altitude of the drone needs to be increased.

[0020] As an optional implementation method, the process of determining the optimal path planning based on Bayesian decision making and adjusting the optimal obstacle avoidance path of the UAV flight includes:

[0021] Taking the drone as a Bayesian cell, the probability of candidate nodes is determined within the flyable area;

[0022] Estimate the distance between each candidate node and obstacles in the flyable area;

[0023] According to the estimated distance between the candidate node and the obstacle, determine whether the corresponding candidate node can safely avoid the obstacle;

[0024] The posterior probability of the candidate node is calculated according to the judgment result, and the candidate node finally selected is determined according to the posterior probability.

[0025] As a further step, the process of estimating the distance between each candidate node and the obstacle in the flyable area includes:

[0026] For the feasible region N1(S n ) any candidate node W i , assuming that the visual angle of the image obtained by the drone camera is the predictable area of ​​the candidate node, the threshold is set according to the grayscale value, and the image is divided into foreground and background using the histogram-based threshold method. The foreground is the obstacle, and the distance is estimated using the estimation method based on visual time difference.

[0027] As a further example, the distance is the product of the distance moved by the drone along the flight direction between two frames of images and the focal length of the camera, divided by the horizontal displacement of the object in the image.

[0028] As a further step, the process of judging whether the corresponding candidate node can safely avoid the obstacle according to the estimated distance between the candidate node and the obstacle includes:

[0029]

[0030] Where D0 is the set flight safety distance, dist(W i ,obstacle) represents the distance between the candidate node and the obstacle. The smaller the distance value is, the smaller the conditional probability value is;

[0031] The conditional probability is:

[0032]

[0033] Among them, G represents the planning event.

[0034] As a further step, the posterior probability of the Bayesian cell candidate node is:

[0035]

[0036] A high-altitude autonomous return-to-home obstacle avoidance system for unmanned aerial vehicles, comprising:

[0037] An image acquisition module is configured to acquire images captured by the drone;

[0038] An image segmentation module is configured to set a rectangle on the captured image and segment it into a grid;

[0039] A grayscale processing module is configured to process the image to obtain a grayscale value of each segmented area;

[0040] The preliminary obstacle avoidance module is configured to preliminarily determine the location and size of obstacles based on the grayscale value and control the flight altitude of the UAV;

[0041] The optimal obstacle avoidance module is configured to determine the optimal path planning based on Bayesian decision making when the grayscale value of the segmented area corresponding to the front of the drone changes, and adjust the optimal obstacle avoidance path for the drone to fly.

[0042] As an optional implementation, the system is deployed on an edge computing module, and the module is provided on a drone.

[0043] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0044] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention innovatively provides a method for autonomous high-altitude return and obstacle avoidance of a UAV, and adopts a grayscale value judgment method to deal with the high-altitude return and obstacle avoidance problem. The background environment of the high-altitude image is single, and the grayscale value scheme algorithm is more efficient and has better effect. The Bayesian decision algorithm framework is used to determine the optimal obstacle avoidance route of the UAV. Different from the previous mechanical methods of raising the height to avoid obstacles and circumventing obstacles, the obstacle avoidance strategy is more flexible and efficient.

[0047] The present invention innovatively provides a high-altitude autonomous return and obstacle avoidance system for unmanned aerial vehicles. It uses the camera built into the unmanned aerial vehicle with low hardware cost. Adding an edge computing module on the unmanned aerial vehicle side can process video stream data in real time without uploading all data to the cloud for processing, thereby reducing network bandwidth consumption, reducing latency and improving privacy and security.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 is a schematic diagram of image segmentation in an embodiment;

[0051] Figure 2 is a grayscale schematic diagram of an image of an embodiment;

[0052] Figure 3 The diagram is a flow chart of the optimal obstacle avoidance path selection process according to an embodiment. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0054] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0056] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0057] Embodiment 1

[0058] A method for autonomously returning to the air and avoiding obstacles for a UAV at high altitude, such as Figure 3 As shown, the following steps are included:

[0059] Obtain images taken by drones;

[0060] Set the captured image into rectangles and divide it into grids;

[0061] Process the image to obtain the grayscale value of each segmented area;

[0062] Preliminarily determine the location and size of obstacles based on the grayscale value and control the flight altitude of the drone;

[0063] When the grayscale value of the segmented area corresponding to the front of the UAV changes, the optimal path planning is determined according to the Bayesian decision, and the optimal obstacle avoidance path of the UAV is adjusted.

[0064] This embodiment relies on the built-in visual module of the drone's camera and the built-in edge computing chip of the drone to obtain images captured by the drone's camera video stream, and automatically plans the drone's obstacle avoidance path based on the Bayesian decision algorithm to determine obstacles ahead when returning and avoid them autonomously.

[0065] Here are the specific steps:

[0066] 1. Get the image

[0067] Adding an edge computing module on the drone side can process video stream data in real time without uploading all data to the cloud for processing, which can reduce network bandwidth consumption, reduce latency, and improve privacy security. Some high-performance computing modules are suitable for complex artificial intelligence and machine learning tasks, such as Jetson Xavier NX of the NVIDIA Jetson series. The edge computing module obtains images from the drone camera by accessing the camera, reading the video stream, and capturing images from the video frame.

[0068] 2. Image Processing

[0069] First, the image taken by the drone is set as a rectangle and its grid is discretized. Figure 1 As shown in the figure, the pictures taken by the drone are discretized into 5*5 rectangular areas.

[0070] The front video stream image obtained by the drone and the image processing can obtain the grayscale values ​​of different areas of the image. The grayscale value can be used to determine whether there are obstacles ahead.

[0071] In this embodiment, when the drone returns at a high altitude, it obtains a picture of the position directly in front. According to the application scenario, it can be considered that if there is an obstacle in front, its grayscale value is very high, because the sky is bright and its grayscale value is low. The grayscale value can clearly determine whether there is an obstacle in front.

[0072] like Figure 2 As shown in the figure, according to the actual high-altitude application of obstacle avoidance, the grayscale value image shows that the lower part of the image is darker, and the middle and upper parts are whiter. This is because when the drone returns from high altitude, the front view is wide, while the lower part is close to the ground.

[0073] Next is the specific process of selecting the optimal obstacle avoidance path:

[0074] 1. Determine the Bayesian prior probability

[0075] In the Bayesian cell (in this article, the cell refers to the drone) the flight area N1 (S n ), the probability of candidate nodes is determined based on the existing derived data. This probability is the prior probability in the optimal path selection, and the formula is:

[0076] Indicates that the Bayesian cell can fly to area N1(S n ) within the pheromone.

[0077] The Bayesian algorithm is a traditional machine learning algorithm that uses prior probabilities to derive posterior probabilities. Before a machine learning algorithm is used, a training set and a test set are usually required. The algorithm is trained with the training set, and the algorithm effect is tested with the test set.

[0078] In this embodiment, it is necessary to take images from a high altitude and process them as a training set, that is, existing derived data, which should include images with obstacles and without obstacles mixed in a certain proportion. The trained Bayesian algorithm can be used in practice to determine the probability of obstacles in different segmentation frames based on the images.

[0079] 2. Calculate the distance between the candidate node and the obstacle

[0080] For the feasible region N1(Sn ) any candidate node W i , the visual angle of the image obtained by the drone camera is assumed to be the predictable area of ​​the candidate node. The threshold is set according to the gray value, and the image is segmented into foreground (obstacle) and background using the threshold method based on histogram.

[0081] In this embodiment, a global threshold method may be used to segment the image into a foreground and a background.

[0082] Step 1: Use relevant image processing algorithm libraries in Python or other languages.

[0083] Step 2: Convert the color image to a grayscale image. This can usually be achieved by calculating the weighted average of the red, green, and blue components of each pixel. The formula used in this embodiment is: I = 0.299*R + 0.587*G + 0.114*B, where R, G, and B represent the red, green, and blue components respectively.

[0084] Step 3: Use Otsu's method to automatically determine the optimal segmentation threshold.

[0085] Step 4: Apply global threshold segmentation and apply the selected threshold to the grayscale image to segment the image into foreground and background. Pixels above the threshold are marked as foreground, and pixels below the threshold are marked as background.

[0086] Next, the distance D is estimated using a visual time difference based estimation method.

[0087]

[0088] in:

[0089] D is the distance from the object to the drone.

[0090] b is the baseline length, which is the distance the drone moves along the flight direction between two frames.

[0091] f is the focal length of the camera.

[0092] Δx is the parallax of the object between the two frames, that is, the horizontal displacement of the object on the image (in pixels).

[0093] 3. Determine the conditional probability of the Bayesian cell candidate node

[0094] The distance between the candidate node and the obstacle is used to determine whether the candidate node can safely avoid the obstacle.

[0095]

[0096] Among them, mindist(W i ,obstacle) represents the candidate node Wi The minimum distance between the obstacle and the obstacle. The smaller the distance value, the greater the potential risk. The conditional probability can be obtained as:

[0097]

[0098] Among them, G represents the planning event, P(W i |G) represents the posterior probability.

[0099] 4. Determine the posterior probability of the Bayesian cell candidate node

[0100] The posterior probability of the Bayesian cell candidate node is:

[0101]

[0102] The posterior probability can be used to select alternative nodes within the flight area where the drone can fly.

[0103] In this embodiment, the result obtained by the naive Bayesian algorithm is in the form of probability value. For example: 0.9, 0.25, 0.66, etc. A limit value can be given to select the candidate node, and the maximum safe probability path is selected among those above the limit value. If there is no path above the limit value, it means that none of the paths ahead meet the requirements and it is necessary to hover and wait for rescue.

[0104] 5. Update drone execution information.

[0105] In summary, in the method of this embodiment, the drone first makes a judgment based on the grayscale value distribution of the acquired image segmentation area. If the grayscale value of the segmentation area on the lower side of the image is large, it means that the drone's flight altitude is close to the ground and may have encountered areas with large terrain fluctuations such as mountains and high-rise buildings. At this time, the drone needs to increase its flight altitude. If the grayscale value of the segmentation area in front of the drone suddenly changes, it means that there is an obstacle in front. The optimal path planning is determined based on the Bayesian decision, and the optimal obstacle avoidance path of the drone is adjusted.

[0106] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A method for autonomously returning to the air and avoiding obstacles at high altitudes for a UAV, characterized in that: The following steps are involved: Obtain images taken by drones; Set the captured image into rectangles and divide it into grids; Process the image to obtain the grayscale value of each segmented area; Preliminarily determine the location and size of obstacles based on the grayscale value and control the flight altitude of the drone; When the grayscale value of the segmented area corresponding to the front of the UAV changes, the optimal path planning is determined according to the Bayesian decision, and the optimal obstacle avoidance path of the UAV is adjusted.

2. The method for autonomously returning to high altitude and avoiding obstacles for a drone as claimed in claim 1, characterized in that: The drone is provided with a camera, and the camera has a built-in visual module for acquiring the shot video and capturing it as an image.

3. The high-altitude autonomous return and obstacle avoidance method for a UAV as claimed in claim 1 is characterized in that: The process of setting a rectangle for the captured image and dividing it into grids includes correcting the image and dividing the image into n*m multi-grid areas according to a preset grid size, where n and m are positive integers.

4. The method for autonomously returning to high altitude and avoiding obstacles for a drone as claimed in claim 1, characterized in that: The location and size of the obstacle are preliminarily determined based on the grayscale value. The process of controlling the flight altitude of the drone includes: If the grayscale value of the segmented area at the bottom is greater than the grayscale value of the segmented area at the top, and the difference exceeds the set value, it is considered that there is an obstacle at the bottom and the flight altitude of the drone needs to be increased.

5. The high-altitude autonomous return and obstacle avoidance method for a UAV as claimed in claim 1 is characterized in that: The process of determining the optimal path planning based on Bayesian decision making and adjusting the optimal obstacle avoidance path for UAV flight includes: Taking the drone as a Bayesian cell, the probability of candidate nodes is determined within the flyable area; Estimate the distance between each candidate node and obstacles in the flyable area; According to the estimated distance between the candidate node and the obstacle, determine whether the corresponding candidate node can safely avoid the obstacle; The posterior probability of the candidate node is calculated according to the judgment result, and the candidate node finally selected is determined according to the posterior probability.

6. A method for avoiding obstacles when autonomously returning to high altitude by a UAV as claimed in claim 5, characterized in that: The process of estimating the distance between each candidate node and obstacles in the flyable area includes: For the feasible region N1(S n ) any candidate node W i , assuming that the visual angle of the image obtained by the drone camera is the predictable area of ​​the candidate node, the threshold is set according to the gray value, and the image is divided into foreground and background using the threshold method based on the histogram. The foreground is the obstacle, and the distance is estimated using the estimation method based on visual time difference; The distance is the product of the distance moved by the drone along the flight direction between two frames of images and the focal length of the camera, divided by the horizontal displacement of the object in the image.

7. A method for avoiding obstacles when an unmanned aerial vehicle returns to high altitude autonomously as claimed in claim 5 or 6, characterized in that: According to the estimated distance between the candidate node and the obstacle, the process of judging whether the corresponding candidate node can safely avoid the obstacle includes: Where D0 is the set flight safety distance, dist(W i ,obstacle) represents the distance between the candidate node and the obstacle. The smaller the distance value is, the smaller the conditional probability value is; The conditional probability is: Where G represents the planning event.

8. The high-altitude autonomous return and obstacle avoidance method for a UAV as claimed in claim 5 is characterized by: The posterior probability of the Bayesian cell candidate node is:

9. A high-altitude autonomous return-to-home obstacle avoidance system for unmanned aerial vehicles, characterized in that: include: An image acquisition module is configured to acquire images captured by the drone; An image segmentation module is configured to set a rectangle on the captured image and segment it into a grid; A grayscale processing module is configured to process the image to obtain a grayscale value of each segmented area; The preliminary obstacle avoidance module is configured to preliminarily determine the location and size of obstacles based on the grayscale value and control the flight altitude of the UAV; The optimal obstacle avoidance module is configured to determine the optimal path planning based on Bayesian decision making when the grayscale value of the segmented area corresponding to the front of the drone changes, and adjust the optimal obstacle avoidance path for the drone to fly.

10. The high-altitude autonomous return-to-home obstacle avoidance system for unmanned aerial vehicles as claimed in claim 9, characterized in that: The system is deployed on an edge computing module, and the edge computing module is arranged on a drone.

Images