A method for autonomous detection and optimization of landing points based on drone perspective

Abstract

The present invention proposes an autonomous landing point detection and optimization method based on the perspective of a drone. First, the perspective transformation matrix of the drone's oblique perspective-orthogonal perspective is constructed with the help of the angle parameters of the drone's multi-axis gimbal to obtain an orthogonal perspective image. Then, the depth of the orthogonal perspective image is estimated to obtain a depth matrix. The "height-width" adaptive sliding window variance minimization selection algorithm is used to obtain the candidate area with the smallest depth value variance as the landing area reference plane. A dynamic threshold function is designed, and an adaptive plane growth strategy with dynamic threshold adjustment is adopted. The growth kernel size is adjusted in stages to achieve adaptive and precise division of the landing area in the process of step-by-step boundary convergence. This method can detect static unknown landing areas using a single drone perspective image, and has good robustness and accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robotics, and in particular to a method for autonomously detecting and optimizing a landing point based on the perspective of an unmanned aerial vehicle (UAV). Background Art

[0002] Rotary-wing unmanned aerial vehicles (UAVs) possess high autonomy, flexibility, and maneuverability, as well as payload capacity. They play a vital role in disaster assessment and search and rescue missions in the wild, as well as in urban traffic monitoring and power line inspections. In responding to natural disasters and emergencies, UAVs have, to a certain extent, overcome the limitations of traditional emergency rescue methods in terms of response speed and disposal costs, effectively opening up air rescue channels. For example, during field search and rescue operations, UAVs play a vital role in the transportation and delivery of emergency supplies. Automatically identifying the landing area for UAVs is crucial for unmanned, large-scale rescue and supply delivery missions.

[0003] For the autonomous identification of drone landing areas, existing technologies are mainly divided into three categories: vision-based, non-vision-based, and combined methods. The attributes of outdoor landing areas are divided into static known landing areas, static unknown landing areas, dynamic known landing areas, and dynamic unknown landing areas. For known landing areas, target guidance is usually used to guide the drone to land. Landing area detection methods for unknown areas include: 1) semantic map-based, key information such as trees and vehicles is identified before selecting the landing area; 2) filter transformation-based methods are used to directly obtain candidate flat areas; 3) multi-image matching is used to obtain elevation and depth information; 4) LiDAR point cloud data is used to obtain digital elevation maps for depth analysis.

[0004] However, existing methods for landing zone detection in static, unknown areas still face three core challenges: First, the demarcated landing area boundaries are rough, paying little attention to local terrain; second, existing methods mostly utilize machine learning methods such as Fourier transforms, Bayesian probability, and static threshold functions, which have poor robustness. Third, methods based on multi-view matching and LiDAR-based depth information acquisition are costly, primarily in terms of sensor and computing power costs, and affect the drone's flight time. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and propose an autonomous detection and optimization method for landing points based on the perspective of a drone. It uses a low-cost camera to realize autonomous detection of the optimal landing area of ​​a drone in an unknown scene, thereby improving the efficiency of unmanned delivery of rescue supplies in practical applications.

[0006] In order to achieve the above object, the technical solutions specifically adopted by the present invention are as follows:

[0007] A method for autonomously detecting and optimizing landing points based on the perspective of a drone includes the following steps:

[0008] Step 1: Obtain a single-frame RGB original image containing PTZ information;

[0009] Step 2: Perform perspective transformation on the original image according to the PTZ information to obtain an orthographic image;

[0010] Step 3: Use the depth estimation module to perform depth estimation on the orthographic image to obtain a depth matrix;

[0011] Step 4: Use the adaptive sliding window variance minimization selection algorithm to calculate the candidate area with the smallest depth variance and use it as the seed plane;

[0012] Step 5: Construct a dynamic threshold function, control the growth range of the seed plane according to the dynamic threshold function, and obtain the final landing area.

[0013] Preferably, the gimbal information includes the pitch angle, roll angle and calibrated camera intrinsic parameter matrix of the drone gimbal used to capture the original image.

[0014] Preferably, in step 2, a perspective transformation matrix of the oblique viewing angle to the orthographic viewing angle of the drone is constructed according to the gimbal information, and perspective transformation is performed using the perspective transformation matrix.

[0015] Preferably, step 2 further includes adjusting the image size, removing blank spaces and centering the image.

[0016] Preferably, the perspective transformation matrix construction method is:

[0017] ; ; ;

[0018] According to the pitch angle Build Matrix, according to the roll angle Build matrix, Refers to the camera intrinsic parameter matrix, Multiply the four matrices to construct the perspective transformation matrix

[0019] Preferably, in step 3, the depth estimation method is the Depth-Anything-V2 model

[0020] Preferably, step 4 includes the following sub-steps:

[0021] Step 4-1: Based on the length of the orthographic image and width And the altitude information of the drone , define the side length of the candidate box The solution range of satisfies the following inequality:

[0022] ;

[0023] This inequality constraint ensures that the length of the candidate box and the flight height maintain a geometric proportional relationship, where the parameter 、 and Three parameters are used to adjust the length of the candidate box and height information The functional relationship of yes A solution set of ;

[0024] Step 4-2: Based on the length of the orthographic image and width , and combined with the drone's running memory , define the side length of the candidate box and sliding step length , and satisfies the following inequality:

[0025] ;

[0026] Indicates the maximum number of candidate boxes. This number is limited by the running memory of the onboard computing device. The running memory of the drone is Related, and Positively correlated. Pick Ensuring that adjacent candidate boxes have 50% overlap not only improves traversal efficiency but also increases the number of candidate planes.

[0027] Step 4-3: Solve the double-constrained maximum that satisfies steps 4-1 and 4-2. If there is no suitable value If the value is set, adjust the shooting height of the drone image or increase the storage capacity of the drone;

[0028] Step 4-4, according to the obtained selection box side length and sliding step length , traverse the entire depth matrix through the sliding window and get Each candidate box, for each candidate box , Indicates the upper left corner. Indicates the lower right corner point, from The one with the smallest depth variance among the candidate frames is selected and used as the reference plane of the landing area, that is, the seed plane.

[0029] Preferably, in step 5, the dynamic threshold function expression is as follows:

[0030] ;

[0031] The dynamic threshold function can be based on the number of iterations and the depth value of the current neighborhood Continuously adjust the threshold ;in, Indicates the maximum roughness allowed for the surface, used to control noise tolerance; It represents the rate at which the threshold changes with the growth stage and determines the speed of exponential decay; 、 is a constant related to depth, used to adjust the effect of depth on the threshold; Indicates the height and width of the depth map; is the depth value of the current neighborhood; is the number of iterations in the current growth phase. If the difference between the mean depth of each pixel in the growth core and the mean depth of the seed plane is less than the dynamic threshold in the current growth phase, the growth core is added to the landing area plane.

[0032] Preferably, in step 5, the planar growth process adopts a segmented iterative growth strategy, and the growth core size is adjusted in stages through a dynamic parameter compensation mechanism. The specific process is as follows:

[0033] When the system detects the current number of iterations iter exceeds the preset threshold When max_iters is reached, the parameter adaptive adjustment is automatically triggered. Gstep is refined and reduced according to the exponential decay law, and is adjusted according to the complexity of the terrain. max_iters implements a gradient increasing strategy. Through this bidirectional parameter adjustment to form negative feedback control, the system uses the generated plane area as the new reference surface to reinitialize the growth process until the dual constraints of plane curvature continuity and regional area convergence are met, and then the iteration is terminated, thus achieving a progressive optimal plane approximation in complex terrain environments.

[0034] Preferably, in step 5, after the seed plane growth is completed, a two-stage operation of erosion-dilation is used for morphological refinement. The erosion stage eliminates scattered noise points and small spaces, and the dilation stage reconstructs the continuity of the plane to obtain the final landing area.

[0035] The present invention has the following characteristics and beneficial effects:

[0036] The present invention converts the original image into an oblique viewing angle image, so even with a low-cost camera, it can achieve autonomous detection of the best landing point in a field scene, reducing labor costs.

[0037] The present invention enables autonomous detection and optimization of landing points, thus supporting unmanned delivery of supplies in large-scale natural disaster scenarios.

[0038] A hierarchical analysis method combining global macro-planning with local fine identification is used to achieve multi-scale and accurate division of the aircraft landing area.

[0039] This method can detect static unknown landing areas using a single UAV-view image with good robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of a method for autonomously detecting and optimizing a drone landing point based on drone-viewing imagery according to an embodiment of the present invention.

[0041] Figure 2 4 is a flowchart of a depth estimation module in an embodiment of the present invention.

[0042] Figure 3 Flowchart of the adaptive plane growth module in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0044] A method for autonomous detection and optimization of landing points based on the perspective of UAV, such as Figure 1 As shown, the following steps are included:

[0045] Step 1: Obtain a single-frame RGB original image containing gimbal information, wherein the gimbal information includes the pitch angle, roll angle, and calibrated camera intrinsic parameter matrix of the drone gimbal used to capture the original image.

[0046] Step 2: Perform perspective transformation on the original image according to the pan / tilt information to obtain an orthographic image.

[0047] Specifically, the multi-axis gimbal angle parameters of the drone are used to construct the perspective transformation matrix of the drone's oblique view to the orthographic view, and the collected drone's perspective image is transformed to convert the oblique view image to the orthographic view image. The drone's perspective image is an outdoor image. The build process includes:

[0048] First, the intrinsic parameter matrix of the airborne camera is obtained through the calibration board ,The collected UAV perspective images are standardized and the lens distortion is corrected through camera calibration parameters.

[0049] Secondly, obtain the gimbal angle parameters of the drone image (including: pitch angle and roll angle ). Among them, the pitch angle Characterizes the inclination parameter between the lens optical axis and the horizontal reference plane at the moment of shooting (looking up is positive, looking down is negative), roll angle The spatial rotation state of the imaging plane relative to the horizontal reference plane is quantified (approaching zero under conventional aerial photography conditions, tilting to the right is positive, and tilting to the left is negative).

[0050] ; ; ;

[0051] Perform matrix operations according to the above formula to obtain the corresponding perspective transformation matrix .

[0052] Step 3: Use the depth estimation module to perform depth estimation on the orthographic image to obtain a depth matrix. It can be understood that the depth matrix stores relative depth values ​​for subsequent judgment of whether it meets the terrain requirements of the landing area.

[0053] In this embodiment, Figure 2 As shown in Figure 1, the depth estimation module adopts the Depth-Anything-V2 model, which includes an encoder, a decoder, and a teacher model.

[0054] Furthermore, the Depth-Anything-V2 model takes into account the uniqueness of image data in drone flight scenes. The flight altitude and angle of the drone usually lead to the complexity of the perspective and depth information. Therefore, in view of this characteristic, the input orthographic image data is processed in this embodiment to constrain its dynamic range. In order to ensure the accuracy of depth estimation, the predicted depth range is limited to between 0.1 meters and 20 meters, which covers most of the actual scenes captured by drones. Through the linear scaling algorithm, the pre-trained parameters of the model can be effectively adapted to avoid the numerical overflow problem that may occur when predicting distant targets.

[0055] Furthermore, the pre-trained weights come from the general feature extraction capabilities obtained from training on large-scale data sets, which can significantly improve the adaptability and robustness of the model in different scenarios. By introducing these pre-trained weights, the model can better learn the depth information in the image, thereby providing a high-quality depth matrix. This process not only improves the efficiency of the algorithm, but also reduces the deviation caused by insufficient data or special scenarios to a certain extent. Through the above operations, this embodiment successfully constructed an efficient and stable Depth-Anything-V2 model, which can accurately predict depth in drone flight scenarios and provide reliable depth information support for subsequent applications.

[0056] Step 4: Use the adaptive sliding window variance minimization selection algorithm to calculate the candidate area with the smallest depth variance and use it as the seed plane, that is, the landing area reference plane.

[0057] Specifically, the following sub-steps are included:

[0058] Step 4-1: Based on the length of the orthographic image and width And the altitude information of the drone , define the side length of the candidate box The solution range of satisfies the following inequality:

[0059] ;

[0060] This inequality constraint ensures that the length of the candidate box and the flight height maintain a geometric proportional relationship, where the parameter 、 and Three parameters are used to adjust the length of the candidate box and height information The functional relationship of yes A solution set of , understandable, parameters 、 and Need to be adjusted according to actual situation.

[0061] Step 4-2: Based on the length of the orthographic image and width , and combined with the drone's running memory , define the side length of the candidate box and sliding step length , and satisfies the following inequality:

[0062] ;

[0063] Indicates the maximum number of candidate boxes. This number is limited by the running memory of the onboard computing device. and positively correlated, Pick Ensuring that adjacent candidate boxes have 50% overlap improves traversal efficiency and increases the number of candidate planes;

[0064] Step 4-3: Solve the double-constrained maximum that satisfies steps 4-1 and 4-2. If there is no suitable value If the value is set, adjust the shooting height of the drone image or increase the storage capacity of the drone;

[0065] It should be noted that the two inequalities in steps 4-1 and 4-2 are empirical formulas based on the practical process. The joint restriction of the two inequalities finds the maximum The value and description have been modified. Details of the actual storage situation have been added.

[0066] Step 4-4: The side length of the candidate box obtained in step 4-3 and sliding step length , traverse the entire depth matrix through the sliding window and get candidate boxes, for each candidate box , Indicates the upper left corner. Indicates the lower right corner point, from The one with the smallest variance is selected from the candidate frames and used as the reference plane of the landing area, that is, the seed plane.

[0067] Specifically, the detailed process of calculating the landing area reference plane is as follows

[0068] Each candidate box is represented as:

[0069] ;

[0070] in: The upper left corner point, The lower right corner

[0071] For each candidate box , the set of pixels it contains is defined as:

[0072] ;

[0073] Get the depth value of each pixel according to the depth matrix , calculate the candidate box The mean depth of all internal pixels:

[0074] ;

[0075] Then calculate the variance of the depth value:

[0076] ;

[0077] Finally, the candidate box with the smallest variance is selected As the landing area reference plane:

[0078] .

[0079] Step 5: Construct a dynamic threshold function, control the growth range of the seed plane according to the dynamic threshold function, and obtain the final landing area.

[0080] Specifically, in this embodiment, the dynamic threshold function is expressed as:

[0081] ;

[0082] The dynamic threshold function can be based on the number of iterations and the depth value of the current neighborhood Continuously adjust the threshold ;in, Indicates the maximum roughness allowed for the surface, used to control noise tolerance; It represents the rate at which the threshold changes with the growth stage and determines the speed of exponential decay; 、 is a constant related to depth, used to adjust the effect of depth on the threshold; Indicates the height and width of the depth map; is the depth value of the current neighborhood; is the number of iterations in the current growth phase. If the difference between the mean depth of each pixel in the growth core and the mean depth of the seed plane is less than the dynamic threshold in the current growth phase, , then add the growth core to the landing area plane.

[0083] Furthermore, the planar growth process adopts a segmented iterative growth strategy, and adjusts the growth core size in stages through a dynamic parameter compensation mechanism. The specific process is as follows:

[0084] When the system detects the current number of iterations Exceeding the preset threshold When , the parameter adaptive adjustment is automatically triggered. The refinement is carried out according to the exponential decay law, and the terrain complexity is adjusted according to the terrain complexity. A gradient increasing strategy is implemented, and negative feedback control is formed through this two-way parameter adjustment. The system uses the generated plane area as the new reference surface to reinitialize the growth process until the dual constraints of plane curvature continuity and regional area convergence are met, and then the iteration is terminated, thereby achieving progressive optimal plane approximation in complex terrain environments.

[0085] Specifically, such as Figure 3 As shown, the following steps are included:

[0086] (1) Initialization: Landing area reference plane As the initial plane;

[0087] (2) Iterative growth: In the iteration, calculate the current dynamic threshold , combined with the depth value and the number of iterations . Check the current plane 8-neighborhood points of , excluding holes and points in the assigned plane. The fitting error , add it to the plane. Update the plane equation and root mean square error .

[0088] (3) When the system detects the current number of iterations Exceeding the preset threshold When the parameter adaptive adjustment is automatically triggered: the growth step size The refinement is carried out according to the exponential decay law, and the terrain complexity is adjusted according to the terrain complexity. Implement the gradient increasing strategy. At the same time, the plane area at this time is used as the reference plane of the landing area. , repeat step (2).

[0089] (4) When the neighborhood point set is empty or When it reaches 0, it stops growing, which means that the plane now meets the dual constraints of plane curvature continuity and regional area convergence.

[0090] (5) Finally, this embodiment adopts a custom core matrix configuration , morphological optimization is performed by implementing the opening operation processing method, and the initial growth plane is morphologically refined through the two-stage operation of corrosion and expansion: the first stage corrosion operation eliminates discrete noise points and small holes, and the second stage expansion operation reconstructs the continuity of the terrain structure, and realizes edge smoothing and regional closure under the constraint of morphological gradient, and finally outputs the optimized landing area that meets the requirements of geometric morphological integrity. B. This process achieves a balance between noise suppression and feature preservation by controlling the scale of structural elements, ensuring that the landing area contour meets the engineering standards for safe UAV landing.

[0091] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for autonomous detection and optimization of landing points based on the perspective of a drone, characterized in that: The steps include: Step 1: Obtain a single-frame RGB original image containing PTZ information; Step 2: Perform perspective transformation on the original image according to the PTZ information to obtain an orthographic image; Step 3: Use the depth estimation module to perform depth estimation on the orthographic image to obtain a depth matrix; Step 4: Use the adaptive sliding window variance minimization selection algorithm to calculate the candidate area with the smallest depth variance and use it as the seed plane; The following sub-steps are included: Step 4-1: Based on the length of the orthographic image and width And the altitude information of the drone , define the side length of the candidate box The solution range of satisfies the following inequality: ； This inequality constraint ensures that the length of the candidate box and the flight height maintain a geometric proportional relationship, where the parameter 、 and Three parameters are used to adjust the length of the candidate box and height information The functional relationship of yes A solution set of ; Step 4-2: Based on the length of the orthographic image and width , and combined with the drone's running memory , define the side length of the candidate box and sliding step length , and satisfies the following inequality: ； Indicates the maximum number of candidate boxes, which is limited by the running memory of the onboard computing device. Running memory with drone Positive correlation; Pick Ensuring that adjacent candidate boxes have 50% overlap improves traversal efficiency and increases the number of candidate planes; Step 4-3: Solve the double-constrained maximum that satisfies steps 4-1 and 4-2. If there is no suitable value If the value is set, adjust the shooting height of the drone image or increase the storage capacity of the drone; Step 4-4: The side length of the candidate box obtained in step 4-3 and sliding step length , traverse the entire depth matrix through the sliding window and get candidate boxes, for each candidate box , Indicates the upper left corner. Indicates the lower right corner point, from Select the one with the smallest variance from the candidate boxes and use it as the reference plane of the landing area, i.e. the seed plane Step 5: Construct a dynamic threshold function, control the growth range of the seed plane according to the dynamic threshold function, and obtain the final landing area.

2. The method according to claim 1, characterized in that The gimbal information includes the pitch angle, roll angle, and calibrated camera intrinsic parameter matrix of the drone gimbal used to capture the original image.

3. The method according to claim 2, characterized in that In step 2, a perspective transformation matrix of the oblique perspective to the orthographic perspective of the drone is constructed according to the gimbal information, and perspective transformation is performed using the perspective transformation matrix.

4. The method according to claim 2, characterized in that Step 2 also includes adjusting the image size, removing blank spaces and centering the image.

5. The method according to claim 3, characterized in that The perspective transformation matrix construction method is: ； According to the pitch angle Build Matrix, according to the roll angle Build matrix, Refers to the camera intrinsic parameter matrix, Multiply the four matrices to construct the perspective transformation matrix .

6. The method according to claim 1, characterized in that In step 3, the depth estimation module adopts the Depth-Anything-V2 model.

7. The method according to claim 1, characterized in that In step 5, the dynamic threshold function expression is as follows: ； The dynamic threshold function can be based on the number of iterations and the depth value of the current neighborhood Continuously adjust the threshold ;in, Indicates the maximum roughness allowed for a surface; represents the rate at which the threshold changes with the growth stage; 、 is a constant related to depth; Indicates the height and width of the depth map; is the depth value of the current neighborhood; is the number of iterations in the current growth phase. If the difference between the mean depth of each pixel in the growth core and the mean depth of the seed plane is less than the dynamic threshold in the current plane growth phase, , then add the growth core to the landing area plane.

8. The method according to claim 1, characterized in that In step 5, the planar growth process adopts a segmented iterative growth strategy, and adjusts the growth core size in stages through a dynamic parameter compensation mechanism. The specific process is as follows: When the system detects the current number of iterations Exceeding the preset threshold When the parameter is automatically triggered to adjust adaptively, the growth step The refinement is carried out according to the exponential decay law, and the terrain complexity is adjusted according to the terrain complexity. A gradient increasing strategy is implemented, and negative feedback control is formed through this two-way parameter adjustment. The system uses the generated plane area as the new reference surface to reinitialize the growth process until the dual constraints of plane curvature continuity and regional area convergence are met, and then the iteration is terminated, thereby achieving progressive optimal plane approximation in complex terrain environments.

9. The method according to claim 1, characterized in that In step 5, after the seed plane growth is completed, a two-stage operation of erosion and dilation is used to perform morphological refinement. The erosion stage eliminates scattered noise points and small spaces, and the dilation stage reconstructs the continuity of the plane to obtain the final landing area.

Images