An unmanned aerial vehicle dynamic platform autonomous landing method and system based on multi-modal visual guidance

Abstract

The application relates to the technical field of unmanned aerial vehicle autonomous landing, and discloses a multi-modal visual guidance unmanned aerial vehicle dynamic platform autonomous landing system. In view of the problem of insufficient reliability of a single sensor in a dynamic environment, a three-stage positioning strategy is adopted: in the long-distance stage (> 2.25 m), a composite landmark is detected through an improved YOLO v5 algorithm, and a barometer and laser ranging are combined to realize fusion positioning; in the medium-distance stage (0.5-2.25 m), the positioning precision is improved by adopting elliptical contour fitting, and the optical flow speed estimation and IMU angular velocity compensation are fused; in the short-distance stage (< 0.5 m), sub-pixel level alignment is realized through multi-layer AprilTag coding. The system adopts complementary filtering to fuse multi-sensor data, adjusts the flight attitude through a PID controller, and cooperates with a silica gel-butterfly spring cushioning platform to reduce the landing impact.

Description

Technical Field

[0001] This invention belongs to the field of autonomous landing technology for unmanned aerial vehicles (UAVs), specifically relating to an autonomous landing method and system for a dynamic UAV platform based on multimodal vision guidance. Background Technology

[0002] With the rapid development of drone technology and its expanding application scenarios, the accuracy and reliability of landing have become crucial. Among the many explorations of drone landing technologies, vision-based landing technology is gradually emerging.

[0003] Early drone landings largely relied on operators visually observing and manually controlling remote controls to guide the drone to a designated location. This method not only demanded extremely high skill levels from the operators, requiring extensive professional training, but was also limited by human visual range, reaction speed, and environmental factors (such as strong light, smog, sandstorms, and other severe weather), making landing errors highly likely, leading to drone crashes and damage, causing economic losses and even safety hazards.

[0004] To overcome the drawbacks of manual landing, vision-based drone landing technology has emerged in recent years, driven by the rapid development of sensor technology. Just as humans rely on vision to perceive their surroundings and make spatial positioning and action decisions, visual sensors mounted on drones simulate human visual functions. They can capture rich image information of the landing area and, through the extraction and analysis of image features—such as identifying landing markers, landmark outlines, and specific patterns—provide crucial relative position and attitude data for precise drone positioning.

[0005] Current drone landing technologies often rely on a single sensor (such as GPS or vision), which is susceptible to environmental interference on dynamically moving platforms, leading to positioning drift or loss. For example:

[0006] Single-stage vision methods, such as QR code-based positioning, are prone to failure at medium to long distances and do not solve the problem of dynamic compensation.

[0007] Traditional landing platforms enhance stability through physical structure, but lack active guidance capabilities and cannot adapt to platform movement.

[0008] Insufficient sensor fusion: Although a distance threshold triggering mechanism was introduced, no multimodal visual collaboration strategy was designed, resulting in poor robustness. Summary of the Invention

[0009] The main objective of this invention is to provide a method and system for autonomous landing of a dynamic UAV platform based on multimodal vision guidance. It includes the following steps:

[0010] S1. Pre-collect photos of landmarks at different altitudes and create a landmark dataset;

[0011] S2. Using the landmark dataset and combining it with the YOLO v5 neural network, a landmark detection model is trained.

[0012] S3. A control module that mounts a landmark detection model onto a drone;

[0013] S4. The landing process of the drone is divided into three stages based on the distance to the landmark: long distance stage (>2.25m), medium distance stage (0.5m–2.25m), and short distance stage (<0.5m).

[0014] S5. In the long-range phase (>2.25m), the YOLO v5 algorithm is used to detect the landmarks of the dynamic platform. The relative position of the platform is calculated based on the center coordinates of the detection frame and the fused height (barometer + laser rangefinder), and the UAV is guided to land in the mid-range phase.

[0015] S6. Detection altitude: If the mid-range stage (0.5m–2.25m) is reached, elliptical detection is used to improve accuracy. Positioning is performed through sensor fusion, platform speed is estimated through optical flow, and speed compensation is performed based on IMU angular velocity. The lateral speed of the UAV is adjusted through the PID controller to guide the UAV to land in the close-range stage.

[0016] S7. Detect altitude. If the distance reaches the close range stage (<0.5m), use multi-layer composite QR code AprilTag encoding and combine it with the terminal precision alignment strategy for landing.

[0017] Furthermore, step S2 specifically includes the following steps:

[0018] S21. Pre-collect landmark photos at different heights, scale down the landmarks proportionally, collect landmark images at different heights and orientations, divide them into training and validation sets according to the proportions, and create a landmark dataset.

[0019] S22. In order to identify cooperative landmarks more quickly, the YOLO v5 algorithm optimized with the PP-LCNet model is trained on the landmark dataset to obtain a landmark detection model.

[0020] Furthermore, the landmark consists of an H-shaped geometric pattern surrounded by concentric rings and an AprilTag code, with the AprilTag code located in the center of the H-shaped geometric pattern.

[0021] Furthermore, step S4 specifically includes the following steps:

[0022] S41. Landmark detection is performed using the YOLO v5 algorithm optimized with the PP-LCNet model;

[0023] S42. When the UAV takes off, establish the transformation relationship from image pixel coordinates to UAV body coordinates by using known reference objects;

[0024] S43. The relative position of the platform is calculated based on the center coordinates of the detection box and the fusion height.

[0025] S44. Land the drone at the mid-range stage.

[0026] Furthermore, in step S43, the relative position of the barometer and the laser ranging platform is calculated, where the coordinate transformation formula is:

[0027]

[0028] Where (u,v) represents the center pixel coordinates of the detection box, h is the fusion height (barometer + laser), f is the camera focal length, and (c x ,c y () is the center of the image.

[0029] Furthermore, step S6 includes the following steps:

[0030] S61. Ellipse detection is used for fitting. The concentric circular elliptical contour of the detection platform is detected. Edge detection is used to improve the center positioning accuracy and construct three-dimensional estimated coordinates.

[0031] S62. Positioning is achieved through multi-sensor fusion, including: visual positioning, optical flow velocity estimation, and IMU angular velocity measurement, with data acquisition using a hardware clock synchronization mechanism;

[0032] S63. Multi-source coordinates are fused using complementary filtering, and optical flow velocity is compensated using IMU data;

[0033] S64. Adjust the drone's lateral speed using the PID controller to guide the drone to land at close range.

[0034] Further, in step S61, edge detection is performed using the Canny algorithm, and then an ellipse is fitted using the contour extracted from the edges. The general equation of the ellipse is:

[0035] x 2 +Axy+By 2 +Cx+Dy+E=0 (2)

[0036] Multiple measurement points were collected, and the objective function fitted using the least squares method is:

[0037]

[0038] To minimize S, we set the partial derivatives of all terms to 0, that is:

[0039]

[0040] Substituting the values ​​of the edge points yields the optimal solution to the system of equations, obtaining the values ​​of A, B, C, D, and E. Substituting these values ​​into the general equation of the ellipse yields the basic parameters of the ellipse, namely: the center point of the ellipse (x0, y0), the major and minor radii of the ellipse a and b, and the deflection angle θ of the ellipse.

[0041] Furthermore, in step S63, the LK optical flow method is used to calculate the platform speed and IMU data is used for compensation;

[0042] Complementary filtering is used to fuse visual results and locate coordinates.

[0043] The final output, after IMU compensation, is the platform velocity estimate and coordinates;

[0044] The visual localization result is (x) vision ,y vision Corrected optical flow velocity

[0045]

[0046] (x,y) are the positioning coordinates, α is the filter coefficient, and Δt is the time step.

[0047] Furthermore, in step S7, the landmark is encoded using a multi-layer composite QR code AprilTag, and the pattern is as shown in claim 3;

[0048] When the altitude reading is ≤10cm and the deviation between the center point of the QR code and the center of the image is <5 pixels, the magnetic attraction function is triggered, and the drone is landed on the dynamic platform.

[0049] Furthermore, a multimodal vision-guided autonomous landing method and system for a dynamic UAV platform is characterized in that the platform has a shock-absorbing structure, which consists of a silicone damping pad and a butterfly spring assembly.

[0050] The beneficial effects of this invention are:

[0051] I. This invention provides a multimodal vision-guided autonomous landing method and system for a dynamic UAV platform. The landing process is divided into three stages, guided by multiple modalities. First, landmarks are detected visually. In the long-range stage, the platform's relative position is calculated using barometers and laser ranging. In the mid-range stage, elliptic detection is used to improve accuracy, and multi-sensor fusion is employed for positioning, with IMU angular velocity compensation. In the close-range stage, AprilTag encoding combined with terminal alignment is used for precise landing. This method improves the accuracy of the autonomous landing of the UAV platform and enhances the robustness of the entire process.

[0052] Second, this invention optimizes the platform by designing a landmark consisting of an H-shaped geometric pattern enclosed by concentric circles and an AprilTag code. Different patterns are detected at different heights. At the same time, a shock-absorbing structure is added to the platform, consisting of a silicone damping pad and a disc spring. During the final landing, the drone is magnetically secured, reducing the shaking and errors when the drone lands on the platform. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0054] Figure 2 This is a schematic diagram of the image processing flow;

[0055] Figure 3 This is a schematic diagram of the PP-LCNet model structure;

[0056] Figure 4 This is a schematic diagram of the platform's landmarks; Detailed Implementation

[0057] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0058] like Figure 1 As shown, a multimodal vision-guided autonomous landing method and system for a dynamic UAV platform includes the following steps:

[0059] S1. Pre-collect photos of landmarks at different altitudes and create a landmark dataset;

[0060] S2. Using the landmark dataset and combining it with the YOLO v5 neural network, a landmark detection model is trained.

[0061] S3. A control module that mounts a landmark detection model onto a drone;

[0062] S4. The landing process of the drone is divided into three stages based on the distance to the landmark: long distance stage (>2.25m), medium distance stage (0.5m–2.25m), and short distance stage (<0.5m).

[0063] S5. In the long-range phase (>2.25m), the YOLO v5 algorithm is used to detect the landmarks of the dynamic platform. The relative position of the platform is calculated based on the center coordinates of the detection frame and the fused height (barometer + laser rangefinder), and the UAV is guided to land in the mid-range phase.

[0064] S6. Detection altitude: If the mid-range stage (0.5m–2.25m) is reached, elliptical detection is used to improve accuracy. Positioning is performed through sensor fusion, platform speed is estimated through optical flow, and speed compensation is performed based on IMU angular velocity. The lateral speed of the UAV is adjusted through the PID controller to guide the UAV to land in the close-range stage.

[0065] S7. Detect altitude. If the distance reaches the close range stage (<0.5m), use multi-layer composite QR code AprilTag encoding and combine it with the terminal precision alignment strategy for landing.

[0066] The drone carries an onboard image processing module, and the specific process is as follows: Figure 2 and Figure 3 As shown, it mainly includes:

[0067] 1. Collect 400 dynamic platform landmark images at different altitudes (0.5m-10m) and under different lighting conditions (sunny / cloudy / nighttime supplemental lighting). The landmark consists of an H-shaped geometric pattern (1.2m in diameter) surrounded by concentric rings and an embedded AprilTag code (0.3m × 0.3m in size). A multi-angle simulation device is used to generate distorted images with pitch angles of ±45° and yaw angles of ±60°.

[0068] Second, 100 images were used as the test set for algorithm verification, and the rest were used as the training set. Based on the training and verification sets, the YOLO v5 algorithm neural network was trained and verified to obtain the landmark detection model.

[0069] Third, the PP-LCNet lightweight network was selected to improve the mesh structure of the YOLO v5 algorithm, such as... Figure 3 As shown, the depthwise separable convolution uses the H-Swish activation function and employs a larger convolutional kernel, with a 5x5 kernel replacement operation at the end of the network. This results in better landmark detection performance.

[0070] Fourth, in addition, a 1x1 convolutional kernel of size 1280 is inserted between the final global average pooling (GAP) layer and the fully connected layer to address the negative impacts of inaccurate recognition and classification caused by directly outputting the GAP layer. The SE module is moved to the end of the network to improve the accuracy and recognition speed of the network model.

[0071] 5. During the landing process, the UAV detects the distance to the platform by fusing barometer and laser rangefinder, and hovers by controlling the motor.

[0072] 6. The UAV uses a PID control strategy to control its lateral speed, thereby tracking the dynamic platform and ensuring an accurate landing.

[0073] In this example, the image preprocessing workflow is a crucial step in ensuring the accuracy of target recognition. The following is a summary of the preprocessing workflow for cooperative landmark detection: The image is converted to grayscale, transforming the RGB three-channel image into a single-channel grayscale image. Gaussian filtering is used for noise reduction to suppress image noise and prevent interference with subsequent edge detection.

[0074] In the mid-range phase, edge detection is performed using the Canny algorithm, and the image is blurred using a 5×5 Gaussian kernel to smooth noise. Then, an ellipse is fitted using the contour extracted from the edges.

[0075] The general equation of an ellipse is:

[0076] x 2 +Axy+By 2 +Cx+Dy+E=0 (6)

[0077] Multiple measurement points were collected, and the objective function fitted using the least squares method is:

[0078]

[0079] To minimize S, we set the partial derivatives of all terms to 0, that is:

[0080]

[0081] Substituting the values ​​of the edge points yields the optimal solution to the system of equations, obtaining the values ​​of A, B, C, D, and E. Substituting these values ​​into the general equation of the ellipse yields the basic parameters of the ellipse, namely: the center point of the ellipse (x0, y0), the major and minor radii of the ellipse a and b, and the deflection angle θ of the ellipse.

[0082] In this example, the landmarks on the platform are as follows: Figure 4 As shown, its pattern consists of an H-shaped geometric pattern (1.2m in diameter) surrounded by concentric rings and an embedded AprilTag code (0.3m × 0.3m in size), which can carry a variety of information.

[0083] During the close-range phase, the target is detected by calling the preset AprilTag encoding package, and the AprilTag encoding open-source vision library is called to perform real-time decoding of the target image captured by the airborne camera. When the height reading is ≤10cm and the deviation between the center point of the QR code and the center of the image is <5 pixels, the magnetic attraction function is triggered to land the drone on the dynamic platform.

[0084] In summary, this technical solution designs an autonomous landing method and system for UAVs on a dynamic platform based on multimodal vision guidance. On the other hand, it designs a new landmark design, which makes the autonomous landing of UAVs on the dynamic platform more efficient and robust.

Claims

1. A method and system for autonomous landing of a dynamic UAV platform based on multimodal vision guidance, comprising the following steps: S1. Pre-collect photos of landmarks at different altitudes and create a landmark dataset; S2. Using the landmark dataset and combining it with the YOLO v5 neural network, a landmark detection model is trained. S3. A control module that mounts a landmark detection model onto a drone; S4. The landing process of the drone is divided into three stages based on the distance to the landmark: long distance stage (>2.25m), medium distance stage (0.5m–2.25m), and short distance stage (<0.5m). S5. In the long-range phase (>2.25m), the YOLO v5 algorithm is used to detect the landmarks of the dynamic platform. The relative position of the platform is calculated based on the center coordinates of the detection frame and the fused height (barometer + laser rangefinder), and the UAV is guided to land in the mid-range phase. S6. Detection altitude: If the mid-range stage (0.5m–2.25m) is reached, elliptical detection is used to improve accuracy. Positioning is performed through multi-sensor fusion. The platform speed is estimated by optical flow method and speed compensation is performed based on IMU angular velocity. The lateral speed of the UAV is adjusted by PID controller to guide the UAV to land in the close-range stage. S7. Detect altitude. If the distance reaches the close range stage (<0.5m), land using a multi-layer composite QR code AprilTag in conjunction with a precise end-of-line alignment strategy.

2. The multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 1, characterized in that, Step S2 includes the following steps: S21. Pre-collect landmark photos at different heights, scale down the landmarks proportionally and collect landmark images at different heights and directions, divide them into training set and validation set according to the proportions, and create a landmark dataset. S22. In order to identify cooperative landmarks more quickly, the YOLO v5 algorithm optimized with the PP-LCNet model is trained on the landmark dataset to obtain a landmark detection model.

3. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 1, characterized in that, The landmark consists of an H-shaped geometric pattern surrounded by concentric rings and an AprilTag code, with the AprilTag code located in the center of the H-shaped geometric pattern.

4. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 1, characterized in that, Step S5 includes the following steps: S41. Landmark detection is performed using the YOLO v5 algorithm optimized with the PP-LCNet model; S42. When the UAV takes off, establish the transformation relationship from image pixel coordinates to UAV body coordinates by using known reference objects; S43. The relative position of the platform is calculated based on the center coordinates of the detection box and the fusion height. S44. Land the drone at medium range.

5. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 4, characterized in that, In step S43, the relative position of the platform is calculated using the barometer and the laser ranging, where the coordinate transformation formula is: Where (u,v) represents the center pixel coordinates of the detection box, h is the fusion height (barometer + laser), f is the camera focal length, and (c x ,c y () is the center of the image.

6. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 1, characterized in that, Step S6 includes the following steps: S61. Ellipse detection is used for fitting. The concentric circular elliptical contour of the detection platform is detected. Edge detection is used to improve the center positioning accuracy and construct three-dimensional estimated coordinates. S62. Positioning is achieved through multi-sensor fusion, including: visual positioning, optical flow velocity estimation, and IMU angular velocity measurement, with data acquisition using a hardware clock synchronization mechanism; S63. Multi-source coordinates are fused using complementary filtering, and optical flow velocity is compensated using IMU data; S64. Adjust the drone's lateral speed using the PID controller to guide the drone to land at close range.

7. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 6, characterized in that, In step S61, edge detection is performed using the Canny algorithm, and then an ellipse is fitted using the contour extracted from the edges. The general equation of the ellipse is: x 2 +Axy+By 2 +Cx+Dy+E=0 (2) Multiple measurement points were collected, and the objective function fitted using the least squares method is: To minimize S, we set the partial derivatives of all terms to 0, that is: Substituting the values ​​of the edge points yields the optimal solution to the system of equations, obtaining the values ​​of A, B, C, D, and E. Substituting these values ​​into the general equation of the ellipse yields the basic parameters of the ellipse, namely: the center point of the ellipse (x0, y0), the major and minor radii of the ellipse a and b, and the deflection angle θ of the ellipse.

8. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 6, characterized in that, In step S63, the platform velocity is calculated using the LK optical flow method and compensated using IMU data; Complementary filtering is used to fuse visual results and locate coordinates. The final output, after IMU compensation, is the platform velocity estimate and coordinates; The visual localization result is (x) vision ,y vision Corrected optical flow velocity (x,y) are the positioning coordinates, α is the filter coefficient, and Δt is the time step.

9. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 1, characterized in that, In step S7, the landmark adopts a multi-layer composite QR code AprilTag, and the pattern is as shown in claim 3; When the altitude reading is ≤10cm and the deviation between the center point of the QR code and the center of the image is <5 pixels, the magnetic attraction function is triggered, and the drone is landed on the dynamic platform.

10. A multimodal vision-guided autonomous landing method and system for a dynamic UAV platform according to claim 1, characterized in that, The platform has a shock-absorbing structure, consisting of silicone damping pads and disc springs.

Images