An unmanned aerial vehicle pinpoint landing method based on laser radar and camera data fusion

Abstract

The application relates to the technical field of unmanned aerial vehicle positioning, in particular to an unmanned aerial vehicle pinpoint landing method based on laser radar and camera data fusion, which comprises the following steps: a camera recognition image is obtained to obtain Aruco code two-dimensional point coordinates and establish an XOY plane; a laser radar scans below an unmanned aerial vehicle to obtain three-dimensional point cloud, reflection intensity data and Aruco code three-dimensional point cloud coordinates; a transformation matrix M is calculated to project the three-dimensional point cloud to the XOY plane; the center position of the Aruco code is calculated and taken as a coordinate origin to establish a coordinate system on the XOY plane; the projected three-dimensional point cloud data is mapped to the coordinate system; an Aruco code corner point is recognized, the relative position of the unmanned aerial vehicle and the Aruco code corner point is calculated, and pinpoint landing is completed. The application has the beneficial technical effects of fusing laser radar and camera data to recognize the Aruco code corner point, improving the environmental adaptability of unmanned aerial vehicle pinpoint landing, and expanding the ranging range.

Description

Technical Field

[0001] This invention relates to the field of UAV positioning technology, specifically to a method for UAV fixed-point landing based on the fusion of lidar and camera data. Background Technology

[0002] Rotary-wing drones are characterized by their small size, simple structure, and flexible control. They can take off and land vertically, hover freely, and adapt to various natural environments. They possess advantages such as autonomous flight and landing, allowing them to operate in complex and dangerous environments unsuitable for human access. In recent years, they have seen increasingly important applications in research institutions, government agencies, broadcasting media, personal applications, and the military. Drone precision landing refers to a drone flying at high altitude, using a downward-looking camera to capture ground images. A human selects a landing target point on the ground image, and the drone automatically flies to and precisely lands at the designated landing point. Drone precision landing can be applied to disaster relief and rescue, automated logistics docking, and emergency drone landings.

[0003] Traditional drones only carry cameras to identify landing reference points. However, cameras have narrow fields of view at close range, are prone to overexposure in strong light, and cannot receive low-light conditions. This results in limitations in application scenarios, susceptibility to failure, and inability to operate at night. In recent years, lidar has gradually become miniaturized and has become a necessary sensor for many functional drones. LiDAR uses the principle of actively emitting and receiving lasers to perceive the three-dimensional environment, offering greater spatial coverage and being unaffected by natural factors such as daytime, nighttime, and light intensity. This provides a reliable option for drones to identify landing reference points.

[0004] The Random Sampling Consensus (RANSAC) algorithm, first proposed by Fischler and Bolles in 1981, uses an iterative approach to estimate the parameters of a mathematical model from a set of observed data containing outliers. RANSAC is a nondeterministic algorithm; in a sense, it produces a result that is reasonable under a certain probability, and more iterations increase this probability. Therefore, this algorithm is often used to process lidar point clouds to form an estimation plane. The random sampling algorithm includes the following steps: (1) randomly select several points in the data and set them as inclusors; (2) calculate the model that fits the inclusors; (3) substitute the other points that were not selected earlier into the model established earlier and calculate whether they are inclusors; (4) record the number of inclusors; (5) repeat the above steps several times and compare which calculation produces the most inclusors. The model established in the calculation with the most inclusors is the required estimation plane.

[0005] However, current UAV precision landing methods still face technical challenges, such as the lack of true visible light color in the reference objects used for identifying landing points, low point cloud density, and short ranging range. For example, the UAV precision landing method proposed by Weimin Wang et al., which uses a printed chessboard with a reflection intensity-assisted 3D lidar and panoramic camera for automatic and accurate external calibration, employs a Velodyne lidar. However, due to the extremely low point cloud density at long distances, effective feature information cannot be extracted, resulting in a maximum ranging range of only 2.6 meters in their experiments, limiting the UAV's usability. Therefore, it is necessary to research UAV precision landing methods that can improve the accuracy of reference object identification for landing points.

[0006] For example, Chinese patent CN113946157A, published on January 18, 2022, discloses a method and system for fixed-point landing of a drone based on multi-functional identification and positioning. This method involves the drone receiving an automatic return command from a control terminal located on the landing platform, and hovering at a first preset distance altitude above the platform using GPS navigation. It also uses dual cameras to acquire real-time image data from below and transmits it to the control terminal. The control terminal uses contour extraction to identify and lock onto a target on the landing platform, guiding the drone to gradually descend. When the drone is at a second preset distance altitude above the target's center, it automatically lands, completing a fixed-point landing. This achieves high accuracy in identifying the landing reference point and short landing time for the drone. However, this technical solution does not solve the problems of the drone's fixed-point landing being susceptible to environmental factors and its short ranging range. Summary of the Invention

[0007] The technical problem this invention aims to solve is that current UAVs, when achieving precise landing, suffer from susceptibility to environmental factors and short ranging range. This invention proposes a UAV precise landing method based on the fusion of lidar and camera data. This method can integrate lidar and camera data to identify Aruco code corner points, improving the environmental adaptability of UAV precise landing and expanding the ranging range.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for fixed-point landing of a UAV based on the fusion of lidar and camera data, used for landing of a UAV using a Livox lidar, comprising:

[0009] The camera captures an image below the drone, identifies the Aruco code in the image, obtains the two-dimensional point coordinates of the Aruco code, and establishes the XOY plane based on the two-dimensional point coordinates.

[0010] The lidar scans the area beneath the drone to obtain 3D point cloud and reflection intensity data. Based on the reflection intensity data, the Aruco code is identified, and the 3D point cloud coordinates of the Aruco code are obtained.

[0011] Calculate the transformation matrix M between the 2D point coordinates of the camera and the 3D point cloud coordinates of the lidar, and project the 3D point cloud onto the XOY plane;

[0012] Calculate the center position of the Aruco code based on the projected 3D point cloud;

[0013] Establish a coordinate system in the XOY plane with the center position as the origin;

[0014] The projected 3D point cloud data is mapped onto the coordinate system;

[0015] Identify Aruco code corner points, calculate the relative position of the drone with the Aruco code corner points, and generate drone movement data;

[0016] Repeat the above steps until the drone is aligned with the corner point of the Aruco code, completing the targeted landing.

[0017] Preferred methods for identifying Aruco codes based on reflection intensity data include:

[0018] R L R represents the maximum reflectance of the black pattern. H This represents the minimum reflectance of the white pattern.

[0019] Define an intermediate interval [β1, β2], β1 = ((σ-1)R L +R H ) / σ,β2=(R L +(σ-1)R H ) / σ, where σ is a preset constant value;

[0020] If the reflection intensity of a three-dimensional point is less than β1, the three-dimensional point is included in the black pattern area; if the reflection intensity of a three-dimensional point is greater than β2, the three-dimensional point is included in the white pattern area.

[0021] The Aruco code is identified based on the black and white patterned areas.

[0022] Preferably, the cost function between the reflection intensity data and the pattern color of the Aruco code is:

[0023]

[0024] Where, x i f1(x) represents the reflection intensity of the i-th three-dimensional point. i This is used to determine whether the reflection intensity of a three-dimensional point is within the intermediate interval [β1, β2]. C represents the square grid of the Aruco code. Representing a three-dimensional point, Used to determine whether square grid C contains three-dimensional points. V i c represents the four vertices of the cell corresponding to the i-th point. i This indicates that based on the reflection intensity x i Estimated color value, Representing a three-dimensional point The corresponding pattern colors are 0 for black and 1 for white. f3 is used to determine the location of 3D points that are incorrectly mapped in the 3D point cloud. f3 = min(Δx1,Δx2) + min(Δy1,Δy2).

[0025] As a preferred method, the location of 3D points with erroneous 3D point cloud mapping is determined as follows:

[0026] When a point cloud incorrectly falls into a white square, the position of the point cloud that incorrectly falls into a white square is determined by f3 = min(Δx1, Δx2) + min(Δy1, Δy2), where Δx1 represents the point cloud... To V i The distance to the left of a defined square grid, Δx2, represents the point cloud. To V i The distance to the right of a defined square grid, Δy1 represents the point cloud. To V i The distance from the top edge of a defined square grid, Δy2 represents the point cloud. To V i The distance from the bottom edge of a defined square grid;

[0027] When a point cloud erroneously falls outside the Aruco code, the location of the point cloud erroneously falling outside the Aruco code is determined by f3 = min(Δx1, Δx2) + min(Δy1, Δy2), where Δx1 represents the point cloud... The distance to the left side of square grid C, Δx2 represents the point cloud. The distance to the right side of square grid C, Δy1 represents the point cloud. The distance to the top edge of the square grid C, Δy2 represents the point cloud. The distance to the bottom edge of square grid C.

[0028] As a preferred option, the set of points in the XOY plane is:

[0029]

[0030] Among them, T r (θ M ,T M ,P i M ) represents the lidar matrix.

[0031] As a preferred method, the transformation matrix M is calculated as follows:

[0032]

[0033] Where (u1, v1) represents the coordinates of a two-dimensional point detected by the camera, and (x, y, z) represents the coordinates of a three-dimensional point cloud captured by the lidar. and Both represent the camera's focal length, R1 is the rotation matrix, and T1 is the translation matrix.

[0034] As a preferred method, the center position of the Aruco code is calculated as follows:

[0035] The 3D point cloud matrix of the Aruco code is mapped along the coordinate system P of the lidar. b =(a1,a2,a3) T The three basis vectors a1, a2 and a3 in the equation are decomposed;

[0036] Two three-dimensional points are combined and connected to form a line segment. The ratios of the line segment to the basis vectors a1, a2, and a3 are calculated and denoted as Δ1, Δ2, and Δ3, respectively.

[0037] The point cloud is processed using a random sampling consensus algorithm to form an estimated plane;

[0038] Line segments with a ratio Δ3 less than 0.01 are considered to be in the same plane, and points in the same plane are projected onto the estimated plane;

[0039] The coordinates of all points projected onto the estimation plane are rotated to the XOY plane through a matrix transformation, and Δ1, Δ2 are then correlated with the X and Y axes through the rotation point. Alignment, where P is defined n×3,f For the fitting point, Let P be the fitting point n×3,f The rotation matrix;

[0040] Calculate the rotation point P n×3,fc rotation matrix This is the center position of the Aruco code.

[0041] As a preferred method, the coordinate system is established in the XOY plane with the center position as the origin:

[0042] Align the center position with the origin of the XOY coordinate system;

[0043] Subtract the value of the center position from the value of the rotation point P. n×3,fc The average value is used to obtain the coordinate system of the origin of the XOY coordinate system after rotation.

[0044] The beneficial technical effects of this invention include: employing a drone positioning landing method based on the fusion of lidar and camera data, which combines the advantages of camera and lidar in identifying landing reference objects, and proposing a cost function to establish the relationship between the color of real objects and the reflection intensity data of lidar point clouds, thereby improving the drone's recognition effect of landing reference objects and further optimizing the drone's positioning landing; based on the fusion of lidar and camera data, and leveraging the reflection intensity advantage of lidar, the maximum distance measurement range of drone positioning landing can reach more than 27 meters, greatly expanding the application range of drones.

[0045] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0046] The invention will be further described below with reference to the accompanying drawings:

[0047] Figure 1 This is a schematic diagram of the drone fixed-point landing method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the method for calculating the center position of the Aruco code according to an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0050] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to indicate orientation or positional relationship for the convenience of describing the embodiments and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0051] A method for precise landing of UAVs based on lidar and camera data fusion, applicable to UAVs using Livox lidar; please refer to the appendix. Figure 1 ,include:

[0052] Step A01) The camera captures an image below the drone, identifies the Aruco code in the image, obtains the two-dimensional point coordinates of the Aruco code, and establishes the XOY plane based on the two-dimensional point coordinates.

[0053] Step A02) The lidar scans the area below the drone to obtain three-dimensional point cloud and reflection intensity data. Based on the reflection intensity data, the Aruco code is identified, and the three-dimensional point cloud coordinates of the Aruco code are obtained.

[0054] Methods for identifying Aruco codes based on reflection intensity data include:

[0055] R L R represents the maximum reflectance of the black pattern. H This represents the minimum reflectance of the white pattern.

[0056] Define an intermediate interval [β1, β2], β1 = ((σ-1)R L +R H ) / σ,β2=(R L +(σ-1)R H ) / σ, where σ is a preset constant value;

[0057] If the reflection intensity of a three-dimensional point is less than β1, the three-dimensional point is included in the black pattern area; if the reflection intensity of a three-dimensional point is greater than β2, the three-dimensional point is included in the white pattern area.

[0058] The Aruco code is identified based on the black and white patterned areas.

[0059] The cost function between the reflection intensity data and the pattern color of the Aruco code is:

[0060]

[0061] Where, x i f1(x) represents the reflection intensity of the i-th three-dimensional point. i This is used to determine whether the reflection intensity of a three-dimensional point is within the intermediate interval [β1, β2]. C represents the square grid of the Aruco code. Representing a three-dimensional point, Used to determine whether square grid C contains three-dimensional points. V i c represents the four vertices of the cell corresponding to the i-th point. i This indicates that based on the reflection intensity x i Estimated color value, Representing a three-dimensional point The corresponding pattern colors are 0 for black and 1 for white. f3 is used to determine the location of 3D points that are incorrectly mapped in the 3D point cloud. f3 = min(Δx1,Δx2) + min(Δy1,Δy2).

[0062] Based on the cost function between the reflection intensity data and the pattern color of the Aruco code, the set of points on the XOY plane can be obtained as follows:

[0063]

[0064] Among them, T r (θ M ,T M ,P i M ) represents the lidar matrix.

[0065] The method for determining the location of 3D points with mapping errors in a 3D point cloud is as follows:

[0066] When a point cloud incorrectly falls into a white square, the position of the point cloud that incorrectly falls into a white square is determined by f3 = min(Δx1, Δx2) + min(Δy1, Δy2), where Δx1 represents the point cloud... To V i The distance to the left of a defined square grid, Δx2, represents the point cloud. To V i The distance to the right of a defined square grid, Δy1 represents the point cloud. To V i The distance from the top edge of a defined square grid, Δy2 represents the point cloud. To V i The distance from the bottom edge of a defined square grid;

[0067] When a point cloud erroneously falls outside the Aruco code, the location of the point cloud erroneously falling outside the Aruco code is determined by f3 = min(Δx1, Δx2) + min(Δy1, Δy2), where Δx1 represents the point cloud... The distance to the left side of square grid C, Δx2 represents the point cloud. The distance to the right side of square grid C, Δy1 represents the point cloud. The distance to the top edge of the square grid C, Δy2 represents the point cloud. The distance to the bottom edge of square grid C.

[0068] Step A03) Calculate the transformation matrix M between the camera's two-dimensional point coordinates and the lidar's three-dimensional point cloud coordinates, and project the three-dimensional point cloud onto the XOY plane;

[0069] The method for calculating the transformation matrix M is as follows:

[0070]

[0071] Where (u1, v1) represents the coordinates of a two-dimensional point detected by the camera, and (x, y, z) represents the coordinates of a three-dimensional point cloud captured by the lidar. and Both represent the camera's focal length, R1 is the rotation matrix, and T1 is the translation matrix.

[0072] Step A04) Calculate the center position of the Aruco code based on the projected 3D point cloud;

[0073] Step A05) Establish a coordinate system in the XOY plane with the center position as the origin;

[0074] The method for establishing a coordinate system in the XOY plane by taking the center position as the origin is as follows:

[0075] Align the center position with the origin of the XOY coordinate system;

[0076] Subtract the value of the center position from the value of the rotation point P. n×3,fc The average value is used to obtain the coordinate system of the origin of the XOY coordinate system after rotation.

[0077] Step A06) Map the projected 3D point cloud data onto the coordinate system;

[0078] Step A07) Identify Aruco code corner points, calculate the relative position of the UAV and the Aruco code corner points, and generate UAV movement data;

[0079] Step A08) Repeat the above steps until the drone is aligned with the corner point of the Aruco code, and complete the drone's pinpoint landing.

[0080] Please see the appendix Figure 2 The method for calculating the center position of the Aruco code is as follows:

[0081] Step B01) Transfer the 3D point cloud matrix of the Aruco code along the coordinate system P of the lidar. b =(a1,a2,a3) T The three basis vectors a1, a2 and a3 in the equation are decomposed;

[0082] Step B02) Combine two three-dimensional points and connect them to form a line segment. Calculate the ratio of the line segment to the basis vectors a1, a2 and a3, and denote them as Δ1, Δ2 and Δ3 respectively.

[0083] Step B03) Process the point cloud using a random sampling consensus algorithm to form an estimation plane;

[0084] Step B04) Line segments with a ratio Δ3 less than 0.01 are considered to be in the same plane, and points in the same plane are projected onto the estimated plane;

[0085] Step B05) Rotate the coordinates of all points projected onto the estimation plane to the XOY plane using a matrix transformation, and reconnect Δ1, Δ2 with the X and Y axes through the rotation point. Alignment, where P is defined n×3,fFor the fitting point, Let P be the fitting point n×3,f The rotation matrix;

[0086] Step B06) Calculate the rotation point P n×3,fc rotation matrix This is the center position of the Aruco code.

[0087] The beneficial technical effects of this embodiment include: adopting a drone positioning landing method based on the fusion of lidar and camera data, which combines the advantages of camera and lidar in identifying landing reference objects, and proposing a cost function between the color of real objects and the reflection intensity data of lidar point clouds, thereby improving the drone's recognition effect of landing reference objects and further optimizing the drone's positioning landing; based on the fusion of lidar and camera data, and with the advantage of the reflection intensity of lidar, the maximum distance measurement range of drone positioning landing can reach more than 27 meters, greatly expanding the application range of drones.

[0088] The landing point reference object identified in this embodiment is the Aruco code. However, the landing point reference objects identified by fusing LiDAR point cloud and camera image data, such as H-shaped markers, Apriltags patterns, and custom special patterns, are all similar to those in this embodiment and should all be within the protection scope of this invention.

[0089] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method for drone pinpoint landing based on fusion of lidar and camera data, for landing of a drone using livox lidar, characterized in that, include: The camera captures an image below the drone, identifies the Aruco code in the image, obtains the two-dimensional point coordinates of the Aruco code, and establishes the XOY plane based on the two-dimensional point coordinates. The lidar scans the area beneath the drone to obtain 3D point cloud and reflection intensity data. Based on the reflection intensity data, the Aruco code is identified, and the 3D point cloud coordinates of the Aruco code are obtained. Calculate the transformation matrix M between the 2D point coordinates of the camera and the 3D point cloud coordinates of the lidar, and project the 3D point cloud onto the XOY plane; Calculate the center position of the Aruco code based on the projected 3D point cloud; Establish a coordinate system in the XOY plane with the center position as the origin; The projected 3D point cloud data is mapped onto the coordinate system; Identify Aruco code corner points, calculate the relative position of the drone with the Aruco code corner points, and generate drone movement data; Repeat the above steps until the drone is aligned with the corner point of the Aruco code, and complete the pinpoint landing; The cost function between the reflection intensity data and the pattern color of the Aruco code is: ， wherein, represents the reflection intensity of the i-th three-dimensional point, for determining whether the reflection intensity of a three-dimensional point is in the range of the intermediate interval , C represents a square grid of an Aruco code, represents a three-dimensional point, for determining whether the square grid C contains the three-dimensional point , represents the four vertices of the grid corresponding to the i-th point, represents the color value estimated from the reflection intensity , , represents the pattern color corresponding to the three-dimensional point , 0 represents black and 1 represents white, to determine the three-dimensional point position of the three-dimensional point cloud mapping error, .

2. The method for UAV fixed-point landing based on lidar and camera data fusion according to claim 1, characterized in that, Methods for identifying Aruco codes based on reflection intensity data include: This represents the maximum reflectance of the black pattern. This represents the minimum reflectance of the white pattern. Define an intermediate interval , , ,in This is a preset constant value; If the reflection intensity of the three-dimensional point is less than Then the three-dimensional point will be included in the black pattern area. If the reflection intensity of the three-dimensional point is greater than 1, the three-dimensional point will be included in the black pattern area. Then the three-dimensional points will be included in the white pattern area; The Aruco code is identified based on the black and white patterned areas.

3. The method for UAV fixed-point landing based on lidar and camera data fusion according to claim 1, characterized in that, The method for determining the location of 3D points with mapping errors in a 3D point cloud is as follows: When a point cloud error occurs and falls into a white square, it is achieved through... Determine the point cloud location where the error falls into a white square, among which Point cloud arrive The distance to the left of a given square grid. Point cloud arrive The distance to the right of a given square grid. Point cloud arrive The distance from the top edge of a given square grid. Point cloud arrive The distance from the bottom edge of a defined square grid; When a point cloud error falls outside the Aruco code, it is handled by... Determine the point cloud location where the error falls outside the Aruco code, where Point cloud The distance to the left side of square grid C, Point cloud The distance to the right side of square grid C, Point cloud The distance to the top edge of square grid C. Point cloud The distance to the bottom edge of square grid C.

4. The method for UAV fixed-point landing based on lidar and camera data fusion according to claim 1, characterized in that, The set of points in the XOY plane is: ， in, This is a lidar matrix.

5. A method for UAV fixed-point landing based on lidar and camera data fusion according to claim 1, characterized in that, The method for calculating the transformation matrix M is as follows: ， in, This represents the coordinates of the two-dimensional point detected by the camera. This represents the coordinates of the three-dimensional point cloud captured by the lidar. and Both refer to the camera's focal length. Let be a rotation matrix. It is a translation matrix.

6. A method for UAV precise landing based on lidar and camera data fusion according to claim 1, characterized in that, The method for calculating the center position of the Aruco code is as follows: The 3D point cloud matrix of the Aruco code is mapped along the coordinate system of the lidar. The three basis vectors Decompose; Combine two 3D points and connect them to form a line segment. Calculate the relationship between the line segment and the basis vectors. The ratios of are denoted as . ; The point cloud is processed using a random sampling consensus algorithm to form an estimated plane; ratio Line segments with a value less than 0.01 are considered to be in the same plane, and points in the same plane are projected onto the estimated plane; The coordinates of all points projected onto the estimation plane are rotated to the XOY plane through a matrix transformation, and then... Rotation point with respect to the X and Y axes Alignment, where, is defined For the fitting point, For the fitting point The rotation matrix; Calculate the rotation point rotation matrix This is the center position of the Aruco code.

7. A method for UAV fixed-point landing based on lidar and camera data fusion according to claim 1, characterized in that, The method for establishing a coordinate system in the XOY plane with the center position as the origin is as follows: Align the center position with the origin of the XOY coordinate system; Subtract the rotation point from the value at the center position. The average value is used to obtain the coordinate system of the origin of the XOY coordinate system after rotation.

Images