Unmanned aerial vehicle camera and laser radar calibration method and system based on structural features

Abstract

The application provides a kind of unmanned aerial vehicle camera and laser radar calibration method and system based on structural features, wherein the method comprises projecting point cloud structural feature characteristics onto the pixel coordinate system of the image as a template image according to the initial extrinsic parameter before correction;Image structural features are used as the image to be searched;Slide the template image in the image to be searched to obtain the pixel coordinates of the highest similarity in the search image;Determine the corrected initial extrinsic parameter according to the pixel coordinates of the highest similarity;Determine the point of the target point cloud structural feature point projected in the pixel coordinate system as the target projection point according to the corrected initial extrinsic parameter;Determine the distance between the target number of image structural feature points and the target projection point respectively;Determine the final extrinsic parameter according to the distance between the target number of image structural feature points and the target projection point respectively.The calibration method proposed in the application effectively improves the success rate of calibration under the condition of large initial extrinsic parameter deviation, and improves the robustness of the feature-based calibration method.

Description

Technical Field

[0001] This invention belongs to the field of UAV positioning technology, and particularly relates to a UAV camera and lidar calibration method and system based on structural features. Background Technology

[0002] For drones, accurate perception and positioning in complex environments are fundamental for obstacle avoidance planning and mission execution. Drones require accurate perception and positioning information to improve safety, enhance efficiency, and ensure stable operation. Cameras and LiDAR complement each other and are often used together in perception and positioning algorithms. The extrinsic parameters of both cameras and LiDAR map their data to the same space, which is the basis for achieving high-precision perception and positioning. Therefore, accurate extrinsic parameters are crucial for subsequent perception and positioning.

[0003] The extrinsic parameters of the camera and LiDAR can be obtained through calibration algorithms. Traditional camera and LiDAR calibration methods are mainly based on calibration boards. This method requires placing a calibration board in the scene and calculating the transformation relationship between the camera and LiDAR by selecting corresponding corner points in the point cloud and image. The calibration board-based method is simple and easy to understand and can be implemented using Autoware or MATLAB toolkits. However, manually selecting feature points cannot achieve high accuracy, and the process is tedious and time-consuming.

[0004] Therefore, feature-based calibration methods have gradually emerged. Cameras and LiDAR simultaneously capture data in a scene, calculating their extrinsic relationships by sharing feature points within the scene. These methods do not require specific calibration boards, are applicable to a wider range of scenarios, and are convenient and fast. Common features include point features, line features, surface features, and higher-order semantic features. However, when feature points are mismatched, the extrinsic parameters calculated by these methods have large errors and poor robustness. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for calibrating UAV cameras and lidar based on structural features.

[0006] In a first aspect, the present invention provides a method for calibrating a UAV camera and lidar based on structural features, comprising:

[0007] Extracting point cloud structural features based on adaptive voxels obtained by lidar; Extract structural features from images acquired by the camera; Based on the initial extrinsic parameters before correction, the point cloud structural features are projected onto the pixel coordinate system of the image acquired by the camera to serve as a template image; where the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera; Use image structural features as the image to be searched; Slide the template image in the image to be searched, determine the similarity between the sliding area in the image to be searched and the target area in the template image, and obtain the pixel coordinates of the point with the highest similarity in the search image; The PnP algorithm is used to determine the corrected initial extrinsic parameters based on the pixel coordinates of the highest similarity point in the search image. The points on the pixel coordinate system that are projected onto the structural feature points of the target point cloud based on the corrected initial extrinsic parameters are used as the target projection points. Search for the number of image structural feature points in the image to be searched that are closest to the target projection point; Determine the distances between the target's image structural feature points and the target's projection points; The final extrinsic parameters are determined based on the distances between the target's image structural feature points and the target's projection points; where the final extrinsic parameters represent the final pose transformation relationship between the lidar and the camera.

[0008] Furthermore, the extraction of point cloud structural features based on adaptive voxels obtained by the lidar includes: The decentralized covariance matrix of the point cloud in each voxel is calculated using the following formula. C : ; in, n The number of points in each voxel; p i In each voxel the first i One point; In each voxel n The mean of the points; T This is the transpose of the matrix; Obtain the covariance matrix of the target voxel C The target voxel is divided into multiple feature values. The first condition is that the ratio of the maximum feature value to the minimum feature value is greater than a preset threshold, so as to determine the points in the target voxel form a plane. The second condition is that the ratio of the maximum feature value to the minimum feature value is not greater than a preset threshold, so as to determine the target voxel is divided into multiple smaller voxels until the maximum number of divisions is reached. By iterating through the voxels corresponding to the first condition, multiple first planes are obtained; Merge any two first planes according to the following constraints: ; in, n 1 and n 2 are the normal vectors of the two planes, respectively; c 1 and c 2 are the center points of the two planes, respectively; Traverse all merged second planes and determine the angle between any two second planes. Obtain the intersection line corresponding to the included angle of the target degree; Points on all intersection lines are taken as structural features of the point cloud in all voxels.

[0009] Furthermore, the extraction of structural features from the image acquired by the camera includes: The pixel coordinates of the image pixels acquired by the camera are calculated using the following formula. x shaft and y Pixel gradient on the axis: ; in, g x ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system x Pixel gradient on the axis; g y ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system y Pixel gradient on the axis; i ( x j , y j ) is in the pixel coordinate system ( x j , y j The pixel value at position (). Calculate the pixel coordinate system using the following formula ( x j , y j gradient magnitude of pixel at position ) G ( x j , y j and gradient direction LLA : ; ; In this process, the line segment support region, composed of multiple pixels, obtains a line segment support region gradient direction through the gradient directions of the pixels within the region. The line segment support region is surrounded by multiple isolated pixels. One isolated pixel is randomly selected from the sorted list, and the orientation tolerance between the gradient direction of the target isolated pixel and the gradient direction of the line segment support region is calculated to determine if it is less than a threshold. r If the tolerance value meets the threshold r Then, the isolated target pixel is included in the line segment support domain, and the line segment support domain after the isolated target pixel is added is updated. The updated line segment support domain is rectangularized to obtain a rectangular support domain. Calculate the coordinates of the center point of the rectangular support area using the following formula: l x , l y ): ; Wherein, Region is the rectangular support domain; The principal direction of the rectangular support region after the line segment support region is rectangularized is calculated using the following formula. M : ; Multiple rectangular support regions that meet preset conditions are used as image structural features.

[0010] Further, the step of sliding the template image in the image to be searched, determining the similarity between the sliding region in the image to be searched and the target region in the template image, and obtaining the pixel coordinates of the point with the highest similarity in the search image includes: The similarity between the sliding region in the search image and the target region in the template image is calculated using the following formula: ; in, R sq,diff The similarity between the sliding region on the image to be searched and the target region of the template image; K Template image; I The image to be searched; The coordinates of the matrix elements of the template image and the image to be searched enclosed in the search box; x 1, y 1) is the coordinate of the top-left element of the current search box in the image matrix to be searched.

[0011] Further, the step of determining the points projected onto the target point cloud structural feature points in the pixel coordinate system based on the corrected initial extrinsic parameters, as the target projection points, includes: The points projected onto the pixel coordinate system from the structural feature points of the target point cloud are calculated using the following formula: ; in, For the first i' Each point cloud structure feature point; f 1(·) represents the camera distortion model; π (·) represents a pinhole camera model; These are the corrected initial extrinsic parameters; for The point projected onto the pixel coordinate system.

[0012] Further, determining the distances between the target number of image structural feature points and the target projection point includes: The distance between structural feature points of the target image and the target projection points is calculated using the following formula: ; in, Among the n image structural feature points in the image to be searched that are closest to the target projection point, the nth... i'' Image structural feature points and The distance; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

[0013] Furthermore, the step of determining the final extrinsic parameters based on the distances between the target number of image structural feature points and the target projection point includes: The final external parameters are calculated using the following formula. : ; in, External parameters to be optimized; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

[0014] Secondly, the present invention provides a method for calibrating UAV cameras and lidar based on structural features, comprising: The first extraction module is used to extract the point cloud structural features based on adaptive voxels obtained by the lidar. The second extraction module is used to extract structural features from the images acquired by the camera; The projection module is used to project the point cloud structural features onto the pixel coordinate system of the image acquired by the camera based on the initial extrinsic parameters before correction, so as to serve as a template image; where the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera; The first determining module is used to use image structural features as the image to be searched; The second determining module is used to slide the template image in the image to be searched, determine the similarity between the sliding area in the image to be searched and the target area of ​​the template image, and obtain the pixel coordinates of the point with the highest similarity in the search image. The third determination module is used to determine the corrected initial extrinsic parameters based on the pixel coordinates of the highest similarity point in the search image using the PnP algorithm. The fourth determination module is used to determine the points in the pixel coordinate system projected onto the structural feature points of the target point cloud based on the corrected initial extrinsic parameters, so as to serve as the target projection points; The search module is used to search for a number of image structural feature points in the image to be searched that are closest to the target projection point. The fifth determining module is used to determine the distances between the target's image structural feature points and the target's projection points. The sixth determining module is used to determine the final extrinsic parameters based on the distances between the target's image structural feature points and the target's projection points; wherein, the final extrinsic parameters are the final pose transformation relationship between the lidar and the camera.

[0015] Thirdly, the present invention provides a computer device including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the UAV camera and lidar calibration method based on structural features described in the first aspect.

[0016] Fourthly, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the UAV camera and lidar calibration method based on structural features described in the first aspect.

[0017] This invention provides a method and system for calibrating UAV cameras and LiDAR based on structural features. The method includes: extracting point cloud structural features based on adaptive voxels acquired by the LiDAR; extracting image structural features acquired by the camera; projecting the point cloud structural features onto the pixel coordinate system of the camera-acquired image according to initial extrinsic parameters before calibration, using this as a template image; wherein the initial extrinsic parameters are the initial pose transformation relationship between the LiDAR and the camera; using the image structural features as the search image; sliding the template image within the search image, determining the similarity between the sliding region in the search image and the target region in the template image, and obtaining the search result. The method involves identifying the pixel coordinates of the highest similarity point in the image, determining the initial extrinsic parameters using the PnP algorithm based on these coordinates, projecting the target point cloud structural feature points into the pixel coordinate system, and using these initial extrinsic parameters as the target projection points. The method then searches for a number of target image structural feature points in the search image that are closest to the target projection points. The distances between these target image structural feature points and the target projection points are determined. Finally, the extrinsic parameters are determined based on these distances, where the final extrinsic parameters represent the final pose transformation relationship between the LiDAR and the camera. This invention effectively improves the calibration success rate when initial extrinsic parameter deviations are large and enhances the robustness of feature-based calibration methods. Attached Figure Description

[0018] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for calibrating a UAV camera and lidar based on structural features, provided for an embodiment of the present invention; Figure 2 This is a diagram illustrating the structural feature extraction and matching of a traditional calibration method provided in this embodiment of the invention. Figure 3 A diagram illustrating the structural feature extraction and matching process in the UAV camera and lidar calibration method based on structural features provided in this embodiment of the invention. Figure 4 A comparison diagram of the reprojection error between the conventional algorithm and the method of the present invention provided in the embodiments of the present invention; Figure 5 This is a schematic diagram of a UAV camera and lidar calibration system based on structural features, provided as an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In one embodiment, such as Figure 1 As shown, this embodiment of the invention provides a method for calibrating UAV cameras and lidar based on structural features, including: Step 101: Extract the point cloud structure features based on adaptive voxels obtained by the lidar.

[0022] In this step, the point cloud structural feature extraction based on adaptive voxels first segments the planar features in the point cloud, and then calculates the intersection lines between different planes as structural features.

[0023] For example, the decentralized covariance matrix of the point cloud in each voxel is calculated according to the following formula. C : .

[0024] in, n The number of points in each voxel; p i In each voxel the first i One point; In each voxel n The mean of the points; T This is the transpose of the matrix.

[0025] Obtain the covariance matrix of the target voxel C Multiple eigenvalues, such as Among them, the first condition is that the ratio of the largest eigenvalue to the smallest eigenvalue is greater than a preset threshold. , τ A preset threshold is used to determine the points in the target voxel to form a plane, and the target voxel is marked as PLANE. The ratio of the maximum eigenvalue to the minimum eigenvalue is not greater than the preset threshold as a second condition to determine that the target voxel is divided into multiple smaller voxels until the maximum number of divisions is reached. For example, the target voxel is further subdivided into eight smaller voxels with a length, width and height of half that of the original voxel until the maximum number of layers is reached.

[0026] Traverse the voxels corresponding to the first condition, that is, traverse the voxels marked as PLANE, to obtain multiple first planes.

[0027] Find if there are any planes that can be merged, and merge any two first planes according to the following constraints: .

[0028] in, n 1 and n 2 are the normal vectors of the two planes, respectively; c 1 and c 2 are the center points of the two planes, respectively.

[0029] Traverse all merged second planes and determine the angle between any two second planes. Obtain the intersection line corresponding to the included angle of the target degree; Points on all intersection lines are taken as structural features of the point cloud in all voxels.

[0030] For example, this step concludes by traversing the merged list of planes, calculating the angle between two planes, and selecting two planes with an angle close to 90 degrees. The points on the intersection line are then calculated based on the plane equations, representing the structural features of the point cloud.

[0031] Step 102: Extract the structural features of the image acquired by the camera.

[0032] In this step, for example, the image pixels acquired by the camera are calculated in pixel coordinate system according to the following formula. x shaft and y Pixel gradient on the axis: .

[0033] in, g x ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system x Pixel gradient on the axis; g y ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system y Pixel gradient on the axis; i ( x j , y j ) is in the pixel coordinate system ( x j , yj The pixel value at point ().

[0034] Calculate the pixel coordinate system using the following formula ( x j , y j gradient magnitude of pixel at position ) G ( x j , y j and gradient direction LLA : .

[0035] .

[0036] The baseline length is determined by the angle between pixels in the image and the baseline. Then, a region growing algorithm is used to merge approximately identical pixel regions to obtain the baseline support region. The line segment support region, composed of multiple pixels, has a gradient direction obtained from the gradient directions of the pixels within that region. If there are multiple isolated pixels surrounding the line segment support region, one isolated pixel (NOT USED) is randomly selected from the sorted list. The tolerance value between the gradient direction of the target isolated pixel and the gradient direction of the line segment support region is calculated to see if it is less than a threshold. r If the tolerance value meets the threshold r Then change the target isolated pixel to USED and include it in the line segment support domain, and update the line segment support domain after adding the target isolated pixel.

[0037] The updated line segment support domain is rectangularized to obtain a rectangular support domain.

[0038] Each line segment support region may correspond to a line segment, requiring determination based on each rectangular support region. The pixel density within the rectangular support region determines whether to divide it into multiple rectangular regions. For example, the center point coordinates of the rectangular support region are calculated using the following formula (…). l x , l y ): .

[0039] Wherein, Region is a rectangular support region.

[0040] The principal direction of the rectangular support region after the line segment support region is rectangularized is calculated using the following formula. M : .

[0041] Multiple rectangular support regions that meet preset conditions are used as image structural features.

[0042] Step 103: Project the point cloud structural features onto the pixel coordinate system of the image acquired by the camera based on the initial extrinsic parameters before correction, so as to use it as a template image; wherein, the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera.

[0043] Step 104: Use the image structural features as the image to be searched.

[0044] Step 105: Slide the template image in the image to be searched, determine the similarity between the sliding area in the image to be searched and the target area in the template image, and obtain the pixel coordinates of the point with the highest similarity in the search image.

[0045] In steps 103-105, for example, the similarity between the sliding region on the image to be searched and the target region of the template image is calculated according to the following formula: .

[0046] in, R sq,diff The similarity between the sliding region on the image to be searched and the target region of the template image; K Template image; I The image to be searched; The coordinates of the matrix elements of the template image and the image to be searched enclosed in the search box; x 1, y 1) is the coordinate of the top-left element of the current search box in the image matrix to be searched.

[0047] Step 106: Determine the corrected initial extrinsic parameters using the PnP algorithm based on the pixel coordinates of the highest similarity point in the search image.

[0048] Step 107: Determine the points in the pixel coordinate system that project the target point cloud structure feature points into the pixel coordinate system based on the corrected initial extrinsic parameters, and use them as the target projection points.

[0049] In this step, the LM (Levenberg-Marquardt) algorithm is used to optimize the residuals, which are the distances from the point cloud structural feature points to the corresponding image structural features.

[0050] For example, the points projected onto the target point cloud structural feature points in the pixel coordinate system are calculated according to the following formula: .

[0051] in, For the first i' Each point cloud structure feature point; f 1(·) represents the camera distortion model; π(·) represents a pinhole camera model; These are the corrected initial extrinsic parameters; for The point projected onto the pixel coordinate system.

[0052] Step 108: Search for the number of image structure feature points in the image to be searched that are closest to the target projection point.

[0053] Step 109: Determine the distances between the target number of image structural feature points and the target projection point.

[0054] In this step, for example, the distance between the structural feature points of the target image and the target projection points is calculated according to the following formula: .

[0055] in, Among the n image structural feature points in the image to be searched that are closest to the target projection point, the nth... i'' Image structural feature points and The distance; n 3 is the target quantity ( The covariance matrix of image structural feature points The eigenvector corresponding to the smallest eigenvalue; ; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

[0056] Step 1010: Determine the final extrinsic parameters based on the distances between the target number of image structural feature points and the target projection point; wherein, the final extrinsic parameters are the final pose transformation relationship between the lidar and the camera.

[0057] In this step, for example, the final extrinsic parameters are calculated according to the following formula. : .

[0058] in, External parameters to be optimized; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

[0059] To verify the effectiveness of the structural feature-based UAV camera and LiDAR calibration method proposed in this embodiment of the invention, a comparison was made between the traditional feature-based calibration method and the method proposed in this embodiment of the invention in a real-world scenario. The sensors used in practice and their parameters are shown in Table 1. Table 1 Sensor Parameters

[0060] For data from the same scenario, 20 experiments were conducted with randomly set initial values, and the reprojection error was calculated to measure the calibration effect. For example... Figure 2 and Figure 3 As shown, traditional feature-based calibration methods are prone to mismatches when the initial values ​​are not ideal. However, the method proposed in this embodiment of the invention reduces mismatches through coarse registration and improves the robustness of the algorithm.

[0061] The algorithm proposed in this embodiment of the invention selects matching point pairs in the point cloud and the image, and calculates the reprojection error. For example... Figure 4 The figure shows a box plot of the reprojection error, with the values ​​from top to bottom representing the maximum, upper quartile, median, lower quartile, and minimum. Figure 3 It can be seen that the maximum, upper quartile, median, lower quartile, and minimum reprojection error of the traditional method are 14.93853, 13.3681, 10.05339, 9.34369, and 6.35100, respectively. The maximum, upper quartile, median, lower quartile, and minimum reprojection error of the method proposed in this embodiment are 12.75247, 11.06358, 7.68666, 6.49076, and 4.55181, respectively. Therefore, the method proposed in this embodiment has better robustness and higher accuracy.

[0062] The UAV camera and lidar calibration method based on structural features provided in this invention effectively improves the calibration success rate when the initial external parameter deviation is large, and improves the robustness of feature-based calibration methods.

[0063] Based on the same inventive concept, this invention also provides a UAV camera and lidar calibration system based on structural features. Since the principle of this system in solving the problem is similar to that of the UAV camera and lidar calibration method based on structural features, the implementation of this system can refer to the implementation of the UAV camera and lidar calibration method based on structural features, and the repeated parts will not be described again.

[0064] In another embodiment, the UAV camera and lidar calibration system based on structural features provided in this invention, such as... Figure 5 As shown, it includes: The first extraction module 10 is used to extract the point cloud structural features based on adaptive voxels obtained by the lidar.

[0065] The second extraction module 20 is used to extract the structural features of the images acquired by the camera.

[0066] The projection module 30 is used to project the point cloud structural features onto the pixel coordinate system of the image acquired by the camera based on the initial extrinsic parameters before correction, so as to serve as a template image; wherein, the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera.

[0067] The first determining module 40 is used to use image structural features as the image to be searched.

[0068] The second determining module 50 is used to slide the template image in the image to be searched, determine the similarity between the sliding area in the image to be searched and the target area of ​​the template image, and obtain the pixel coordinates of the point with the highest similarity in the search image.

[0069] The third determining module 60 is used to determine the corrected initial extrinsic parameters based on the pixel coordinates of the highest similarity point in the search image using the PnP algorithm; wherein, the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera.

[0070] The fourth determining module 70 is used to determine the points in the pixel coordinate system projected onto the structural feature points of the target point cloud based on the corrected initial extrinsic parameters, so as to serve as the target projection points.

[0071] Search module 80 is used to search for a number of image structural feature points in the image to be searched that are closest to the target projection point.

[0072] The fifth determining module 90 is used to determine the distances between the target number of image structural feature points and the target projection point.

[0073] The sixth determining module 100 is used to determine the final extrinsic parameters based on the distances between the target number of image structural feature points and the target projection point; wherein, the final extrinsic parameters are the final pose transformation relationship between the lidar and the camera.

[0074] For example, the first extraction module includes: The first computational unit is used to calculate the decentralized covariance matrix of the point cloud in each voxel according to the following formula. C : .

[0075] in, n The number of points in each voxel; p i In each voxel the first i One point; In each voxel n The mean of the points; T This is the transpose of the matrix.

[0076] The first acquisition unit is used to acquire the covariance matrix of the target voxel. CThe target voxel is divided into multiple feature values. The first condition is that the ratio of the maximum feature value to the minimum feature value is greater than a preset threshold, so as to determine the points in the target voxel form a plane. The second condition is that the ratio of the maximum feature value to the minimum feature value is not greater than a preset threshold, so as to determine the target voxel to be divided into multiple smaller voxels until the maximum number of divisions is reached.

[0077] The first traversal unit is used to traverse the voxels corresponding to the first condition to obtain multiple first planes.

[0078] The merging unit is used to merge any two first planes according to the following constraints: .

[0079] in, n 1 and n 2 are the normal vectors of the two planes, respectively; c 1 and c 2 are the center points of the two planes, respectively.

[0080] The second traversal unit is used to traverse all the merged second planes and determine the included angle between any two second planes.

[0081] The second acquisition unit is used to acquire the intersection line corresponding to the included angle of the target degree.

[0082] The first determining unit is used to treat all points on the intersection lines as structural features of the point cloud in all voxels.

[0083] For example, the second extraction module includes: The second calculation unit is used to calculate the image pixels acquired by the camera in the pixel coordinate system according to the following formula. x shaft and y Pixel gradient on the axis: .

[0084] in, g x ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system x Pixel gradient on the axis; g y ( x j , y j )for( x j , yj The pixel at position ) in the pixel coordinate system y Pixel gradient on the axis; i ( x j , y j ) is in the pixel coordinate system ( x j , y j The pixel value at point ().

[0085] The third calculation unit is used to calculate the pixel coordinates in the pixel coordinate system according to the following formula ( x j , y j gradient magnitude of pixel at position ) G ( x j , y j and gradient direction LLA : .

[0086] .

[0087] In this process, the line segment support region, composed of multiple pixels, obtains a line segment support region gradient direction through the gradient directions of the pixels within the region. The line segment support region is surrounded by multiple isolated pixels. One isolated pixel is randomly selected from the sorted list, and the orientation tolerance between the gradient direction of the target isolated pixel and the gradient direction of the line segment support region is calculated to determine if it is less than a threshold. r If the tolerance value meets the threshold r Then, the isolated target pixel is included in the line segment support domain, and the line segment support domain after the isolated target pixel is added is updated. The rectangularization unit is used to rectangularize the updated line segment support domain to obtain a rectangular support domain.

[0088] The fourth calculation unit is used to calculate the coordinates of the center point of the rectangular support domain according to the following formula ( l x , l y ): .

[0089] Wherein, Region is a rectangular support region.

[0090] The fifth calculation unit is used to calculate the principal direction of the rectangular support domain after the line segment support domain is rectangularized, according to the following formula. M : .

[0091] The second determining unit is used to take multiple rectangular support domains that meet preset conditions as image structural features.

[0092] For example, the second determining module includes: The sixth calculation unit is used to calculate the similarity between the sliding region on the image to be searched and the target region of the template image according to the following formula: .

[0093] in, R sq,diff The similarity between the sliding region on the image to be searched and the target region of the template image; K Template image; I The image to be searched; The coordinates of the matrix elements of the template image and the image to be searched enclosed in the search box; x 1, y 1) is the coordinate of the top-left element of the current search box in the image matrix to be searched.

[0094] For example, the fourth determining module includes: The seventh calculation unit is used to calculate the points projected onto the pixel coordinate system by the structural feature points of the target point cloud according to the following formula: .

[0095] in, For the first i' Each point cloud structural feature point; f 1(·) represents the camera distortion model; π (·) represents a pinhole camera model; These are the corrected initial extrinsic parameters; for The point projected onto the pixel coordinate system.

[0096] For example, the fifth determining module includes: The eighth calculation unit is used to calculate the distance between the structural feature points of the target image and the target projection point according to the following formula: .

[0097] in, Among the n image structural feature points in the image to be searched that are closest to the target projection point, the nth... i'' Image structural feature points and The distance; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

[0098] For example, the sixth determining module includes: The ninth calculation unit is used to calculate the final extrinsic parameters according to the following formula. : .

[0099] in, External parameters to be optimized; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

[0100] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0101] In another embodiment, the present invention provides a computer device including a processor and a memory; wherein the processor executes a computer program stored in the memory to implement the steps of the above-described method for calibrating UAV cameras and lidar based on structural features.

[0102] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0103] In another embodiment, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the above-described method for calibrating UAV cameras and lidar based on structural features.

[0104] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The systems, devices, and storage media disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant details can be found in the method section.

[0106] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0107] The present invention has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A method for calibrating UAV cameras and lidar based on structural features, characterized in that, include: Extracting point cloud structural features based on adaptive voxels obtained by lidar; Extract structural features from images acquired by the camera; The point cloud structural features are projected onto the pixel coordinate system of the image acquired by the camera based on the initial extrinsic parameters before correction, serving as a template image; where the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera; Use image structural features as the image to be searched; Slide the template image in the image to be searched, determine the similarity between the sliding area in the image to be searched and the target area in the template image, and obtain the pixel coordinates of the point with the highest similarity in the image to be searched. The PnP algorithm is used to determine the corrected initial extrinsic parameters based on the pixel coordinates of the highest similarity point in the image to be searched. The points on the pixel coordinate system that are projected onto the structural feature points of the target point cloud based on the corrected initial extrinsic parameters are used as the target projection points. Search for the number of image structural feature points in the image to be searched that are closest to the target projection point; Determine the distances between the target's image structural feature points and the target's projection points; The final extrinsic parameters are determined based on the distances between the target's image structural feature points and the target's projection points; where the final extrinsic parameters represent the final pose transformation relationship between the lidar and the camera.

2. The UAV camera and lidar calibration method based on structural features according to claim 1, characterized in that, The extraction of point cloud structural features based on adaptive voxels obtained by lidar includes: The decentralized covariance matrix of the point cloud in each voxel is calculated using the following formula. C : ； in, n The number of points in each voxel; p i In each voxel the first i One point; In each voxel n The mean of the points; T This is the transpose of the matrix; Obtain the covariance matrix of the target voxel C The target voxel is divided into multiple feature values. The first condition is that the ratio of the maximum feature value to the minimum feature value is greater than a preset threshold, so as to determine the points in the target voxel form a plane. The second condition is that the ratio of the maximum feature value to the minimum feature value is not greater than a preset threshold, so as to determine the target voxel is divided into multiple smaller voxels until the maximum number of divisions is reached. By iterating through the voxels corresponding to the first condition, multiple first planes are obtained; Merge any two first planes according to the following constraints: ； in, n 1 and n 2 are the normal vectors of the two planes, respectively; c 1 and c 2 are the center points of the two planes, respectively; Traverse all merged second planes and determine the angle between any two second planes. Obtain the intersection line corresponding to the included angle of the target degree; Points on all intersection lines are taken as structural features of the point cloud in all voxels.

3. The UAV camera and lidar calibration method based on structural features according to claim 1, characterized in that, The extraction of structural features from images acquired by the camera includes: The pixel coordinates of the image pixels acquired by the camera are calculated using the following formula. x shaft and y Pixel gradient on the axis: ； in, g x ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system x Pixel gradient on the axis; g y ( x j , y j )for( x j , y j The pixel at position ) in the pixel coordinate system y Pixel gradient on the axis; i ( x j , y j ) is in the pixel coordinate system ( x j , y j The pixel value at position (). Calculate the pixel coordinate system using the following formula ( x j , y j gradient magnitude of pixel at position ) G ( x j , y j and gradient direction LLA : ； ； In this process, the line segment support region, composed of multiple pixels, obtains a line segment support region gradient direction through the gradient directions of the pixels within the region. The line segment support region is surrounded by multiple isolated pixels. One isolated pixel is randomly selected from the sorted list, and the orientation tolerance between the gradient direction of the target isolated pixel and the gradient direction of the line segment support region is calculated to determine if it is less than a threshold. r If the tolerance value meets the threshold r Then, the isolated target pixel is included in the line segment support domain, and the line segment support domain after the isolated target pixel is added is updated. The updated line segment support domain is rectangularized to obtain a rectangular support domain. Calculate the coordinates of the center point of the rectangular support area using the following formula: l x , l y ): ； Wherein, Region is the rectangular support domain; The principal direction of the rectangular support region after the line segment support region is rectangularized is calculated using the following formula. M : ； Multiple rectangular support regions that meet preset conditions are used as image structural features.

4. The UAV camera and lidar calibration method based on structural features according to claim 1, characterized in that, The step of sliding a template image in the image to be searched, determining the similarity between the sliding region in the image to be searched and the target region in the template image, and obtaining the pixel coordinates of the point with the highest similarity in the image to be searched includes: The similarity between the sliding region in the search image and the target region in the template image is calculated using the following formula: ； in, R sq,diff The similarity between the sliding region on the image to be searched and the target region of the template image; K Template image; I The image to be searched; The coordinates of the matrix elements of the template image and the image to be searched enclosed in the search box; x 1, y 1) is the coordinate of the top-left element of the current search box in the image matrix to be searched.

5. The UAV camera and lidar calibration method based on structural features according to claim 4, characterized in that, The step of determining the points projected onto the target point cloud structural feature points in the pixel coordinate system based on the corrected initial extrinsic parameters, as the target projection points, includes: The points projected onto the pixel coordinate system from the structural feature points of the target point cloud are calculated using the following formula: ； in, For the first i' Each point cloud structural feature point; f 1(·) represents the camera distortion model; π (·) represents a pinhole camera model; These are the corrected initial extrinsic parameters; for The point projected onto the pixel coordinate system.

6. The UAV camera and lidar calibration method based on structural features according to claim 5, characterized in that, Determining the distances between the target number of image structural feature points and the target projection point includes: The distance between structural feature points of the target image and the target projection points is calculated using the following formula: ； in, Among the n image structural feature points in the image to be searched that are closest to the target projection point, the nth... i'' Image structural feature points and The distance; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

7. The UAV camera and lidar calibration method based on structural features according to claim 6, characterized in that, The step of determining the final extrinsic parameters based on the distances between the target number of image structural feature points and the target projection points includes: The final external parameters are calculated using the following formula. : ； in, External parameters to be optimized; n 3 represents the eigenvector corresponding to the eigenvalue with the smallest covariance matrix of the target number of image structural feature points; T This is the transpose of the matrix; The target number of image structural feature points is the center point.

8. A UAV camera and lidar calibration system based on structural features, characterized in that, include: The first extraction module is used to extract the point cloud structural features based on adaptive voxels obtained by the lidar. The second extraction module is used to extract structural features from the images acquired by the camera; The projection module is used to project the point cloud structural features onto the pixel coordinate system of the image acquired by the camera based on the initial extrinsic parameters before correction, so as to serve as a template image; wherein, the initial extrinsic parameters are the initial pose transformation relationship between the lidar and the camera; The first determining module is used to use image structural features as the image to be searched; The second determining module is used to slide the template image in the image to be searched, determine the similarity between the sliding area in the image to be searched and the target area of ​​the template image, and obtain the pixel coordinates of the point with the highest similarity in the image to be searched. The third determination module is used to determine the corrected initial extrinsic parameters based on the pixel coordinates of the highest similarity point in the image to be searched using the PnP algorithm. The fourth determination module is used to determine the points in the pixel coordinate system projected onto the structural feature points of the target point cloud based on the corrected initial extrinsic parameters, so as to serve as the target projection points; The search module is used to search for a number of image structural feature points in the image to be searched that are closest to the target projection point. The fifth determining module is used to determine the distances between the target's image structural feature points and the target's projection points. The sixth determining module is used to determine the final extrinsic parameters based on the distances between the target's image structural feature points and the target's projection points; wherein, the final extrinsic parameters are the final pose transformation relationship between the lidar and the camera.

9. A computer device, characterized in that, It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the UAV camera and lidar calibration method based on structural features as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; when executed by a processor, the computer programs implement the steps of the UAV camera and lidar calibration method based on structural features as described in any one of claims 1-7.

Images