An improved posture correction method for unmanned vehicles based on depth estimation

Abstract

The present invention discloses an improved unmanned vehicle posture correction method based on depth estimation, comprising the following steps: collecting point clouds and images respectively, and obtaining depth images using the Depth Angthing algorithm; inputting point clouds, images and depth images into a posture correction network to obtain a point cloud-image set; solving the point cloud-image set using the EPnP algorithm to obtain an initial posture matrix; extracting depth features from local features of the mixed image, and optimizing the initial posture matrix using the depth features to obtain a final camera posture matrix. The present invention calculates the overlapping area by extracting depth maps, point clouds and image features, and performs preliminary posture estimation, completes secondary registration through the depth map, establishes a polynomial model using the depth value information of each pixel and the distance information of the point cloud projected on the image, and searches for the best position within a small range, thereby driving the movement of all points and the adjustment of the camera posture.

Description

Technical Field

[0001] The invention relates to an unmanned vehicle posture correction method based on depth estimation improvement, belonging to the technical field of unmanned driving. Background Art

[0002] Point cloud and image registration plays a key role in attitude and position correction in unmanned driving. By accurately aligning the data collected by the lidar and camera, it ensures that the vehicle can accurately perceive the three-dimensional structure and visual information of the surrounding environment, thereby achieving precise position positioning and vehicle attitude correction to ensure safe driving of the vehicle. It is also one of the important issues in the fields of computer vision, robotics and augmented reality.

[0003] It involves aligning 2D images with 3D point clouds to achieve accurate scene understanding, virtual object overlay, 3D reconstruction and other applications. In recent years, with the development of deep learning technology, deep learning-based image to point cloud registration methods have gradually become a research hotspot. These methods use deep learning models to learn the complex mapping relationship between images and point clouds to achieve an end-to-end registration process.

[0004] Although there has been great progress in the research on the accuracy of point cloud and image registration, it still does not meet the current development needs, and there is still a lot of room for exploration and utilization of image information. At present, many studies on point cloud and image registration focus on the corresponding matching based on feature extraction in local space, while ignoring the fact that the depth value of the pixel obtained by depth estimation of the monocular camera is directly related to the distance of the point cloud. If this relationship can be added to the process of posture estimation, the errors caused by various factors in current research can be reduced, and the accuracy of registration can be improved to a certain extent.

[0005] Therefore, an improved unmanned vehicle posture correction method based on depth estimation is needed to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide an improved unmanned vehicle posture correction method based on depth estimation to solve the problems raised in the above background technology.

[0007] An improved unmanned vehicle posture correction method based on depth estimation, wherein the unmanned vehicle is provided with a laser radar and a camera, comprising the following steps:

[0008] Step 1: Use a laser radar and a camera to collect a three-dimensional point cloud and a two-dimensional image respectively, and use the DepthAnything algorithm to obtain a depth estimation image from the two-dimensional image;

[0009] Step 2: Input the 3D point cloud, 2D image and depth estimation image into the posture correction network.

[0010] The posture correction network includes a feature embedding module, an overlapping area detection module and a posture estimation module connected in sequence;

[0011] The feature embedding module includes a first ResNet network, a first maximum pooling layer, a first SO-Net network and a second maximum pooling layer, wherein the first maximum pooling layer is arranged after the first ResNet network, and the second maximum pooling layer is arranged after the first SO-Net network;

[0012] The two-dimensional image and the depth estimation image are connected to obtain a mixed image and input into the first ResNet network, and the first ResNet network is used to extract local image features of the mixed image The first maximum pooling layer is used to obtain the global image features of the mixed image ;

[0013] The three-dimensional point cloud is input into the first SO-Net network, and the first SO-Net network is used to extract local point cloud features of the three-dimensional point cloud The second maximum pooling layer is used to obtain the global point cloud features of the three-dimensional point cloud ;

[0014] The overlapping area detection module includes a point cloud operation module, an image operation module and an overlapping detection module.

[0015] The point cloud operation module includes an image to point cloud attention fusion module I2PAF and a point cloud decoder;

[0016] The image operation module includes a point cloud to image attention fusion module P2IAF and an image decoder;

[0017] Global image features of mixed images And local point cloud features of 3D point clouds After connection, the image weight feature is obtained through the image to point cloud attention fusion module I2PAF ;

[0018] Global point cloud features of 3D point cloud and image features of mixed images After the connection, the point cloud weight feature is obtained through the point cloud to image attention fusion module P2IAF ;

[0019] The point cloud weight feature and local image features of mixed images After the connection, the image score is obtained through the image decoder;

[0020] The image weight feature Local point cloud features of 3D point clouds After the connection, the point cloud score is obtained through the point cloud decoder;

[0021] The overlap detection module finds the value that is closest to the point cloud score and the image score, and generates a point cloud-image set with the corresponding three-dimensional point cloud and two-dimensional image;

[0022] Step 3: Use the EPnP algorithm to solve the point cloud-image set obtained in step 2 to obtain the initial posture matrix ;

[0023] Step 4: Local image features from the mixed image in step 2 Extract deep features from the image and combine the corresponding 3D point cloud, 2D image and deep features into a new set , using the collection Get the transformation matrix , using the transformation matrix For the initial posture matrix obtained in step 3 Optimize and get the final camera pose matrix.

[0024] Furthermore, the image to point cloud attention fusion module I2PAF in step 2 includes an image attention weight generation module and an image feature weight generation module, wherein the image attention weight generation module includes an MLP function and a first Softmax function for generating a global image feature according to the global image feature of the connected mixed image. And local point cloud features of 3D point clouds The image attention weight is obtained, and the image feature weight generation module is used to combine the image attention weight and the local features of the mixed image Dot multiplication to get image weight features ;

[0025] The point cloud to image attention fusion module P2IAF in step 2 includes a point cloud attention weight generation module and a point cloud feature weight generation module. The point cloud attention weight generation module includes a CNNS network and a second Softmax function for generating a point cloud feature according to the global point cloud features of the connected three-dimensional point cloud. and local image features of mixed images The point cloud attention weight is obtained, and the point cloud feature weight generation module is used to combine the point cloud attention weight and the local point cloud feature of the three-dimensional point cloud After point multiplication, we get the point cloud weight feature .

[0026] Furthermore, the point cloud decoder in step 2 includes a first eigenvalue module and a point cloud score module, wherein the first eigenvalue module includes a first PointNet network and a first upsampling module for obtaining a point cloud score based on the weighted features of the connected mixed image and the local point cloud features of the three-dimensional point cloud. The first eigenvalue is obtained, and the point cloud score module includes a second PointNet network and a second upsampling module for obtaining the first eigenvalue and the image weight feature according to the connection. And local point cloud features of 3D point clouds Get the point cloud score;

[0027] The image decoder in step 2 includes a second eigenvalue module and an image score module, wherein the second eigenvalue module includes a second ResNet network and a third upsampling module, which is used to obtain the weight features of the connected three-dimensional point cloud and the local features of the mixed image. The second eigenvalue is obtained, and the image score module includes a third ResNet network and a fourth upsampling module for obtaining the second eigenvalue and the point cloud weight feature according to the connection. and local image features of mixed images Get the image score.

[0028] Furthermore, the initial posture matrix in step 3 Calculated by the following formula:

[0029] ,

[0030] In the formula, Represents the coordinates of a three-dimensional reference point in the world coordinate system, Represents the coordinates of the projection of the three-dimensional reference point in the camera reference coordinate system, , n is the total number of three-dimensional reference points, and Respectively represent the coordinates of the four control points in the world coordinate system and the camera coordinate system, , are homogeneous barycentric coordinates, and are the center of gravity of the three-dimensional reference point in the world coordinate system and the camera reference coordinate system, respectively. is the external parameter of the camera, is the rotation matrix, The vectors U and V are obtained by performing SVD decomposition on the matrix H.

[0031] Furthermore, the loss function of the EPnP algorithm in step 3 is expressed by the following formula:

[0032] ,

[0033] In the formula, and Respectively represent the coordinates of the point cloud in the world coordinate system and the camera coordinate system, and Represents the coordinates of the four control points in the world coordinate system and the camera coordinate system respectively.

[0034] Furthermore, in step 4, we use the collection Get the transformation matrix , including the following steps:

[0035] Step 41: Use the camera intrinsic parameter matrix K to project the three-dimensional point cloud, convert the three-dimensional coordinates of the three-dimensional point cloud into two-dimensional pixel coordinates on the two-dimensional image, obtain the projected point cloud, and obtain the depth value information of the projected point cloud at the projection position of the depth map and the point cloud projection distance;

[0036] Step 42: Use the least squares method to establish a fitting function of the point cloud projection distance and depth value information, and then use the average pooling method to calculate the optimal displacement point with the minimum loss value;

[0037] Step 43: Using the optimal displacement point to adjust the displacement of the projected point cloud;

[0038] Step 44, posture transformation: According to the projection displacement of the three-dimensional point cloud, it is converted into the posture transformation of the three-dimensional point cloud in three-dimensional space to obtain the transformation matrix .

[0039] Furthermore, the final camera pose matrix in step 4 is It is expressed by the following formula:

[0040] ,

[0041] In the formula, is the optimized posture matrix, is the transformation matrix, is the initial posture matrix, , and They are the camera postures during the displacement of the 3D point cloud. axis, Axis and The angle selected on the axis, , and They are , and The rotation matrix of .

[0042] Furthermore, the loss function of the posture correction network is expressed as follows:

[0043] ,

[0044] In the formula, is a hyperparameter, is the descriptor loss function, is the detector loss function, is the depth loss function, is a positive boundary, is a negative boundary, is the characteristic distance facing each other, is the characteristic distance of the negative pair, R is the set safety distance, and n is the set of 3D point clouds and 2D images in the overlapping area All the numbers in and are the scores of pixels and point clouds in the overlapping area, and is the score of pixels and point clouds in non-overlapping areas, is the distance from the i-th point cloud to the camera, and are the maximum and minimum values ​​of the point cloud distance respectively; is the size of the estimated value of the i-th depth map, and are the maximum and minimum values ​​of the depth image estimation, respectively.

[0045] Beneficial effect: The improved unmanned vehicle posture correction method based on depth estimation of the present invention calculates the overlapping area by extracting the depth map, point cloud and image features, and performs preliminary posture estimation. The secondary alignment is completed through the depth map, and a polynomial model is established using the depth value information of each pixel and the distance information of the point cloud projected on the image. The optimal position is then found within a small range, thereby driving the movement of all points and the adjustment of the camera posture, thereby improving the accuracy of point cloud and image alignment in multiple complex environments, and obtaining more accurate position and posture information. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural schematic diagram of the posture correction network of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the image to point cloud attention fusion module I2PAF;

[0048] Figure 3 Schematic diagram of the structure of the point cloud to image attention fusion module P2IAF;

[0049] Figure 4 It is a structural diagram of an image decoder;

[0050] Figure 5 It is a schematic diagram of the structure of the point cloud decoder;

[0051] Figure 6 A schematic diagram of the process of optimizing the initial pose matrix using depth information;

[0052] Figure 7 Schematic diagram of displacement adjustment;

[0053] Figure 8 is the error histogram of different networks;

[0054] Fig. 9 This is a comparison diagram of the errors between the present invention and the CorrI2P network;

[0055] Fig.10 2D visualization of point cloud and image registration;

[0056] Fig.11 3D visualization of point cloud and image registration. DETAILED DESCRIPTION

[0057] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0058] See also Figure 1-Figure 7 As shown, the present invention calculates the overlapping area by extracting depth map, point cloud and image features, uses EPNP for preliminary posture estimation, completes secondary alignment through the depth map, and uses the depth value information of each pixel and the distance information of the point cloud projected on the image to establish a polynomial model, and finds the best position in a small range, thereby driving the movement of all points and the adjustment of the camera posture, thereby improving the accuracy of point cloud and image alignment in multiple complex environments, and obtaining more accurate position and posture information.

[0059] In the first step, the RGB image, the 3D point cloud, and the depth data generated by the depth anything algorithm are input into the network together, the image data and the point cloud data are connected to form new image data, and ResNet and SO-Net are used to extract the local feature values ​​of each layer of the point cloud and image respectively, and average pooling is used to obtain the global features of the image and point cloud.

[0060] In the second step, the feature map of the point cloud and image generated in the previous step is used to obtain the overlapping area by the following method:

[0061] Point cloud operation: The local features of the point cloud are linked to the global features of the image. After passing through the MLP and Softmax functions, the image attention weight is obtained, and the weight information is multiplied by the image feature points to obtain the feature weight of the image.

[0062] The point cloud weight features and image features are connected, passed into the ResNet network, and upsampled. The obtained feature values ​​are connected with the point cloud weight features and image features, and ResNet and upsampling operations are performed again for the second time to obtain the score information of the image pixels.

[0063] Image operation: The local features of the image are linked to the global features of the point cloud. After passing through the CNNS network and the Softmax function, the point cloud attention weight is obtained, and the weight information is multiplied by the point cloud feature to obtain the feature weight of the point cloud.

[0064] The image weight features and point cloud features are connected, passed into the pointnet network, and upsampled. The obtained feature values ​​are connected with the image weight features and point cloud features, and then passed into the piontnet and upsampled for the second time to obtain the point cloud score information.

[0065] Whether it is in the overlapping area can be determined based on the score.

[0066] The third step is to find the value closest to the point cloud score and the image score as the point cloud and image set, and then use EPnP to optimize the posture and obtain the initial posture matrix.

[0067] Both the EPnP algorithm and the PnP algorithm are algorithms for calculating camera poses. The EPnP algorithm is designed as an efficient solution that converts the PnP problem into a linear equations problem and solves the equations through methods such as SVD, so that it can handle larger-scale problems while maintaining high accuracy.

[0068] Unlike the EPnP algorithm, most PnP algorithms use the depth of feature points as a breakthrough to obtain the 3D coordinates in the camera coordinate system. The EPnP algorithm uses four non-coplanar control points. When the external parameters are unknown, that is, the coordinates of these four control points in the camera reference coordinate system are unknown, the 3D coordinates of the point cloud in the world coordinate system are considered to be their weighted sum. After solving the coordinates of these four control points, the camera's position and posture can be calculated.

[0069] The coordinates of the point cloud in the world coordinate system are , then the coordinates of the four control points in the world coordinate system and the camera coordinate system can be expressed as . The coordinates of the point cloud projected onto the 2D image through the transformation matrix can be expressed as , then the four control points in the world coordinate system.

[0070] Assuming that the external parameter of the camera is [R| t], the relationship between the virtual control points and can be deduced:

[0071] ,

[0072] When the control point is determined and satisfies the premise of non-coplanarity, it is also uniquely determined, and the formula can be further obtained:

[0073]

[0074] The optimization function can be expressed by the distance between the control points, because it will not change due to coordinate transformation, further:

[0075] According to the coordinates of the 3D reference point in the camera coordinate system, we can get:

[0076] ,,

[0077] use , The value of the center of gravity is calculated , , we can get matrix A and matrix B, and then calculate H=B T A, the final rotation in the posture and pan :

[0078] ,

[0079] Among them, vectors U and V are obtained by performing SVD decomposition on matrix H.

[0080] Step 4: Please refer to Figure 2 and Figure 3 As shown, while performing the third step, the point cloud, image and corresponding depth information are taken as a new combination, the point cloud is projected using the camera intrinsic parameter matrix K, the three-dimensional coordinates in the point cloud are converted into two-dimensional pixel coordinates on the image, and the surrounding depth value information of the point cloud at the projection position of the depth map and the projection distance of the point cloud are obtained.

[0081] Next, by analyzing the linear or nonlinear relationship between the two values, we find the best position near the point that meets this relationship, and use the least squares method to establish a fitting function for the point cloud distance and depth value information. Then, we use the average pooling method to calculate the position with the minimum loss value. This method is used to find the best displacement point in the 7*7 window that minimizes the error between the projection distance and the pixel value on the depth image, and optimize the matrix , and finally and Multiply the matrices to get the final camera pose.

[0082] The specific steps are as follows:

[0083] Step 1: Determine the set of point clouds, images, and depths in the overlapping area The position in the depth map, N is the set of overlapping areas , and obtain a 7*7 window centered on the corresponding pixel of the point cloud on the depth map, then obtain 4*4 windows in order from the upper left, upper right, lower left, and lower right, and use average pooling to obtain 4 depth values, respectively establish 4 groups of N corresponding values ​​between depth values ​​and point cloud distances, and use least squares to complete the fitting of the second-order function in turn. Taking the average distance of all points to the fitting curve as the standard, select the 4*4 window with the smallest distance mean as the first displacement result.

[0084] Step 2: Based on the first step, continue to obtain 2*2 windows in the upper left, upper right, lower left, and lower right order, use average pooling to obtain 4 depth values, establish 4 groups of corresponding values ​​between N depth values ​​and point cloud distances, and use least squares to complete the fitting of the second-order function. Taking the average distance from all points to the fitting curve as the standard, select the 2*2 window with the smallest distance mean as the second displacement result.

[0085] Step 3: Repeat the above steps to divide the 2*2 window into 4 1*1 windows, and then complete the fitting of the second-order function. The 1*1 window with the smallest distance mean is the optimal projection position in the 7*7 window, completing the third displacement.

[0086] Step 4: Construct the plane equation. Get the The spatial positions (x, y, z) of the point clouds corresponding to the maximum and minimum values ​​of the image pixels in the horizontal and vertical directions, a total of four groups, three of which are randomly selected =( , , ), i=1,2,3. Then the normal vector can be obtained :

[0087] ,,

[0088] Further construct the space equation:

[0089] ,

[0090] Step 5: Calculate the displacement error. The displacement transformation of the projection point on the image is based on pixels as the basic unit. It is necessary to determine the corresponding (∆X, ∆Y, ∆Z) of the displacement transformation on the image in space.

[0091] Step 6: Calculate the angle error. When calculating the angle error between before and after displacement, use the cosine similarity formula and the inverse cosine function to calculate the angle error. :

[0092] ,

[0093] in, represents the dot product operator of vectors, Represents the arc cosine function, which is used to calculate the radian value of the angle.

[0094] Step 7: Posture transformation: According to the projection displacement of the point cloud on the 2D image, it is converted into the posture transformation of the point cloud in the 3D space, and the transformation matrix M is obtained. Finally, according to the projection displacement of the point cloud on the 2D image, it is converted into the transformation of the posture of the point cloud in the 3D space, and the transformation matrix M is obtained:

[0095] ,

[0096] In the formula, , and are the angles selected by the camera posture on the x, y and z axes during the displacement of the point cloud, , and They are , and The rotation matrix of .

[0097] Finally, the new The matrix can be replaced by the original Matrices and Transformation Matrices To express:

[0098]

[0099] In this way, the accuracy of point cloud registration can be effectively improved, making the point cloud more consistent with the depth image, and providing more accurate data support for subsequent applications.

[0100] In addition, there is a newly designed loss function:

[0101] The loss function is mainly composed of descriptor loss, detector loss and depth loss, which are used for matching and detection tasks of image and point cloud data respectively.

[0102] The main task of the descriptor is to ensure that corresponding pixel pairs and point pairs have similar representations in the feature space to promote matching accuracy. An acceptable feature distance threshold is set in the network, and samples are divided into positive samples and negative samples. The cosine distance is used to calculate the feature distance for positive samples (known correspondence), and the minimum feature distance is selected for negative samples (no correspondence).

[0103] ,

[0104] In the formula, are the feature vectors of the image and point cloud respectively.

[0105] In order to speed up training, positive boundaries are introduced and negative margins , and finally the descriptor loss is:

[0106] ,

[0107] is the characteristic distance facing each other, is the characteristic distance of the negative pair, and R is the set safety distance.

[0108] The detector loss ensures that the model can accurately detect the overlapping regions and give higher scores to these regions. The detector loss is defined by a weighted sum, where the score inside the overlapping region should be higher than the score outside the overlapping region.

[0109] ,

[0110] In the formula, , are the scores of pixels and point clouds in the overlapping area, , is the score of pixels and point clouds in non-overlapping areas.

[0111] The depth loss mainly describes the distance difference between the distance value of the registered point cloud and the pixel value of the depth image.

[0112] ,

[0113] In the formula, and are the maximum and minimum values ​​of the point cloud distance respectively; and are the maximum and minimum distances of the point cloud, respectively.

[0114] Finally, we introduce a hyperparameter that balances the three values , the loss function is:

[0115] .

[0116] Embodiment 1:

[0117] This embodiment mainly selects the NuScenes dataset, and adds depth map data on the basis of the original dataset, using 850 scenes for training and 100 scenes for testing (the training and testing datasets include point clouds, images, and K matrices).

[0118] Following the official data segmentation of NuScenes, use the official SDK to obtain the corresponding point cloud and image. The point cloud data is obtained by the 360° field of view of the LiDAR and accumulated from nearby frames; the image is obtained frame by frame from the single-sided camera, with a pixel size of 160×320.

[0119] The training batch size is 8, the initial learning rate is 0.001, and it is multiplied by 0.25 every 5 iterations until it reaches 0.00001.

[0120] In addition, for the depth estimation displacement optimization module, in order to prevent the imported depth map from having different scales, all values ​​are normalized to the fifth decimal place. In the process of fitting the point cloud distance information and depth information, 100 point cloud information were randomly selected to participate in the fitting, the polynomial coefficient was set to 4, and the minimum value of all points to the fitting curve was selected as the displacement object, and the variable (minimum distance sum) was automatically saved as the model threshold. In order to prevent the error from increasing due to poor registration effect during the registration process, when the minimum distance of the fitting is greater than 2 times the model threshold or the rotation angle is greater than 2°, the M matrix will become a unit matrix.

[0121] Verification results:

[0122] The present invention shows the results of the registration results with displacement errors within 5 meters and angle errors within 10 degrees. The specific results are shown in Table 1. It can be found from the table that the method of the present invention is superior to other methods.

[0123] In terms of displacement error, the mean value of the network experiment designed by the present invention is 1.89 meters, which is lower than 2.13 meters of CorrI2P; in terms of angle error, the experimental result of the present invention is 2.71°, which is also lower than 3.32° of CorrI2P. In addition, the variance of the experiment of the present invention is also the smallest among several groups of data, and the stability is higher.

[0124] Table 1 Registration accuracy on NuScenes, ” Downward means smaller is better, and the best results are in bold

[0125]

[0126] For the distribution of registration error, the RTE and RRE results of the network designed by the present invention and the CorrI2P network are statistically analyzed. The results are as follows: Figure 8 and Fig. 9 As shown in the figure, from the distribution results, the network designed by the present invention is superior to the CorrI2P network in all aspects, especially in terms of the angle error. Figure 4 Parts a and b are the error histogram distribution of the posture correction network experiment using the present invention. Figure 8 Parts c and d are the error histogram distributions of the CorrI2P network experiment.

[0127] As shown in Table 2, the comparison of the accuracy, recall rate and F2 score between the CorrI2P network and the network of the present invention is shown. When the depth value variation difference in the image is large, that is, when the reference object and the background can be clearly distinguished, the aspects of the present invention can reduce the error loss to a great extent. From the final results, it can be seen that the method of the present invention is better than CorrI2P.

[0128] Table 2 Comparison of recall, precision and F2 score of two different methods on the NuScenes dataset

[0129]

[0130] Detection of overlapping areas is the key to completing the registration task, such as Fig.10 and Fig.11 As shown in Figure 2, the method proposed by the present invention is the most accurate in predicting the overlapping area. Fig.10 In the present invention, the point cloud is colored according to the distance from the sensor, and the 3D point cloud is projected onto the image to observe the registration result. By comparing and observing, it can be found that the registration result completed by the method proposed by the present invention is closer to the correct registration result.

[0131] exist Fig.11 The present invention can more clearly observe the gap between the two methods. Fig.11 Part a is based on the correct registration result, and the grayscale of the pixels corresponding to the point cloud is assigned to the point cloud. Fig.11 Part b and part c are the overlapping areas shown by the CorrI2P registration result and the overlapping areas shown by the registration result of the method proposed in the present invention, respectively. The point cloud in the right box is not detected in the overlapping area, and the point cloud in the left box is detected outside the overlapping area due to displacement and angle errors. Fig.11 The number of point clouds selected in part b is obviously more than that in part c, which indirectly proves that the method proposed in the present invention is better than CorrI2P.

[0132] In Table 2, the present invention compares the recall, precision and F2 score of two different methods on the NuScenes dataset, which once again confirms that the method proposed in the present invention is better than CorrI2P in predicting overlapping areas.

Claims

1. An improved unmanned vehicle posture correction method based on depth estimation, characterized in that: The unmanned vehicle is provided with a laser radar and a camera, and the following steps are included: Step 1: Use a laser radar and a camera to collect a three-dimensional point cloud and a two-dimensional image respectively, and use a Depth Anything algorithm to obtain a depth estimation image from the two-dimensional image; Step 2: Input the 3D point cloud, 2D image and depth estimation image into the posture correction network. The posture correction network includes a feature embedding module, an overlapping area detection module and a posture estimation module connected in sequence; The feature embedding module includes a first ResNet network, a first maximum pooling layer, a first SO-Net network and a second maximum pooling layer, wherein the first maximum pooling layer is arranged after the first ResNet network, and the second maximum pooling layer is arranged after the first SO-Net network; The two-dimensional image and the depth estimation image are connected to obtain a mixed image and input into the first ResNet network, and the first ResNet network is used to extract local image features of the mixed image The first maximum pooling layer is used to obtain the global image features of the mixed image ; The three-dimensional point cloud is input into the first SO-Net network, and the first SO-Net network is used to extract local point cloud features of the three-dimensional point cloud The second maximum pooling layer is used to obtain the global point cloud features of the three-dimensional point cloud ; The overlapping area detection module includes a point cloud operation module, an image operation module and an overlapping detection module. The point cloud operation module includes an image to point cloud attention fusion module I2PAF and a point cloud decoder; The image operation module includes a point cloud to image attention fusion module P2IAF and an image decoder; Global image features of mixed images And local point cloud features of 3D point clouds After connection, the image weight feature is obtained through the image to point cloud attention fusion module I2PAF ; Global point cloud features of 3D point cloud and image features of mixed images After the connection, the point cloud weight feature is obtained through the point cloud to image attention fusion module P2IAF ; The point cloud weight feature and local image features of mixed images After the connection, the image score is obtained through the image decoder; The image weight feature Local point cloud features of 3D point clouds After the connection, the point cloud score is obtained through the point cloud decoder; The overlap detection module finds the value that is closest to the point cloud score and the image score, and generates a point cloud-image set with the corresponding three-dimensional point cloud and two-dimensional image; The loss function of the posture correction network is expressed as follows: ， In the formula, is a hyperparameter, is the descriptor loss function, is the detector loss function, is the depth loss function, is a positive boundary, is a negative boundary, is the characteristic distance facing each other, is the characteristic distance of the negative pair, R is the set safety distance, and n is the set of 3D point clouds and 2D images in the overlapping area All the numbers in and are the scores of pixels and point clouds in the overlapping area, respectively. and is the score of pixels and point clouds in non-overlapping areas, is the distance from the i-th point cloud to the camera, and are the maximum and minimum values ​​of the point cloud distance respectively; is the size of the estimated value of the i-th depth map, and are the maximum and minimum values ​​of the depth image estimation respectively; Step 3: Use the EPnP algorithm to solve the point cloud-image set obtained in step 2 to obtain the initial posture matrix ; Step 4: Local image features from the mixed image in step 2 Extract deep features from the image and combine the corresponding 3D point cloud, 2D image and deep features into a new set , using the collection Get the transformation matrix , using the transformation matrix For the initial posture matrix obtained in step 3 Optimize and get the final camera pose matrix.

2. The unmanned vehicle posture correction method based on depth estimation improvement as claimed in claim 1, characterized in that: The image to point cloud attention fusion module I2PAF in step 2 includes an image attention weight generation module and an image feature weight generation module, wherein the image attention weight generation module includes an MLP function and a first Softmax function for generating a global image feature according to the global image feature of the connected mixed image. And local point cloud features of 3D point clouds The image attention weight is obtained, and the image feature weight generation module is used to combine the image attention weight and the local features of the mixed image Dot multiplication to get image weight features ; The point cloud to image attention fusion module P2IAF in step 2 includes a point cloud attention weight generation module and a point cloud feature weight generation module. The point cloud attention weight generation module includes a CNNS network and a second Softmax function for generating a point cloud feature according to the global point cloud features of the connected three-dimensional point cloud. and local image features of mixed images The point cloud attention weight is obtained, and the point cloud feature weight generation module is used to combine the point cloud attention weight and the local point cloud feature of the three-dimensional point cloud After point multiplication, we get the point cloud weight feature .

3. The unmanned vehicle posture correction method based on depth estimation improvement as claimed in claim 1, characterized in that: The point cloud decoder in step 2 includes a first eigenvalue module and a point cloud score module, wherein the first eigenvalue module includes a first PointNet network and a first upsampling module, which is used to obtain the local point cloud features of the three-dimensional point cloud according to the weight features of the connected mixed image. The first eigenvalue is obtained, and the point cloud score module includes a second PointNet network and a second upsampling module for obtaining the first eigenvalue and the image weight feature according to the connection. And local point cloud features of 3D point clouds Get the point cloud score; The image decoder in step 2 includes a second eigenvalue module and an image score module, wherein the second eigenvalue module includes a second ResNet network and a third upsampling module, which is used to obtain the weight features of the connected three-dimensional point cloud and the local features of the mixed image. The second eigenvalue is obtained, and the image score module includes a third ResNet network and a fourth upsampling module for obtaining the second eigenvalue and the point cloud weight feature according to the connection. and local image features of mixed images Get the image score.

4. The improved unmanned vehicle posture correction method based on depth estimation as claimed in claim 1, characterized in that: The initial posture matrix in step 3 Calculated by the following formula: ， In the formula, Represents the coordinates of a three-dimensional reference point in the world coordinate system, Represents the coordinates of the projection of the three-dimensional reference point in the camera reference coordinate system, , n is the total number of three-dimensional reference points, and Respectively represent the coordinates of the four control points in the world coordinate system and the camera coordinate system, , are homogeneous barycentric coordinates, and are the center of gravity of the three-dimensional reference point in the world coordinate system and the camera reference coordinate system, respectively. is the external parameter of the camera, is the rotation matrix, The vectors U and V are obtained by performing SVD decomposition on the matrix H.

5. The unmanned vehicle posture correction method based on depth estimation improvement as claimed in claim 1, characterized in that: The loss function of the EPnP algorithm described in step 3 is expressed by the following formula: ， In the formula, and Respectively represent the coordinates of the point cloud in the world coordinate system and the camera coordinate system, and Represents the coordinates of the four control points in the world coordinate system and the camera coordinate system respectively.

6. The unmanned vehicle posture correction method based on depth estimation improvement as claimed in claim 1, characterized in that: In step 4, use the collection Get the transformation matrix , including the following steps: Step 41: Use the camera intrinsic parameter matrix K to project the three-dimensional point cloud, convert the three-dimensional coordinates of the three-dimensional point cloud into two-dimensional pixel coordinates on the two-dimensional image, obtain the projected point cloud, and obtain the depth value information of the projected point cloud at the projection position of the depth map and the point cloud projection distance; Step 42: Use the least squares method to establish a fitting function of the point cloud projection distance and depth value information, and then use the average pooling method to calculate the optimal displacement point with the minimum loss value; Step 43: Using the optimal displacement point to adjust the displacement of the projected point cloud; Step 44, posture transformation: According to the projection displacement of the three-dimensional point cloud, it is converted into the posture transformation of the three-dimensional point cloud in three-dimensional space to obtain the transformation matrix .

7. The improved unmanned vehicle posture correction method based on depth estimation as claimed in claim 1, characterized in that: The final camera pose matrix in step 4 It is expressed by the following formula: ， In the formula, is the optimized posture matrix, is the transformation matrix, is the initial posture matrix, , and They are the camera postures during the displacement of the 3D point cloud. axis, Axis and The angle selected on the axis, , and They are , and The rotation matrix of .

Images