Calibration object-free camera-laser radar relative pose on-line estimation method and calibration object-free camera-laser radar relative pose on-line estimation system

Abstract

The invention relates to a calibration object-free camera-laser radar relative pose online estimation method and a calibration object-free camera-laser radar relative pose online estimation system. The method comprises the steps of firstly collecting images and point clouds by using a camera and a laser radar; performing feature extraction and fusion on the image and the point cloud by using a neural network and an attention mechanism to respectively obtain a point cloud cross-modal feature and an image cross-modal feature; pixels and points in the common-view area are extracted from the common-view area, and 2D-3D matching is established between the pixels and the points based on a similarity maximization strategy; and finally, a re-projection equation is established based on the established 2D-3D matching, and the optimal relative pose is obtained by using EPnP-RANSAC optimization, so that the calibration object-free camera-laser radar relative pose online estimation is completed. Compared with the prior art, the method has the advantages that online estimation of the relative pose of the non-calibration object can be achieved, and the pose estimation precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-sensor fusion, and in particular to an online estimation method and system for relative pose of a camera-lidar without calibration objects. Background Art

[0002] As an indispensable component of robot vision and autonomous driving systems, more and more smart devices are equipped with cameras and lidar. Cameras and lidar usually work together to achieve comprehensive perception of the surrounding environment. Cameras can capture rich color and texture information, while lidar can provide accurate three-dimensional structural information. By fusing and calibrating the camera and lidar data, the relative position between them can be determined, and then a high-precision environmental perception image can be synthesized in real time. This fused perception system can not only provide robots and autonomous vehicles with a wider field of view and eliminate perception blind spots, but can also be used in key tasks such as target detection, obstacle recognition, and path planning. In the field of robot vision, it helps robots to more accurately identify objects, locate their own positions and navigate, greatly improving the robot's ability to operate and autonomy in complex environments. In the field of autonomous driving, the combination of cameras and lidar provides vehicles with reliable environmental perception, enabling vehicles to detect other vehicles, pedestrians, traffic signs, obstacles, etc. in a timely manner, so as to make safer and more efficient driving decisions, which plays a vital role in promoting the development of autonomous driving technology.

[0003] When there is an obvious posture deviation in the perception image fused by the camera and the lidar, timely correction of the relative posture of the camera and the lidar is of great significance to the safety of robot vision and autonomous driving. Unfortunately, so far, academia and industry still lack solutions designed for cameras and lidar that can effectively and online correct the relative posture. At present, the existing research solutions in related fields have the following limitations:

[0004] 1. Most existing camera-lidar relative pose estimation methods rely on the use of calibration objects, which limits the use scenarios of such methods and reduces the flexibility and real-time performance of the methods. Specifically, first of all, the calibration-based methods rely on specific calibration objects, such as chessboards, markers, etc. These calibration objects need to be arranged in a specific environment, and may be damaged, moved or blocked in actual applications, affecting the accuracy of the calibration. Secondly, the pose estimation process of the calibration-based method is complicated, requiring professional equipment and technicians to operate, which is time-consuming and costly.

[0005] 2. Existing methods cannot handle the task of online correction of calibration parameters. Collisions and bumps that occur during the operation of vehicles or robots, or thermal expansion and contraction caused by temperature changes may destroy the original spatial structure between the surround camera and lidar system, causing changes in external parameters. At this time, if the original external parameters are still used to synthesize the fused perception image without corresponding adjustments, there will be obvious misalignment in the generated fused perception image, that is, the perception of the camera-lidar system is abnormal. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and system for online estimation of relative pose of camera-lidar without calibration objects.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to one aspect of the present invention, a method for online estimation of relative pose of a camera-lidar without a calibration object is provided, and the method comprises the following steps:

[0009] S1, using cameras and lidar to collect images and point clouds;

[0010] S2. Use neural network and attention mechanism to extract and fuse features of images and point clouds, and obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features respectively;

[0011] S3, according to the fine-grained point cloud cross-modal features and the fine-grained image cross-modal features, determine the pixels and points in the common view area, and establish 2D-3D matching between pixels and points based on the similarity maximization strategy;

[0012] S4. Based on the established 2D-3D matching, a reprojection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the camera-lidar relative pose without calibration objects.

[0013] As a preferred technical solution, the image and point cloud are collected in S1 by using a downsampling method.

[0014] As a preferred technical solution, in S2, feature extraction of images and point clouds is performed by using ResNet to extract multi-scale features of images and using PointNet to extract multi-scale features of point clouds.

[0015] As a preferred technical solution, the specific steps of obtaining fine-grained point cloud cross-modal features and fine-grained image cross-modal features in S2 are:

[0016] S21, expand the multi-scale features of the image into a one-dimensional tensor and concatenate it with the multi-scale features of the point cloud and use the attention mechanism to fuse the two features to obtain the coarse-grained point cloud fusion features;

[0017] S22. Extract local features of point cloud from multi-scale features of point cloud based on KNN algorithm, and use attention mechanism to fuse local features of point cloud with multi-scale features of image to obtain coarse-grained image fusion features.

[0018] S23. Based on the twin neural network, the coarse-grained point cloud fusion features and the coarse-grained image fusion features are mapped to the same feature space to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features.

[0019] As a preferred technical solution, the specific method for determining the pixels and points in the common view area in S3 is: inputting the cross-modal features of the pixels and point clouds into the common view area detector respectively to generate the corresponding common view scores of the pixels and point cloud points. If the common view score of a pixel or point cloud point exceeds a preset threshold, the pixel or point cloud point is considered to be in the common view area.

[0020] As a preferred technical solution, the specific process of establishing 2D-3D matching in S3 is:

[0021] S31, calculating the cosine similarity between pixels and point cloud points in the common view area;

[0022] S32, a matching matrix constructed by cosine similarity, where is the number of pixels and is the number of point cloud points;

[0023] S33, optimizing the matching matrix according to the similarity maximization strategy, finding the most matching point cloud point for each pixel, and completing the establishment of 2D-3D matching.

[0024] As a preferred technical solution, the similarity maximization strategy in S3 is specifically as follows:

[0025] Feature prediction is performed on point cloud points and pixels respectively, and a pixel feature vector corresponds to multiple point feature vectors. The similarity between a pixel feature vector and multiple point feature vectors is calculated respectively, and the point with the largest similarity is taken as the point cloud point that best matches the pixel.

[0026] As a preferred technical solution, the specific formula of the reprojection equation in S4 is:

[0027]

[0028] In the formula, T / T * is the pose to be optimized, K is the camera internal parameter, p i is the matching 3D point, u i is the matching 2D pixel.

[0029] As a preferred technical solution, the specific steps of using EPnP-RANSAC optimization in S4 to obtain the optimal relative pose are:

[0030] S41, randomly initialize the pose;

[0031] S42, calculating a reprojection error based on the pose;

[0032] S43, using the reprojection error to solve the error gradient;

[0033] S44. Optimize the posture parameters according to the direction of the error gradient, return to S42 and repeat S42-S44 until the set number of iterations is reached.

[0034] According to another aspect of the present invention, a system for online estimation of relative pose of a camera-lidar without a calibration object is provided. The system is operated by the online estimation method of relative pose of a camera-lidar without a calibration object as described above. The system includes an information acquisition module, a feature extraction and fusion module, a 2D-3D matching module, and a relative pose estimation module.

[0035] The information acquisition module is used to acquire images and point clouds;

[0036] The feature extraction and fusion module uses a neural network and an attention mechanism to extract and fuse features of images and point clouds to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features;

[0037] The 2D-3D matching module uses a similarity maximization strategy to establish 2D-3D matching between pixels and points based on fine-grained point cloud cross-modal features and fine-grained image cross-modal features;

[0038] The relative pose estimation module establishes a reprojection equation based on 2D-3D matching and uses EPnP-RANSAC optimization to obtain the optimal relative pose.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention uses a camera and a laser radar to collect images and point clouds, and then uses a neural network and an attention mechanism to extract and fuse features from the images and point clouds, respectively obtaining fine-grained point cloud cross-modal features and fine-grained image cross-modal features; then extracts pixels and points in the common view area, and establishes 2D-3D matching between pixels and points based on a similarity maximization strategy; finally, a reprojection equation is established based on the established 2D-3D matching, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the relative pose of the camera-lidar without a calibration object. For the camera-lidar system, this method can not only complete the initial external parameter estimation, but also realize the online estimation of the relative pose without a calibration object.

[0041] 2. The present invention extracts fine-grained point cloud cross-modal features and fine-grained image cross-modal features without relying on specific calibration objects and calibration scenes. As long as there is a certain amount of texture in the scene, the online estimation of extrinsic parameters can be completed.

[0042] 3. The present invention integrates the information acquisition module, feature extraction and fusion module, 2D-3D matching module, and relative posture estimation module into the system. Only a pair of point clouds and images need to be input into the system to automatically complete the subsequent processes. The process is simple and easy to operate.

[0043] 4. The present invention integrates a feature fusion module and a similarity maximization strategy. The feature fusion module uses an attention mechanism, a KNN algorithm, and a twin neural network to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features; the similarity maximization strategy predicts features for point cloud points and pixels respectively, obtains a pixel feature vector corresponding to multiple point feature vectors, and calculates the similarity between a pixel feature vector and multiple point feature vectors respectively, and the point with the largest similarity is used as the point cloud point that best matches the pixel. It can reduce the consumption of computing resources, improve the accuracy of matching establishment, and improve the accuracy of pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the steps of a method for online estimation of relative pose of a camera-lidar without calibration objects in the present invention;

[0045] Figure 2 This is a flow chart of a method for online estimation of relative pose of a camera-lidar without calibration objects in the present invention;

[0046] Figure 3 Schematic diagram of the similarity maximization strategy flow in the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] In this embodiment, a camera-lidar relative pose online estimation method without calibration objects is adopted. Figure 1 As shown, it specifically includes the following steps:

[0049] S1, using cameras and lidar to collect images and point clouds;

[0050] S2. Use neural network and attention mechanism to extract and fuse features of images and point clouds, and obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features respectively;

[0051] S3, using fine-grained point cloud cross-modal features and fine-grained image cross-modal features to extract pixels and points in the common view area, and establish 2D-3D matching between pixels and points based on a similarity maximization strategy;

[0052] S4. Based on the established 2D-3D matching, a reprojection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the camera-lidar relative pose without calibration objects.

[0053] The specific process of this method is as follows Figure 2 As shown in the flowchart. In this embodiment, the camera-lidar system is taken as an example. Figure 2 A detailed description of the process:

[0054] In this embodiment, the camera and lidar are used to collect images and point clouds. Downsampling point clouds and images reduces computing requirements. In the feature extractor, first check whether the image resolution ratio and point cloud data density are sufficient. Each frame of point cloud should contain at least 20480 3D points, and the resolution of each image cannot be less than 160512.

[0055] In this embodiment, if the resolution of the current image and the data density of the point cloud meet the requirements, the residual neural network (Residual Neural Network, ResNet) and PointNet are used to extract the multi-scale features of the image and point cloud respectively. The multi-scale features of the image are expanded into a one-dimensional tensor and spliced ​​with the point cloud features, and feature fusion is performed based on the attention mechanism to obtain the cross-modal features of the point cloud. Specifically, the multi-scale features of the image include feature maps and global feature vectors with resolutions scaled by 2, 4, and 8 times, and the multi-scale features of the point cloud include two features containing 256 key points, a point-wise feature tensor, and a global feature vector.

[0056] In this embodiment, in the coarse-grained feature processing stage of scale feature aggregation, after the multi-scale image and point cloud features are extracted, the feature fusion step begins. First, the image features are expanded into a one-dimensional tensor, and the expanded tensor is concatenated with the point cloud features, and feature weighted fusion is performed using the attention mechanism to obtain the coarse-grained point cloud fusion feature f (coarse，P) Based on the KNN algorithm, the local features of the point cloud are extracted, and the attention mechanism is used to fuse the local features with the image features to obtain the cross-modal features of the image. At the same time, the multi-scale point cloud features are spliced ​​with the image features and fused with the attention mechanism to obtain the coarse-grained image fusion features f (coarse，I) The attention mechanism is implemented with a multi-layer perceptron, and the convolution kernel size is 1×1.

[0057] Subsequently, in the fine-grained feature processing stage, the coarse-grained image fusion features and point cloud fusion features are mapped to the same feature space to obtain fine-grained cross-modal features f with stronger representation capabilities. (fine，P) and f (fine，I) , and then enter the 2D-3D matching establishment stage. Taking the cross-modal features of images and point clouds as input, based on the twin network module, the two are mapped to the same feature space to obtain a more consistent cross-modal feature representation of images and point clouds. Specifically, the feature space is constructed through the twin network structure, and PointNet is the basic unit in the network structure.

[0058] In this embodiment, in the process of establishing 2D-3D matching, the common view area detector (Confocal Volumetric Display, CVD) for images and point clouds is first used to detect pixels and point cloud points in the common view area. Taking the cross-modal features of the image and point cloud as input, the common view area detector is used to generate the common view score at the pixel / point cloud level. Among them, the common view area detector for the image is implemented with 3 convolutional layers and a Softmax layer. The final common view score of the pixel and the point cloud point is between 0-1. Based on the threshold discrimination method, the pixels and point cloud points in the common view area are screened out. Specifically, if the common view score of a pixel or point cloud point exceeds the threshold, the pixel / point cloud point is considered to be in the common view area. Pixels and point cloud points above the threshold are considered to be in the common view area, and the threshold is set to 0.95.

[0059] Secondly, the cosine distance of fine-grained cross-modal features between pixels and point cloud points in the common view area is calculated. An M×N matching matrix is ​​constructed as a correlation map, which is specifically:

[0060]

[0061] Where M is the number of pixels in the common view area, N is the number of point cloud points in the common view area, δ(·) is the cosine distance calculator, and u and p are the matched 2D pixels and 3D points.

[0062] In this embodiment, Figure 2 and Figure 3 The similarity maximization strategy shown is used to optimize the correlation graph to obtain the final 2D-3D correspondence. First, for each pixel, select the point cloud point with the highest similarity, that is, the lowest cosine distance, as the point cloud point matching the pixel, and establish a 2D-3D match in the form of pixel-point cloud point. Then, check the number of matching pixel-point pairs. If it is less than 10 pairs, the frame is abandoned. The resulting 2D-3D correspondence is as follows:

[0063]

[0064] Where M is the number of pixels in the common view area, N is the number of point cloud points in the common view area, δ(·) is the cosine distance calculator, and u and p are the matched 2D pixels and 3D points.

[0065] In this embodiment, in the PnP solver, first, for each pair of matching pixels and points, based on the 3×3 camera intrinsic parameter matrix, 2×1 pixel coordinates, and 3×1 point coordinates, a series of reprojection equations are constructed with the reprojection error as the final error. The specific formula of the constructed reprojection equation is:

[0066]

[0067] In the formula, T / T * is the pose to be optimized, K is the camera internal parameter, p i is the matching 3D point, u i For matching 2D pixels, the transformation process between homogeneous coordinates and non-homogeneous coordinates is omitted.

[0068] In this embodiment, the reprojection equation constructed by iterative optimization using the random sampling consensus algorithm (EPnP-RANSAC) is iteratively optimized, and the number of iterations is 500. The specific optimization steps are: first randomly initialize the pose; then calculate the reprojection error based on the pose; then solve the error gradient; finally optimize the pose parameters according to the gradient direction; and calculate the reprojection error again based on the pose;

[0069] In this embodiment, the EPnP-RANSAC algorithm is used to perform 500 iterative optimizations on the pose and output a Boolean estimated marker value, that is, the camera internal parameters and matching point / pixel coordinates are fixed, and only the camera pose is optimized; after the iterative optimization is completed, if the estimated marker value is true, the optimized relative pose is read according to the optimization result, and the relative position between the camera and the radar is updated.

[0070] In summary, this method is designed for the camera-lidar system, which can complete the initial extrinsic parameter estimation as well as realize the online estimation of relative pose; it does not rely on specific calibration objects and calibration scenes, and can complete the online estimation of extrinsic parameters as long as there are a certain number of textures in the scene; the process is simple, and only a pair of point clouds and images need to be input to automatically complete the subsequent process; the integration of similarity maximization strategy and feature fusion module can reduce the consumption of computing resources and improve the accuracy of matching establishment, thereby improving the accuracy of pose estimation.

[0071] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for online estimation of relative pose of camera-lidar without calibration objects, characterized in that: The method comprises the following steps: S1, using cameras and lidar to collect images and point clouds; S2. Use neural network and attention mechanism to extract and fuse features of images and point clouds, and obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features respectively; S3, according to the fine-grained point cloud cross-modal features and the fine-grained image cross-modal features, determine the pixels and points in the common view area, and establish 2D-3D matching between pixels and points based on the similarity maximization strategy; S4. Based on the established 2D-3D matching, a reprojection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the camera-lidar relative pose without calibration objects.

2. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: In the S1, the image and point cloud are collected by using a downsampling method.

3. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: In S2, the feature extraction of the image and the point cloud is performed by using ResNet to extract the multi-scale features of the image, and using PointNet to extract the multi-scale features of the point cloud.

4. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: The specific steps of obtaining the fine-grained point cloud cross-modal features and the fine-grained image cross-modal features in S2 are: S21, expand the multi-scale features of the image into a one-dimensional tensor and concatenate it with the multi-scale features of the point cloud and use the attention mechanism to fuse the two features to obtain the coarse-grained point cloud fusion features; S22. Extract local features of point cloud from multi-scale features of point cloud based on KNN algorithm, and use attention mechanism to fuse local features of point cloud with multi-scale features of image to obtain coarse-grained image fusion features. S23. Based on the twin neural network, the coarse-grained point cloud fusion features and the coarse-grained image fusion features are mapped to the same feature space to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features.

5. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: The specific method for determining the pixels and points in the common view area in S3 is: inputting the cross-modal features of the pixels and point clouds into the common view area detector respectively to generate the common view scores of the corresponding pixels and point cloud points; if the common view score of a pixel or point cloud point exceeds a preset threshold, the pixel or point cloud point is considered to be in the common view area.

6. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: The specific process of establishing 2D-3D matching in S3 is as follows: S31, calculating the cosine similarity between pixels and point cloud points in the common view area; S32, constructing an M×N matching matrix by cosine similarity, where M is the number of pixels and N is the number of point cloud points; S33, optimizing the matching matrix according to the similarity maximization strategy, finding the most matching point cloud point for each pixel, and completing the establishment of 2D-3D matching.

7. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: The similarity maximization strategy in S3 is specifically as follows: Feature prediction is performed on point cloud points and pixels respectively, and a pixel feature vector corresponds to multiple point feature vectors. The similarity between a pixel feature vector and multiple point feature vectors is calculated respectively, and the point with the largest similarity is taken as the point cloud point that best matches the pixel.

8. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: The specific formula of the reprojection equation in S4 is: In the formula, T / T * is the pose to be optimized, K is the camera internal parameter, p i is the matching 3D point, u i is the matching 2D pixel.

9. The method for online estimation of relative pose of camera-lidar without calibration object according to claim 1, characterized in that: The specific steps of using EPnP-RANSAC optimization in S4 to obtain the optimal relative pose are: S41, randomly initialize the pose; S42, calculating a reprojection error based on the pose; S43, using the reprojection error to solve the error gradient; S44. Optimize the posture parameters according to the direction of the error gradient, return to S42 and repeat S42-S44 until the set number of iterations is reached.

10. A camera-lidar relative pose online estimation system without calibration objects, characterized in that: The system is applied to work by a camera-lidar relative pose online estimation method without calibration objects as described in any one of claims 1 to 9, and the system includes an information acquisition module, a feature extraction and fusion module, a 2D-3D matching module, and a relative pose estimation module; The information acquisition module is used to acquire images and point clouds; The feature extraction and fusion module uses a neural network and an attention mechanism to extract and fuse features of images and point clouds to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features; The 2D-3D matching module uses a similarity maximization strategy to establish 2D-3D matching between pixels and points based on fine-grained point cloud cross-modal features and fine-grained image cross-modal features; The relative pose estimation module establishes a reprojection equation based on 2D-3D matching and uses EPnP-RANSAC optimization to obtain the optimal relative pose.

Images