A method and system for online estimation of camera-lidar relative pose without calibration objects

Abstract

The present application relates to a kind of camera-laser radar relative pose online estimation method and system without calibration object.The method first utilizes camera and laser radar to collect image and point cloud;Then neural network and attention mechanism are used to extract and fuse features of image and point cloud, respectively to obtain point cloud cross-modal feature and image cross-modal feature;From it, pixel and point in the co-view area are extracted, and 2D-3D matching is established between pixel and point based on similarity maximization strategy;Finally, based on the 2D-3D matching established, re-projection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, so as to complete the online estimation of camera-laser radar relative pose without calibration object.Compared with prior art, the present application has the advantages of realizing the online estimation of relative pose without calibration object, improving the pose estimation precision and the like.

Description

Technical Field

[0001] The present invention relates to the field of multi-sensor fusion, and in particular to a method and system for online estimation of relative pose of a camera and a laser radar without a calibration object. Background Art

[0002] As integral components of robotic vision and autonomous driving systems, cameras and lidar are increasingly being equipped in smart devices. These sensors typically work in tandem to achieve comprehensive environmental awareness. Cameras capture rich color and texture information, while lidar provides precise three-dimensional structural information. By fusing and calibrating camera and lidar data, their relative pose can be determined, and a highly accurate image of the environment can be synthesized in real time. This fused perception system not only provides robots and autonomous vehicles with a wider field of view and eliminates blind spots, but can also be used for critical tasks such as object detection, obstacle recognition, and path planning. In the field of robotic vision, it helps robots more accurately identify objects, locate their own position, and navigate, significantly improving their operational capabilities and autonomy in complex environments. In the field of autonomous driving, the combination of cameras and lidar provides vehicles with reliable environmental perception, enabling them to detect other vehicles, pedestrians, traffic signs, and obstacles in a timely manner, enabling safer and more efficient driving decisions. This plays a crucial role in advancing autonomous driving technology.

[0003] When significant pose deviations appear in the fused imagery of a camera and lidar, timely correction of their relative pose is crucial for ensuring the safety of robotic vision and autonomous driving. Unfortunately, to date, academia and industry still lack effective, online solutions for correcting the relative pose of cameras and lidar. Existing research solutions in this area suffer from the following limitations:

[0004] 1. Most existing camera-lidar relative pose estimation methods rely on the use of calibration objects, which limits their application scenarios and reduces their flexibility and real-time performance. Specifically, first, calibration-based methods rely on specific calibration objects, such as checkerboards and markers. These calibration objects need to be arranged in a specific environment and may be damaged, moved, or obscured in actual application, affecting the accuracy of the calibration. Second, the pose estimation process of calibration-based methods is complex and requires specialized equipment and technicians to operate, which is time-consuming and costly.

[0005] 2. Existing methods are unable to handle the task of online correction of calibration parameters. Collisions and bumps during vehicle or robot operation, as well as thermal expansion and contraction caused by temperature fluctuations, can disrupt the original spatial structure between the surround-view camera and lidar system, causing changes in extrinsic parameters. In this case, if the original extrinsic parameters are still used to synthesize the fused perception image without appropriate adjustments, the resulting fused perception image will show obvious misalignment, indicating that the camera-lidar system's perception is abnormal. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and system for online estimation of camera-lidar relative pose without calibration objects.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to one aspect of the present invention, a method for online estimation of relative pose of a camera and a lidar without a calibration object is provided, the method comprising the following steps:

[0009] S1, using cameras and lidar to collect images and point clouds;

[0010] S2. Use neural networks and attention mechanisms to extract and fuse features from images and point clouds, obtaining fine-grained point cloud cross-modal features and fine-grained image cross-modal features respectively;

[0011] S3. Determine the pixels and points within the common view area based on the fine-grained point cloud cross-modal features and fine-grained image cross-modal features, and establish 2D-3D matching between pixels and points based on a similarity maximization strategy;

[0012] S4. Based on the established 2D-3D matching, a reprojection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the camera-lidar relative pose without calibration objects.

[0013] As a preferred technical solution, the image and point cloud collected in S1 are down-sampling.

[0014] As a preferred technical solution, in S2, feature extraction of images and point clouds is performed by using ResNet to extract multi-scale features of images and PointNet to extract multi-scale features of point clouds.

[0015] As a preferred technical solution, the specific steps of obtaining fine-grained point cloud cross-modal features and fine-grained image cross-modal features in S2 are:

[0016] S21, expand the multi-scale features of the image into a one-dimensional tensor and concatenate it with the multi-scale features of the point cloud and use the attention mechanism to fuse the two features to obtain the coarse-grained point cloud fusion features;

[0017] S22. Extract local features of the point cloud from the multi-scale features of the point cloud based on the KNN algorithm, and use the attention mechanism to fuse the local features of the point cloud with the multi-scale features of the image to obtain coarse-grained image fusion features;

[0018] S23. Based on the twin neural network, the coarse-grained point cloud fusion features and the coarse-grained image fusion features are mapped to the same feature space to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features.

[0019] As a preferred technical solution, the specific method for determining the pixels and points in the common view area in S3 is: inputting the cross-modal features of the pixels and point cloud into the common view area detector respectively to generate the common view scores of the corresponding pixels and point cloud points. If the common view score of a pixel or point cloud point exceeds a preset threshold, the pixel or point cloud point is considered to be in the common view area.

[0020] As a preferred technical solution, the specific process of establishing 2D-3D matching in S3 is as follows:

[0021] S31, calculating the cosine similarity between pixels and point cloud points in the common view area;

[0022] S32, a matching matrix constructed by cosine similarity, where is the number of pixels and is the number of point cloud points;

[0023] S33. Optimize the matching matrix according to the similarity maximization strategy to find the most matching point cloud point for each pixel, thus completing the 2D-3D matching.

[0024] As a preferred technical solution, the similarity maximization strategy in S3 is as follows:

[0025] Feature prediction is performed on point cloud points and pixels respectively, and a pixel feature vector is obtained corresponding to multiple point feature vectors. The similarity between a pixel feature vector and multiple point feature vectors is calculated respectively, and the point with the largest similarity is regarded as the point cloud point that best matches the pixel.

[0026] As a preferred technical solution, the specific formula of the reprojection equation in S4 is:

[0027]

[0028] Where, T / T * is the pose to be optimized, K is the camera internal parameter, p i is the matching 3D point, u i is the matching 2D pixel.

[0029] As a preferred technical solution, the specific steps of using EPnP-RANSAC optimization in S4 to obtain the optimal relative pose are:

[0030] S41, randomly initialize the pose;

[0031] S42, calculating a reprojection error based on the pose;

[0032] S43, using the reprojection error to solve the error gradient;

[0033] S44. Optimize the posture parameters according to the direction of the error gradient, return to S42 and repeat S42-S44 until the set number of iterations is reached.

[0034] According to another aspect of the present invention, a system for online estimation of relative pose of a camera and a lidar without a calibration object is provided. The system operates using the method for online estimation of relative pose of a camera and a lidar without a calibration object as described above. The system includes an information acquisition module, a feature extraction and fusion module, a 2D-3D matching module, and a relative pose estimation module.

[0035] The information acquisition module is used to collect images and point clouds;

[0036] The feature extraction and fusion module uses a neural network and an attention mechanism to extract and fuse features from images and point clouds to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features;

[0037] The 2D-3D matching module establishes 2D-3D matching between pixels and points using a similarity maximization strategy based on fine-grained point cloud cross-modal features and fine-grained image cross-modal features;

[0038] The relative pose estimation module establishes a reprojection equation based on 2D-3D matching and uses EPnP-RANSAC optimization to obtain the optimal relative pose.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This method uses a camera and lidar to capture images and point clouds. A neural network and an attention mechanism are then used to extract and fuse features from the images and point clouds, yielding fine-grained point cloud cross-modal features and fine-grained image cross-modal features, respectively. Pixels and points within the common view area are then extracted, and 2D-3D matching is established between the pixels and points based on a similarity maximization strategy. Finally, a reprojection equation is established based on the established 2D-3D matching, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing online estimation of the camera-lidar relative pose without a calibration object. For camera-lidar systems, this method can both complete initial extrinsic parameter estimation and achieve online estimation of relative pose without a calibration object.

[0041] 2. The present invention extracts fine-grained point cloud cross-modal features and fine-grained image cross-modal features, which does not rely on specific calibration objects and calibration scenes. As long as there is a certain amount of texture in the scene, the online estimation of extrinsic parameters can be completed.

[0042] 3. The present invention integrates the information acquisition module, feature extraction and fusion module, 2D-3D matching module, and relative pose estimation module into the system. Only a pair of point clouds and images need to be input into the system to automatically complete the subsequent process. The process is simple and easy to operate.

[0043] 4. This invention integrates a feature fusion module and a similarity maximization strategy. The feature fusion module uses an attention mechanism, a KNN algorithm, and a twin neural network to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features. The similarity maximization strategy performs feature prediction on both point cloud points and pixels, obtaining a pixel feature vector corresponding to multiple point feature vectors. The similarity between a pixel feature vector and multiple point feature vectors is calculated, and the point with the greatest similarity is used as the point cloud point that best matches the pixel. This can reduce the consumption of computing resources, improve the accuracy of matching, and enhance the precision of pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the steps of a method for online estimation of relative pose of a camera and a lidar without a calibration object in the present invention;

[0045] Figure 2 This is a flow chart of a method for online estimation of relative pose of a camera and a lidar without a calibration object in the present invention;

[0046] Figure 3 Schematic diagram of the similarity maximization strategy flow in the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0048] In this embodiment, a camera-lidar relative pose online estimation method without calibration objects is adopted. Figure 1 As shown, it specifically includes the following steps:

[0049] S1, using cameras and lidar to collect images and point clouds;

[0050] S2. Use neural networks and attention mechanisms to extract and fuse features from images and point clouds, obtaining fine-grained point cloud cross-modal features and fine-grained image cross-modal features respectively;

[0051] S3. Utilize fine-grained point cloud cross-modal features and fine-grained image cross-modal features to extract pixels and points within the common view area, and establish 2D-3D matching between pixels and points based on a similarity maximization strategy.

[0052] S4. Based on the established 2D-3D matching, a reprojection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the camera-lidar relative pose without calibration objects.

[0053] The specific process of this method is as follows Figure 2 As shown in the flow chart. In this embodiment, the camera-lidar system is taken as an example. Figure 2 A detailed description of the process:

[0054] In this embodiment, cameras and lidar are used to capture images and point clouds. Downsampling the point clouds and images reduces computational requirements. The feature extractor first checks whether the image resolution and point cloud data density are sufficient. Each point cloud frame should contain at least 20,480 3D points, and the resolution of each image should not be less than 160,512.

[0055] In this embodiment, if the resolution of the current image and the data density of the point cloud meet the requirements, the Residual Neural Network (ResNet) and PointNet are used to extract the multi-scale features of the image and point cloud respectively. The multi-scale features of the image are expanded into a one-dimensional tensor and spliced ​​with the point cloud features. Feature fusion is performed based on the attention mechanism to obtain the cross-modal features of the point cloud. Specifically, the multi-scale features of the image include feature maps with resolutions scaled by 2, 4, and 8 times and a global feature vector. The multi-scale features of the point cloud include two features containing 256 key points, a point-wise feature tensor, and a global feature vector.

[0056] In this embodiment, in the coarse-grained feature processing stage of scale feature aggregation, after the multi-scale image and point cloud features are extracted, the feature fusion step begins. First, the image features are expanded into a one-dimensional tensor, and the expanded tensor is concatenated with the point cloud features. The feature weighted fusion is performed using the attention mechanism to obtain the coarse-grained point cloud fusion feature f (coarse，P) Based on the KNN algorithm, the local features of the point cloud are extracted and the attention mechanism is used to fuse the local features with the image features to obtain the cross-modal features of the image. At the same time, the multi-scale point cloud features are spliced ​​with the image features and fused with the attention mechanism to obtain the coarse-grained image fusion features f (coarse，I) The attention mechanism is implemented with a multi-layer perceptron and the convolution kernel size is 1×1.

[0057] Subsequently, in the fine-grained feature processing stage, the coarse-grained image fusion features and point cloud fusion features are mapped to the same feature space to obtain fine-grained cross-modal features f with stronger representation capabilities. (fine，P) and f (fine，I) , then enters the 2D-3D matching establishment phase. Taking the cross-modal features of the image and point cloud as input, the Siamese network module maps them into the same feature space, obtaining a more consistent cross-modal representation of the image and point cloud. Specifically, this feature space is constructed using the Siamese network structure, with PointNet as the basic unit in the network structure.

[0058] In this embodiment, during the 2D-3D matching process, a confocal volumetric display (CVD) for images and point clouds is first used to detect pixels and point cloud points in the confocal area. Taking the cross-modal features of the image and point cloud as input, the confocal area detector is used to generate a pixel / point cloud level confocal score. Among them, the confocal area detector for the image is implemented with 3 convolutional layers and a Softmax layer. The final confocal score of the pixel and point cloud point is between 0 and 1. Based on the threshold discrimination method, the pixels and point cloud points in the confocal area are screened out. Specifically, if the confocal score of a pixel or point cloud point exceeds the threshold, the pixel / point cloud point is considered to be in the confocal area. Pixels and point cloud points above the threshold are considered to be in the confocal area, and the threshold is set to 0.95.

[0059] Secondly, the cosine distance of fine-grained cross-modal features between pixels and point cloud points in the common view area is calculated. An M×N matching matrix is ​​constructed as the correlation map, which is specifically:

[0060]

[0061] Where M is the number of pixels in the common view area, N is the number of point cloud points in the common view area, δ(·) is the cosine distance operator, and u and p are the matched 2D pixels and 3D points.

[0062] In this embodiment, Figure 2 and Figure 3 The presented similarity maximization strategy is used to optimize the correlation graph to obtain the final 2D-3D correspondence. First, for each pixel, the point cloud point with the highest similarity, i.e., the lowest cosine distance, is selected as the point cloud point matching the pixel, and a 2D-3D matching is established in the form of pixel-point cloud points. Next, the number of matching pixel-point pairs is checked. If there are less than 10 pairs, the frame is discarded. The resulting 2D-3D correspondence is as follows:

[0063]

[0064] Where M is the number of pixels in the common view area, N is the number of point cloud points in the common view area, δ(·) is the cosine distance operator, and u and p are the matched 2D pixels and 3D points.

[0065] In this embodiment, in the PnP solver, for each pair of matched pixels and points, a series of reprojection equations are constructed based on the 3×3 camera intrinsic parameter matrix, 2×1 pixel coordinates, and 3×1 point coordinates, with the reprojection error as the final error. The specific formula of the constructed reprojection equation is:

[0066]

[0067] Where, T / T * is the pose to be optimized, K is the camera internal parameter, p i is the matching 3D point, u i For matching 2D pixels, the transformation process between homogeneous coordinates and non-homogeneous coordinates is omitted.

[0068] In this embodiment, the reprojection equation constructed is iteratively optimized using the random sampling consensus algorithm (EPnP-RANSAC) with 500 iterations. The specific optimization steps are: first, randomly initialize the pose; then calculate the reprojection error based on the pose; then solve the error gradient; finally, optimize the pose parameters according to the gradient direction; and calculate the reprojection error again based on the pose;

[0069] In this embodiment, the EPnP-RANSAC algorithm is used to perform 500 iterative pose optimizations and output a Boolean estimated marker value. That is, the camera intrinsic parameters and matching point / pixel coordinates are fixed, and only the camera pose is optimized. After the iterative optimization is completed, if the estimated marker value is true, the optimized relative pose is read according to the optimization result, and the relative position between the camera and radar is updated.

[0070] In summary, this method is designed for camera-lidar systems, which can not only complete the initial extrinsic parameter estimation but also realize the online estimation of relative pose. It does not rely on specific calibration objects and calibration scenes, and can complete the online estimation of extrinsic parameters as long as there are a certain number of textures in the scene. The process is simple, and only a pair of point clouds and images need to be input to automatically complete the subsequent processes. The integration of similarity maximization strategy and feature fusion module can reduce the consumption of computing resources and improve the accuracy of matching establishment, thereby improving the accuracy of pose estimation.

[0071] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for online estimation of relative pose of camera and lidar without calibration objects, characterized in that: The method comprises the following steps: S1, using cameras and lidar to collect images and point clouds; S2. Use neural networks and attention mechanisms to extract and fuse features from images and point clouds, obtaining fine-grained point cloud cross-modal features and fine-grained image cross-modal features respectively; S3. Determine the pixels and points within the common view area based on the fine-grained point cloud cross-modal features and fine-grained image cross-modal features, and establish 2D-3D matching between pixels and points based on a similarity maximization strategy; S4. Based on the established 2D-3D matching, a reprojection equation is established, and EPnP-RANSAC optimization is used to obtain the optimal relative pose, thereby completing the online estimation of the camera-lidar relative pose without calibration objects; The specific steps of obtaining the fine-grained point cloud cross-modal features and the fine-grained image cross-modal features in S2 are: S21, expand the multi-scale features of the image into a one-dimensional tensor and concatenate it with the multi-scale features of the point cloud and use the attention mechanism to fuse the two features to obtain the coarse-grained point cloud fusion features; S22. Extract local features of the point cloud from the multi-scale features of the point cloud based on the KNN algorithm, and use the attention mechanism to fuse the local features of the point cloud with the multi-scale features of the image to obtain coarse-grained image fusion features; S23. Based on the twin neural network, the coarse-grained point cloud fusion features and the coarse-grained image fusion features are mapped to the same feature space to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features; The specific process of establishing 2D-3D matching in S3 is as follows: S31, calculating the cosine similarity between pixels and point cloud points in the common view area; S32. Construct an M×N matching matrix based on cosine similarity, where M is the number of pixels and N is the number of point cloud points. S33, optimizing the matching matrix according to the similarity maximization strategy, finding the most matching point cloud point for each pixel, and completing the 2D-3D matching; The specific formula of the reprojection equation in S4 is: Where, T / T * is the pose to be optimized, K is the camera internal parameter, p i is the matching 3D point, u i is the matching 2D pixel.

2. The method for online estimation of relative pose of a camera and a laser radar without a calibration object according to claim 1, characterized in that: The image and point cloud collected in S1 are down-sampled.

3. The method for online estimation of relative pose of camera and laser radar without calibration object according to claim 1, characterized in that: In S2, the feature extraction of the image and the point cloud is performed by using ResNet to extract the multi-scale features of the image and using PointNet to extract the multi-scale features of the point cloud.

4. The method for online estimation of relative pose of camera and laser radar without calibration objects according to claim 1, characterized in that: The specific method for determining the pixels and points in the common view area in S3 is: inputting the cross-modal features of the pixels and point cloud into the common view area detector respectively to generate the common view scores of the corresponding pixels and point cloud points. If the common view score of a pixel or point cloud point exceeds a preset threshold, the pixel or point cloud point is considered to be in the common view area.

5. The method for online estimation of relative pose of camera and laser radar without calibration objects according to claim 1, characterized in that: The similarity maximization strategy in S3 is specifically as follows: Feature prediction is performed on point cloud points and pixels respectively, and a pixel feature vector is obtained corresponding to multiple point feature vectors. The similarity between a pixel feature vector and multiple point feature vectors is calculated respectively, and the point with the largest similarity is regarded as the point cloud point that best matches the pixel.

6. The method for online estimation of relative pose of camera and laser radar without calibration objects according to claim 1, characterized in that: The specific steps of using EPnP-RANSAC optimization in S4 to obtain the optimal relative pose are: S41, randomly initialize the pose; S42, calculating a reprojection error based on the pose; S43, using the reprojection error to solve the error gradient; S44. Optimize the posture parameters according to the direction of the error gradient, return to S42 and repeat S42-S44 until the set number of iterations is reached.

7. A camera-lidar relative pose online estimation system without calibration objects, characterized in that: The system is operated by a method for online estimation of relative pose of a camera and a lidar without a calibration object as described in any one of claims 1 to 6, and the system includes an information acquisition module, a feature extraction and fusion module, a 2D-3D matching module, and a relative pose estimation module; The information acquisition module is used to collect images and point clouds; The feature extraction and fusion module uses a neural network and an attention mechanism to extract and fuse features from images and point clouds to obtain fine-grained point cloud cross-modal features and fine-grained image cross-modal features; The 2D-3D matching module establishes 2D-3D matching between pixels and points using a similarity maximization strategy based on fine-grained point cloud cross-modal features and fine-grained image cross-modal features; The relative pose estimation module establishes a reprojection equation based on 2D-3D matching and uses EPnP-RANS AC optimization to obtain the optimal relative pose.

Images