Vehicle attitude estimation method based on image and point cloud feature alignment and attitude fine adjustment

Abstract

The invention discloses a vehicle attitude estimation method based on image and point cloud feature alignment and attitude fine adjustment. The invention discloses a vehicle attitude estimation method based on image and point cloud feature alignment and attitude fine adjustment, and mainly solves the problems of low accuracy of a vehicle attitude estimation result and mismatching of image feature and point cloud feature representation information in the prior art. The implementation scheme is as follows: 1) acquiring a point cloud and an image data set; 2) constructing a pose estimation model; 3) constructing a model loss function; 4) training a pose estimation model; and 5) reasoning and obtaining an attitude transformation result. According to the vehicle pose estimation model constructed by the invention, the enhancement of the multi-modal common feature representation capability is realized through the image and point cloud feature matching alignment mode and the cross-modal attention mechanism, the fine adjustment of the pose estimation result is further realized, the accuracy of the vehicle pose estimation is improved, and the practical application value is relatively high.

Description

Technical Field

[0001] The present invention belongs to the technical field of real-time mapping and positioning, and in particular relates to a vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning. Background Art

[0002] With the continuous development of unmanned driving technology, real-time mapping and positioning has become one of the core technologies for achieving stable operation of autonomous driving systems. Accurate vehicle posture estimation is crucial in real-time mapping and positioning technology, especially in complex dynamic environments. Accurately grasping the vehicle's posture information is the basis for path planning, obstacle detection and automatic control system decision-making. Traditional posture estimation methods usually rely on single sensor data, such as inertial measurement units (IMUs) or monocular cameras, but these methods have certain limitations in practical applications and are difficult to meet high-precision requirements. In recent years, the fusion technology of laser radar (LiDAR) and cameras has received widespread attention and has become an effective means to improve the accuracy and robustness of posture estimation. LiDAR provides high-precision spatial information through point cloud data generated by laser scanning, which can effectively assist posture estimation; while cameras provide rich visual information and have significant advantages in scene understanding and object recognition. However, how to efficiently align the feature information of these two heterogeneous sensors and achieve high-precision vehicle posture estimation based on this is still a technical challenge. In addition, existing vehicle posture estimation algorithms usually adopt a single model structure and only estimate the posture transformation results in the output stage, which makes it impossible to perform more refined optimization of the model in the training stage, thus affecting the accuracy of the estimation results. Summary of the invention

[0003] The present invention fully considers the shortcomings of the existing methods, and its purpose is to provide a vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning. Through the image and point cloud feature matching and alignment and the cross-modal attention mechanism, the multi-modal common feature representation capability is enhanced, and then the posture estimation result is fine-tuned, thereby improving the accuracy of vehicle posture estimation.

[0004] 1. Technical Principle

[0005] At present, unsupervised vehicle attitude estimation models based on learning usually use a single sensor input, which has certain limitations. For example, the visual attitude estimation model is susceptible to the influence of depth information ambiguity and dynamic object interference, while the point cloud attitude estimation model is limited by sparsity and high computational overhead. In order to fully combine the advantages of the two types of sensor data and improve the accuracy and robustness of the odometer, the present invention proposes an unsupervised attitude estimation framework based on image and point cloud feature alignment and attitude fine-tuning, which effectively combines the information of the two sensors and uses unsupervised learning to reduce the learning cost. Due to the differences in the data organization form and field of view of the camera and the lidar, the features corresponding to the two modes also have differences in the areas or targets concerned when representing the scene information, resulting in the instability of the model during the learning process, thereby affecting the accuracy of the attitude estimation results. To address this problem, the present invention designs a feature alignment and posture refinement module, which improves the consistency of feature representation through matching point alignment module, cross-modal attention mechanism, and cross-frame attention mechanism, thereby achieving fine-tuning of the initial attitude transformation matrix and further improving the accuracy of the attitude transformation results.

[0006] 2. According to the above principle, the present invention is implemented by the following scheme:

[0007] A vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning comprises the following steps:

[0008] (1) Obtaining point cloud and image datasets; collecting images and point cloud datasets collected by the unmanned vehicle during a continuous driving process as a training set. The dataset contains N frames of image data {I1, I2, …, I N}, point cloud data {PC1, PC2, ..., PC N} and the corresponding calibration parameter set, the sampling frequency of the image and point cloud data is the same, and I t and PC t ,{t=1,2,…,N} have the same acquisition timestamp, the image data is collected by a single camera, and the point cloud data is collected by a single rotating lidar; the same method is used to collect the image and point cloud data sets of the corresponding unmanned vehicle during a continuous driving process as the test set.

[0009] (2) Constructing a pose estimation model; the model consists of a data preprocessing module, a depth estimation module, a multimodal feature encoding module, a feature alignment and pose refinement module. The specific construction process includes the following steps:

[0010] (2-a) Construct a data preprocessing module to calculate the PC of two consecutive point clouds at t-1 and t t-1 and PC t As input, the PC t-1 and PCt Generate point cloud projection V through cylindrical projection and coordinate transformation respectively t-1 and V t , where the coordinates are transformed from the point cloud coordinate system to the camera coordinate system.

[0011] (2-b) Construct a depth estimation module based on two consecutive frames of image I at t-1 and t t-1 and I t As input, resize the image to H×W and get the resized image and Processing using the depth estimation neural network Monodepth2 and The relative depth map D is obtained respectively t-1 and D t ; Where H and W are positive integers, and H∈[1,10000], W∈[1,10000].

[0012] (2-c) Construct a multimodal feature encoding module, which consists of an image encoder, a depth encoder, a radar encoder, and an initial regressor; convert the adjusted image obtained in step (2-b) and The concatenation is performed in the channel dimension and then input into the image encoder with ResNet-18 as the feature extractor to obtain the image feature E img ; The relative depth map D obtained in step (2-b) t-1 and D t The concatenation is performed in the channel dimension and then input into the deep encoder with ResNet-18 as the feature extractor to obtain the deep feature E depth ; Project the point cloud V obtained in step (2-a) t-1 and V t The concatenation is performed in the channel dimension, and then the radar encoder with ResNet-18 as the feature extractor is input. The output of the radar encoder is pooled into the same shape as E using an adaptive average pooling operation. depth Consistent, get the point cloud feature E lidar ; E img 、E depth and E lidar Use the initial regressor to estimate their corresponding posture estimation results respectively, and perform weighted summation on these results to obtain the initial posture transformation matrix The initial regressor consists of three fully connected layers.

[0013] (2-d) Construct a feature alignment and posture refinement module, and transform the image two-stage feature E obtained in step (2-c) img And point cloud two-stage feature E lidarAfter being flattened, they are input into the fully connected layer to reduce the feature dimension to M, and the and Where M is a positive integer and M∈[1,10000].

[0014] The adjusted image And point cloud projection map V t-1 The obtained product in step (II-3) Perform the distortion operation to obtain the distorted images And point cloud projection Input them together into the matching point alignment module 1 to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t-1,img ,Will and Concatenate in the channel dimension to get F t-1,lidar ; F t-1,img and F t-1,lidar As the image feature input and point cloud feature input of the cross-modal attention module 1, the fusion feature F is obtained. t-1,fusion .

[0015] The adjusted image And point cloud projection map V t Input the matching point alignment module 2 together to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t,img ,Will and Concatenate in the channel dimension to get F t,lidar ; F t,img and F t,lidar As the image feature input and point cloud feature input of the cross-modal attention module 2, the fusion feature F is obtained. t,fusion .

[0016] F t-1,fusion and F t,fusion Input the cross-frame attention module to obtain the fine-tuned feature F final ; F final Input the fine-tuning regressor to get the posture fine-tuning matrix ΔT, and use ΔT to transform the initial posture matrix Perform posture fine-tuning to obtain the final posture transformation result The calculation formula is:

[0017]

[0018] The fine-tuning regressor consists of two fully connected layers and one regression layer; This is the output of the pose estimation model.

[0019] The matching point alignment module 1 and the matching point alignment module 2 in this step have the same structure and are constructed as follows:

[0020] (2-d-1) Construct a matching point alignment module; calculate the position of each 3D point in the point cloud projection image in the pixel coordinate system of the adjusted image, extract the RGB value of the position in the adjusted image as the image matching point corresponding to the 3D point, and randomly sample all valid matching point pairs to obtain M pairs of matching image points and 3D points, which are the output of the matching point alignment module.

[0021] The cross-modal attention module 1 and the cross-modal attention module 2 in this step have the same structure and are constructed as follows:

[0022] (2-d-2) Construct a cross-modal attention module; calculate image features F img and point cloud features F lidar The similarity matrix S is calculated as follows:

[0023]

[0024] in Indicates F img The transpose of Represents matrix multiplication; the softmax function is used on the similarity matrix S to obtain the weighted matrix W lidar , W lidar Multiply the matrix with the point cloud features to get the point cloud weighted features; transpose the similarity matrix S and then use the softmax function to get the weighted matrix W img , W img The image weighted features are obtained by matrix multiplication with the image features; the point cloud weighted features and the image weighted features are concatenated in the channel dimension to obtain the fused features, which are the output of the cross-modal attention module.

[0025] The cross-frame attention module in this step is constructed as follows:

[0026] (2-d-3) Construct a cross-frame attention module to calculate the fusion feature F of the t-1th frame t-1,fusion And the fusion feature F of the tth frame t,fusion The similarity matrix S fusion , the calculation process is:

[0027]

[0028] in Ft,fusion The transpose of Represents matrix multiplication; for the similarity matrix S fusion Use the softmax function to get the weight matrix W fusion , W fusion and F t-1,fusion Perform matrix multiplication to obtain the fine-tuning feature F final , which is the output of the cross-frame attention module.

[0029] (3) Constructing the model loss function; the specific construction process includes the following steps:

[0030] (3-a) Construct image photometric reconstruction loss L img :

[0031]

[0032] where ||·||1 is the L-1 regularization constraint, is the adjusted image of the tth frame, for The t-th frame pseudo image is obtained after photometric reconstruction (see the method: Zhou, T., Brown, M., Snavely, N., & Lowe, DG (2017). Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1851-1858).).

[0033] (3-b) Construct point cloud reconstruction loss L LiDAR :

[0034]

[0035] where ||·||2 represents L-2 regularization, p t-1,k Represents the point cloud projection image V t-1 3D points in Indicates that the point cloud projection image V t Point p in t,k The 3D point obtained after the distortion operation, N represents p t-1,k and The number of valid matching pairs in .

[0036] (3-c) Construct the total loss function L total :

[0037] L total =Limg +L LiDAR

[0038] (4) training the pose estimation model; using the training set obtained in step (1) to train the pose estimation model constructed in step (2); using the loss function L constructed in step (3) total Calculate the model loss; during the training process, use the Adam algorithm to update the model parameters and use L-2 regularization as a constraint until the loss value no longer decreases or reaches the maximum number of training times to obtain a trained pose estimation model.

[0039] (5) Input the test set data obtained in step (1) into the trained pose estimation model. The output of the model is This is the final result of the posture transformation.

[0040] In step (2-b), H and W are preferably 192 and 640, respectively.

[0041] In step (2-d), M is preferably 4096.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] (1) The unsupervised pose estimation framework based on image and point cloud feature alignment and pose fine-tuning constructed by the present invention can efficiently process data from multiple sensors, effectively avoid the pose estimation failure problem caused by the limitations of a single sensor, and achieve accurate pose estimation. At the same time, the framework uses self-supervised learning, does not require expensive labels, and reduces the learning cost.

[0044] (2) The feature alignment and posture refinement module constructed by the present invention obtains matching image points and 3D points through a matching point alignment strategy, and uses effective matching point pairs to guide feature alignment, so that the features of the image and point cloud focus on the same area or target in the environment, thereby improving the consistency of feature representation.

[0045] (3) The feature alignment and posture refinement module constructed by the present invention realizes the mutual weighting of multimodal features within a frame and the weighting of features between frames through a cross-modal attention module and a cross-frame attention module, respectively, so that the model can autonomously learn the same scene information from different sensors and different frame data, thereby improving the scene representation ability of the feature and further realizing the fine-tuning of the initial posture transformation matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flow chart of a vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning according to an embodiment of the present invention;

[0047] Figure 2 A structural diagram of a pose estimation model constructed in an embodiment of the present invention;

[0048] Figure 3 A structural diagram of the feature alignment and posture refinement module in the overall model of an embodiment of the present invention;

[0049] Figure 4 A structural diagram of a matching point alignment module in the overall model of an embodiment of the present invention;

[0050] Figure 5 The structure diagram of the cross-modal attention module in the overall model of the embodiment of the present invention;

[0051] Figure 6 A structural diagram of the cross-frame attention module in the overall model of an embodiment of the present invention;

[0052] Figure 7 Comparison chart of the vehicle posture estimation method according to the embodiment of the present invention and other methods on sequence 9 of the KITTIOdometry dataset;

[0053] Figure 8 A comparison chart of the vehicle posture estimation method according to an embodiment of the present invention and other methods on sequence 10 of the KITTIOdometry dataset. DETAILED DESCRIPTION

[0054] The specific implementation mode of the present invention is described below:

[0055] Example 1

[0056] Figure 1 The flowchart of the vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning implemented by the present invention is shown, and the specific steps are as follows:

[0057] Step 1, obtain point cloud and image datasets:

[0058] The image and point cloud dataset collected by the unmanned vehicle during a continuous driving process is collected as a training set. The dataset contains N frames of image data {I1, I2, …, I N}, point cloud data {PC1, PC2, ..., PC N} and the corresponding calibration parameter set, the sampling frequency of the image and point cloud data is the same, and I t and PC t ,{t=1,2,…,N} have the same acquisition timestamp, the image data is collected by a single camera, and the point cloud data is collected by a single rotating lidar; the same method is used to collect the image and point cloud data sets of the corresponding unmanned vehicle during a continuous driving process as the test set.

[0059] Step 2: The model consists of a data preprocessing module, a depth estimation module, a multimodal feature encoding module, a feature alignment and posture refinement module. The specific construction process includes the following steps:

[0060] Figure 2 The figure shows a structural diagram of a pose estimation model constructed in an embodiment of the present invention, and the specific steps are as follows:

[0061] (2-a) Constructing data preprocessing module

[0062] Take two consecutive frames of point cloud PC at t-1 and t t-1 and PC t As input, the PC t-1 and PC t Generate point cloud projection V through cylindrical projection and coordinate transformation respectively t-1 and V t , where the coordinates are transformed from the point cloud coordinate system to the camera coordinate system.

[0063] (2-b) Constructing the depth estimation module

[0064] Take two consecutive frames of image I at t-1 and t t-1 and I t As input, resize the image to H×W and get the resized image and Processing using the depth estimation neural network Monodepth2 and The relative depth map D is obtained respectively t-1 and D t ; where H and W are 192 and 640 respectively.

[0065] (2-c) Constructing a multimodal feature encoding module

[0066] This module consists of an image encoder, a depth encoder, a radar encoder, and an initial regressor; the adjusted image obtained in step (2-b) is and The concatenation is performed in the channel dimension and then input into the image encoder with ResNet-18 as the feature extractor to obtain the image feature E img ; The relative depth map D obtained in step (2-b) t-1 and D t The concatenation is performed in the channel dimension and then input into the deep encoder with ResNet-18 as the feature extractor to obtain the deep feature E depth ; Project the point cloud V obtained in step (2-a) t-1 and V tThe concatenation is performed in the channel dimension, and then the radar encoder with ResNet-18 as the feature extractor is input. The output of the radar encoder is pooled into the same shape as E using an adaptive average pooling operation. depth Consistent, get the point cloud feature E lidar ; E img 、E depth and E lidar Use the initial regressor to estimate their corresponding posture estimation results respectively, and perform weighted summation on these results to obtain the initial posture transformation matrix The initial regressor consists of three fully connected layers.

[0067] (2-d) Construct feature alignment and pose refinement module

[0068] Figure 3 The figure shows the structure of the feature alignment and posture refinement module in the pose estimation model in the embodiment of the present invention. The image two-stage feature E obtained in step (2-c) is img And point cloud two-stage feature E lidar After being flattened, they are input into the fully connected layer to reduce the feature dimension to M, and the and Where M=4096.

[0069] The adjusted image And point cloud projection map V t-1 The obtained product in step (II-3) Perform the distortion operation to obtain the distorted images And point cloud projection Input them together into the matching point alignment module 1 to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t-1,img ,Will and Concatenate in the channel dimension to get F t-1,lidar ; F t-1,img and F t-1,lidar As the image feature input and point cloud feature input of the cross-modal attention module 1, the fusion feature F is obtained. t-1,fusion ;

[0070] The adjusted image And point cloud projection map V t Input the matching point alignment module 2 together to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get Ft,img ,Will and Concatenate in the channel dimension to get F t,lidar ; F t,img and F t,lidar As the image feature input and point cloud feature input of the cross-modal attention module 2, the fusion feature F is obtained. t,fusion ;

[0071] F t-1,fusion and F t,fusion Input the cross-frame attention module to obtain the fine-tuned feature F final ; F final Input the fine-tuning regressor to get the posture fine-tuning matrix ΔT, and use ΔT to transform the initial posture matrix Perform posture fine-tuning to obtain the final posture transformation result The calculation formula is:

[0072]

[0073] The fine-tuning regressor consists of two fully connected layers and one regression layer; This is the output of the pose estimation model.

[0074] The matching point alignment module 1 and the matching point alignment module 2 in this step have the same structure and are constructed as follows:

[0075] (2-d-1) Construct matching point alignment module

[0076] Figure 4 The figure shows the structure diagram of the matching point alignment module in the feature alignment and posture refinement module in the embodiment of the present invention; the position of each 3D point in the point cloud projection image in the pixel coordinate system of the adjusted image is calculated, and the RGB value of the position in the adjusted image is extracted as the image matching point corresponding to the 3D point. All valid matching point pairs are randomly sampled to obtain M pairs of matching image points and 3D points, which are the output of the matching point alignment module.

[0077] The cross-modal attention module 1 and the cross-modal attention module 2 in this step have the same structure and are constructed as follows:

[0078] (2-d-2) Constructing a cross-modal attention module

[0079] Figure 5 The cross-modal attention module structure diagram in the feature alignment and posture refinement module of the embodiment of the present invention is shown as follows: Calculate the image feature F img and point cloud features F lidar The similarity matrix S is calculated as follows:

[0080]

[0081] in Indicates F img The transpose of Represents matrix multiplication; the softmax function is used on the similarity matrix S to obtain the weighted matrix W lidar , W lidar Multiply the matrix with the point cloud features to get the point cloud weighted features; transpose the similarity matrix S and then use the softmax function to get the weighted matrix W img , W img The image weighted features are obtained by matrix multiplication with the image features; the point cloud weighted features and the image weighted features are concatenated in the channel dimension to obtain the fused features, which are the output of the cross-modal attention module.

[0082] The cross-frame attention module in this step is constructed as follows:

[0083] (2-d-3) Constructing a cross-frame attention module

[0084] Figure 6 The figure shows the structure of the cross-frame attention module in the feature alignment and posture refinement module of the embodiment of the present invention, which calculates the fusion feature F of the t-1th frame. t-1,fusion And the fusion feature F of the tth frame t,fusion The similarity matrix S fusion , the calculation process is:

[0085]

[0086] in F t,fusion The transpose of Represents matrix multiplication; for the similarity matrix S fusion Use the softmax function to get the weight matrix W fusion , W fusion and F t-1,fusion Perform matrix multiplication to obtain the fine-tuning feature F final , which is the output of the cross-frame attention module.

[0087] Step 3: construct the model loss function. The specific construction process includes the following steps:

[0088] (3-a) Construct image photometric reconstruction loss L img :

[0089]

[0090] where ||·||1 is the L-1 regularization constraint, is the adjusted image of the tth frame, for The t-th frame pseudo image is obtained after photometric reconstruction.

[0091] (3-b) Construct point cloud reconstruction loss L LiDAR :

[0092]

[0093] where ||·||2 represents L-2 regularization, p t-1,k Represented in the point cloud projection V t-1 3D points in Indicates that the point cloud projection image V t Point p in t,k The 3D point obtained after the distortion operation, N represents p t-1,k and The number of valid matching pairs in .

[0094] (3-c) Construct the total loss function L total :

[0095] L total =L img +L LiDAR

[0096] Step 4: Train the pose estimation model:

[0097] Use the training set obtained in step 1 to train the pose estimation model constructed in step 2; use the loss function L constructed in step 3 total Calculate the model loss; during the training process, use the Adam algorithm to update the model parameters and use L-2 regularization as a constraint until the loss value no longer decreases or reaches the maximum number of training times to obtain a trained pose estimation model.

[0098] Step 5: Infer and obtain the posture transformation result:

[0099] The test set data obtained in step 1 is input into the trained pose estimation model. The output of the model is This is the final result of the posture transformation.

[0100] Example 2

[0101] The method in Example 1 was used to conduct a self-supervised multi-sensor fusion mileage estimation method experiment on the public dataset KITTIOdometry. The experimental operating system was Ubuntu 18.04, based on the PyTorch 1.8.0 framework of CUDA 11.3 and cuDNN 8.2.1, and trained and tested on a personal computer equipped with Intel (R) Xeon (R) W-2123 CPU (3.60 GHz) and NVIDIA GeForce RTX 3090 (24 GB) hardware.

[0102] This embodiment uses the translation error t err (%), rotation error r err The two indicators (deg / 100m) are used to compare the six odometer methods, Monodepth2, SC-SfMLearner, FeatDepth, Zhou et al., MLF-VO, and DeLORA, with the method of the present invention on the KITTIOdometry dataset. err (%) and r err (deg / 100m) is calculated in units of 100 meters, which respectively represent the average translation error of a trajectory length of 100 meters and the average rotation error of a subsequence of a trajectory length of 100 meters. The unit is angle, and the calculation formula is as follows:

[0103]

[0104] In the formula Respectively represent the true value of the translation vector and the true value of the rotation matrix of the i-th frame, denote the translation vector and rotation matrix of the i-th frame calculated by the odometer method, respectively. N denotes the number of frames in the 100-meter subsequence trajectory. ||·|| denotes the Euclidean norm. trace(·) denotes the matrix trace operation.

[0105] The comparison results are shown in Table 1. It can be found that compared with other methods, the present invention can obtain accurate odometer results on multiple sequences of the KITTIOdometry dataset. In all two sequences, 5 evaluation indicators reach the optimal level.

[0106] Figure 7 , Figure 8 The figure shows the comparison of the vehicle posture estimation method of the embodiment of the present invention and other methods on sequence 09 and sequence 10 of the KITTIOdometry dataset. The results show that the posture prediction results of the model designed by the present invention are closer to the true value results than the image-based odometer method (MLF-VO) and the odometer method based on point cloud data (DeLORA); the visualization results of the whole sequence have a smaller error than the posture estimation method based on a single sensor; the present invention can achieve better results in both the sequence with rich scene features (sequence 09) and the sequence with sparse scene features (sequence 10).

[0107] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made based on the structure and principle of the present invention should be included in the protection scope of the present invention.

[0108] Table 1

[0109]

Claims

1. A vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning, characterized in that: The following steps are involved: (I) Obtain point cloud and image datasets: The image and point cloud datasets collected by the unmanned vehicle during a continuous driving process are collected as training sets. The dataset contains N frames of image data {I1,I 2, …,I N }、Point cloud data{PC1,PC 2, …,PC N } and the corresponding calibration parameter set, the sampling frequency of the image and point cloud data is the same, and I t and PC t ,{t=1,2,…,N} have the same acquisition timestamp, the image data is collected by a single camera, and the point cloud data is collected by a single rotating laser radar; the image and point cloud data sets collected by the corresponding unmanned vehicle during a continuous driving process are used as the test set in the same way; (II) Constructing a pose estimation model: The model consists of a data preprocessing module, a depth estimation module, a multimodal feature encoding module, a feature alignment and posture refinement module. The specific construction process includes the following steps: (II-1) Constructing data preprocessing module: Take two consecutive frames of point cloud PC at t-1 and t t-1 and PC t As input, the PC t-1 and PC t Generate point cloud projection V through cylindrical projection and coordinate transformation respectively t-1 and V t , where the coordinates are transformed from the point cloud coordinate system to the camera coordinate system; (II-2) Constructing the depth estimation module: Take two consecutive frames of image I at t-1 and t t-1 and I t As input, resize the image to H×W and get the resized image and Processing using the depth estimation neural network Monodepth2 and The relative depth map D is obtained respectively t-1 and D t ; Where H and W are positive integers, and H∈[1,10000], W∈[1,10000]; (II-3) Constructing a multimodal feature encoding module: This module consists of an image encoder, a depth encoder, a radar encoder, and an initial regressor; the adjusted image obtained in step (II-2) and The concatenation is performed in the channel dimension and then input into the image encoder with ResNet-18 as the feature extractor to obtain the image feature E img ; The relative depth map D obtained in step (II-2) t-1 and D t The concatenation is performed in the channel dimension and then input into the deep encoder with ResNet-18 as the feature extractor to obtain the deep feature E depth ; Project the point cloud V obtained in step (II-1) t-1 and V t The concatenation is performed in the channel dimension, and then the radar encoder with ResNet-18 as the feature extractor is input. The output of the radar encoder is pooled into the same shape as E using an adaptive average pooling operation. depth Consistent, get the point cloud feature E lidar ; E img 、E depth and E lidar Use the initial regressor to estimate their corresponding posture estimation results respectively, and perform weighted summation on these results to obtain the initial posture transformation matrix The initial regressor consists of three fully connected layers; (II-4) Construct feature alignment and pose refinement module: The image feature E obtained in step (II-3) img and point cloud feature E lidar After being flattened, they are input into the fully connected layer to reduce the feature dimension to M, and the and Where M is a positive integer and M∈[1,10000]; The adjusted image And point cloud projection map V t-1 The obtained product in step (II-3) Perform the distortion operation to obtain the distorted images And point cloud projection Input them together into the matching point alignment module 1 to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t-1,img ,Will and Concatenate in the channel dimension to get F t-1,lidar ; F t-1,img and F t-1,lidar As the image feature input and point cloud feature input of the cross-modal attention module 1, the fusion feature F is obtained. t-1,fusion ; The adjusted image And point cloud projection map V t Input the matching point alignment module 2 together to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t,img ,Will and Concatenate in the channel dimension to get F t,lidar ; F t,img and F t,lidar As the image feature input and point cloud feature input of the cross-modal attention module 2, the fusion feature F is obtained. t,fusion ; F t-1,fusion and F t,fusion Input the cross-frame attention module to obtain the fine-tuned feature F final ; F final Input the fine-tuning regressor to get the posture fine-tuning matrix ΔT, and use ΔT to transform the initial posture matrix Perform posture fine-tuning to obtain the final posture transformation result The calculation formula is: The fine-tuning regressor consists of two fully connected layers and one regression layer; This is the output of the pose estimation model; (III) Construct the model loss function. The specific construction process is as follows: (III-1) Constructing image photometric reconstruction loss L img : where ||·||1 is the L-1 regularization constraint, is the adjusted image of the tth frame, for The pseudo image of the tth frame obtained after photometric reconstruction; (III-2) Constructing point cloud reconstruction loss L LiDAR : where ||·||2 represents L-2 regularization, p t-1,k Represents the point cloud projection image V t-1 3D points in Indicates that the point cloud projection image V t Point p in t,k The 3D point obtained after the distortion operation, N represents p t-1,k and The number of valid matching pairs in ; (III-3) Constructing the total loss function L total : L total =L img +L LiDAR (IV) Training pose estimation model: Use the training set obtained in step (I) to train the pose estimation model constructed in step (II); use the loss function L constructed in step (III) total Calculate the model loss; during the training process, use the Adam algorithm to update the model parameters and use L-2 regularization as a constraint until the loss value no longer decreases or reaches the maximum number of training times, and then obtain a trained pose estimation model; (V) Reasoning and obtaining posture transformation results: The test set data obtained in step (I) is input into the trained pose estimation model, and the output of the model is This is the final result of the posture transformation.

2. A vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning as claimed in claim 1, characterized in that: The matching point alignment module 1 and the matching point alignment module 2 in step (II-4) have the same structure and are constructed as follows: Calculate the position of each 3D point in the point cloud projection image in the pixel coordinate system of the adjusted image, extract the RGB value of the position in the adjusted image as the image matching point corresponding to the 3D point, randomly sample all valid matching point pairs, and obtain M pairs of matching image points and 3D points, which are the output of the matching point alignment module.

3. A vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning as claimed in claim 1, characterized in that: The cross-modal attention module 1 and the cross-modal attention module 2 in step (II-4) have the same structure and are constructed as follows: Calculate image features F img and point cloud features F lidar The similarity matrix S is calculated as follows: in Indicates F img The transpose of Represents matrix multiplication; Use the softmax function on the similarity matrix S to get the weighted matrix W lidar , W lidar Perform matrix multiplication with point cloud features to obtain point cloud weighted features; Transpose the similarity matrix S and then use the softmax function to get the weighted matrix W img , W img The image weighted features are obtained by matrix multiplication with the image features; the point cloud weighted features and the image weighted features are concatenated in the channel dimension to obtain the fused features, which are the output of the cross-modal attention module.

4. The vehicle posture estimation method based on image and point cloud feature alignment and posture fine-tuning as claimed in claim 1, characterized in that: The cross-frame attention module in step (II-4) is constructed as follows: Calculate the fusion feature F of the t-1th frame t-1,fusion And the fusion feature F of the tth frame t,fusion The similarity matrix S fusion , the calculation process is: in F t,fusion The transpose of Represents matrix multiplication; for the similarity matrix S fusion Use the softmax function to get the weight matrix W fusion , W fusion and F t-1,fusion Perform matrix multiplication to obtain fine-tuning feature F final , which is the output of the cross-frame attention module.

Images