Vehicle attitude estimation method fusing image and point cloud data

Abstract

The invention discloses a vehicle attitude estimation method fusing an image and point cloud data, and mainly solves the problems that an existing attitude estimation method is low in accuracy, a visual attitude estimation method is fuzzy in scale, and a point cloud attitude estimation method is difficult to match corresponding points. The implementation scheme is as follows: 1) acquiring a point cloud and an image data set; 2) constructing a pose estimation model; 3) constructing a model loss function; 4) training a pose estimation model; and 5) reasoning and obtaining an attitude transformation result. According to the pose estimation model constructed by the invention, full fusion of multi-modal features is realized through image and point cloud feature multi-level channel exchange, feature matching alignment, a cross-modal attention mechanism and a cross-frame attention mechanism, the accuracy of vehicle pose estimation is effectively improved, the problem of misalignment of multi-modal heterogeneous data information is solved, and the accuracy of vehicle pose estimation is improved. The practical application value is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of real-time mapping and positioning, and in particular relates to a vehicle posture estimation method that fuses image and point cloud data. Background Art

[0002] With the development of unmanned driving technology, real-time mapping and localization has become one of the core technologies of autonomous driving systems. Accurate vehicle pose estimation is crucial in path planning, obstacle detection and control decision-making. Especially in complex dynamic environments, accurate pose information is the basis for the stable operation of real-time mapping and localization systems. Traditional pose estimation methods usually rely on a single sensor, such as an inertial measurement unit (IMU) or a monocular camera, but these methods have limitations in complex environments, especially in scenes with low texture or large illumination changes. In recent years, the fusion technology of laser radar (LiDAR) and camera has become an effective means to improve the accuracy and robustness of pose estimation. LiDAR provides high-precision spatial information, while the camera provides rich visual information. The fusion of the two can improve the estimation accuracy. How to efficiently align the features of these two heterogeneous sensors and achieve high-precision pose estimation based on this is still a technical challenge. In the fusion methods of images and point clouds, existing research mainly focuses on fusion at the data level or result level. Fusion at the data level is limited by the heterogeneity of multi-sensor data, and often requires data from different sensors to be converted into a unified space or format. This process inevitably discards a lot of useful information. The fusion at the result level is usually to fuse the results from different sources through weighted averaging and other methods after performing attitude estimation separately. This method can easily lead to unstable final results, especially in the presence of large noise or errors. The fusion results may be greatly affected, thereby reducing the estimation accuracy and robustness. In addition, the existing vehicle attitude estimation algorithms usually adopt a single model structure and only estimate the attitude transformation results in the output stage, which makes it impossible to perform more refined optimization of the model during the training stage, thus affecting the accuracy of the attitude estimation results. Summary of the invention

[0003] The present invention fully considers the shortcomings of the existing methods, and its purpose is to provide a vehicle posture estimation method that integrates image and point cloud data. Through channel exchange, the fusion of image and point cloud features at multiple levels is realized. Through the matching and alignment of image and point cloud features and cross-modal and cross-frame attention mechanisms, the multi-modal common feature representation capability is enhanced and the posture estimation results are fine-tuned, thereby improving the accuracy of posture estimation.

[0004] 1. Technical Principle

[0005] At present, unsupervised vehicle attitude estimation models based on learning usually use a single sensor input, which has certain limitations. For example, visual attitude estimation models are susceptible to the influence of depth information ambiguity and dynamic object interference, while point cloud attitude estimation models are limited by sparsity and high computational overhead. In order to fully combine the advantages of the two types of sensor data, the present invention proposes a vehicle attitude estimation method that fuses image and point cloud data, and uses an unsupervised learning method to reduce the learning cost. Existing image and point cloud data fusion methods mainly focus on fusion at the data level or result level. The former is prone to information loss, and the latter leads to unstable results when the noise is large. To this end, the present invention designs a multimodal feature extraction and fusion module, which uses homogeneous and heterogeneous feature encoders and feature channel exchange to efficiently fuse image and point cloud features at multiple levels, thereby enhancing the model's overall understanding of the scene. Due to the differences in the organization form and field of view of camera and lidar data, the model learning process is unstable, which affects the accuracy of attitude estimation. To this end, the present invention uses a feature alignment and pose refinement module, combined with point alignment and cross-modal attention mechanism, to enhance the consistency of feature representation and further improve the accuracy of attitude estimation.

[0006] 2. According to the above principle, the present invention is implemented by the following scheme:

[0007] A vehicle posture estimation method fusing image and point cloud data comprises the following steps:

[0008] (1) Obtaining point cloud and image datasets; collecting images and point cloud datasets collected by the unmanned vehicle during a continuous driving process as a training set. The dataset contains N frames of image data {I1, I2, …, I N}, point cloud data {PC1, PC2, ..., PC N} and the corresponding calibration parameter set, the sampling frequency of the image and point cloud data is the same, and I t and PC t ,{t=1,2,…,N} have the same acquisition timestamp, the image data is collected by a single camera, and the point cloud data is collected by a single rotating lidar; the same method is used to collect the image and point cloud data sets of the corresponding unmanned vehicle during a continuous driving process as the test set.

[0009] (2) Constructing a pose estimation model; the model consists of a data preprocessing module, a depth estimation module, a multimodal feature extraction and fusion module, and a feature alignment and pose refinement module. The specific construction process includes the following steps:

[0010] (2-a) Construct a data preprocessing module to calculate the PC of two consecutive point clouds at t-1 and t t-1 and PC tAs input, the PC t-1 and PC t Generate point cloud projection V through cylindrical projection and coordinate transformation respectively t-1 and V t , where the coordinates are transformed from the point cloud coordinate system to the camera coordinate system.

[0011] (2-b) Construct a depth estimation module based on two consecutive frames of image I at t-1 and t t-1 and I t As input, resize the image to H×W and get the resized image and Processing using the depth estimation neural network Monodepth2 and The relative depth map D is obtained respectively t-1 and D t ; Where H and W are positive integers, and H∈[1,10000], W∈[1,10000].

[0012] (2-c) Construct a multimodal feature extraction and fusion module, which consists of a homogeneous feature fusion encoder, a heterogeneous feature fusion encoder, a radar encoder, an image encoder, and an initial regressor; the adjusted image obtained in step (2-b) is and Splice in the channel dimension and convert the relative depth map D obtained in step (2-b) t-1 and D t The splicing is performed in the channel dimension, and the spliced ​​results are input into the homogeneous feature fusion encoder to obtain the first-stage image features E with the same shape. img,1 and the deep one-stage feature E depth,1 ; Project the point cloud V obtained in step (2-a) t-1 and V t The concatenation is performed in the channel dimension, and then the radar encoder with ResNet-18 as the feature extractor is input. The output of the radar encoder is pooled into the same shape as E using an adaptive average pooling operation. depth,1 Consistent, we get the first-stage feature E of the point cloud lidar,1 ; E depth,1 and E lidar,1 Input heterogeneous features to fusion encoder to obtain deep two-stage features E depth,2 And point cloud two-stage feature E lidar,2 ; E img,1 Input the image encoder composed of two residual convolution blocks to obtain the image two-stage feature E img,2 ; E img,2 、E depth,2 and E lidar,2Use the initial regressor to estimate their corresponding posture estimation results respectively, and perform weighted summation on these results to obtain the initial posture transformation matrix The initial regressor consists of three fully connected layers.

[0013] The homogeneous feature fusion encoder and heterogeneous feature fusion encoder in this step are constructed as follows:

[0014] (2-c-1) Construct a homogeneous feature fusion encoder; the encoder consists of P feature fusion modules with the same structure, where P is a positive integer and P∈[1,10000]; the spliced ​​image and the spliced ​​relative depth Figure 1 Input P feature fusion modules in sequence to obtain the first-stage image feature E img,1 and the deep one-stage feature E depth,1 , which is the output of the homogeneous feature fusion encoder.

[0015] (2-c-2) Construct a heterogeneous feature fusion encoder; the encoder consists of Q feature fusion modules with the same structure, where Q is a positive integer and Q∈[1,10000], and the deep one-stage feature E depth,1 And the point cloud first stage feature E lidar,1 Input Q feature fusion modules together to obtain the deep two-stage feature E depth,2 And point cloud two-stage feature E lidar,2 , which is the output of the heterogeneous feature fusion encoder.

[0016] The fusion module constructed in this step (2-c) is composed of convolution layer 1, activation layer, convolution layer 2, batch normalization layer 1 and batch normalization layer 2; the two inputs of the module are sequentially passed through convolution layer 1, activation layer and convolution layer 2 to obtain features corresponding to the two inputs respectively, and the two features are processed by batch normalization layers 1 and 2 respectively and then channel exchange is performed to obtain exchanged features 1 and exchanged features 2, which are the outputs of the feature fusion module.

[0017] (2-d) Construct a feature alignment and posture refinement module, and transform the image two-stage feature E obtained in step (2-c) img,2 And point cloud two-stage feature E lidar,2 After being flattened, they are input into the fully connected layer to reduce the feature dimension to M, and the and Where M is a positive integer, and M∈[1,10000];

[0018] The adjusted image And point cloud projection map V t-1 The obtained product in step (II-3) Perform the distortion operation to obtain the distorted images And point cloud projection Input them together into the matching point alignment module 1 to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t-1,img ,Will and Concatenate in the channel dimension to get F t-1,lidar ; F t-1,img and F t-1,lidar As the image feature input and point cloud feature input of the cross-modal attention module 1, the fusion feature F is obtained. t-1,fusion ;

[0019] The adjusted image And point cloud projection map V t Input the matching point alignment module 2 together to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t,img ,Will and Concatenate in the channel dimension to get F t,lidar ; F t,img and F t,lidar As the image feature input and point cloud feature input of the cross-modal attention module 2, the fusion feature F is obtained. t,fusion ;

[0020] F t-1,fusion and F t,fusion Input the cross-frame attention module to obtain the fine-tuned feature F final ; F final Input the fine-tuning regressor to get the posture fine-tuning matrix ΔT, and use ΔT to transform the initial posture matrix Perform posture fine-tuning to obtain the final posture transformation result The calculation formula is:

[0021]

[0022] The fine-tuning regressor consists of two fully connected layers and one regression layer; This is the output of the pose estimation model.

[0023] The matching point alignment module 1 and the matching point alignment module 2 in this step have the same structure and are constructed as follows:

[0024] (2-d-1) Construct a matching point alignment module; calculate the position of each 3D point in the point cloud projection image in the pixel coordinate system of the adjusted image, extract the RGB value of the position in the adjusted image as the image matching point corresponding to the 3D point, and randomly sample all valid matching point pairs to obtain M pairs of matching image points and 3D points, which are the output of the matching point alignment module.

[0025] The cross-modal attention module 1 and the cross-modal attention module 2 in this step have the same structure and are constructed as follows:

[0026] (2-d-2) Construct a cross-modal attention module; calculate image features F img and point cloud features F lidar The similarity matrix S is calculated as follows:

[0027]

[0028] in Indicates F img The transpose of Represents matrix multiplication; the softmax function is used on the similarity matrix S to obtain the weighted matrix W lidar , W lidar Multiply the matrix with the point cloud features to get the point cloud weighted features; transpose the similarity matrix S and then use the softmax function to get the weighted matrix W img , W img The image weighted features are obtained by matrix multiplication with the image features; the point cloud weighted features and the image weighted features are concatenated in the channel dimension to obtain the fused features, which are the output of the cross-modal attention module.

[0029] The cross-frame attention module in this step is constructed as follows:

[0030] (2-d-3) Construct a cross-frame attention module to calculate the fusion feature F of the t-1th frame t-1,fusion And the fusion feature F of the tth frame t,fusion The similarity matrix S fusion , the calculation process is:

[0031]

[0032] in F t,fusion The transpose of Represents matrix multiplication; for the similarity matrix S fusion Use the softmax function to get the weight matrix W fusion , W fusion and F t-1,fusion Perform matrix multiplication to obtain the fine-tuning feature F final, which is the output of the cross-frame attention module.

[0033] (3) Constructing the model loss function; the specific construction process includes the following steps:

[0034] (3-a) Construct image photometric reconstruction loss L img :

[0035]

[0036] where ||·||1 is the L-1 regularization constraint, is the adjusted image of the tth frame, for The t-th frame pseudo image is obtained after photometric reconstruction (see Zhou, T., Brown, M., Snavely, N., & Lowe, DG (2017). Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1851-1858).).

[0037] (3-b) Construct point cloud reconstruction loss L LiDAR :

[0038]

[0039] where ||·||2 represents L-2 regularization, p t-1,k Represents the point cloud projection image V t-1 3D points in Indicates that the point cloud projection image V t Point p in t,k The 3D point obtained after the distortion operation, N represents p t-1,k and The number of valid matching pairs in .

[0040] (3-c) Construct feature fusion loss L CE :

[0041]

[0042] where γ m,i,l represents the offset of the i-th exchangeable channel in the output of the m-th batch normalization layer in the l-th feature fusion module, where l, m, i are all positive integers and l∈[1,10000], m∈[1,2], i∈[1,10000].

[0043] (3-d) Construct the total loss function L total :

[0044] L total =L img +L LiDAR +L CE

[0045] (4) training the pose estimation model; using the training set obtained in step (1) to train the pose estimation model constructed in step (2); using the loss function L constructed in step (3) total Calculate the model loss; during the training process, use the Adam algorithm to update the model parameters and use L-2 regularization as a constraint until the loss value no longer decreases or reaches the maximum number of training times to obtain a trained pose estimation model.

[0046] (5) Input the test set data obtained in step (1) into the trained pose estimation model. The output of the model is This is the final result of the posture transformation.

[0047] In step (2-b), H and W are preferably 192 and 640, respectively.

[0048] In step (2-c-1), P is preferably 4.

[0049] In step (2-c-2), Q is preferably 2.

[0050] In step (2-d), M is preferably 4096.

[0051] In step (3-c), l is preferably 6, m is preferably 2, and i is preferably 512.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] (1) The self-supervised image and point cloud fusion pose estimation framework constructed by the present invention can efficiently process data from multiple sensors, effectively avoid the pose estimation failure problem caused by the limitations of single sensor data, and achieve accurate pose estimation. At the same time, the framework uses self-supervised learning, does not require expensive labels, and reduces learning costs.

[0054] (2) The multimodal feature extraction and fusion module constructed by the present invention adaptively fuses features from different sensors through a channel exchange strategy without significantly increasing the complexity of the model, thereby improving the robustness of the model while ensuring computational efficiency.

[0055] (3) The feature alignment and posture refinement module constructed by the present invention constructs matching image points and 3D points through a matching point alignment strategy, and uses effective matching point pairs to guide feature alignment, so that the features of the image and point cloud focus on the same area or target in the environment, thereby improving the consistency of feature representation.

[0056] (4) The feature alignment and posture refinement module constructed by the present invention realizes the mutual weighting of multimodal features within a frame and the weighting of features between frames through a cross-modal attention module and a cross-frame attention module, respectively, so that the model can autonomously learn the same scene information from different sensors and different frame data, thereby improving the scene representation ability of the feature and further realizing the fine-tuning of the initial posture transformation matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Flow chart of a vehicle posture estimation method that fuses image and point cloud data according to an embodiment of the present invention;

[0058] Figure 2 A structural diagram of a pose estimation model constructed in an embodiment of the present invention;

[0059] Figure 3 A structural diagram of a homogeneous feature fusion encoder in the overall model of an embodiment of the present invention;

[0060] Figure 4 A structural diagram of a heterogeneous feature fusion encoder in the overall model of an embodiment of the present invention;

[0061] Figure 5 A structural diagram of a feature fusion module in the overall model of an embodiment of the present invention;

[0062] Figure 6 A structural diagram of the feature alignment and posture refinement module in the overall model of an embodiment of the present invention;

[0063] Figure 7 A structural diagram of a matching point alignment module in the overall model of an embodiment of the present invention;

[0064] Figure 8 The structure diagram of the cross-modal attention module in the overall model of the embodiment of the present invention;

[0065] Fig. 9 A structural diagram of the cross-frame attention module in the overall model of an embodiment of the present invention;

[0066] Fig.10 A comparison chart of the results of the posture estimation method according to the embodiment of the present invention and other methods on sequence 9 of the KITTIOdometry dataset;

[0067] Fig.11A comparison chart of the results of the posture estimation method according to an embodiment of the present invention and other methods on sequence 10 of the KITTIOdometry dataset. DETAILED DESCRIPTION

[0068] The specific implementation mode of the present invention is described below:

[0069] Example 1

[0070] Figure 1 The flowchart of the vehicle posture estimation method of fusing image and point cloud data implemented by the present invention is shown, and the specific steps are as follows:

[0071] Step 1, obtain point cloud and image datasets:

[0072] The image and point cloud dataset collected by the unmanned vehicle during a continuous driving process is collected as a training set. The dataset contains N frames of image data {I1, I2, …, I N}, point cloud data {PC1, PC2, ..., PC N} and the corresponding calibration parameter set, the sampling frequency of the image and point cloud data is the same, and I t and PC t ,{t=1,2,…,N} have the same acquisition timestamp, the image data is collected by a single camera, and the point cloud data is collected by a single rotating lidar; the same method is used to collect the image and point cloud data sets of the corresponding unmanned vehicle during a continuous driving process as the test set.

[0073] Step 2: The model consists of a data preprocessing module, a depth estimation module, a multimodal feature extraction and fusion module, a feature alignment and posture refinement module. The specific construction process includes the following steps:

[0074] Figure 2 The figure shows a structural diagram of a pose estimation model constructed in an embodiment of the present invention, and the specific steps are as follows:

[0075] (2-a) Constructing data preprocessing module

[0076] Take two consecutive frames of point cloud PC at t-1 and t t-1 and PC t As input, the PC t-1 and PC t Generate point cloud projection V through cylindrical projection and coordinate transformation respectively t-1 and V t , where the coordinates are transformed from the point cloud coordinate system to the camera coordinate system.

[0077] (2-b) Constructing the depth estimation module

[0078] Take two consecutive frames of image I at t-1 and t t-1 and I t As input, resize the image to H×W and get the resized image and Processing using the depth estimation neural network Monodepth2 and The relative depth map D is obtained respectively t-1 and D t , where H and W are 192 and 640 respectively.

[0079] (2-c) Constructing multimodal feature extraction and fusion module

[0080] This module consists of a homogeneous feature fusion encoder, a heterogeneous feature fusion encoder, a radar encoder, an image encoder, and an initial regressor; the adjusted image obtained in step (2-b) is and Splice in the channel dimension and convert the relative depth map D obtained in step (2-b) t-1 and D t The splicing is performed in the channel dimension, and the spliced ​​results are input into the homogeneous feature fusion encoder to obtain the first-stage image features E with the same shape. img,1 and the deep one-stage feature E depth,1 ; Project the point cloud V obtained in step (2-a) t-1 and V t The concatenation is performed in the channel dimension, and then the radar encoder with ResNet-18 as the feature extractor is input. The output of the radar encoder is pooled into the same shape as E using an adaptive average pooling operation. depth,1 Consistent, we get the first-stage feature E of the point cloud lidar,1 ; E depth,1 and E lidar,1 Input heterogeneous features to fusion encoder to obtain deep two-stage features E depth,2 And point cloud two-stage feature E lidar,2 ; E img,1 Input the image encoder composed of two residual convolution blocks to obtain the image two-stage feature E img,2 ; E img,2 、E depth,2 and E lidar,2 Use the initial regressor to estimate their corresponding posture estimation results respectively, and perform weighted summation on these results to obtain the initial posture transformation matrix The initial regressor consists of three fully connected layers.

[0081] The homogeneous feature fusion encoder and heterogeneous feature fusion encoder in this step are constructed as follows:

[0082] (2-c-1) Constructing a homogeneous feature fusion encoder

[0083] Figure 3 The figure shows the structure of the homogeneous feature fusion encoder in the pose estimation model according to the embodiment of the present invention; the encoder is composed of four feature fusion modules with the same structure; the spliced ​​image and the spliced ​​relative depth are Figure 1 Then, the four feature fusion modules are input in sequence to obtain the first-stage feature E of the image. img,1 and the deep one-stage feature E depth,1 , which is the output of the homogeneous feature fusion encoder.

[0084] (2-c-2) Constructing a heterogeneous feature fusion encoder

[0085] Figure 4 The structure diagram of the heterogeneous feature fusion encoder in the pose estimation model according to the embodiment of the present invention is shown. The encoder is composed of two feature fusion modules with the same structure. depth,1 And the point cloud first stage feature E lidar,1 Input two feature fusion modules together to obtain the deep two-stage feature E depth,2 And point cloud two-stage feature E lidar,2 , which is the output of the heterogeneous feature fusion encoder.

[0086] The feature fusion module constructed in this step (2-c) consists of convolution layer 1, activation layer, convolution layer 2, batch normalization layer 1 and batch normalization layer 2.

[0087] Figure 5 The figure shows the structure of the feature fusion module in the feature fusion encoder of the embodiment of the present invention; the two inputs of the module are sequentially passed through the convolution layer 1, the activation layer, and the convolution layer 2 to obtain the features corresponding to the two inputs respectively, and the two features are processed by the batch normalization layer 1 and 2 respectively and then the channels are exchanged to obtain the exchanged feature 1 and the exchanged feature 2, which are the outputs of the feature fusion module.

[0088] (2-d) Construct feature alignment and pose refinement module

[0089] Figure 6 The figure shows the structure of the feature alignment and posture refinement module in the pose estimation model in the embodiment of the present invention. The image two-stage feature E obtained in step (2-c) is img,2 And point cloud two-stage feature E lidar,2 After being flattened, they are input into the fully connected layer to reduce the feature dimension to M, and the and Where M=4096.

[0090] The adjusted image And point cloud projection map Vt-1 The obtained product in step (II-3) Perform the distortion operation to obtain the distorted images And point cloud projection Input them together into the matching point alignment module 1 to obtain 4096 pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t-1,img ,Will and Concatenate in the channel dimension to get F t-1,lidar ; F t-1,img and F t-1,lidar As the image feature input and point cloud feature input of the cross-modal attention module 1, the fusion feature F is obtained. t-1,fusion .

[0091] The adjusted image And point cloud projection map V t Input the matching point alignment module 2 together to obtain 4096 pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t,img ,Will and Concatenate in the channel dimension to get F t,lidar ; F t,img and F t,lidar As the image feature input and point cloud feature input of the cross-modal attention module 2, the fusion feature F is obtained. t,fusion .

[0092] F t-1,fusion and F t,fusion Input the cross-frame attention module to obtain the fine-tuned feature F final ; F final Input the fine-tuning regressor to get the posture fine-tuning matrix ΔT, and use ΔT to transform the initial posture matrix Perform posture fine-tuning to obtain the final posture transformation result The calculation formula is:

[0093]

[0094] The fine-tuning regressor consists of two fully connected layers and one regression layer; This is the output of the pose estimation model.

[0095] The matching point alignment module 1 and the matching point alignment module 2 in this step have the same structure and are constructed as follows:

[0096] (2-d-1) Construct matching point alignment module

[0097] Figure 7 The figure shows the structure diagram of the matching point alignment module in the feature alignment and posture refinement module in the implementation mode of the present invention; the position of each 3D point in the point cloud projection image in the pixel coordinate system of the adjusted image is calculated, and the RGB value of the position in the adjusted image is extracted as the image matching point corresponding to the 3D point. All valid matching point pairs are randomly sampled to obtain 4096 pairs of matching image points and 3D points, which are the output of the matching point alignment module.

[0098] The cross-modal attention module 1 and the cross-modal attention module 2 in this step have the same structure and are constructed as follows:

[0099] (2-d-2) Constructing a cross-modal attention module

[0100] Figure 8 The cross-modal attention module structure diagram in the feature alignment and posture refinement module of the embodiment of the present invention is shown as follows: Calculate the image feature F img and point cloud features F lidar The similarity matrix S is calculated as follows:

[0101]

[0102] in Indicates F img The transpose of Represents matrix multiplication; the softmax function is used on the similarity matrix S to obtain the weighted matrix W lidar , W lidar Multiply the matrix with the point cloud features to get the point cloud weighted features; transpose the similarity matrix S and then use the softmax function to get the weighted matrix W img , W img The image weighted features are obtained by matrix multiplication with the image features; the point cloud weighted features and the image weighted features are concatenated in the channel dimension to obtain the fused features, which are the output of the cross-modal attention module.

[0103] The cross-frame attention module in this step is constructed as follows:

[0104] (2-d-3) Constructing a cross-frame attention module

[0105] Fig. 9 The cross-frame attention module structure diagram in the feature alignment and posture refinement module of the embodiment of the present invention is shown, which calculates the fusion feature F of the t-1th frame. t-1,fusion And the fusion feature F of the tth frame t,fusion The similarity matrix S fusion, the calculation process is:

[0106]

[0107] in F t,fusion The transpose of Represents matrix multiplication; for the similarity matrix S fusion Use the softmax function to get the weight matrix W fusion , W fusion and F t-1,fusion Perform matrix multiplication to obtain the fine-tuning feature F final , which is the output of the cross-frame attention module.

[0108] Step 3: construct the model loss function. The specific construction process includes the following steps:

[0109] (3-a) Construct image photometric reconstruction loss L img :

[0110]

[0111] where ||·||1 is the L-1 regularization constraint, is the adjusted image of the tth frame, for The t-th frame pseudo image is obtained after photometric reconstruction.

[0112] (3-b) Construct point cloud reconstruction loss L LiDAR :

[0113]

[0114] where ||·||2 represents L-2 regularization, p t-1,k Represented in the point cloud projection V t-1 3D points in Indicates that the point cloud projection image V t Point p in t,k The 3D point obtained after the distortion operation, N represents p t-1,k and The number of valid matching pairs in .

[0115] (3-c) Construct feature fusion loss L CE :

[0116]

[0117] where γ m,i,l denotes the offset of the i-th exchangeable channel in the output of the m-th batch normalization layer in the l-th feature fusion module, l=6, m=2, i=512.

[0118] (3-d) Construct the total loss function L total :

[0119] L total =L img +L LiDAR +L CE

[0120] Step 4: Train the pose estimation model:

[0121] Use the training set obtained in step 1 to train the pose estimation model constructed in step 2; use the loss function L constructed in step 3 total Calculate the model loss; during the training process, use the Adam algorithm to update the model parameters and use L-2 regularization as a constraint until the loss value no longer decreases or reaches the maximum number of training times to obtain a trained pose estimation model.

[0122] Step 5: Infer and obtain the pose transformation result:

[0123] The test set data obtained in step 1 is input into the trained pose estimation model. The output of the model is This is the final result of the posture transformation.

[0124] Example 2

[0125] The method in Example 1 was used to conduct a self-supervised multi-sensor fusion mileage estimation method experiment on the public dataset KITTIOdometry. The experimental operating system was Ubuntu 18.04, based on the PyTorch 1.8.0 framework of CUDA 11.3 and cuDNN 8.2.1, and trained and tested on a personal computer equipped with Intel (R) Xeon (R) W-2123 CPU (3.60 GHz) and NVIDIA GeForce RTX 3090 (24 GB) hardware.

[0126] This embodiment uses the translation error t err (%), rotation error r err The two indicators (deg / 100m) are used to compare the six odometer methods, Monodepth2, SC-SfMLearner, FeatDepth, Zhou et al., MLF-VO, and DeLORA, with the method of the present invention on the KITTIOdometry dataset. err (%) and r err (deg / 100m) is calculated in units of 100 meters, which respectively represent the average translation error of a trajectory length of 100 meters and the average rotation error of a subsequence of a trajectory length of 100 meters. The unit is angle, and the calculation formula is as follows:

[0127]

[0128] In the formula Respectively represent the true value of the translation vector and the true value of the rotation matrix of the i-th frame, denote the translation vector and rotation matrix of the i-th frame calculated by the odometer method, respectively. N denotes the number of frames in the 100-meter subsequence trajectory. ||·|| denotes the Euclidean norm. trace(·) denotes the matrix trace operation.

[0129] The comparison results are shown in Table 1. It can be found that compared with other methods, the present invention can obtain accurate odometer results on multiple sequences of the KITTIOdometry dataset. In all two sequences, 6 evaluation indicators reach the optimal level.

[0130] Fig.10 , Fig.11 The figure shows the comparison of the pose estimation method of the embodiment of the present invention and other methods on sequence 09 and sequence 10 of the KITTI Odometry dataset. The results show that the pose prediction results of the model designed by the present invention are closer to the true value results than the image-based odometer method (MLF-VO) and the point cloud data-based odometer method (DeLORA); the present invention has a smaller error than the pose estimation method based on a single sensor in the visualization results of the entire sequence; the present invention can achieve better results in both the sequence with rich scene features (sequence 09) and the sequence with sparse scene features (sequence 10).

[0131] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made based on the structure and principle of the present invention should be included in the protection scope of the present invention.

[0132] Table 1

[0133]

Claims

1. A vehicle posture estimation method integrating image and point cloud data, characterized in that: The following steps are involved: (I) Obtain point cloud and image datasets: The image and point cloud dataset collected by the unmanned vehicle during a continuous driving process is collected as a training set. The dataset contains N frames of image data {I1, I2, …, I N }, point cloud data {PC1, PC2, ..., PC N } and the corresponding calibration parameter set, the sampling frequency of the image and point cloud data is the same, and I t and PC t ,{t=1,2,…,N} have the same acquisition timestamp, the image data is collected by a single camera, and the point cloud data is collected by a single rotating laser radar; the image and point cloud data sets collected by the corresponding unmanned vehicle during a continuous driving process are used as the test set in the same way; (II) Constructing a pose estimation model: The model consists of a data preprocessing module, a depth estimation module, a multimodal feature extraction and fusion module, a feature alignment and posture refinement module. The specific construction process includes the following steps: (II-1) Constructing data preprocessing module: Take two consecutive frames of point cloud PC at t-1 and t t-1 and PC t As input, the PC t-1 and PC t Generate point cloud projection V through cylindrical projection and coordinate transformation respectively t-1 and V t , where the coordinates are transformed from the point cloud coordinate system to the camera coordinate system; (II-2) Constructing the depth estimation module: Take two consecutive frames of image I at t-1 and t t-1 and I t As input, resize the image to H×W and get the resized image and Processing using the depth estimation neural network Monodepth2 and The relative depth map D is obtained respectively t-1 and D t ; Where H and W are positive integers, and H∈[1,10000], W∈[1,10000]; (II-3) Construct multimodal feature extraction and fusion module: This module consists of a homogeneous feature fusion encoder, a heterogeneous feature fusion encoder, a radar encoder, an image encoder, and an initial regressor; the adjusted image I obtained in step (II-2) is t-1 and I t Splice in the channel dimension and convert the relative depth map D obtained in step (II-2) t-1 and D t The splicing is performed in the channel dimension, and the spliced ​​results are input into the homogeneous feature fusion encoder to obtain the first-stage image features E with the same shape. img,1 and the deep one-stage feature E depth,1 ; Project the point cloud V obtained in step (II-1) t-1 and V t The concatenation is performed in the channel dimension, and then the radar encoder with ResNet-18 as the feature extractor is input. The output of the radar encoder is pooled into the same shape as E using an adaptive average pooling operation. depth,1 Consistent, we get the first-stage feature E of the point cloud lidar,1 ; E depth,1 and E lidar,1 Input heterogeneous features to fusion encoder to obtain deep two-stage features E depth,2 And point cloud two-stage feature E lidar,2 ; E img,1 Input the image encoder composed of two residual convolution blocks to obtain the image two-stage feature E img,2 ; E img,2 、E depth,2 and E lidar,2 Use the initial regressor to estimate their corresponding posture estimation results respectively, and perform weighted summation on these results to obtain the initial posture transformation matrix The initial regressor consists of three fully connected layers; (II-4) Construct feature alignment and pose refinement module: The image two-stage feature E obtained in step (II-3) img,2 And point cloud two-stage feature E lidar,2 After being flattened, they are input into the fully connected layer to reduce the feature dimension to M, and the and Where M is a positive integer, and M∈[1,10000]; The adjusted image And point cloud projection map V t-1 The obtained product in step (II-3) Perform the distortion operation to obtain the distorted images And point cloud projection Input them together into the matching point alignment module 1 to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t-1,img ,Will and Concatenate in the channel dimension to get F t-1,lidar ; F t-1,img and F t-1,lidar As the image feature input and point cloud feature input of the cross-modal attention module 1, the fusion feature F is obtained. t-1,fusion ; The adjusted image And point cloud projection map V t Input the matching point alignment module 2 together to obtain M pairs of matching image points and 3D points Will and Concatenate in the channel dimension to get F t,img ,Will and Concatenate in the channel dimension to get F t,lidar ; F t,img and F t,lidar As the image feature input and point cloud feature input of the cross-modal attention module 2, the fusion feature F is obtained. t,fusion ; F t-1,fusion and F t,fusion Input the cross-frame attention module to obtain the fine-tuned feature F final ; F final Input the fine-tuning regressor to get the posture fine-tuning matrix ΔT, and use ΔT to transform the initial posture matrix Perform posture fine-tuning to obtain the final posture transformation result The calculation formula is: The fine-tuning regressor consists of two fully connected layers and one regression layer; This is the output of the pose estimation model; (III) Construct the model loss function. The specific construction process is as follows: (III-1) Constructing image photometric reconstruction loss L img : where ||·||1 is the L-1 regularization constraint, is the adjusted image of the tth frame, for The pseudo image of the tth frame obtained after photometric reconstruction; (III-2) Constructing point cloud reconstruction loss L LiDAR : where ||·||2 represents L-2 regularization, p t-1,k Represents the point cloud projection image V t-1 3D points in Indicates that the point cloud projection image V t Point p in t,k The 3D point obtained after the distortion operation, N represents p t-1,k and The number of valid matching pairs in ; (III-3) Constructing feature fusion loss L CE : where γ m,i,l represents the offset of the i-th exchangeable channel in the output of the m-th batch normalization layer in the l-th feature fusion module, where l, m, i are all positive integers and l∈[1,10000], m∈[1,2], i∈[1,10000]; (III-4) Constructing the total loss function L total : L total =L img +L LiDAR +L CE (IV) Training pose estimation model: Use the training set obtained in step (I) to train the pose estimation model constructed in step (II); use the loss function L constructed in step (III) total Calculate the model loss; during the training process, use the Adam algorithm to update the model parameters and use L-2 regularization as a constraint until the loss value no longer decreases or reaches the maximum number of training times, and then obtain a trained pose estimation model; (V) Reasoning and obtaining posture transformation results: The test set data obtained in step (I) is input into the trained pose estimation model, and the output of the model is This is the final result of the posture transformation.

2. A vehicle posture estimation method for fusing image and point cloud data as claimed in claim 1, characterized in that: The homogeneous feature fusion encoder in step (II-3) is composed of P feature fusion modules with the same structure, where P is a positive integer and P∈[1,10000]; the spliced ​​image and the spliced ​​relative depth map are sequentially input into the P feature fusion modules to obtain the first-stage feature E of the image img,1 and the deep one-stage feature E depth,1 , which is the output of the homogeneous feature fusion encoder.

3. The vehicle posture estimation method of fusing image and point cloud data as claimed in claim 1, characterized in that: The heterogeneous feature fusion encoder in step (II-3) is composed of Q feature fusion modules with the same structure, where Q is a positive integer and Q∈[1,10000], and the deep one-stage feature E depth,1 And the point cloud first stage feature E lidar,1 Input Q feature fusion modules together to obtain the deep two-stage feature E depth,2 And point cloud two-stage feature E lidar,2 , which is the output of the heterogeneous feature fusion encoder.

4. The vehicle posture estimation method of fusing image and point cloud data as claimed in claim 1, characterized in that: The feature fusion module is composed of convolution layer 1, activation layer, convolution layer 2, batch normalization layer 1 and batch normalization layer 2; the two inputs of the module are sequentially processed by convolution layer 1, activation layer and convolution layer 2 to obtain features corresponding to the two inputs respectively, and the two features are processed by batch normalization layers 1 and 2 respectively and then channel exchange is performed to obtain exchanged features 1 and exchanged features 2, which are the outputs of the feature fusion module.

5. The vehicle posture estimation method of fusing image and point cloud data as claimed in claim 1, characterized in that: The matching point alignment module 1 and the matching point alignment module 2 in step (II-4) have the same structure and are constructed as follows: Calculate the position of each 3D point in the point cloud projection image in the pixel coordinate system of the adjusted image, extract the RGB value of the position in the adjusted image as the image matching point corresponding to the 3D point, randomly sample all valid matching point pairs, and obtain M pairs of matching image points and 3D points, which are the output of the matching point alignment module.

6. The vehicle posture estimation method of fusing image and point cloud data as claimed in claim 1, characterized in that: The cross-modal attention module 1 and the cross-modal attention module 2 in step (II-4) have the same structure and are constructed as follows: Calculate image features F img and point cloud features F lidar The similarity matrix S is calculated as follows: in Indicates F img The transpose of Represents matrix multiplication; Use the softmax function on the similarity matrix S to get the weighted matrix W lidar , W lidar Perform matrix multiplication with point cloud features to obtain point cloud weighted features; After transposing the similarity matrix S, the softmax function is used to obtain the weighted matrix W. img , W img The image weighted features are obtained by matrix multiplication with the image features; the point cloud weighted features and the image weighted features are concatenated in the channel dimension to obtain the fused features, which are the output of the cross-modal attention module.

7. The vehicle posture estimation method of fusing image and point cloud data as claimed in claim 1, characterized in that: The cross-frame attention module in step (II-4) is constructed as follows: Calculate the fusion feature F of the t-1th frame t-1,fusion And the fusion feature F of the tth frame t,fusion The similarity matrix S fusion , the calculation process is: in F t,fusion The transpose of Represents matrix multiplication; for the similarity matrix S fusion Use the softmax function to get the weight matrix W fusion , W fusion and F t-1,fusion Perform matrix multiplication to obtain fine-tuning feature F final , which is the output of the cross-frame attention module.

Images