A robust camera-lidar multimodal contrastive pre-training method and system against spatio-temporal bias

Abstract

The application discloses a kind of robust camera-lidar multimodal contrast pre-training method and system of anti spacetime bias.The method comprises: generating distance projection map to lidar point cloud;Multi-scale representation is extracted using image encoder and distance projection map encoder respectively;The hierarchical alignment of global vector and regional level block vector is completed by semantic perception branch and geometric perception branch;Space consistency constraint and sparsity regularization are introduced to suppress the representation collapse of low information area;Multi-scale attention aggregation is used to form the final cross-modal representation and is trained with bidirectional contrast loss, and finally a general camera-lidar data representation with robust migration capability for downstream three-dimensional perception tasks is obtained.The method of the application can more robustly provide fused features for downstream three-dimensional perception tasks, and the features can be extended to various severe weather scenarios and road conditions, improving the reliability of autonomous driving in severe scenarios.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving multi-sensor fusion, cross-modal representation learning and self-supervised comparative learning technology, and in particular, a robust camera-LiDAR multi-modal comparative pre-training method and system resistant to spatiotemporal bias. Background Technology

[0002] With the rapid development of autonomous driving and intelligent connected vehicle technologies, safe driving in complex road environments increasingly relies on highly reliable environmental perception capabilities. Cameras and LiDAR, as mainstream vehicle sensors, are widely used in tasks such as 3D target detection and tracking. Camera images provide rich color, texture, and semantic cues, facilitating the identification of traffic participant categories and road features; LiDAR provides accurate distance measurements and 3D geometry, exhibiting strong robustness to changes in illumination. How to fully utilize the complementary advantages of these two modalities to learn stable and consistent cross-modal representations and improve downstream perception performance has become an important research direction in the field of multimodal perception for autonomous driving.

[0003] Existing camera-LiDAR fusion methods typically rely on the assumption of "accurate cross-modal spatiotemporal registration," meaning that the camera and LiDAR have reliable extrinsic parameter calibration and strict time synchronization, allowing point clouds to be accurately projected onto a shared bird's-eye view space. Under this premise, early fusion methods can achieve fine-grained interaction through pixel-to-point correspondence, mid-stage fusion methods can perform feature alignment and aggregation within a unified space, and late-stage fusion methods can complement information at the decision level. However, in real-world road scenarios, differences in sensor sampling frequencies, hardware delays, and buffer transmission delays can cause time asynchrony; bumps, vibrations, and assembly deviations during vehicle movement can cause extrinsic parameter drift and attitude disturbances; coupled with the effects of dynamic targets and occlusion, cross-modal data inevitably suffers from spatiotemporal bias, making the aforementioned "accurate registration" assumption difficult to hold in the long term. This leads to semantic mismatch and degradation of fused features, resulting in a significant decline in downstream perception performance.

[0004] To reduce the geometric complexity of cross-modal alignment, a technique has emerged in recent years that projects 3D LiDAR point clouds into 2D range images. This approach can reuse mature 2D vision backbone networks, facilitating contrastive learning or feature interaction with the camera branch using similar coding structures. However, projecting point clouds to range images inevitably introduces many-to-one mappings, sparse holes, and nonlinear geometric distortions, especially pronounced in long-range regions. These projection errors and sparsity further amplify the differences between the camera and LiDAR, making cross-modal local alignment more unstable, and the training process more susceptible to noise in low-information regions, resulting in unreliable representations.

[0005] While existing research has yielded numerous studies on mid-range camera-LiDAR fusion, most still require precise estimation of registration relationships. In real-world scenarios, various external factors inevitably influence the perception fusion model, such as differences in sensor sampling frequencies, extrinsic parameter drift during vehicle movement, and attitude perturbations, ultimately leading to a decline in the performance of the perceptual fusion model. Summary of the Invention

[0006] The purpose of this invention is to address the problems existing in the prior art by providing a robust multimodal contrastive pre-training method and system for cameras and lidar that resists spatiotemporal bias. This invention improves cross-modal mutual retrieval capabilities and enhances the stability and robustness of downstream 3D perception tasks under spatiotemporal bias conditions by employing global and local hierarchical contrastive alignment, introducing a bidirectional cross-modal attention mechanism with geometric bias modulation, and combining regularization constraints and multi-scale aggregation strategies for long-range / holey / low-information regions.

[0007] The technical solution to achieve the purpose of this invention is: a robust camera-lidar multimodal contrastive pre-training method resistant to spatiotemporal bias, the method comprising the following steps:

[0008] Step 1: Acquire time-synchronized or near-synchronized camera images and LiDAR point clouds in the same scene, and generate a LiDAR distance image by distance projection of the LiDAR point clouds. Simultaneously acquire camera images ;

[0009] Step 2, using a dual-branch encoder to... and Multimodal feature encoding is performed to obtain the camera's global features. Camera tile features and global features of lidar LiDAR patch features ;

[0010] Step 3, for and Perform global contrastive alignment to learn scene-level cross-modal semantic consistency and obtain global contrastive loss. ;

[0011] Step 4, for and Perform tile-level contrast alignment to learn local cross-modal correspondences and obtain tile contrast loss. ;

[0012] Step 5: The patch-level enhancement features obtained in Step 4 are used to perform region aggregation using a learnable pooling function to obtain the region-level sequences of the image. Regional sequence with lidar Furthermore, a bidirectional cross-modal multi-head attention mechanism is constructed based on semantic-aware transformers and geometric-aware transformers, and a geometric bias matrix is ​​introduced. Attention weights are modulated to obtain region features of the image after cross-modal alignment. and the regional characteristics of lidar ;

[0013] Step 6: Introduce regularization constraints to suppress unstable representations in distant, hollow, or low-information regions, including sparsity regularization terms. Spatial consistency constraints ;

[0014] Step 7: Employ a multi-scale attention aggregation mechanism, using the initial global features as the query vector to... and The final global features of the image are obtained by aggregation. and the final global features of lidar The final projected representation of the image is obtained through a shared projection head. The final projection representation of the lidar ;

[0015] Step 8: Construct and minimize the total loss function to pre-train and optimize the model. The total loss function includes at least the global contrastive loss. Patch contrast loss sparse regularization term Spatial consistency constraints Compared with the final loss Thus, cross-modal representation embedding is obtained. D is the dimension of the embedding vector.

[0016] Furthermore, step 1 specifically includes:

[0017] Step 1-1: Acquire camera images ;

[0018] Steps 1-2: Generate a lidar range image by projecting the lidar point cloud over distance. Furthermore, the sparse depth observation value of the lidar projection at pixel p is defined as... Where H and W are the number of pixels in the vertical and horizontal directions of the image, respectively, and B is the batch size;

[0019] Steps 1-3, when pixel p has no valid echo, set ;

[0020] This is used to characterize the hollow region and suppress it in subsequent feature learning and regularization constraints;

[0021] Steps 1-4, convert the single channel Copy the data to a three-channel format or perform equivalent channel expansion before inputting it into the LiDAR encoder;

[0022] Steps 1-5, for and Perform timestamp pairing or near-synchronous pairing.

[0023] Furthermore, step 2 specifically includes:

[0024] Step 2-1, using a camera encoder right Perform tile division and encoding to satisfy:

[0025]

[0026] In the formula, D is the dimension of the embedding vector, and N is the number of partitioned graph tiles;

[0027] Step 2-2, using a lidar encoder right Perform tile division and encoding to satisfy:

[0028]

[0029] In the formula, ;

[0030] Steps 2-3, during the pre-training phase... and The parameters are jointly trained or fine-tuned to make the two modalities comparable in a shared comparison space.

[0031] Furthermore, both the camera encoder and the lidar encoder are vision transformer structures or equivalent self-attention-based encoder structures.

[0032] Furthermore, step 3 specifically includes:

[0033] Step 3-1: Project the global features of the i-th pair of matching samples onto the shared projection head P to obtain the features of the i-th pair of matching samples in the image. Features of the i-th pair of matching samples with LiDAR ,satisfy:

[0034]

[0035] Step 3-2: Construct a bidirectional contrastive learning objective based on matching sample pairs with a batch size of B, and use sample pairs with the same index within the batch as positive sample pairs and the remaining samples within the batch as negative sample pairs.

[0036] Step 3-3, Define the global contrast loss for:

[0037]

[0038] In the formula, This is a temperature coefficient used to adjust the sharpness of the similarity distribution; , These are the features of the j-th pair of matching samples in the image and the features of the j-th pair of matching samples in the LiDAR, respectively.

[0039] Furthermore, step 4 specifically includes:

[0040] Step 4-1: Organize the camera patch features and LiDAR patch features according to their spatial location and perform intra-batch comparison learning so that cross-modal patch features at the same location form positive sample pairs, and patch features from different samples form negative sample pairs.

[0041] Step 4-2: Calculate the intra-batch similarity matrix for the k-th position. :

[0042]

[0043] In the formula, These are the camera patch features and the LiDAR patch features at the k-th position, respectively;

[0044] Step 4-3, Define the patch contrast loss for:

[0045]

[0046] In the formula, This is the mutual information loss function.

[0047] Furthermore, step 5 specifically includes:

[0048] Step 5-1, using learnable pooling functions The aggregation yields a sequence of regions that satisfy:

[0049]

[0050]

[0051] In the formula, This indicates the number of representative regions defined under the attention mechanism;

[0052] Step 5-2, using a shared linear mapping The features of the two modal regions are aligned in distribution, and the query, key, and value of bidirectional cross-modal attention are constructed respectively:

[0053]

[0054]

[0055]

[0056] In the formula, These represent the query, key, and value of an image under the attention mechanism, respectively. Representing the query, key, and value of the LiDAR under the attention mechanism, respectively, and W is the attention matrix. For the region features of the image, represents the regional features of the lidar; D is the dimension of the embedding vector. The scale used to define the sampling distance of the lidar;

[0057] Step 5-3: Introduce the geometric bias matrix and calculate the bidirectional multi-head attention output:

[0058]

[0059]

[0060] In the formula, These represent the key and value, respectively, for querying image information at a distance h under the attention mechanism. These represent the key and value, respectively, for querying LiDAR information at a distance of h under the attention mechanism. , These represent the bidirectional multi-head attention outputs of the camera and the LiDAR, respectively.

[0061] Step 5-4: Concatenate attention heads and perform linear mapping to obtain cross-modal fusion output, and obtain cross-modal aligned region features through gated residual connections.

[0062]

[0063]

[0064] In the formula, These represent the attention aggregation matrices for the image and radar, respectively. For learnable scalar gating parameters, As a standardization layer for attention mechanisms, , These are the intermediate attention results for the image and the LiDAR, respectively; (*) indicates that the outputs of H attention heads are concatenated along the feature dimension.

[0065] Furthermore, step 6 specifically includes:

[0066] Step 6-1, distance to pixel p Define geometric weights :

[0067]

[0068] In the formula, Indicates the effective distance threshold. The temperature coefficient represents the degree of smoothness in controlling the weight decay. (*) indicates the Sigmoid activation function;

[0069] Step 6-2, Aggregate within a tile to form a block level. With the overall And let the feature of the lidar branch sample b in patch k be... Define sparse regularization terms :

[0070]

[0071] In the formula, Indicates the sample batch size;

[0072] Step 6-3, Define the within-batch mean With dispersion :

[0073]

[0074] Step 6-4, Define spatial consistency constraints :

[0075] .

[0076] Furthermore, step 7 specifically includes:

[0077] Step 7-1: Use a multi-scale attention aggregation mechanism to obtain the final global features of the camera.

[0078]

[0079] In the formula, These represent the query, key, and value, respectively.

[0080] Step 7-2: Use a multi-scale attention aggregation mechanism to obtain the final global features of the LiDAR:

[0081]

[0082] Step 7-3, via shared projection head The final projected representation is obtained:

[0083] .

[0084] Furthermore, step 8 specifically includes:

[0085] Step 8-1, Define the final stage bidirectional contrast loss :

[0086]

[0087] In the formula, , These represent the final projected representation of the image corresponding to the i-th patch and the final projected representation of the LiDAR, respectively; , These represent the final projected representation of the image corresponding to the j-th tile and the final projected representation of the lidar, respectively.

[0088] Step 8-2, Construct the total loss function :

[0089]

[0090] In the formula, They are respectively The weights;

[0091] Step 8-3: Iteratively update the pre-trained model parameters using gradient descent or its equivalent optimization algorithm to minimize... ;

[0092] Step 8-4: Use the synchronous or near-synchronous sample pairs obtained in Step 1 as positive sample sources and the remaining samples in the batch as negative sample sources to form a discriminative cross-modal attitude quantity space without additional manual annotation.

[0093] Step 8-5: After optimization, output the cross-modal representation embedding. .

[0094] Compared with the prior art, the significant advantages of this invention are:

[0095] 1) Mid-term fusion of multimodal data was achieved through global and local comparative learning methods without the need for camera-lidar extrinsic parameters.

[0096] 2) Robust interaction and alignment of cross-modal features are achieved through region aggregation and bidirectional cross-modal multi-head attention mechanism, which alleviates cross-modal mismatch caused by spatiotemporal bias.

[0097] 3) By using sparsity and spatial consistency constraints, unstable representations and intra-batch fluctuations caused by low-information regions of lidar are suppressed, thereby improving the convergence speed of the model.

[0098] 4) Through a multi-scale attention aggregation mechanism, the final features simultaneously contain scene semantics and local geometric details, which facilitates transfer to downstream tasks and improves their stability under spatiotemporal bias.

[0099] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0100] Figure 1 This is a flowchart of a robust camera-lidar multimodal comparative pre-training method to resist spatiotemporal bias in one embodiment.

[0101] Figure 2 This is a similarity matrix comparison diagram for a cross-modal retrieval task in one embodiment, wherein... Figure 2 In the diagram, (a) to (d) represent the visualization results of the baseline model ResNet, AECMLoc, Lip-Loc, and the cosine similarity matrix generated by this method, respectively; where the diagonal color is closer to green, the higher the confidence of the correct match, and the off-diagonal area color is closer to yellow, the lower the noise of the mismatch.

[0102] Figure 3 This is a comparison chart of cosine similarity distribution on a cross-modal retrieval task in one embodiment, wherein... Figure 3 In the figure, (a) to (d) represent the frequency distribution statistics of cosine similarity corresponding to ResNet, AECMLoc, Lip-Loc and the present method, respectively; the green bar represents the distribution of correctly matched samples, and the red area represents the distribution of incorrectly matched samples. The more obvious the separation between the two, the stronger the cross-modal alignment discriminative ability of the model. Detailed Implementation

[0103] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0104] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0105] In one embodiment, combined Figure 1 This paper presents a robust camera-LiDAR multimodal contrastive pre-training method to combat spatiotemporal bias. Addressing the representation bias problem caused by spatial sampling inconsistencies, temporal perturbations, and differences in regional information density between camera and LiDAR data in real-world autonomous driving scenarios, the method constructs a multimodal input encoding mechanism for a "distance projection map" and camera images. It also designs hierarchical contrastive alignment and cross-modal refinement modules to achieve consistent alignment from global semantics to local regions. The method includes the following steps:

[0106] Step 1: Acquire time-synchronized or near-synchronized camera images and LiDAR point clouds in the same scene, and generate a LiDAR distance image by distance projection of the LiDAR point clouds. Simultaneously acquire camera images ;

[0107] Here, the camera image can be a forward-view image; the lidar point cloud generates a distance image corresponding to the camera view through distance projection, so as to perform cross-modal alignment learning.

[0108] Step 2, using a dual-branch encoder to... and Multimodal feature encoding is performed to obtain the camera's global features. Camera tile features and global features of lidar LiDAR patch features ;

[0109] Step 3, for and Perform global contrastive alignment to learn scene-level cross-modal semantic consistency and obtain global contrastive loss. ;

[0110] Step 4, for and Perform tile-level contrast alignment to learn local cross-modal correspondences and obtain tile contrast loss. ;

[0111] Step 5: The patch-level enhancement features obtained in Step 4 are used to perform region aggregation using a learnable pooling function to obtain the region-level sequences of the image. Regional sequence with lidar Furthermore, a bidirectional cross-modal multi-head attention mechanism is constructed based on semantic-aware transformers and geometric-aware transformers, and a geometric bias matrix is ​​introduced. Attention weights are modulated to obtain region features of the image after cross-modal alignment. and the regional characteristics of lidar ;

[0112] Step 6: Introduce regularization constraints to suppress unstable representations in distant, hollow, or low-information regions, including sparsity regularization terms. Spatial consistency constraints ;

[0113] Step 7: Employ a multi-scale attention aggregation mechanism, using the initial global features as the query vector to... and The final global features of the image are obtained by aggregation. and the final global features of lidar The final projected representation of the image is obtained through a shared projection head. The final projection representation of the lidar ;

[0114] Step 8: Construct and minimize the total loss function to pre-train and optimize the model. The total loss function includes at least the global contrastive loss. Patch contrast loss sparse regularization term Spatial consistency constraints Compared with the final loss Thus, cross-modal representation embedding is obtained. D is the dimension of the embedding vector.

[0115] Furthermore, in one embodiment, step 1 specifically includes:

[0116] Step 1-1: Acquire camera images ;

[0117] Steps 1-2: Generate a lidar range image by projecting the lidar point cloud over distance. Furthermore, the sparse depth observation value of the lidar projection at pixel p is defined as... Where H and W are the number of pixels in the vertical and horizontal directions of the image, respectively, and B is the batch size;

[0118] Steps 1-3, when pixel p has no valid echo, set ;

[0119] This is used to characterize the hollow region and suppress it in subsequent feature learning and regularization constraints;

[0120] Steps 1-4, convert the single channel Copy the data to a three-channel format or perform equivalent channel expansion before inputting it into the LiDAR encoder;

[0121] Steps 1-5, for and Timestamp pairing or near-synchronous pairing can be performed to reduce positive sample noise caused by sampling timing deviation.

[0122] Furthermore, in one embodiment, step 2 specifically includes:

[0123] Step 2-1, using a camera encoder right Perform tile division and encoding to satisfy:

[0124]

[0125] In the formula, D is the dimension of the embedding vector, and N is the number of partitioned graph tiles;

[0126] Step 2-2, using a lidar encoder right Perform tile division and encoding to satisfy:

[0127]

[0128] In the formula, ;

[0129] Steps 2-3, during the pre-training phase... and The parameters are jointly trained or fine-tuned to make the two modalities comparable in a shared comparison space.

[0130] Preferably, in some embodiments, the camera encoder and the lidar encoder are both vision transformer structures or equivalent self-attention-based encoder structures.

[0131] Furthermore, in one embodiment, step 3 specifically includes:

[0132] Step 3-1: Project the global features of the i-th pair of matching samples onto the shared projection head P to obtain the features of the i-th pair of matching samples in the image. Features of the i-th pair of matching samples with LiDAR ,satisfy:

[0133]

[0134] Step 3-2: Construct a bidirectional contrastive learning objective based on matching sample pairs with a batch size of B, and use sample pairs with the same index within the batch as positive sample pairs and the remaining samples within the batch as negative sample pairs.

[0135] Step 3-3, Define the global contrast loss for:

[0136]

[0137] In the formula, This is a temperature coefficient used to adjust the sharpness of the similarity distribution; , These are the features of the j-th pair of matching samples in the image and the features of the j-th pair of matching samples in the LiDAR, respectively.

[0138] Here, we learn scene-level cross-modal semantic consistency through global contrastive alignment to improve cross-modal retrieval under conditions of illumination variation, weather variation, and sampling time sequence deviation.

[0139] Here, for and Perform tile-level contrast alignment to learn local cross-modal correspondences and obtain tile contrast loss. .

[0140] Furthermore, in one embodiment, step 4 specifically includes:

[0141] Step 4-1: Organize the camera patch features and LiDAR patch features according to their spatial location and perform intra-batch comparison learning so that cross-modal patch features at the same location form positive sample pairs, and patch features from different samples form negative sample pairs.

[0142] Step 4-2: Calculate the intra-batch similarity matrix for the k-th position. :

[0143]

[0144] In the formula, These are the camera patch features and the LiDAR patch features at the k-th position, respectively;

[0145] Step 4-3, Define the patch contrast loss for:

[0146]

[0147] In the formula, This is the mutual information loss function.

[0148] Here, through the analysis of and The bidirectional constraints enable the model to simultaneously optimize the local retrieval consistency from camera to lidar and from lidar to camera, thereby improving robustness to cross-modal biases caused by local misalignment, occlusion, and dynamic targets.

[0149] Here, tile-level alignment is used as the input basis for cross-modal refinement in step 5 to reduce the accumulation of region mismatches in the subsequent cross-modal attention learning process.

[0150] Furthermore, in one embodiment, step 5 specifically includes:

[0151] Step 5-1, using learnable pooling functions The aggregation yields a sequence of regions that satisfy:

[0152]

[0153]

[0154] In the formula, This indicates the number of representative regions defined under the attention mechanism;

[0155] Step 5-2, using a shared linear mapping The features of the two modal regions are aligned in distribution, and the query, key, and value of bidirectional cross-modal attention are constructed respectively:

[0156]

[0157]

[0158]

[0159] In the formula, These represent the query, key, and value of an image under the attention mechanism, respectively. Representing the query, key, and value of the LiDAR under the attention mechanism, respectively, and W is the attention matrix. For the region features of the image, represents the regional features of the lidar; D is the dimension of the embedding vector. The scale used to define the sampling distance of the lidar;

[0160] Step 5-3: Introduce the geometric bias matrix and calculate the bidirectional multi-head attention output:

[0161]

[0162]

[0163] In the formula, These represent the key and value, respectively, for querying image information at a distance h under the attention mechanism. These represent the key and value, respectively, for querying LiDAR information at a distance of h under the attention mechanism. , These represent the bidirectional multi-head attention outputs of the camera and the LiDAR, respectively.

[0164] Step 5-4: Concatenate attention heads and perform linear mapping to obtain cross-modal fusion output, and obtain cross-modal aligned region features through gated residual connections.

[0165]

[0166]

[0167] In the formula, These represent the attention aggregation matrices for the image and radar, respectively. For learnable scalar gating parameters, As a standardization layer for attention mechanisms, , These are the intermediate attention results for the image and the LiDAR, respectively; (*) indicates that the outputs of H attention heads are concatenated along the feature dimension.

[0168] Here, cross-modal mismatch caused by spatiotemporal bias is suppressed and cross-modal alignment robustness is improved by using geometric bias and gated residuals.

[0169] Furthermore, in one embodiment, step 6 specifically includes:

[0170] Step 6-1, distance to pixel p Define geometric weights :

[0171]

[0172] In the formula, Indicates the effective distance threshold. The temperature coefficient represents the degree of smoothness in controlling the weight decay. (*) indicates the Sigmoid activation function;

[0173] Step 6-2, Aggregate within a tile to form a block level. With the overall And let the feature of the lidar branch sample b in patch k be... Define sparse regularization terms :

[0174]

[0175] In the formula, Indicates the sample batch size;

[0176] Step 6-3, Define the within-batch mean With dispersion :

[0177]

[0178] Step 6-4, Define spatial consistency constraints :

[0179] .

[0180] Here, by and By incorporating the total loss function, the representation amplitude and intra-batch fluctuations in distant sparse regions, void regions, and low-information regions are suppressed to improve the stability of cross-modal aligned representations.

[0181] Furthermore, in one embodiment, step 7 specifically includes:

[0182] Step 7-1: Use a multi-scale attention aggregation mechanism to obtain the final global features of the camera.

[0183]

[0184] In the formula, These represent the query, key, and value, respectively.

[0185] Step 7-2: Use a multi-scale attention aggregation mechanism to obtain the final global features of the LiDAR:

[0186]

[0187] Step 7-3, via shared projection head The final projected representation is obtained:

[0188] .

[0189] Here, the multi-scale attention aggregation mechanism gathers regional-level refined information into global features in an attention-weighted manner, so that the final projection representation simultaneously includes scene-level semantics and local geometric details.

[0190] Here, will and The final contrastive learning objective in step 8 is used to form a unified cross-modal embedding space.

[0191] Furthermore, in one embodiment, step 8 specifically includes:

[0192] Step 8-1, Define the final stage bidirectional contrast loss :

[0193]

[0194] In the formula, , These represent the final projected representation of the image corresponding to the i-th patch and the final projected representation of the LiDAR, respectively; , These represent the final projected representation of the image corresponding to the j-th tile and the final projected representation of the lidar, respectively.

[0195] Step 8-2, Construct the total loss function :

[0196]

[0197] In the formula, They are respectively The weights;

[0198] Step 8-3: Iteratively update the pre-trained model parameters using gradient descent or its equivalent optimization algorithm to minimize... ;

[0199] Step 8-4: Use the synchronous or near-synchronous sample pairs obtained in Step 1 as positive sample sources and the remaining samples in the batch as negative sample sources to form a discriminative cross-modal attitude quantity space without additional manual annotation.

[0200] Step 8-5: After optimization, output the cross-modal representation embedding. ,in:

[0201]

[0202] .

[0203] In one embodiment, a robust camera-LiDAR multimodal contrastive pre-training system resistant to spatiotemporal bias is provided, the system comprising sequentially executing:

[0204] The first module is used to acquire time-synchronized or near-synchronized camera images and LiDAR distance images in the same scene;

[0205] The second module is used to input the camera image and the lidar distance image into a pre-trained model to obtain a cross-modal representation embedding, wherein the pre-trained model consists of an input encoding module, a global contrast alignment module, a tile-level contrast alignment module and a cross-modal refinement module;

[0206] The third module is used to extract global features and camera patch features, as well as global features and lidar patch features from the input encoding module.

[0207] The fourth module is used to perform global contrast alignment on global features through the global contrast alignment module to obtain global contrast loss. ;

[0208] The fifth module is used to perform tile-level contrast alignment on tile features through the tile-level contrast alignment module to obtain tile contrast loss. And obtain tile-level augmented features;

[0209] The sixth module is used to obtain cross-modal aligned region features by using a bidirectional cross-modal attention mechanism and modulating the attention weights based on the geometric bias matrix through the cross-modal refinement module.

[0210] The seventh module is used to obtain the final global features by employing a multi-scale attention aggregation mechanism and to obtain the final projected representation by sharing a projection head;

[0211] The eighth module is used to pre-train and optimize the model using a total loss function, which includes global contrast loss, patch contrast loss, sparse regularization term, spatial consistency constraint term, and final contrast loss.

[0212] Specific limitations regarding the robust camera-LiDAR multimodal contrastive pre-training system resistant to spatiotemporal bias can be found in the limitations of the robust camera-LiDAR multimodal contrastive pre-training method resistant to spatiotemporal bias described above, and will not be repeated here. Each module in the aforementioned robust camera-LiDAR multimodal contrastive pre-training system resistant to spatiotemporal bias can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0213] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0214] Step 1: Acquire time-synchronized or near-synchronized camera images and LiDAR point clouds in the same scene, and generate a LiDAR distance image by distance projection of the LiDAR point clouds. Simultaneously acquire camera images ;

[0215] Step 2, using a dual-branch encoder to... and Multimodal feature encoding is performed to obtain the camera's global features. Camera tile features and global features of lidar LiDAR patch features ;

[0216] Step 3, for and Perform global contrastive alignment to learn scene-level cross-modal semantic consistency and obtain global contrastive loss. ;

[0217] Step 4, for and Perform tile-level contrast alignment to learn local cross-modal correspondences and obtain tile contrast loss. ;

[0218] Step 5: The patch-level enhancement features obtained in Step 4 are used to perform region aggregation using a learnable pooling function to obtain the region-level sequences of the image. Regional sequence with lidar Furthermore, a bidirectional cross-modal multi-head attention mechanism is constructed based on semantic-aware transformers and geometric-aware transformers, and a geometric bias matrix is ​​introduced. Attention weights are modulated to obtain region features of the image after cross-modal alignment. and the regional characteristics of lidar ;

[0219] Step 6: Introduce regularization constraints to suppress unstable representations in distant, hollow, or low-information regions, including sparsity regularization terms. Spatial consistency constraints ;

[0220] Step 7: Employ a multi-scale attention aggregation mechanism, using the initial global features as the query vector to... and The final global features of the image are obtained by aggregation. and the final global features of lidar The final projected representation of the image is obtained through a shared projection head. The final projection representation of the lidar ;

[0221] Step 8: Construct and minimize the total loss function to pre-train and optimize the model. The total loss function includes at least the global contrastive loss. Patch contrast loss sparse regularization term Spatial consistency constraints Compared with the final loss Thus, cross-modal representation embedding is obtained. D is the dimension of the embedding vector.

[0222] For specific limitations on each step, please refer to the limitations on the multimodal contrastive pre-training method for robust cameras-LiDAR against spatiotemporal bias mentioned above, which will not be repeated here.

[0223] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0224] Step 1: Acquire time-synchronized or near-synchronized camera images and LiDAR point clouds in the same scene, and generate a LiDAR distance image by distance projection of the LiDAR point clouds. Simultaneously acquire camera images ;

[0225] Step 2, using a dual-branch encoder to... and Multimodal feature encoding is performed to obtain the camera's global features. Camera tile features and global features of lidar LiDAR patch features ;

[0226] Step 3, for and Perform global contrastive alignment to learn scene-level cross-modal semantic consistency and obtain global contrastive loss. ;

[0227] Step 4, for and Perform tile-level contrast alignment to learn local cross-modal correspondences and obtain tile contrast loss. ;

[0228] Step 5: The patch-level enhancement features obtained in Step 4 are used to perform region aggregation using a learnable pooling function to obtain the region-level sequences of the image. Regional sequence with lidar Furthermore, a bidirectional cross-modal multi-head attention mechanism is constructed based on semantic-aware transformers and geometric-aware transformers, and a geometric bias matrix is ​​introduced. Attention weights are modulated to obtain region features of the image after cross-modal alignment. and the regional characteristics of lidar ;

[0229] Step 6: Introduce regularization constraints to suppress unstable representations in distant, hollow, or low-information regions, including sparsity regularization terms. Spatial consistency constraints ;

[0230] Step 7: Employ a multi-scale attention aggregation mechanism, using the initial global features as the query vector to... and The final global features of the image are obtained by aggregation. and the final global features of lidar The final projected representation of the image is obtained through a shared projection head. The final projection representation of the lidar ;

[0231] Step 8: Construct and minimize the total loss function to pre-train and optimize the model. The total loss function includes at least the global contrastive loss. Patch contrast loss sparse regularization term Spatial consistency constraints Compared with the final loss Thus, cross-modal representation embedding is obtained. D is the dimension of the embedding vector.

[0232] For specific limitations on each step, please refer to the limitations on the multimodal contrastive pre-training method for robust cameras-LiDAR against spatiotemporal bias mentioned above, which will not be repeated here.

[0233] As a specific example, the invention will be further verified and illustrated in one embodiment.

[0234] This embodiment is experimentally validated on a multimodal camera-LiDAR dataset, which provides multiple camera views, LiDAR point clouds, and corresponding pose information. The forward-view camera image and the LiDAR point cloud within the corresponding view range are selected, and distance projection is performed according to step 1 to obtain the LiDAR distance image. Training and evaluation are then performed following a training / validation / test partitioning. During training, the global contrast alignment, tile-level contrast alignment, region aggregation and bidirectional cross-modal multi-head attention refinement, sparsity regularization and spatial consistency constraints, and multi-scale attention aggregation mechanism described in the claims are employed, and optimization is performed according to the total loss function described in step 8.

[0235] Regarding evaluation metrics, for cross-modal retrieval tasks, Top@K accuracy is used to evaluate bidirectional retrieval performance, which is defined as:

[0236]

[0237] in Indicates a query Return the set of the K most similar samples. Indicates and The corresponding set of real matches is given, where N is the number of query samples. Experimental results are shown in Table 1, where 0-shot represents the experiment conducted in a test scenario completely disjoint from the training set, and the Top@k value represents the proportion of real matching data in the top K most similar samples returned for each query data in the test dataset after performing a cross-modal retrieval task. Experimental results show that this method can effectively encode the latent unified space of cross-modal data, possessing strong cross-modal continuous data encoding and discriminative capabilities.

[0238] Table 1. Performance comparison of different methods on cross-modal retrieval tasks (Top@k)

[0239]

[0240] To verify the robustness of this invention to spatiotemporal bias, yaw, translation, and pitch angle perturbations were further applied to the lidar data, and range projection was performed after the perturbations to form a misaligned test set. Comparative experiments show that, under the above misalignment conditions, the method of this invention exhibits better stability and smaller performance degradation in cross-modal retrieval accuracy. Figure 2 In the comparison, ResNet, AECMLoc, Lip-Loc and our method are compared. Figure 2 The cosine similarity matrix heatmap generated by ResNet (d) shows significant differences. Figure 2 In example a), the boundaries between intra-class (diagonal green area) and inter-class (off-diagonal area) regions are relatively blurred, indicating low feature separation. In contrast, while AECMLoc and Lip-Loc show some clustering ability, this method ( Figure 2 In example d), the most distinct blocky diagonal structure is observed: the diagonal regions are dark green, representing extremely high intra-class similarity; while the off-diagonal regions are clearly colored, representing lower inter-class similarity. This intuitively demonstrates that the features extracted by this method have excellent intra-class compactness and inter-class separability. Figure 3 This observation was further quantified. In the figure, the red areas represent the similarity distribution of dissimilar sample pairs, and the green areas represent the similarity distribution of similar sample pairs. Compared to the baseline model ( Figure 3 (a to c in the original text), this method ( Figure 3 In example d), a significant performance advantage is demonstrated: the distribution peak of similar samples (green) is extremely high and narrow (Y-axis density as high as 40, far exceeding the baseline's 12-16), and they are tightly clustered near a similarity of 1.0; at the same time, dissimilar samples (red) are effectively suppressed in the low similarity range. These results indicate that this method can greatly optimize the distribution of the feature space, effectively pushing away dissimilar features while narrowing the distance between similar features, thus achieving a superior feature discrimination capability compared to existing baselines.

[0241] Furthermore, to verify the generalization ability and robustness of the cross-modal representation extracted in this invention in actual perception tasks, this embodiment uses the features output by this invention as the front-end input of the downstream 3D target detection model (perception model), and performs quantitative evaluation under a scenario simulating sensor extrinsic calibration error (i.e., misalignment condition). The experiment uses mean accuracy (mAP) and normalized detection score (NDS) as core evaluation metrics, where mAP is defined as:

[0242]

[0243] In the formula, C represents the number of categories, and T represents the number of distance thresholds. Let be the average accuracy of category c at threshold t. The experimental results are shown in Table 2, where Car, Truc., Bus, Trail., CV, Ped., MC, Bike, TC, and Bar. represent the AP of categories car, bus, truck, trailer, construction vehicle, pedestrian, motorcycle, bicycle, traffic cone, and guardrail, respectively.

[0244] Table 2. Comparison of cross-modal fusion target detection performance of different methods under misalignment conditions (mAP, NDS, AP)

[0245] Table 2 shows the cross-modal fusion target detection performance comparison between our method (RangeCLIP) and mainstream baseline methods (BEVFusion, CMT) under different types of sensor misalignment conditions.

[0246] The following key conclusions can be drawn from the quantitative analysis in Table 2: (1) Robustness of heading angle offset: When introducing -2 0 Up to 2 0When the heading angle noise is high, the traditional BEVFusion method suffers severe performance degradation (mAP is only 0.264). In contrast, this method exhibits excellent anti-interference ability. Although the mAP is on par with CMT, the method achieves 0.557 in the NDS index, which better reflects the overall detection quality, significantly better than CMT's 0.463. Especially in the detection of large vehicles such as trucks (0.684), buses (0.619), and trailers (0.584), this method significantly outperforms the baseline, demonstrating its ability to align the geometric features of large targets. (2) Azimuth offset robustness: In the azimuth offset scenario, the advantages of this method are further expanded, with mAP increasing to 0.562 and NDS reaching 0.629, comprehensively surpassing BEVFusion (mAP 0.327) and CMT (mAP 0.548). This indicates that the cross-modal refinement mechanism of this invention can effectively correct the rotational deviation of the viewpoint and maintain the spatial consistency of features. (3) Robustness to front and rear position offsets: Experiments show that under position translation disturbances ranging from -0.2m to 0.2m, the CMT model suffers severe performance collapse, with mAP dropping sharply from 0.548 in the azimuth angle to 0.288. However, the proposed method still maintains extremely high stability, with mAP remaining at 0.474 and NDS at 0.556. This result strongly demonstrates that the geometric offset modulation and regularization constraint mechanism introduced in this invention can effectively "absorb" spatial position errors and prevent detection failure caused by feature misalignment.

[0247] In summary, experimental results demonstrate that this invention can significantly improve the stability of downstream detection performance under various sensor misalignment conditions. The robust camera-LiDAR multimodal comparative pre-training method proposed in this invention, which resists spatiotemporal bias, learns stable cross-modal representations even with external parameter perturbations, projection errors, or sampling timing deviations through global and local hierarchical alignment, regional cross-modal refinement and geometric bias modulation, and regularization constraints for sparse and empty regions. Furthermore, it can be seamlessly transferred to cross-modal retrieval and downstream 3D perception tasks, demonstrating extremely high practical value.

[0248] Furthermore, the method of the present invention can not only provide more robust fusion features for downstream 3D perception tasks, but also extend the robustness of these features to various adverse weather scenarios such as rain, fog, and nighttime, as well as complex road conditions, thereby significantly improving the safety and reliability of autonomous driving systems in extreme and long-tail scenarios.

[0249] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A robust camera-LiDAR multimodal contrastive pre-training method resistant to spatiotemporal bias, characterized in that, The method includes the following steps: Step 1: Acquire time-synchronized or near-synchronized camera images and LiDAR point clouds in the same scene, and generate a LiDAR distance image by distance projection of the LiDAR point clouds. Simultaneously acquire camera images ; Step 2, using a dual-branch encoder to... and Multimodal feature encoding is performed to obtain the camera's global features. Camera tile features and global features of lidar LiDAR patch features ; Step 3, for and Perform global contrastive alignment to learn scene-level cross-modal semantic consistency and obtain global contrastive loss. ; Step 4, for and Perform tile-level contrast alignment to learn local cross-modal correspondences and obtain tile contrast loss. ; Step 5: The patch-level enhancement features obtained in Step 4 are used to perform region aggregation using a learnable pooling function to obtain the region-level sequences of the image. Regional sequence with lidar Furthermore, a bidirectional cross-modal multi-head attention mechanism is constructed based on semantic-aware transformers and geometric-aware transformers, and a geometric bias matrix is ​​introduced. Attention weights are modulated to obtain region features of the image after cross-modal alignment. and the regional characteristics of lidar ; Step 6: Introduce regularization constraints to suppress unstable representations in distant, hollow, or low-information regions, including sparsity regularization terms. Spatial consistency constraints ; Step 7: Employ a multi-scale attention aggregation mechanism, using the initial global features as the query vector to... and The final global features of the image are obtained by aggregation. and the final global features of lidar The final projected representation of the image is obtained through a shared projection head. The final projection representation of the lidar ; Step 8: Construct and minimize the total loss function to pre-train and optimize the model. The total loss function includes at least the global contrastive loss. Patch contrast loss sparse regularization term Spatial consistency constraint Compared with the final loss Thus, cross-modal representation embedding is obtained. D is the dimension of the embedding vector.

2. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 1, characterized in that, Step 1 specifically includes: Step 1-1: Acquire camera images ; Steps 1-2: Generate a lidar range image by projecting the lidar point cloud over distance. Furthermore, the sparse depth observation value of the lidar projection at pixel p is defined as... Where H and W are the number of pixels in the vertical and horizontal directions of the image, respectively, and B is the batch size; Steps 1-3, when pixel p has no valid echo, set ; This is used to characterize the hollow region and suppress it in subsequent feature learning and regularization constraints; Steps 1-4, convert the single channel Copy the data to a three-channel format or perform equivalent channel expansion before inputting it into the LiDAR encoder; Steps 1-5, for and Perform timestamp pairing or near-synchronous pairing.

3. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 2, characterized in that, Step 2 specifically includes: Step 2-1, using a camera encoder right Perform tile division and encoding to satisfy: In the formula, D is the dimension of the embedding vector, and N is the number of partitioned graph tiles; Step 2-2, using a lidar encoder right Perform tile division and encoding to satisfy: In the formula, ; Steps 2-3, during the pre-training phase... and The parameters are jointly trained or fine-tuned to make the two modalities comparable in a shared comparison space.

4. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 3, characterized in that, Both the camera encoder and the lidar encoder are vision transformer structures or equivalent self-attention-based encoder structures.

5. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 3, characterized in that, Step 3 specifically includes: Step 3-1: Project the global features of the i-th pair of matching samples onto the shared projection head P to obtain the features of the i-th pair of matching samples in the image. Features of the i-th pair of matching samples with LiDAR ,satisfy: Step 3-2: Construct a bidirectional contrastive learning objective based on matching sample pairs with a batch size of B, and use sample pairs with the same index within the batch as positive sample pairs and the remaining samples within the batch as negative sample pairs. Step 3-3, Define the global contrast loss for: In the formula, This is a temperature coefficient used to adjust the sharpness of the similarity distribution; , These are the features of the j-th pair of matching samples in the image and the features of the j-th pair of matching samples in the LiDAR, respectively.

6. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 5, characterized in that, Step 4 specifically includes: Step 4-1: Organize the camera patch features and LiDAR patch features according to their spatial location and perform intra-batch comparison learning so that cross-modal patch features at the same location form positive sample pairs, and patch features from different samples form negative sample pairs. Step 4-2: Calculate the intra-batch similarity matrix for the k-th position. : In the formula, These are the camera patch features and the LiDAR patch features at the k-th position, respectively; Step 4-3, Define the patch contrast loss for: In the formula, This is the mutual information loss function.

7. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 6, characterized in that, Step 5 specifically includes: Step 5-1, using learnable pooling functions The aggregation yields a sequence of regions that satisfy: In the formula, This indicates the number of representative regions defined under the attention mechanism; Step 5-2, using a shared linear mapping The features of the two modal regions are aligned in distribution, and the query, key, and value of bidirectional cross-modal attention are constructed respectively: In the formula, These represent the query, key, and value of an image under the attention mechanism, respectively. Representing the query, key, and value of the LiDAR under the attention mechanism, respectively, and W is the attention matrix. For the region features of the image, represents the regional features of the lidar; D is the dimension of the embedding vector. The scale used to define the sampling distance of the lidar; Step 5-3: Introduce the geometric bias matrix and calculate the bidirectional multi-head attention output: In the formula, These represent the key and value, respectively, for querying image information at a distance h under the attention mechanism. These represent the key and value, respectively, for querying LiDAR information at a distance of h under the attention mechanism. , These represent the bidirectional multi-head attention outputs of the camera and the LiDAR, respectively. Step 5-4: Concatenate attention heads and perform linear mapping to obtain cross-modal fusion output, and obtain cross-modal aligned region features through gated residual connections. In the formula, These represent the attention aggregation matrices for the image and radar, respectively. For learnable scalar gating parameters, As a standardization layer for attention mechanisms, , These are the intermediate attention results for the image and the LiDAR, respectively; (*) indicates that the outputs of H attention heads are concatenated along the feature dimension.

8. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 7, characterized in that, Step 6 specifically includes: Step 6-1, distance to pixel p Define geometric weights : In the formula, Indicates the effective distance threshold. The temperature coefficient represents the degree of smoothness in controlling the weight decay. (*) indicates the Sigmoid activation function; Step 6-2, Aggregate within a tile to form a block level. With the overall And let the feature of the lidar branch sample b in patch k be... Define sparse regularization terms : In the formula, Indicates the sample batch size; Step 6-3, Define the within-batch mean With dispersion : Step 6-4, Define spatial consistency constraints : 。 9. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 8, characterized in that, Step 7 specifically includes: Step 7-1: Use a multi-scale attention aggregation mechanism to obtain the final global features of the camera. In the formula, These represent the query, key, and value, respectively. Step 7-2: Use a multi-scale attention aggregation mechanism to obtain the final global features of the LiDAR: Step 7-3, via shared projection head The final projected representation is obtained: 。 10. The robust camera-lidar multimodal contrastive pre-training method against spatiotemporal bias according to claim 9, characterized in that, Step 8 specifically includes: Step 8-1, Define the final stage bidirectional contrast loss : In the formula, , These represent the final projected representation of the image corresponding to the i-th patch and the final projected representation of the LiDAR, respectively; , These represent the final projected representation of the image corresponding to the j-th tile and the final projected representation of the lidar, respectively. Step 8-2, Construct the total loss function : In the formula, They are respectively The weights; Step 8-3: Iteratively update the pre-trained model parameters using gradient descent or its equivalent optimization algorithm to minimize... ; Step 8-4: Use the synchronous or near-synchronous sample pairs obtained in Step 1 as positive sample sources and the remaining samples in the batch as negative sample sources to form a discriminative cross-modal attitude quantity space without additional manual annotation. Step 8-5: After optimization, output the cross-modal representation embedding. .

Images