An adaptive calibration and physically guided multi-modal data fusion perception method

Abstract

This invention discloses an adaptive calibration and physically guided multimodal data fusion perception method and system, belonging to the fields of computer vision and autonomous driving technology. Addressing the core problems of projection misalignment caused by long-term sensor extrinsic parameter drift in existing BEV perception systems, and target detection lag in high-speed dynamic scenarios, this invention constructs a perception framework comprising four modules: multi-view image feature projection transformation, LiDAR point cloud feature encoding, multi-sensor online adaptive calibration based on BEV consistency, and physically prior-driven cross-modal deformable Transformer fusion. A closed-loop feedback mechanism is formed through perception and calibration, and sensor extrinsic parameters are corrected in real time by combining visual reprojection, IMU pre-integration, and BEV feature alignment constraints to eliminate projection ghosting. A vehicle kinematics prior is introduced to construct a dynamic query, driving the attention sampling points to adaptively shift along the target motion trajectory, thus solving the detection lag problem.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to an adaptive calibration and physically guided multimodal data fusion sensing method. Background Technology

[0002] With the rapid development of computer vision, many novel methods have been proposed to address bird's-eye view (BEV) perception in multimodal autonomous driving centered on multiple sensors. From this perspective, the vehicle can "see" the surrounding road conditions in a bird's-eye view manner, mapping multi-source data from cameras, LiDAR, and other sources onto a unified BEV plane to form a unified spatial representation that includes semantic information such as road structure, vehicle and pedestrian positions, lane lines, and traffic lights.

[0003] BEV target detection, as an important innovation of the new generation of multimodal autonomous driving perception framework, is no longer limited to the traditional deterministic feature superposition. Instead, it dynamically and adaptively fuses data from different sensors, including image features generated by cameras, three-dimensional spatial structure information provided by LiDAR, and motion state captured by IMU, in a unified bird's-eye view space through unified probabilistic modeling.

[0004] Currently, the core task of perception in autonomous BEVs is to achieve accurate fusion of multimodal data within a unified spatial coordinate system. While existing mainstream solutions have begun to combine probabilistic fusion with cross-modal spatiotemporal modeling, feature fitting relying solely on deep learning models has reached its limits in actual long-term operation and high-speed dynamic scenarios. Existing technologies often treat "sensor calibration" and "perception fusion" as two independent offline or decoupled processes: the front end relies on pre-calibrated extrinsic parameters for projection, while the back end is only responsible for feature processing. This fragmented approach leads to slight drifts in sensor extrinsic parameters during high-speed steering, causing the visual features projected onto the BEV plane to misalign with radar features in space, resulting in "ghosting" or "misalignment," severely limiting the final accuracy of perception. Two key issues arise regarding this: (1) Failure of static calibration: Most existing BEV perception schemes rely on offline calibration. However, during long-term vehicle operation or severe bumps, the extrinsic parameters of the camera and LiDAR will drift slightly, causing the features projected onto the BEV plane to be geometrically misaligned, thus resulting in fusion failure.

[0005] (2) Lag of dynamic features: Traditional Transformer uses random or fixed sampling points during fusion, ignoring the vehicle's own kinematic state. In high-speed or turning scenarios, the perception results often have "mog" or position lag, and cannot accurately capture the instantaneous position of the target.

[0006] Currently, research on existing autonomous driving BEV algorithms mainly focuses on improving the speed of target detection. However, there is still a lack of in-depth research on how to ensure the stability of multimodal fusion in complex scenarios and how to achieve online calibration of multiple sensors during long-term operation. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides an adaptive calibration and physically guided multimodal data fusion sensing method and system.

[0008] To achieve the above objectives, the present invention provides the following technical solution: An adaptive calibration and physically guided multimodal data fusion sensing method includes: S1: Dataset Input: Introduce the nuscenes dataset to obtain RGB image data from multi-view cameras, sparse point cloud data from LiDAR, acceleration and angular velocity data from inertial measurement units, and real-time kinematic state information from vehicle chassis feedback. S2: Dual-stream feature extraction and BEV spatial projection; S21: Image Branch: Input multi-view images into the view encoder, use ResNet as the backbone network to extract deep semantic features; then use the Lift-Splat-Shoot algorithm to estimate the frustum depth, "lift" the two-dimensional image features to 3D space and project them onto a unified BEV plane to generate the initial camera flow BEV feature map. S22: Radar Branch: The original LiDAR point cloud is divided into vertical cylinders. The PointNet network is used to encode the local geometric features within each cylinder to generate fixed-dimensional cylinder features. Then, according to their physical coordinate index on the plane, they are scattered and mapped into a regular two-dimensional grid to generate radar flow BEV feature map. S3: Online calibration of multiple sensors based on BEV consistency: Step S31: Error Calculation: Receive the camera stream BEV feature map and radar stream BEV feature map generated in step S2, and combine them with IMU data to construct a joint optimization objective function that includes visual reprojection error, IMU pre-integration error and BEV spatial alignment error. The BEV spatial alignment error is obtained by calculating the structural difference between the two modal feature maps in the overlapping field of view. Step S32: Closed-loop correction: The above joint error is minimized by using a nonlinear optimization method, the relative pose transformation matrix of the camera to the LiDAR is solved in real time, and the projection matrix in step S21 is dynamically updated using the corrected extrinsic parameters. S4: Physics-prior-driven cross-modal Transformer fusion, specifically: S41: Physical Query Construction: Based on the real-time physical state of the vehicle, a dynamic Query code is constructed. The initial Query code is concatenated with the physical state vector and then subjected to feature mapping and dimensionality reduction processing through an MLP (Multilayer Perceptron) to obtain an enhanced Query code. S42: Inertial Guided Sampling: An inertial guided field mechanism is introduced. When calculating deformable attention, the potential position of the target in the next frame is predicted by the dynamic query. The sampling offset is calculated by MLP. The sampling points of the attention mechanism are adaptively offset along the predicted motion trajectory direction to obtain the corrected key representation. S43: Feature aggregation: Based on the offset sampling points, aggregate key features on the calibrated camera and LiDAR BEV feature maps, calculate attention weights, and perform weighted summation of features; S44: Deep Encoding: The fused features are input into the multi-layer encoder of the Transformer. After self-attention processing, feedforward nonlinear transformation, residual connection and layer normalization, the fused features are output to the detection head to obtain the 3D target detection result.

[0009] The present invention further defines the technical solution as follows: Preferably, the specific process of using the Lift-Splat-Shoot algorithm in step S21 to convert multi-view image features into a unified BEV feature map is as follows: 2D to 3D transformation (Lift): Processing high-dimensional image features to predict a set of discrete depth probability distributions for each pixel location in the image. This depth probability is used as a weighting factor to multiply the semantic features of the pixel, "lifting" the original two-dimensional features into probability-weighted features in a 3D view frustum. Feature projection (Splat): Projects and aggregates all probability-weighted features of a 3D view frustum into a unified 3D mesh space based on their positions in the world coordinate system. BEV Mapping (Shoot): The features in the 3D mesh are compressed and summarized along the direction perpendicular to the ground. The compressed feature layer is then mapped onto the BEV plane to generate a camera flow BEV feature map.

[0010] Preferably, in step S22, the specific processing procedure for the radar branch further includes: Add an extra dimensional feature to each LiDAR point, including the local average coordinate offset of that point relative to all points within the Pillar. and the absolute offset of that point relative to the Pillar geometric center. and ; For each valid Pillar, limit the maximum number of points N it contains, apply a simplified version of the PointNet structure to encode each augmented point in the Pillar and perform multilayer perceptual operations to map the feature dimension of the point from D dimensions to C dimensions, and then aggregate the features of all points in the Pillar through the Max Pooling operation to obtain a fixed-length feature vector. An optimized 2D feature extraction network is used to extract features from pseudo-image feature maps at multiple scales. This network includes a top-down network and a feature pyramid network (FPN). It generates feature maps of different scales by downsampling layer by layer, and then fuses the deep and shallow features through upsampling and feature concatenation operations to improve them to a uniform BEV resolution.

[0011] Preferably, the visual reprojection error in step S3 is constructed as follows: a feature point s is defined as being first observed in frame i and observed again in frame j, and the inverse depth λ of frame i is used... s With relative pose transformation, feature points are projected onto the normalized camera plane of the j-th frame to construct the reprojection error: ; in Let be the observed coordinates of feature point s in the j-th frame of the image. , Let be the projection function of the camera. The extrinsic parameter matrix from the camera coordinate system to the IMU coordinate system. Let be the pose transformation matrix of the IMU in the world coordinate system for the j-th frame and the i-th frame. Let be the coordinates of feature point s in the normalized plane of the i-th frame; IMU pre-integration constraints are implemented in the following way: The relationship between IMU measurements and true values ​​is as follows: ; in and These are the angular velocity and acceleration biases, respectively. and To measure noise, For the rotation from the world coordinate system to the machine coordinate system; The dynamic constraint formula is: ; Motion constraints are obtained by integrating the acceleration and angular velocity of the IMU in two keyframes, where... The quaternion derivative formula reflects the equation of motion for rotation, ultimately yielding the constraint residuals of the integral: .

[0012] Preferably, in step S3, the alignment constraint between LIDAR and BEV is defined as: projecting the radar point cloud onto the BEV mesh to obtain... Projecting camera features onto the BEV mesh yields... Alignment error is defined as the geometric difference between two modal features in the overlapping region: ; Minimizing this error allows the visual features to maintain geometric consistency with the radar features in the BEV space.

[0013] Preferably, the physical state vector in step S4 has the following form: ; in Includes the vehicle's speed in the x and y directions. Includes the vehicle's acceleration in the x and y directions. The vehicle's heading angle. The cosine and sine values ​​represent the heading angle and are used to represent the direction vector; enhanced query encoding. ,in It is the basic positional encoding of the i-th query. For splicing operations; The sampling offset is in the following form: ; in To enhance the query vector, The j-th feature vector to be aggregated contains raw information from both camera stream BEV features and radar stream BEV features; the correction key represents... , This is the original key vector.

[0014] Preferably, in step S4, the attention weights are calculated and the features are weighted and summed, using the following formula: ; in This represents the fused feature vector, which contains feature information from different modalities. The similarity between the query and the offset key determines the query vector. For different key vectors The level of attention; Features from both the camera and radar are incorporated. The similarity between the query and the offset key is calculated, and then weighted and summed with the value vector to obtain the fused features. These fused features are then input into a multi-layer encoder of a Transformer for processing. Each encoder layer further processes the features through self-attention, thereby extracting deeper semantic information and spatial features. The process is as follows: ; Q is the query matrix generated from the fused features, K is the offset key matrix, and V is the value matrix composed of continuously updated camera and radar features. The dimension of the key vector is used to control the scaling factor; After passing through the self-attention layer, the features are fed into a feedforward network for nonlinear transformation to extract more abstract and deeper semantic information and spatial features: ; y is the input of the current layer (i.e., the output of the previous layer). For residual connections, LayerNorm ensures the stability of the mean and variance of features across different dimensions; FFN (Feedforward Network) contains two fully connected layers and a non-linear activation function, extracting deeper semantic information; The final output contains highly fused BEV features with deep semantic representation, providing the most reliable input for the final detection task.

[0015] This invention provides an adaptive calibration and physically guided multimodal data fusion sensing system, implementing the above method, including: Multi-view image feature projection transformation module: used to extract deep semantic features from multi-view cameras, and use depth estimation technology to "enhance" the two-dimensional image features to three-dimensional space, and project to generate camera flow BEV feature maps; The LiDAR point cloud feature encoding module is used to voxelize the original sparse laser point cloud, extract spatial features containing precise geometric structure information, and map them into radar flow BEV feature maps. The multi-sensor online adaptive calibration module based on BEV consistency is used to construct a joint optimization model that includes BEV spatial alignment error, calculate the structural consistency difference between the camera stream BEV feature map and the radar stream BEV feature map in the overlapping area in real time, and correct the extrinsic matrix between the camera and LiDAR in real time by minimizing the alignment error. Physics-prior-driven cross-modal deformable Transformer fusion module: for deep feature fusion within a calibrated unified BEV space.

[0016] Beneficial effects Compared with existing technologies, it has the following advantages: This invention deeply couples sensor calibration with perception fusion. By calculating the spatial alignment error of BEV feature maps from camera stream and radar stream in real time, and combining visual reprojection error and IMU pre-integration error to construct a joint optimization model, it corrects sensor extrinsic parameters in reverse. This eliminates the extrinsic parameter drift problem caused by long-term vehicle operation and turning from the root, solves the "ghosting" and "misalignment" phenomena of BEV spatial projection, and ensures the geometric consistency of multimodal features in a unified space. This invention constructs a "perception-calibration" closed-loop feedback mechanism based on BEV geometric consistency, utilizing alignment errors at the feature map level as real-time monitoring signals. This fundamentally solves the projection misalignment problem caused by sensor extrinsic drift, IMU accumulated errors, and hard synchronization failure in autonomous driving systems operating for extended periods and in complex environments. It achieves continuous adaptive calibration and pixel-level alignment of multimodal data in a unified spatial coordinate system. Simultaneously, this invention innovatively introduces a physical kinematics prior-driven attention sampling strategy. During cross-modal fusion, an inertial guidance field is constructed, forcing the Transformer's feature sampling points to adaptively shift along the target motion prediction trajectory. This mechanism effectively overcomes the perception lag and feature loss problems of traditional algorithms in sparse point clouds, severe occlusion, and high-speed scenarios, thereby significantly improving the detection accuracy, trajectory prediction stability, and all-weather robustness of autonomous driving systems in complex dynamic environments. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall autonomous driving BEV framework of the present invention; Figure 2 This is a flowchart of the feature conversion and extraction module for multi-view images of the present invention; Figure 3 This is a flowchart of the feature extraction and transformation module for the point cloud dataset of the present invention; Figure 4 This is a flowchart of the multi-sensor online calibration module of the present invention; Figure 5 This is a flowchart of the cross-modal deformable Transformer fusion module of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 This embodiment provides an adaptive calibration and physically guided multimodal data fusion sensing method, including: S1: Simultaneously collect multi-source data through the vehicle sensor kit, including: RGB image data from 6 surround view cameras, sparse point cloud data from 16-line LiDAR, acceleration and angular velocity data from IMU, and real-time kinematic state information fed back by the vehicle chassis. These image data need to be preprocessed, and distortion correction, size normalization and pixel value standardization should be performed on the RGB images to eliminate the inherent geometric distortion of the sensor and provide standardized input for feature extraction. Denoising is performed on the LiDAR point cloud to remove invalid points that are too far away or have too low reflection intensity; S2: Dual-stream feature extraction and BEV spatial projection; S21: Image Branch: Multi-view images are input to the view encoder, and ResNet is used as the backbone network to extract deep semantic features. Then, the Lift-Splat-Shoot algorithm is used to estimate the frustum depth, "lifting" the 2D image features to 3D space and projecting them onto a unified BEV plane to generate the initial camera flow BEV feature map. The specific process is as follows: 2D to 3D transformation (Lift): Processing high-dimensional image features to predict a set of discrete depth probability distributions for each pixel location in the image. This depth probability is used as a weighting factor to multiply the semantic features of the pixel, "lifting" the original two-dimensional features into probability-weighted features in a 3D view frustum. Feature projection (Splat): Projects and aggregates all probability-weighted features of a 3D view frustum into a unified 3D mesh space based on their positions in the world coordinate system. BEV Mapping (Shoot): The features in the 3D mesh are compressed and summarized along the direction perpendicular to the ground. The compressed feature layer is then mapped onto the BEV plane to generate a camera flow BEV feature map. S22: Radar Branch: The original LiDAR point cloud is divided into vertical cylinders. The PointNet network is used to encode the local geometric features within each cylinder to generate fixed-dimensional cylinder features. Then, according to their physical coordinate index on the plane, they are scattered and mapped into a regular two-dimensional grid to generate radar flow BEV feature map. The specific processing steps for the radar branch also include: Add an extra dimensional feature to each LiDAR point, including the local average coordinate offset of that point relative to all points within the Pillar. and the absolute offset of that point relative to the Pillar geometric center. and ; For each valid Pillar, limit the maximum number of points N it contains, apply a simplified version of the PointNet structure to encode each augmented point in the Pillar and perform multilayer perceptual operations to map the feature dimension of the point from D dimensions to C dimensions, and then aggregate the features of all points in the Pillar through the Max Pooling operation to obtain a fixed-length feature vector. An optimized 2D feature extraction network is used to extract features from pseudo-image feature maps at multiple scales. This network includes a top-down network and a feature pyramid network (FPN). It generates feature maps at different scales by downsampling layer by layer, and then fuses the deep and shallow features through upsampling and feature concatenation operations to improve them to a uniform BEV resolution. S3: To address the issues of sensor extrinsic drift and timing inconsistency that occur during long-term vehicle operation, environmental temperature changes, or high-speed motion, an online learnable multi-sensor calibration mechanism based on BEV geometric consistency is introduced. To address the aforementioned issues, a joint constraint model is constructed using three types of data: IMU, visual, and LiDAR. First, continuous measurements from the IMU are used to predict pose increments. Then, combined with traditional visual reprojection errors, high-frequency motion constraints are provided for the pose increments of the camera and LiDAR. Crucially, a BEV spatial alignment error is introduced. This error is calculated by directly comparing the structural differences between the camera stream BEV feature map and the radar stream BEV feature map generated in step S2 in the overlapping region, thus directly quantifying the degree of misalignment in the BEV projection. Subsequently, by minimizing the joint residual model, which includes BEV spatial alignment error, the system can utilize geometric deviation information at the perception level to guide the adjustment of the extrinsic parameter matrix. This process achieves a closed-loop feedback of "perception guiding calibration, and calibration enhancing perception," dynamically compensating for errors caused by vibration, drift, or temporal offset. The corrected extrinsic parameter matrix is ​​then fed back to the image BEV conversion module in real time. Through this design, the system achieves continuous pixel-level geometric consistency assurance for multimodal data: it effectively solves the misalignment problem of BEV projection and significantly enhances the system's perception robustness and stability in complex environments, extreme motion, or long-term operation scenarios. The corrected, highly aligned BEV feature map provides the most reliable and accurate input foundation for the physical prior-driven cross-modal Transformer fusion in subsequent step 4; Step S31: To achieve online joint calibration of the camera, IMU, and LIDAR, this invention defines the system state variables as follows: ; in For the camera's posture, Indicates the camera at the f pose at frame time, Indicates the first s The signpost, i.e., the sliding window from the first... f The s-th landmark in frame s is the starting point. N is the number of keyframes in the sliding window, M is the number of landmarks observed by all keyframes in the sliding window, and λ is the inverse depth. include i The position p, attitude q, and velocity v of the IMU body in the inertial coordinate system at any given time, as well as the estimated acceleration bias in the IMU body coordinate system. And angular velocity bias estimation Represents the acceleration and angular velocity of the camera at time i. Let w represent the acceleration and angular velocity of the camera at time i, where w represents the angular velocity and b represents the acceleration. Represents the velocity at time i. This represents the estimated acceleration bias at time i in the IMU body coordinate system. This is expressed as the estimate of the angular velocity offset at time i in the IMU body coordinate system; Step S32: Error Calculation: Receive the camera stream BEV feature map and radar stream BEV feature map generated in step S2, and combine them with IMU data to construct a joint optimization objective function that includes visual reprojection error, IMU pre-integration error and BEV spatial alignment error. The BEV spatial alignment error is obtained by calculating the structural difference between the two modal feature maps in the overlapping field of view. Step S33: Closed-loop correction: The above joint error is minimized by using a nonlinear optimization method, the relative pose transformation matrix of the camera to the LiDAR is solved in real time, and the projection matrix in step S21 is dynamically updated using the corrected extrinsic parameters. The above-mentioned visual reprojection error is constructed as follows: Define feature point s as being first observed in frame i and then observed again in frame j, using the inverse depth λ of frame i. s With relative pose transformation, feature points are projected onto the normalized camera plane of the j-th frame to construct the reprojection error: ; in Let be the observed coordinates of feature point s in the j-th frame of the image. , Let be the projection function of the camera. The extrinsic parameter matrix from the camera coordinate system to the IMU coordinate system. Let be the pose transformation matrix of the IMU in the world coordinate system for the j-th frame and the i-th frame. Let be the coordinates of feature point s in the normalized plane of the i-th frame; Inverse depth parameterization: The relationship between the coordinates of the feature points in the normalized camera coordinate system and their coordinates in the camera coordinate system is as follows: ; Here, λ=1 / z is called the inverse depth. The reason for using the inverse depth of 1 / z is that when z→∞, x / z and y / z cannot be represented, while using the inverse depth, λ→0, which can represent any value well. Moreover, when using the inverse depth, the reprojection error function is closer to the Gaussian distribution. Cross-frame reprojection: The inverse depth of feature points in the th It was first observed in frame 1, and then again in frame 2. It was observed again in the frame and predicted to be in the [frame number]. j The coordinates at time point are: ; Let J be the camera coordinates at time j. The extrinsic parameters from the camera coordinate system to the IMU coordinate system. The first Frame and the The transformation matrix from the IMU coordinate system to the world coordinate system of the frame. Let the inverse depth of the feature point at time i be denoted as . For the first Observations of the frame; IMU pre-integration constraints: The introduction of IMU pre-integration constraints aims to effectively utilize high-frequency IMU measurements in vision-inertial fusion, thereby stably and accurately estimating the camera pose during rapid motion. The relationship between IMU measurements and true values ​​is as follows: ; in and These are the angular velocity and acceleration biases, respectively. and To measure noise, For the rotation from the world coordinate system to the machine coordinate system; The dynamic constraint formula is: ; Motion constraints are obtained by integrating the acceleration and angular velocity of the IMU in two keyframes, where... The quaternion derivative formula reflects the equation of motion for rotation, ultimately yielding the constraint residuals of the integral: ; By mathematically compressing high-speed, continuous acceleration and angular velocity measurements between two keyframes, a "relative motion constraint" independent of the specific initial pose is obtained. Since the IMU's sampling frequency is much higher than that of the camera, pre-integration avoids re-integrating a large amount of original IMU data during each optimization, thus significantly reducing the computational load. Simultaneously, pre-integration explicitly models acceleration bias, gyroscope bias, and noise, ensuring that translation, rotation, and velocity changes between keyframes can be accurately represented. The alignment constraint between LIDAR and BEV is defined as follows: Projecting the radar point cloud onto the BEV mesh yields... Projecting camera features onto the BEV mesh yields... Alignment error is defined as the geometric difference between two modal features in the overlapping region: ; Minimizing this error ensures that the visual features remain geometrically consistent with the radar features in the BEV space; Multi-sensor joint optimization: The system state vector is jointly optimized by globally minimizing all error terms, including those from IMU, vision, and LIDAR, as follows: ; Where β is the set of IMU pre-integration terms, C is the set of visual observation terms, and is the set of all observed feature points. () is the Huber robust kernel function and These are the IMU pre-integration covariance matrix and the visual observation covariance matrix, respectively, used to adjust the weights of each error term. By solving this objective function, the spatial misalignment noise commonly found in feature fusion is eliminated, providing the purest and most reliable structured input for subsequent fusion. This allows the Transformer to focus on learning the semantic and motion features of the target without expending computational power to fit incorrect geometric positions. S4: Physics-prior-driven cross-modal Transformer fusion, specifically: S41: Physics Query Construction: Based on the real-time physical state of the vehicle, a dynamic query is constructed to reflect the motion characteristics of the target object. The physical information of each query is passed to the Transformer as part of the input. By fusing state data such as the target's velocity, acceleration, and heading angle, a physical state vector is constructed, in the following form: ; in Includes the vehicle's speed in the x and y directions. Includes the vehicle's acceleration in the x and y directions. The vehicle's heading angle. The cosine and sine values ​​represent the heading angle and are used to represent the direction vector; The initial query encoding is then combined with the physical state vector. Combined, the final enhanced query encoding is obtained. Attention mechanisms used to guide Transformer, enhancing query encoding. ; in It is the basic positional encoding of the i-th query. For the concatenation operation; MLP (Multilayer Perceptron) is used to perform feature mapping and dimensionality reduction on the concatenated high-dimensional vector, and to fuse heterogeneous physical state and geometric position information into a unified, semantically rich query code; S42: Introduce a deformable attention mechanism, which enables the attention mechanism to dynamically adjust its region of interest according to the spatial distribution of different modal features. Use the dynamic query constructed in step S41 to guide sampling, form an inertial guidance field, and achieve feature foresight. The core of this deformable attention mechanism lies in the introduction of a sampling offset. It enables the alignment between the Query vector and the Key vector to be dynamically adjusted as the target moves and is positioned. By calculating the sampling offset through MLP, the sampling points of the attention mechanism are adaptively offset along the predicted motion trajectory to obtain the corrected key representation. The sampling offset is in the following form: ; in To enhance the query vector, The j-th feature vector to be aggregated contains raw information from both camera stream BEV features and radar stream BEV features. By introducing an offset, we make the spatial relationship between the query and the key no longer fixed, but dynamically adjustable according to the target's motion and position. because It includes the motion state, therefore the calculated With motion foresight, the Transformer can calculate an adaptive displacement based on the motion trend reflected in the Query, shifting the sampling point from its current static position to the predicted position where the target is most likely to appear in the next moment. We add this to the original key vector. The new correction key representation is obtained. : ; The original key vector is used; this correction will make the target dynamic position pointed to by the query after motion compensation more closely in space, thus enabling the attention mechanism to perform cross-modal feature fusion more accurately.

[0020] S43: Feature Aggregation: Based on the offset sampling points, key features are aggregated on the calibrated camera and LiDAR BEV feature maps, attention weights are calculated, and the features are weighted and summed, as shown in the following formula: ; in This represents the fused feature vector, which contains feature information from different modalities. The similarity between the query and the offset key determines the query vector. For different key vectors The level of attention; Features from both the camera and radar are included. The similarity between the query and the offset key is calculated, and then the similarity is weighted and summed with the value vector to obtain the fused features. S44: Deep Encoding: The fused features are input into a multi-layer encoder of the Transformer. Each encoder layer further processes the features through self-attention to extract deeper semantic information and spatial features. The process is as follows: ; Q is the query matrix generated from the fused features, K is the offset key matrix, and V is the value matrix composed of continuously updated camera and radar features. The dimension of the key vector is used to control the scaling factor; After passing through the self-attention layer, the features are fed into the feedforward network for nonlinear transformation to extract more abstract and deeper semantic information and spatial features. To ensure network stability and training efficiency, and to avoid gradient vanishing or exploding in deep networks, standard residual connections and layer normalization are used after each operation. ; y is the input of the current layer (i.e., the output of the previous layer). For residual connections, LayerNorm ensures the stability of the mean and variance of features across different dimensions; FFN (Feedforward Network) contains two fully connected layers and a non-linear activation function, extracting deeper semantic information; The final output contains highly fused BEV features with deep semantic representation, providing the most reliable input for the final detection task.

[0021] This embodiment provides an adaptive calibration and physically guided multimodal data fusion sensing system to implement the above method, including: Multi-view image feature projection transformation module: used to extract deep semantic features from multi-view cameras, and use depth estimation technology to "enhance" the two-dimensional image features to three-dimensional space, and project to generate camera flow BEV feature maps; The LiDAR point cloud feature encoding module is used to voxelize the original sparse laser point cloud, extract spatial features containing precise geometric structure information, and map them into radar flow BEV feature maps. The multi-sensor online adaptive calibration module based on BEV consistency is used to construct a joint optimization model that includes BEV spatial alignment error, calculate the structural consistency difference between the camera stream BEV feature map and the radar stream BEV feature map in the overlapping area in real time, and correct the extrinsic matrix between the camera and LiDAR in real time by minimizing the alignment error. Physics-prior-driven cross-modal deformable Transformer fusion module: for deep feature fusion within a calibrated unified BEV space.

[0022] This invention introduces a deformable attention mechanism, enabling a cross-modal feature fusion module to achieve precise alignment and fusion of features from different modalities. By utilizing physically-enhanced Query and deformable attention, the Transformer can adaptively adjust the attention mechanism according to the dynamic changes of the target object, thereby improving the target detection accuracy of the autonomous driving perception system.

[0023] By implementing this invention, users can alleviate two key problems existing in the current autonomous BEV framework: (1) Failure of static calibration: Most existing BEV perception schemes rely on offline calibration. However, during long-term vehicle operation or severe bumps, the extrinsic parameters of the camera and LiDAR will drift slightly, causing the features projected onto the BEV plane to be geometrically misaligned (Ghosting Effect), which in turn causes fusion failure.

[0024] (2) Lag of dynamic features: Traditional Transformer uses random or fixed sampling points during fusion, ignoring the vehicle's own kinematic state. In high-speed or turning scenarios, the perception results often have "mog" or position lag, and cannot accurately capture the instantaneous position of the target.

[0025] Through these two innovations, this framework ensures high-precision temporal and spatial consistency of the fused features. This enables the BEV algorithm to effectively eliminate the positional lag of dynamic targets during target detection and trajectory prediction, outputting accurate target features after temporal compensation. Ultimately, this makes the algorithm more robust and precise in detecting and predicting target regions, significantly enhancing the perception system's ability to understand complex dynamic environments and fundamentally improving the overall performance and reliability of the BEV detection algorithm.

[0026] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0027] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal data fusion sensing method with adaptive calibration and physical guidance, characterized in that, include: S1: Dataset Input: Introduce the nuscenes dataset to obtain RGB image data from multi-view cameras, sparse point cloud data from LiDAR, acceleration and angular velocity data from the inertial measurement unit, and kinematic state information fed back by the vehicle. S2: Dual-stream feature extraction and BEV spatial projection; S21: Image Branch: Input multi-view images into the view encoder and use ResNet as the backbone network to extract deep semantic features; Subsequently, the Lift-Splat-Shoot algorithm is used to estimate the frustum depth, "lifting" the 2D image features to 3D space and projecting them onto a unified BEV plane to generate the initial camera flow BEV feature map. S22: Radar Branch: The original LiDAR point cloud is divided into vertical cylinders. The PointNet network is used to encode the local geometric features within each cylinder to generate fixed-dimensional cylinder features. Then, according to their physical coordinate index on the plane, they are scattered and mapped into a regular two-dimensional grid to generate radar flow BEV feature map. S3: Online calibration of multiple sensors based on BEV consistency: Step S31: Error Calculation: Receive the camera stream BEV feature map and radar stream BEV feature map generated in step S2, and combine them with IMU data to construct a joint optimization objective function that includes visual reprojection error, IMU pre-integration error and BEV spatial alignment error. The BEV spatial alignment error is obtained by calculating the structural difference between the two modal feature maps in the overlapping field of view. Step S32: Closed-loop correction: The above joint error is minimized by using a nonlinear optimization method, the relative pose transformation matrix of the camera to the LiDAR is solved in real time, and the projection matrix in step S21 is dynamically updated using the corrected extrinsic parameters and applied to the current frame. S4: Physics-prior-driven cross-modal Transformer fusion, specifically: S41: Physical Query Construction: Based on the real-time physical state of the vehicle, a dynamic Query code is constructed. The initial Query code is concatenated with the physical state vector and then subjected to feature mapping and dimensionality reduction processing through an MLP (Multilayer Perceptron) to obtain an enhanced Query code. S42: Inertial Guided Sampling: An inertial guided field mechanism is introduced. When calculating deformable attention, the potential position of the target in the next frame is predicted by the dynamic query. The sampling offset is calculated by MLP. The sampling points of the attention mechanism are adaptively offset along the predicted motion trajectory direction to obtain the corrected key representation. S43: Feature aggregation: Based on the offset sampling points, aggregate key features on the calibrated camera and LiDAR BEV feature maps, calculate attention weights, and perform weighted summation of features; S44: Deep Encoding: The fused features are input into the multi-layer encoder of the Transformer. After self-attention processing, feedforward nonlinear transformation, residual connection and layer normalization, the fused features are output to the detection head to obtain the 3D target detection result.

2. The adaptive calibration and physically guided multimodal data fusion sensing method according to claim 1, characterized in that, The specific process of using the Lift-Splat-Shoot algorithm to transform multi-view image features into a unified BEV feature map in step S21 is as follows: 2D to 3D transformation (Lift): Processes high-dimensional image features to predict a set of discrete depth probability distributions for each pixel location in the image. This depth probability is used as a weighting factor to multiply the semantic features of the pixel, "lifting" the original two-dimensional features into probability-weighted features in a 3D view frustum. Feature projection (Splat): Projects and aggregates all probability-weighted features of a 3D view frustum into a unified 3D mesh space based on their positions in the world coordinate system. BEV Mapping (Shoot): The features in the 3D mesh are compressed and summarized along the direction perpendicular to the ground. The compressed feature layer is then mapped onto the BEV plane to generate a camera flow BEV feature map.

3. The adaptive calibration and physically guided multimodal data fusion sensing method according to claim 1, characterized in that, In step S22, the specific processing procedure for the radar branch also includes: Add an extra dimensional feature to each LiDAR point, including the local average coordinate offset of that point relative to all points within the Pillar. and the absolute offset of that point relative to the Pillar geometric center. and ; For each valid Pillar, limit the maximum number of points N it contains, apply a simplified version of the PointNet structure to encode each augmented point in the Pillar and perform multilayer perceptual operations to map the feature dimension of the point from D dimensions to C dimensions, and then aggregate the features of all points in the Pillar through the Max Pooling operation to obtain a fixed-length feature vector. An optimized 2D feature extraction network is used to extract features from pseudo-image feature maps at multiple scales. This network includes a top-down network and a feature pyramid network (FPN). It generates feature maps of different scales by downsampling layer by layer, and then fuses the deep and shallow features through upsampling and feature concatenation operations to improve them to a uniform BEV resolution.

4. The adaptive calibration and physically guided multimodal data fusion sensing method according to claim 1, characterized in that, The visual reprojection error in step S3 is constructed as follows: Feature point s is defined as being first observed in frame i and observed again in frame j. The inverse depth λ of frame i is then used... s With relative pose transformation, feature points are projected onto the normalized camera plane of the j-th frame to construct the reprojection error: ; in Let be the observed coordinates of feature point s in the j-th frame of the image. , Let be the projection function of the camera. The extrinsic parameter matrix from the camera coordinate system to the IMU coordinate system. Let be the pose transformation matrix of the IMU in the world coordinate system for the j-th frame and the i-th frame. Let be the coordinates of feature point s in the normalized plane of the i-th frame; IMU pre-integration constraints are implemented in the following way: The relationship between IMU measurements and true values ​​is as follows: ; in and These are the angular velocity and acceleration biases, respectively. and To measure noise, For the rotation from the world coordinate system to the machine coordinate system; The dynamic constraint formula is: ; Motion constraints are obtained by integrating the acceleration and angular velocity of the IMU in two keyframes, where... The quaternion derivative formula reflects the equation of motion for rotation, ultimately yielding the constraint residuals of the integral: 。 5. The adaptive calibration and physically guided multimodal data fusion sensing method according to claim 1, characterized in that, In step S3, the alignment constraint between LIDAR and BEV is defined as follows: Projecting the radar point cloud onto the BEV mesh yields... Projecting camera features onto the BEV mesh yields... Alignment error is defined as the geometric difference between two modal features in the overlapping region: ; Minimizing this error allows the visual features to maintain geometric consistency with the radar features in the BEV space.

6. The adaptive calibration and physically guided multimodal data fusion sensing method according to claim 1, characterized in that, The physical state vector in step S4 has the following form: ; in Includes the vehicle's speed in the x and y directions. Includes the vehicle's acceleration in the x and y directions. The vehicle's heading angle. The cosine and sine values ​​of the heading angle are used to represent the direction vector; enhanced query encoding. ,in It is the basic positional encoding of the i-th query. For splicing operations; The sampling offset is in the following form: ; in To enhance the query vector, The j-th feature vector to be aggregated contains raw information from both camera stream BEV features and radar stream BEV features; the correction key represents... , This is the original key vector.

7. The adaptive calibration and physically guided multimodal data fusion sensing method according to claim 1, characterized in that, In step S4, the attention weights are calculated and the features are weighted and summed, using the following formula: ; in This represents the fused feature vector, which contains feature information from different modalities. The similarity between the query and the offset key determines the query vector. For different key vectors The level of attention; Features from both the camera and radar are incorporated. The similarity between the query and the offset key is calculated, and then weighted and summed with the value vector to obtain the fused features. These fused features are then input into a multi-layer encoder of a Transformer for processing. Each encoder layer further processes the features through self-attention, thereby extracting deeper semantic information and spatial features. The process is as follows: ; Q is the query matrix generated from the fused features, K is the offset key matrix, and V is the value matrix composed of continuously updated camera and radar features. The dimension of the key vector is used to control the scaling factor; After passing through the self-attention layer, the features are fed into a feedforward network for nonlinear transformation to extract more abstract and deeper semantic information and spatial features: ; y is the input of the current layer (i.e., the output of the previous layer). For residual connections, LayerNorm ensures the stability of the mean and variance of features across different dimensions; FFN (Feedforward Network) contains two fully connected layers and a non-linear activation function, extracting deeper semantic information; The final output contains highly fused BEV features with deep semantic representation, providing the most reliable input for the final detection task.

8. An adaptive calibration and physically guided multimodal data fusion sensing system, implementing the method of any one of claims 1-7, characterized in that, include: Multi-view image feature projection transformation module: used to extract deep semantic features from multi-view cameras, and use depth estimation technology to "enhance" two-dimensional image features to three-dimensional space, and project to generate camera flow BEV feature maps; The LiDAR point cloud feature encoding module is used to voxelize the original sparse laser point cloud, extract spatial features containing precise geometric structure information, and map them into radar flow BEV feature maps. The multi-sensor online adaptive calibration module based on BEV consistency is used to construct a joint optimization model that includes BEV spatial alignment error, calculate the structural consistency difference between the camera stream BEV feature map and the radar stream BEV feature map in the overlapping area in real time, and correct the extrinsic matrix between the camera and LiDAR in real time by minimizing the alignment error. Physics-prior-driven cross-modal deformable Transformer fusion module: for deep feature fusion within a calibrated unified BEV space.

Images