Three-dimensional target detection method and system based on dynamic weight bidirectional cross-modal fusion

Abstract

The application discloses a kind of bidirectional cross-modal fusion three-dimensional target detection method and system based on dynamic weight, belong to the field of automatic driving technology.The application is by constructing bidirectional cross-modal attention interaction of not sharing parameter, independent learning respectively in BEV space, fully exploit the complementary potential of two kinds of modal;On this basis, the physical quality index corresponding to each other is extracted from original image and original point cloud frame by frame, and is regressed into quality score by light-weight network, and is mapped into double-modal fusion coefficient by smooth gate;Meanwhile, set the lower threshold of quality score, when the quality score of a certain mode is lower than the threshold, its fusion coefficient is smoothly attenuated to zero, and the system is automatically converted into single-mode detection mode dominated by another available mode.The application improves the precision and robustness of three-dimensional target detection in complex traffic environment and sensor degradation condition by bidirectional cross-modal feature enhancement and quality adaptive dynamic fusion.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights. Background Technology

[0002] For autonomous driving systems, accurate perception of the surrounding traffic environment is a fundamental prerequisite for ensuring safe operation. Among these, 3D object detection, as a core component of the perception process, is responsible for accurately quantifying the spatial position, geometric posture, and physical dimensions of traffic participants such as vehicles, pedestrians, and bicycles on the road. Since the output of this component directly provides decision-making support for downstream planning and control modules, its accuracy and reliability directly impact the driving safety of autonomous vehicles in real-world road environments.

[0003] Current mainstream autonomous driving perception solutions generally use vehicle-mounted cameras and LiDAR as the main sensors. Vehicle-mounted cameras can capture images rich in texture, color, and semantic information, offering the advantage of high information density. However, due to limitations in their imaging principles, they have inherent errors in depth estimation and long-range localization, and are susceptible to adverse weather conditions such as low light, strong light, and rain / fog. LiDAR, on the other hand, actively emits laser pulses to directly acquire the three-dimensional geometric coordinates of a target, offering high measurement accuracy and minimal susceptibility to lighting conditions. This effectively compensates for the shortcomings of vehicle-mounted cameras in depth estimation and long-range localization. However, its point cloud data is sparse, making it difficult to provide stable and rich semantic information, and its hardware cost is relatively high. Therefore, vehicle-mounted cameras and LiDAR exhibit a natural complementary relationship in terms of information characteristics. A single-modal sensor cannot simultaneously achieve both semantic richness and geometric accuracy, and its perception accuracy and robustness in complex traffic scenarios are insufficient to meet the practical application requirements of autonomous driving systems.

[0004] To address the aforementioned limitations, cross-modal methods that integrate image semantic information and point cloud geometric information have become the mainstream technical approach for improving the perception accuracy and robustness of autonomous driving. These methods can be mainly categorized as follows: The first category is a single-stage fusion method based on Bird's-Eye View (BEV) space. This type of method maps camera images and LiDAR point clouds to a unified BEV space, aligns and fuses dual-modal features in the BEV space, and achieves end-to-end 3D target detection. For example, BEVFusion uses a dual-branch structure to process images and point clouds separately. It projects image features to the BEV space through Lift-Splat-Shoot and combines voxel features from the point cloud to perform feature stitching and fusion in the BEV space.

[0005] The second category is unidirectional fusion methods based on attention mechanisms. These methods utilize the attention mechanism of Transformers to adaptively determine what information to extract from the image and from what location, achieving robust fusion under conditions of image quality degradation. For example, TransFusion uses query vectors generated from LiDAR features as the primary source and adaptively extracts semantic information from image features through a cross-attention mechanism, achieving unidirectional information supplementation of the point cloud from the image.

[0006] The first and second types of methods mentioned above are essentially unidirectional guided fusion strategies, which can only achieve unidirectional information supplementation from one modality to another. They are difficult to simultaneously enhance the geometric features of point clouds with image semantic information and correct the spatial position of image features with point cloud geometric information.

[0007] To address this, researchers have further developed a third type of fusion method employing a bidirectional interaction strategy. This type of method establishes a bidirectional information flow channel between the image and the point cloud, achieving bidirectional dynamic compensation and symmetry enhancement of geometric information and semantic features. Based on the different implementation levels and focuses of the bidirectional interaction, existing methods can be further divided into cross-modal bidirectional attention interaction and intra-modal bidirectional feature enhancement. The former, such as the Chinese invention patent with authorization announcement number CN120894759A and the MTC-BEV method, establishes bidirectional attention interaction between image features aligned at the feature level and point cloud features, enabling mutual enhancement of information from the two modalities and effectively improving the completeness of small target detection and the stability of distant target perception in occluded scenes. The latter, such as the Chinese invention patent with authorization announcement number CN119445536B, employs a local feature encoding / decoding network in the point cloud feature extraction stage, achieving bidirectional feature enhancement within the point cloud through self-attention encoding and cross-attention decoding, improving the ability to acquire features of foreground targets in complex backgrounds.

[0008] In addition, Chinese invention patent with authorization announcement number CN121837869B introduces the current quality signal constructed from the initial detection results of each modality, and dynamically allocates feature fusion weights in temporal fusion and cross-modal fusion to improve the detection robustness in sensor degradation scenarios; Chinese invention patents with authorization announcement numbers CN116403186B and CN119445534B explore cascaded fusion and temporal information fusion from the perspectives of respectively.

[0009] Despite the significant progress made by the above methods in cross-modal fusion, the following shortcomings still exist: First, the bidirectional interaction capability is limited, making it difficult to achieve differentiated enhancement and correction. Existing bidirectional interaction methods mostly adopt symmetric attention structures with shared parameters, resulting in convergent functional goals in both directions. Specifically, the image-to-point-cloud direction should focus on enhancing the class discrimination of the point cloud using texture semantics, while the point-cloud-to-image direction should focus on correcting the spatial localization deviation of the image using geometric structure. However, existing technologies do not set up mechanisms for non-shared parameters and independent learning for the two directions, resulting in the inability to optimize the functions of the two directions independently according to their respective goals, and the potential for bidirectional complementarity has not been fully explored.

[0010] Second, the fusion weights lack awareness of sensor quality. Specifically, existing methods mostly use globally fixed weights or simple feature splicing for fusion, without incorporating sensor information quality into the fusion model. Although the Chinese invention patent with authorization announcement number CN121837869B introduces a quality signal constructed from the initial detection results for dynamic weight allocation, its quality signal comes from the initial detection results, rather than interpretable physical indicators of the original sensor data. Once a certain mode degrades due to severe weather, sensor anomalies, or sparse point clouds, this method is unlikely to complete pre-detection quality awareness before detection, and may still introduce noise from low-quality modes into the fusion process.

[0011] Third, there is a lack of a continuous and smooth transition mechanism when a mode degrades. Specifically, existing methods usually use hard truncation or simple discarding when a certain mode degrades severely. This approach can lead to abrupt changes in fusion weights and jumps in detection results, which seriously affect the consistency of behavior and the reliability of decision-making of autonomous driving systems under sensor performance boundary conditions.

[0012] Therefore, this invention proposes a bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights, which can effectively solve the problems of limited bidirectional interaction capability, lack of perception of the quality of original sensor data by fusion weights, and lack of continuous and smooth transition mechanism when modal degradation exists in the prior art. Summary of the Invention

[0013] The purpose of this invention is to provide a bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights, which can effectively improve the detection accuracy and robustness of autonomous driving perception systems under complex environments and sensor degradation conditions.

[0014] To achieve the above objectives, the technical solution adopted by this invention is: a bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights, comprising the following steps: S1. Data Acquisition: Acquire images captured by the vehicle-mounted camera and point cloud data collected by the LiDAR; S2. Feature extraction and bird's-eye view spatial alignment: Features are extracted from camera images and LiDAR point clouds respectively, and the two heterogeneous modalities are mapped to a unified bird's-eye view space to obtain strictly spatially aligned image bird's-eye view features and point cloud bird's-eye view features. S3. Bidirectional cross-modal feature enhancement: Construct bidirectional attention interaction, and achieve mutual enhancement of bimodal features through semantically enhanced geometric features and geometrically corrected semantic features. In the bidirectional attention interaction, the two directions do not share parameters and learn independently. S4. Dynamic weighted fusion: The quality scores of the original image and the original point cloud are evaluated frame by frame and mapped to dual-modal fusion coefficients through smooth gating. S5. Dynamic weighted fusion: Based on the dual-modal fusion coefficient, the bird's-eye view features of the image and the bird's-eye view features of the point cloud are fused to obtain a unified fused bird's-eye view feature. S6. 3D Object Detection: The fused bird's-eye view features are input into the 3D object detection head. The target category is predicted and its 3D attribute parameters are regressed through the anchor-free center point detection paradigm. The output is a 3D object detection box containing position, size, category, orientation, and velocity.

[0015] Further, in step S2, the method for generating the image bird's-eye view features is as follows: Multi-scale features are extracted from camera images using a two-dimensional backbone network; The depth interval is discretized into multiple depth intervals, and the discrete depth probability distribution is estimated for each pixel through a depth prediction branch; the two-dimensional features are expanded along the depth direction by an outer product to obtain three-dimensional frustum features; The view frustum feature points are transformed from the camera coordinate system to the vehicle coordinate system using camera intrinsic and extrinsic parameters. In the vehicle coordinate system, the bird's-eye view plane is discretized into a regular grid. All view frustum feature points falling into the same bird's-eye view grid are aggregated along the height direction to obtain the bird's-eye view features of the image.

[0016] Furthermore, in step S2, the method for generating the point cloud bird's-eye view features is as follows: The 3D point cloud collected by the lidar is input into the point cloud branch, and the space is divided into regular grids at a set resolution on the bird's-eye view plane, and then expanded into columnar bodies along the height direction; Each point is assigned to a corresponding column according to its planar coordinates; For each point within the column, based on the original features, the offset relative to the geometric center of the column and the offset relative to the mean of the point cloud within the column are spliced ​​together to obtain the augmented features. The columnar features are obtained by transforming the shared multilayer perceptron and then performing max pooling. Based on the grid index of each column on the plane, its features are scattered to the corresponding positions of the bird's-eye view feature tensor; The feature tensors of the bird's-eye view are encoded at multiple scales and fused through a feature pyramid network to obtain the point cloud bird's-eye view features.

[0017] Furthermore, in step S3, the specific steps of the bidirectional cross-modal feature enhancement are as follows: The aligned image bird's-eye view features and point cloud bird's-eye view features are flattened into a sequence along the spatial dimension to obtain the flattened image features and point cloud features. Using the flattened point cloud features as the query and the flattened image features as the key and value, after learningable linear projection, the scaled dot product of the query and the key and value is calculated. The attention weight is obtained through a normalized exponential function. Then, the values ​​are weighted and aggregated according to the attention weight of each value to obtain the point cloud features after image feature enhancement. Using the flattened image features as the query and the flattened point cloud features as the key and value, after learning linear projection, the scaled dot product of the query and the key and value is calculated. The attention weight is obtained through a normalized exponential function. Then, the values ​​are weighted and aggregated according to the attention weight of each value to obtain the image features after point cloud feature enhancement. The point cloud features enhanced by the image features and the image features enhanced by the point cloud features are subjected to residual connection and normalization processing respectively to obtain the point cloud bird's-eye view features and the image bird's-eye view features after bidirectional enhancement and normalization.

[0018] Furthermore, in step S4, the dynamic weight fusion specifically includes: Image quality index vectors are constructed by extracting contrast, sharpness, and wavelet domain signal-to-noise ratio from image modalities. Voxel point density, echo intensity information entropy, and geometric consistency index are extracted from point cloud modalities to form a point cloud quality index vector; a lightweight quality mapping network performs nonlinear regression on the quality index vector to output image quality score and point cloud quality score frame by frame. A lower threshold for quality score is set, and the quality scores of each modality are processed by a smoothing gating function. The processed quality scores are then normalized to obtain the fusion coefficients of the image modality and the point cloud modality.

[0019] Furthermore, the smoothing gate function is the Sigmoid function, whose expression is:

[0020] in, For the Sigmoid function, A coefficient used to control the steepness of attenuation.

[0021] A bidirectional cross-modal fusion 3D target detection system based on dynamic weights, comprising: The input module, including an onboard camera and a LiDAR, is used to acquire RGB images from multiple perspectives around the autonomous vehicle and 3D point cloud data collected by the LiDAR. The BEV feature extraction and alignment module includes an image branch module and a point cloud branch module, which are used to extract features from camera images and LiDAR point clouds, respectively, and map the two heterogeneous modalities to a unified bird's-eye view space, providing strictly aligned feature representations for subsequent cross-modal interactions. The bidirectional cross-modal feature enhancement module is used to construct bidirectional attention interaction to achieve mutual enhancement of bimodal features; The dynamic weighted fusion module is used to perform quality-adaptive weighted fusion of bi-modal bird's-eye view features after bi-directional enhancement and normalization, and to trigger continuous and stable single-modal degradation when modality degrades. The output module includes a 3D target detection head and a 3D detection box output end, which is used to output the 3D target detection results.

[0022] Furthermore, the bidirectional cross-modal feature enhancement module includes: The first attention unit is used to integrate semantic information of the image into the point cloud features through a cross-attention mechanism, using point cloud bird's-eye view features as queries and image bird's-eye view features as keys and values, thereby achieving semantically enhanced geometry. The second attention unit is used to integrate point cloud geometric information into image features through a cross-attention mechanism, using image bird's-eye view features as queries and point cloud bird's-eye view features as keys and values, to achieve geometric correction semantics. The first attention unit and the second attention unit do not share parameters and learn independently.

[0023] Furthermore, the dynamic weight fusion module includes: The image quality assessment submodule is used to extract contrast, sharpness and wavelet domain signal-to-noise ratio indicators from the original image, and then regress them into an image quality score through a lightweight quality mapping network. The point cloud quality assessment submodule is used to extract voxel density, echo intensity information entropy and geometric consistency index from the original point cloud, and then regress them into point cloud quality score through a lightweight quality mapping network. The fusion coefficient mapping submodule is used to set the lower threshold of the quality score, process and normalize the quality scores of each mode through a smoothing gating function, and obtain the fusion coefficient. The weighted fusion submodule is used to perform element-wise weighted summation of the bidirectional enhanced and normalized image bird's-eye view features and the point cloud bird's-eye view features based on the fusion coefficients, so as to generate fused bird's-eye view features.

[0024] The beneficial effects of the above technical solution are as follows: 1. This invention achieves differentiated feature enhancement and correction through bidirectional cross-modal attention interaction without sharing parameters, fully exploring the complementary potential of images and point clouds. Specifically, existing bidirectional interaction methods mostly adopt symmetrical attention structures with shared parameters, resulting in convergent functional objectives in both directions and an inability to independently optimize for different objectives. To address this issue, this invention sets the attention interaction in both directions to be parameter-free and to learn independently. The first direction uses point cloud features as queries and image features as keys and values, leveraging the texture and semantic information of the image to enhance the class discrimination power of point cloud features, achieving semantic enhancement geometry. The second direction uses image features as queries and point cloud features as keys and values, utilizing the three-dimensional geometric structure of the point cloud to correct the spatial positioning deviation of image features, achieving geometric correction semantics. The two directions are independently optimized for their respective functional objectives, achieving truly differentiated collaborative enhancement.

[0025] 2. This invention achieves quality perception before detection through pre-detection quality assessment, avoiding the introduction of low-quality modal noise. Specifically, existing methods mostly use globally fixed weights or simple feature splicing for fusion, without incorporating sensor information quality into the fusion model. Although some methods introduce quality signals, these signals originate from the initial detection results, i.e., post-detection, and rely on preliminary detection to complete quality judgment. In contrast, this invention directly extracts interpretable physical quality indicators such as contrast, sharpness, wavelet domain signal-to-noise ratio, voxel density, echo intensity information entropy, and geometric consistency from the original sensor data, and regresses them into frame-level quality scores through a lightweight network, thus achieving quality assessment before detection, i.e., pre-detection.

[0026] 3. This invention achieves smooth single-modal degradation with the fusion coefficient continuously decaying to zero through a joint design of Sigmoid smoothing gating and a lower threshold for quality score, ensuring the behavioral continuity of the autonomous driving system under sensor degradation conditions. Specifically, existing methods typically use hard truncation or simple discarding when a certain modality degrades severely, leading to abrupt changes in fusion weights and detection results. To address this issue, this invention sets a lower threshold for quality score and introduces a Sigmoid smoothing gating function to process the quality score. When the quality score is much higher than the threshold, the gating output approaches 1; when the quality score is much lower than the threshold, the gating output continuously approaches 0, exhibiting a smooth transition near the threshold. Simultaneously, when the fusion coefficient smoothly decays to zero with the gating output, the system automatically transforms into a single-modal detection mode dominated by another available modality. This continuous and smooth transition mechanism fundamentally avoids detection result jumps caused by abrupt changes in fusion coefficients, ensuring the behavioral consistency and decision reliability of the autonomous driving system under sensor performance boundary conditions. Attached Figure Description

[0027] Figure 1This is a schematic diagram of a bidirectional cross-modal fusion three-dimensional target detection system based on dynamic weights in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the overall process of a bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0029] It should be noted that, unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] To address the limitations of existing technologies, such as limited bidirectional interaction capabilities, lack of perception of the quality of raw sensor data in fusion weights, and lack of a continuous and smooth transition mechanism during modal degradation, this invention designs a bidirectional cross-modal fusion 3D target detection system based on dynamic weights, the structure of which is as follows: Figure 1 As shown, it includes an input module, a BEV feature extraction and alignment module, a bidirectional cross-modal feature enhancement module, a dynamic weight fusion module, and an output module.

[0031] The input module includes an onboard camera and a LiDAR, used to acquire RGB images from multiple perspectives around the autonomous vehicle, as well as 3D point cloud data collected by the LiDAR. The 3D point cloud contains the 3D spatial coordinates and reflection intensity information of targets in the scene.

[0032] The BEV feature extraction and alignment module includes an image branch module and a point cloud branch module, which are used to extract features from camera images and LiDAR point clouds, respectively, and map the two heterogeneous modalities to a unified BEV space, providing strictly aligned feature representations for subsequent cross-modal interactions. Specifically: The image branching module is used to extract multi-scale semantic features from camera images. It enhances two-dimensional image features to three-dimensional view frustum space through a view frustum feature enhancement method based on depth probability distribution. Then, it generates image BEV features through camera intrinsic and extrinsic parameter transformation and height dimension pooling compression.

[0033] The point cloud branch module is used to encode the LiDAR point cloud into columnar representations, regularizing the disordered point cloud into columnar representations. After the columnar features are extracted by a shared multilayer perceptron, they are scattered to a two-dimensional grid, and then further extracted by a two-dimensional backbone network to generate point cloud BEV features.

[0034] The bidirectional cross-modal feature enhancement module includes a first attention unit and a second attention unit, used to establish attention interactions in two directions within the BEV unified space. The two directions do not share parameters and learn independently. Specifically: The first attention unit is direction one, used to enhance geometric features with semantic features. Specifically, it uses point cloud BEV features as queries and image BEV features as keys and values. Through a cross-attention mechanism, it enables each point cloud location to adaptively aggregate the required semantic information from image features, thereby enhancing the class discrimination power of point cloud features.

[0035] The second attention unit is direction two, used to enhance semantic features with geometric features. Specifically, it uses image BEV features as queries and point cloud BEV features as keys and values. Through a cross-attention mechanism, it enables each image location to adaptively aggregate the required geometric information from the point cloud features, correcting the spatial positioning deviation of the image features.

[0036] Residual connectivity and normalization are introduced in both directions to obtain bidirectional enhanced and normalized point cloud BEV features and image BEV features, respectively.

[0037] The dynamic weighted fusion module performs quality-adaptive weighted fusion of the bidirectional enhanced and normalized bimodal BEV features, and triggers continuous and stable single-modal degradation when modal degradation occurs. It includes: The image quality assessment submodule is used to extract indicators such as contrast, sharpness, and wavelet domain signal-to-noise ratio from the original image, and then regress them to an image quality score between 0 and 1 through a lightweight quality mapping network.

[0038] The point cloud quality assessment submodule is used to extract indicators such as voxel density, echo intensity information entropy, and geometric consistency from the original point cloud, and then regress them to a point cloud quality score between 0 and 1 through a lightweight quality mapping network.

[0039] The fusion coefficient mapping submodule is used to set the lower limit threshold of the quality score, process the quality scores of each modality through the Sigmoid smoothing gating function, and normalize the processed quality scores to obtain the fusion coefficient.

[0040] The weighted fusion submodule is used to perform element-wise weighted summation of the bidirectional enhanced and normalized image BEV features and point cloud BEV features based on the fusion coefficients to generate fused BEV features.

[0041] The output module includes a 3D target detection head and a 3D detection box output end. Specifically: The 3D target detection head receives fused BEV features, adopts an anchorless center point detection paradigm, predicts the category center heatmap through convolutional layers, and regresses the target's center offset, 3D size, orientation angle, height, and velocity parameters.

[0042] The 3D detection box output is used to output the category label of each target and the 3D target detection box containing position, size, orientation and velocity information based on the prediction results of the 3D target detection head.

[0043] The above modules are connected in sequence. The original image and point cloud data enter the system through the input module. The BEV feature extraction and alignment module completes feature extraction and spatial alignment. The bidirectional cross-modal feature enhancement module completes mutual enhancement and correction of features. Then, the dynamic weight fusion module performs quality adaptive weighted fusion. Finally, the output module generates and outputs the 3D target detection results.

[0044] Based on the above system, this invention designs a bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights, such as... Figure 2 As shown, it includes the following steps: Step S1, Data Acquisition By using cameras and LiDAR mounted on autonomous vehicles, RGB images from multiple perspectives around the vehicle and 3D point cloud data containing precise 3D spatial coordinates and reflection intensity information of targets in the scene can be obtained.

[0045] Step S2: Feature Extraction and BEV Space Alignment First, features are extracted from both camera images and LiDAR point clouds. Then, these two heterogeneous modalities are mapped to a unified BEV space, generating strictly aligned image BEV features and point cloud BEV features, providing a unified spatial representation basis for subsequent cross-modal interactions. This process includes two sub-steps: image BEV feature construction and point cloud BEV feature construction.

[0046] S21, Image BEV Feature Construction The image BEV feature construction sub-step is used to extract multi-scale semantic features from camera images and upscale them to the BEV space, providing semantically rich image BEV features for subsequent cross-modal interactions. Specifically: Given an input image Two-dimensional feature representations are extracted through the backbone network:

[0047] In the formula, and These represent the height and width of the input image, respectively. Indicates the downsampling factor. Indicates the number of feature channels.

[0048] Since two-dimensional image features lack depth information and cannot be directly mapped to the BEV space, this sub-step further employs a view frustum feature enhancement method based on depth probability distribution to map the depth interval... Discretized For each depth interval, a discrete depth probability distribution is estimated for each pixel using the depth prediction branch. The expression for this distribution is:

[0049] In the formula, This represents a depth prediction head consisting of several convolutional layers. Represents pixels Two-dimensional feature representation.

[0050] After obtaining the depth probability distribution of each pixel, the two-dimensional features are expanded along the depth direction using an outer product method to obtain the three-dimensional view frustum features, the expression of which is:

[0051] In the formula, Represents pixels In depth Discrete depth probability at a given location. Represents pixels In depth The three-dimensional view frustum features at that location, Represents pixels semantic context features, This indicates the context feature prediction branch.

[0052] After obtaining the 3D view frustum features, they need to be transformed from the camera coordinate system to the vehicle coordinate system to achieve spatial uniformity. This is done using camera intrinsic parameters. With external references Each three-dimensional feature point in the view frustum Transformation from camera coordinate system to vehicle coordinate system:

[0053] In the formula, and Let represent the rotation matrix and translation vector from the camera to the vehicle coordinate system, respectively.

[0054] In the vehicle coordinate system, the bird's-eye view plane is discretized into a regular grid, and the elements falling into the same BEV grid are... All view frustum feature points are aggregated along the height direction to obtain the image BEV feature, which is expressed as:

[0055] In the formula, Indicates BEV grid Image BEV features, Indicates falling into the BEV grid The set of all view frustum feature points.

[0056] S22, Point Cloud BEV Feature Construction The point cloud BEV feature construction sub-step is used to extract BEV features from the original LiDAR point cloud, providing geometrically rich point cloud BEV features for subsequent cross-modal interaction. Specifically: First, given the original point cloud ,in, Let the coordinates of the point be 3D coordinates. To represent the reflection intensity, in terms of resolution on the bird's-eye view plane. Divide the space into a regular grid. It extends into a columnar shape along the height direction.

[0057] Then, each point According to its plane coordinates Distributed to the corresponding columnar bodies middle.

[0058] For each point within the column, based on the original features, the augmented features are obtained by stitching together the offsets relative to the geometric center of the column and the offsets relative to the mean of the point cloud within the column. .

[0059] Subsequently, after transformation using a shared multilayer perceptron and max-pooling aggregation, columnar-level features are obtained, the expression of which is:

[0060] In the formula, Indicates that it is located at the th A set of points within a columnar column. Represents columnar body The columnar level characteristics.

[0061] According to each column Grid index on a plane to its characteristics Scattered to BEV feature tensor The corresponding position:

[0062] Finally, for Multi-scale encoding is performed, and the BEV features of the point cloud branch are obtained after fusion through a Feature Pyramid Network (FPN) structure. .

[0063] Due to the BEV characteristics of the image BEV features of point cloud All are located on the same bird's-eye view plane in the vehicle coordinate system, with the same spatial range and grid resolution. Therefore, for any grid position, and Both represent the semantic and geometric features of the same physical location in the vehicle coordinate system, meaning they are strictly aligned in space, providing a unified spatial representation basis for bidirectional cross-modal attention interaction and fusion in subsequent steps.

[0064] Step S3: Bidirectional cross-modal feature enhancement By establishing two-way attention interaction in the BEV unified space, symmetrical enhancement and correction of semantic and geometric information are achieved, mainly including feature serialization, cross-modal attention interaction and residual normalization.

[0065] S31, Feature Serialization BEV features of the aligned image and point cloud BEV features Flattened along the spatial dimension into a sequence form:

[0066] In the formula, Indicates the number of BEV grids. This represents the features of the flattened image. This represents the features of the flattened point cloud. The unified feature dimensions.

[0067] S32, Cross-modal attention interaction Given query source features Key-value source features Cross-modal attention is defined as follows:

[0068] in, It is a learnable linear projection matrix. Scaling factor This is the attention weight matrix, which represents the degree of attention each query position pays to each key value position.

[0069] Direction 1: Semantic Augmentation Geometry Features of flattened point clouds To query the features of the flattened image. For the key and value, after learningable linear projection, the scaled dot product of the query and the key and value is calculated. The attention weights are obtained through the Softmax function. Then, the values ​​are weighted and aggregated according to the attention weight of each value, so that each point cloud location adaptively extracts the required semantic information from the image features, thereby enhancing the class discriminative power of the point cloud features.

[0070] In the formula, This represents the point cloud features after image feature enhancement.

[0071] Direction 2: Geometric Correction Semantics Based on the flattened image features To query the features of the flattened point cloud. For the key and value, after learningable linear projection, the scaled dot product of the query and the key and value is calculated. The attention weights are obtained through the Softmax function. Then, the values ​​are weighted and aggregated according to the attention weight of each value, so that the image features obtain the precise geometric constraints of the point cloud.

[0072] In the formula, This represents the image features after point cloud feature enhancement.

[0073] It should be noted that both directions use independent learnable linear projection matrices and do not share parameters.

[0074] S33, Residual Normalization Both directions undergo residual connection and normalization after the attention output to alleviate the gradient degradation problem in deep attention, resulting in bidirectional enhanced and normalized point cloud BEV features, respectively. Image BEV features Their expressions are as follows:

[0075] Step S4: Dynamic Weight Fusion The quality scores of the original image and the original point cloud are evaluated frame by frame and then smoothly gated and mapped to bimodal fusion coefficients, providing a weighting basis for subsequent dynamic weighted fusion. This process includes two sub-steps: quality score evaluation and fusion coefficient mapping.

[0076] S41, Quality Score Assessment Quality score assessment includes two stages: quality indicator vector generation and quality score regression.

[0077] (1) Vector generation stage Image quality metrics such as contrast, sharpness, and wavelet domain signal-to-noise ratio are extracted from image modalities to form an image quality metric vector:

[0078] In the formula, Indicates contrast. Indicates sharpness, This represents the wavelet signal-to-noise ratio.

[0079] The voxel density, echo intensity information entropy, and geometric consistency of the point cloud modal are extracted to construct a point cloud quality index vector:

[0080] In the formula, represents the voxel point density, Represents the echo intensity information entropy. Indicates geometric consistency.

[0081] (2) Quality score regression stage A lightweight quality mapping network performs nonlinear regression on the image quality index vector and the point cloud quality index vector respectively, and outputs a quality score with a value between zero and one frame by frame.

[0082]

[0083] in, This represents the image quality score of the current frame, used to characterize the image quality of the current frame. This represents the point cloud quality score for the current frame, used to characterize the point cloud quality of the current frame. Represents an image quality mapping network; This represents the point cloud quality mapping network. The closer the score is to 1, the higher the quality and reliability of the modality data; the closer the score is to 0, the lower the quality and reliability of the modality data.

[0084] S42, Fusion Coefficient Mapping After obtaining the frame-by-frame quality scores of the two modalities, a quality score-driven normalization mapping is used to convert the quality scores into relative fusion coefficients, so that the higher quality modality dominates the fusion and the lower quality modality weakens its contribution accordingly.

[0085] To prevent low-quality modes from introducing noise into the fusion process, a lower limit threshold for the quality score is set. And introduce a smooth gating function. Mass fraction of each mode The processing is performed using the following formula:

[0086] in, For the Sigmoid function, A coefficient used to control the steepness of attenuation.

[0087] Normalize the modal quality scores to obtain the fusion coefficients for image modes and point cloud modes, respectively, as shown in the following formula:

[0088] In the formula, For smoothing terms, The fusion coefficients are the image modalities. denoted as the fusion coefficient of the point cloud modes.

[0089] When the quality score of a certain mode falls below the lower threshold, its output continuously approaches zero after smooth gating. The fusion coefficient of this mode then smoothly decays to zero, and the system automatically transforms into a single-mode detection mode dominated by another available mode. Because smooth gating is used instead of hard zeroing, this transformation process is continuous and stable when the quality score crosses the threshold, avoiding abrupt changes in detection results caused by sudden changes in the fusion coefficient, and ensuring the continuity of the autonomous driving system's behavior under sensor performance boundary conditions.

[0090] Step S5: Dynamic weighted fusion Based on the fusion coefficients, the BEV features of the bidirectional enhanced and normalized image are fused with the BEV features of the point cloud to obtain a unified fused BEV feature:

[0091] Step S6: 3D Target Detection The fused BEV features generated in step S5 are input into the 3D target detection head, and the target category is predicted and its 3D attribute parameters are regressed through the anchorless center point detection paradigm.

[0092] S61. Construct a 3D target detection head The detection head consists of a shared pre-convolutional layer and a multi-branch independent prediction head connected in series. All of them use 3×3 convolution, BN batch normalization and ReLU activation function, and no pooling layer is set to ensure that the spatial resolution of the BEV feature map remains unchanged.

[0093] Specifically, a shared pre-feature enhancement layer is constructed using two 3×3 convolutional layers, with a uniform number of channels. The input fused BEV features are subjected to global contextual smoothing, and the output shared basic feature map is supplied to all prediction branches.

[0094] Set up a category center heatmap prediction branch. This branch contains two 3×3 convolutional layers and one 1×1 output convolutional layer. The number of output channels is the same as the total number of target categories in the dataset. Output heatmap:

[0095] In the formula, This represents the number of target categories.

[0096] A multi-parameter regression branch for 3D bounding boxes is set up, which is split into multiple independent sub-regression branches. Each sub-branch consists of two 3×3 convolutional layers and one 1×1 output convolutional layer. The predicted 3D bounding boxes can be parameterized as follows:

[0097] In the formula, The three-dimensional center position of the target in the vehicle coordinate system. These represent the lateral and longitudinal velocities, respectively. These represent the target's width, length, and height, respectively. The yaw angle of the target in the BEV plane.

[0098] S62. Design a positive and negative sample allocation strategy. The 3D center of each real target is transformed into mesh coordinates using BEV coordinates, and the Gaussian kernel standard deviation is adaptively determined based on the BEV projection size of the target. This allows for a corresponding expansion of the positive sample range for large targets while maintaining a compact size for small targets. The Gaussian response is defined as:

[0099] In the formula, For BEV grid positions, The grid coordinates are the center of the target.

[0100] S63, Design Training Strategies The multi-task joint loss, which is a weighted sum of classification and regression losses, is expressed as follows:

[0101] In the formula, For classifying losses, For bounding box regression loss, This is the balance coefficient between the two.

[0102] The regression loss uses L1 loss to constrain the differences between the parameters of the prediction box and the true values:

[0103] In the formula, The first Predicted and true bounding box parameters for a positive sample.

[0104] The classification loss uses Focal Loss with a Gaussian penalty term:

[0105] In the formula, The total number of all positive sample grids in a single frame is used for loss normalization. The prediction confidence level for grid location. Adjustment factor for easy and difficult samples, The Gaussian penalty factor is used for negative samples.

[0106] S64. Design reasoning strategies.

[0107] During inference, 3×3 3×3 max pooling local peak filtering is used instead of traditional non-maximum suppression (NMS) to avoid missed detections in dense scenes. The specific inference process is as follows: First, perform a 3×3 max pooling operation on each type of heatmap, retaining only the grids with local maximum confidence in their neighborhood as candidate target centers.

[0108] Then, candidate grids with confidence levels higher than a preset threshold are selected, and the grid coordinates are corrected by combining the predicted center offset to restore the accurate BEV center of the target.

[0109] Next, the height, 3D dimensions, yaw angle, and velocity regression parameters are combined to reconstruct the complete 3D bounding box.

[0110] Finally, sort the detection boxes of all valid 3D targets from highest to lowest prediction confidence and output all valid detection boxes, each containing complete information on location, size, category, orientation, and velocity.

[0111] Finally, it should be noted that any parts of this invention not described in detail are prior art. Those skilled in the art will understand that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A bidirectional cross-modal fusion three-dimensional target detection method based on dynamic weights, characterized in that, Includes the following steps: S1. Data Acquisition: Acquire images captured by the vehicle-mounted camera and point cloud data collected by the lidar; S2. Feature extraction and bird's-eye view spatial alignment: Features are extracted from camera images and LiDAR point clouds respectively, and the two heterogeneous modalities are mapped to a unified bird's-eye view space to obtain strictly spatially aligned image bird's-eye view features and point cloud bird's-eye view features. S3. Bidirectional cross-modal feature enhancement: Construct bidirectional attention interaction, and achieve mutual enhancement of bimodal features through semantically enhanced geometric features and geometrically corrected semantic features. In the bidirectional attention interaction, the two directions do not share parameters and learn independently. S4. Dynamic weighted fusion: The quality scores of the original image and the original point cloud are evaluated frame by frame and mapped to dual-modal fusion coefficients through smooth gating. S5. Dynamic weighted fusion: Based on the dual-modal fusion coefficient, the bird's-eye view features of the image and the bird's-eye view features of the point cloud are fused to obtain a unified fused bird's-eye view feature. S6. 3D Object Detection: The fused bird's-eye view features are input into the 3D object detection head. The target category is predicted and its 3D attribute parameters are regressed through the anchor-free center point detection paradigm. The output is a 3D object detection box containing position, size, category, orientation, and velocity.

2. The method for bidirectional cross-modal fusion three-dimensional target detection based on dynamic weights according to claim 1, characterized in that, In step S2, the method for generating the bird's-eye view features of the image is as follows: Multi-scale features are extracted from camera images using a two-dimensional backbone network; The depth interval is discretized into multiple depth intervals, and the discrete depth probability distribution is estimated for each pixel through a depth prediction branch; the two-dimensional features are expanded along the depth direction by an outer product to obtain three-dimensional frustum features; The view frustum feature points are transformed from the camera coordinate system to the vehicle coordinate system using camera intrinsic and extrinsic parameters. In the vehicle coordinate system, the bird's-eye view plane is discretized into a regular grid. All view frustum feature points falling into the same bird's-eye view grid are aggregated along the height direction to obtain the bird's-eye view features of the image.

3. The method for bidirectional cross-modal fusion three-dimensional target detection based on dynamic weights according to claim 1, characterized in that, In step S2, the method for generating the point cloud bird's-eye view features is as follows: The 3D point cloud collected by the lidar is input into the point cloud branch, and the space is divided into regular grids at a set resolution on the bird's-eye view plane, and then expanded into columnar bodies along the height direction; Each point is assigned to a corresponding column according to its planar coordinates; For each point within the column, based on the original features, the offset relative to the geometric center of the column and the offset relative to the mean of the point cloud within the column are spliced ​​together to obtain the augmented features. The columnar features are obtained by transforming the shared multilayer perceptron and then performing max pooling. Based on the grid index of each column on the plane, its features are scattered to the corresponding positions of the bird's-eye view feature tensor; The feature tensors of the bird's-eye view are encoded at multiple scales and fused through a feature pyramid network to obtain the point cloud bird's-eye view features.

4. The method for bidirectional cross-modal fusion three-dimensional target detection based on dynamic weights according to claim 1, characterized in that, In step S3, the specific steps of the bidirectional cross-modal feature enhancement are as follows: The aligned image bird's-eye view features and point cloud bird's-eye view features are flattened into a sequence along the spatial dimension to obtain the flattened image features and point cloud features. Using the flattened point cloud features as the query and the flattened image features as the key and value, after learningable linear projection, the scaled dot product of the query and the key and value is calculated. The attention weight is obtained through a normalized exponential function. Then, the values ​​are weighted and aggregated according to the attention weight of each value to obtain the point cloud features after image feature enhancement. Using the flattened image features as the query and the flattened point cloud features as the key and value, after learning linear projection, the scaled dot product of the query and the key and value is calculated. The attention weight is obtained through a normalized exponential function. Then, the values ​​are weighted and aggregated according to the attention weight of each value to obtain the image features after point cloud feature enhancement. The point cloud features enhanced by the image features and the image features enhanced by the point cloud features are subjected to residual connection and normalization processing respectively to obtain the point cloud bird's-eye view features and the image bird's-eye view features after bidirectional enhancement and normalization.

5. The method for bidirectional cross-modal fusion three-dimensional target detection based on dynamic weights according to claim 1, characterized in that, In step S4, the dynamic weight fusion specifically includes: Image quality index vectors are constructed by extracting contrast, sharpness, and wavelet domain signal-to-noise ratio from image modalities. Voxel point density, echo intensity information entropy, and geometric consistency index are extracted from point cloud modalities to form a point cloud quality index vector; a lightweight quality mapping network performs nonlinear regression on the quality index vector to output image quality score and point cloud quality score frame by frame. A lower threshold for quality score is set, and the quality scores of each modality are processed by a smoothing gating function. The processed quality scores are then normalized to obtain the fusion coefficients of the image modality and the point cloud modality.

6. The method for bidirectional cross-modal fusion three-dimensional target detection based on dynamic weights according to claim 5, characterized in that, The smoothing gate function is the Sigmoid function, and its expression is: In the formula, For the Sigmoid function, A coefficient used to control the steepness of attenuation.

7. A dynamic weight-based bidirectional cross-modal fusion three-dimensional target detection system for implementing the method of any one of claims 1-6, characterized in that, include: The input module, including an onboard camera and a LiDAR, is used to acquire RGB images from multiple perspectives around the autonomous vehicle and 3D point cloud data collected by the LiDAR. The BEV feature extraction and alignment module includes an image branch module and a point cloud branch module, which are used to extract features from camera images and LiDAR point clouds, respectively, and map the two heterogeneous modalities to a unified bird's-eye view space, providing strictly aligned feature representations for subsequent cross-modal interactions. The bidirectional cross-modal feature enhancement module is used to construct bidirectional attention interaction to achieve mutual enhancement of bimodal features; The dynamic weighted fusion module is used to perform quality-adaptive weighted fusion of bi-modal bird's-eye view features after bi-directional enhancement and normalization, and to trigger continuous and stable single-modal degradation when modality degrades. The output module includes a 3D target detection head and a 3D detection box output end, which is used to output the 3D target detection results.

8. The bidirectional cross-modal fusion three-dimensional target detection system based on dynamic weights according to claim 7, characterized in that, The bidirectional cross-modal feature enhancement module includes: The first attention unit is used to integrate semantic information of the image into the point cloud features through a cross-attention mechanism, using point cloud bird's-eye view features as queries and image bird's-eye view features as keys and values, thereby achieving semantically enhanced geometry. The second attention unit is used to integrate point cloud geometric information into image features through a cross-attention mechanism, using image bird's-eye view features as queries and point cloud bird's-eye view features as keys and values, to achieve geometric correction semantics. The first attention unit and the second attention unit do not share parameters and learn independently.

9. A bidirectional cross-modal fusion three-dimensional target detection system based on dynamic weights according to claim 7, characterized in that, The dynamic weight fusion module includes: The image quality assessment submodule is used to extract contrast, sharpness and wavelet domain signal-to-noise ratio indicators from the original image, and then regress them into an image quality score through a lightweight quality mapping network. The point cloud quality assessment submodule is used to extract voxel density, echo intensity information entropy and geometric consistency index from the original point cloud, and then regress them into point cloud quality score through a lightweight quality mapping network. The fusion coefficient mapping submodule is used to set the lower threshold of the quality score, process and normalize the quality scores of each mode through a smoothing gating function, and obtain the fusion coefficient. The weighted fusion submodule is used to perform element-wise weighted summation of the bidirectional enhanced and normalized image bird's-eye view features and the point cloud bird's-eye view features based on the fusion coefficients, so as to generate fused bird's-eye view features.

Images