Heterogeneous sensor perception method based on multi-dimensional physical feature decoupling and double-layer interaction fusion

Abstract

The application provides a heterogeneous sensor perception method based on multi-dimensional physical feature decoupling and double-layer interactive fusion, first constructs a physical feature enhanced coding model, decouples radar measurement data into geometric state components and physical attribute components, combines radar auxiliary loss to adaptively suppress environmental clutter by using a physical confidence gating mechanism, then designs a double-layer interactive fusion architecture, performs dense splicing by constructing pseudo-radar feature maps and multi-scale visual features, and initializes the query vector of the Transformer decoder by using high-confidence radar spatial prior, guides the attention mechanism to lock the spatial area where the physical target exists, finally, combines the detection result and the radar speed enhanced track state prediction, and uses a distance gating and adaptive penalty matching algorithm to realize multi-target tracking. The scheme effectively corrects the depth ambiguity of visual perception, and significantly improves the robustness of three-dimensional target detection and tracking in complex dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving environment perception technology, and in particular to a heterogeneous sensor perception method based on multi-dimensional physical feature decoupling and two-level interactive fusion. Background Technology

[0002] In autonomous driving systems, accurate environmental perception is the cornerstone of ensuring safe vehicle operation. Visual sensors, with their rich semantic information and low cost, have been widely used in 3D object detection. In recent years, Transformer-based visual detection architectures have made significant progress by modeling detection as an ensemble prediction problem. In particular, RayDN (Ray Denoising) networks have pointed out that multi-view visual detection faces the bottleneck of camera ray depth blur, where visual features along the projection ray are identical, causing detectors to easily generate false detections at incorrect depths. Although existing techniques implicitly learn to distinguish targets through negative sample sampling, this data-driven estimation lacks explicit physical depth confirmation, resulting in severe performance degradation in low-light or low-texture scenes.

[0003] Millimeter-wave radar provides direct, weather-independent range and radial velocity measurements, making it an ideal complementary mode for resolving visual depth ambiguity. However, effectively fusing radar with Transformer-based visual networks faces two major challenges: First, the inherent sparsity and clutter (such as multipath reflections) of radar point clouds, if directly fused without filtering, introduce noise that contaminates visual features; second, the spatial misalignment between sparse radar points and dense image pixels makes it difficult to effectively guide the Transformer's attention mechanism. Existing fusion methods often employ simple feature stitching or depth-based projection, ignoring the rich physical properties of radar signals (such as radar cross section (RCS) and Doppler velocity). Furthermore, existing fusion frameworks rarely address 3D detection and multi-target tracking simultaneously in a unified end-to-end process. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a heterogeneous sensor perception method based on multidimensional physical feature decoupling and dual-level interactive fusion. This method constructs a radar coding module with enhanced physical features to decouple the radar's geometric state from its physical signal attributes, and utilizes radar-assisted supervision loss to optimize feature extraction. Simultaneously, it designs a dual-level interactive fusion architecture, injecting radar information into both the dense feature layer and the sparse query layer to achieve physical correction of visual depth blur.

[0005] A heterogeneous sensor sensing method based on multidimensional physical feature decoupling and two-level interactive fusion, the method comprising the following steps:

[0006] Step S1, Spatial Alignment and Radar Point Cloud Representation: Collect image data from multi-view cameras and point cloud data from millimeter-wave radar. For radar data, analyze its measurement vectors. Using a pre-calibrated extrinsic matrix, transform the distributed visual and radar sensor coordinate systems to a unified vehicle body coordinate system. Construct a unified radar point cloud set through homogeneous coordinate transformation.

[0007] Step S2, Radar coding with enhanced physical features: The radar measurement vector is decoupled into a geometric state vector and a physical attribute vector. The geometric state vector is mapped into a high-dimensional motion state feature using a multilayer perceptron, and the physical attribute vector is encoded into a high-dimensional attribute feature. The high-dimensional attribute feature is converted into the probability of the point cloud being an effective target by combining an activation function. The motion state feature is then subjected to element-wise gating modulation to generate physically enhanced radar features. Finally, the radar point cloud is divided into positive and negative samples based on the ground truth box constraint to construct the radar auxiliary loss.

[0008] Step S3, extract multi-scale visual semantic features: use a visual backbone network containing a feature pyramid network to extract multi-scale feature maps of multi-view images;

[0009] Step S4, Dense Feature-Guided Radar and Vision Fusion: Based on the calibration parameters, the encoded radar point cloud is projected onto the image plane, the corresponding physical augmentation radar features are filled at the projected pixel positions, and zeros are padded at the remaining positions. The resolution is dynamically adjusted to match the visual multi-scale feature levels to construct a pseudo radar feature map. The pseudo radar feature map is then concatenated with the corresponding visual feature map in the channel dimension. Feature smoothing and fusion are performed through a convolutional sub-network to generate a fused dense feature map.

[0010] Step S5, Sparse query-guided radar and vision fusion: The effective target scores of the point cloud generated by the radar physical enhancement features are used to filter the top K radar point clouds. The coordinates of the selected radar points are encoded as position embedding features to initialize the partial object query of the Transformer decoder. The initialized query vector is used to perform cross-attention calculation on the fused dense feature map to guide the attention of the decoder to the physically existing area, thereby establishing the radar spatial prior to guide the search process of the Transformer decoder.

[0011] Step S6, 3D target detection and multi-target tracking: Based on the radar and visual sparse fusion features output by the decoder, the target category and 3D bounding box attributes are predicted by the detection head, and a detection-based tracking module is constructed. The position of historical tracks is predicted based on the radar-enhanced velocity estimate. The detection results and track predictions are matched using distance gating and adaptive penalty. Based on the matching results, a hysteresis-based lifecycle management strategy is adopted to confirm continuously matched tracks and remove tracks that have not been matched for a long time, thus obtaining a set of effective matching pairs.

[0012] Furthermore, the step S1 of parsing the radar data measurement vector specifically involves:

[0013] ;

[0014] in, This represents the i-th radar point in the radar point cloud, containing three-dimensional spatial coordinates. velocities in the x and y directions Radar cross section and depth information .

[0015] Furthermore, in step S1, a unified radar point cloud is constructed through homogeneous coordinate transformation, specifically as follows:

[0016] ;

[0017] in, express Radars from different perspectives converge at a single point. This represents the radar point cloud from the k-th viewpoint. Represents the extrinsic parameter matrix. Represents the homogeneous coordinates of the radar point cloud.

[0018] Furthermore, in step S2, a multilayer perceptron is used to map the geometric state vector into high-dimensional motion state features and to encode the physical attribute vector into high-dimensional attribute features, specifically as follows:

[0019] ;

[0020] ;

[0021] in, Represents the characteristics of high-dimensional motion states. Represents the geometric state vector. Represents high-dimensional attribute features. Represents a physical property vector. Indicates a linear layer. This indicates batch normalization processing. This represents the rectified linear unit function.

[0022] Further, in step S2, the high-dimensional attribute features are converted into the probability that the point cloud is an effective target, and the motion state features are subjected to element-wise gated modulation to generate physically enhanced radar features, specifically:

[0023] ;

[0024] ;

[0025] in, This represents the Sigmoid activation function. Represents the probability of an effective target. Indicates radar enhancement characteristics, This represents element-wise product.

[0026] Furthermore, in step S2, the radar point cloud is divided into positive and negative samples based on the truth box constraint to construct the radar-assisted loss, specifically as follows:

[0027] ;

[0028] ;

[0029] in, Indicates radar-assisted loss, Indicates the predicted probability. The mask vector representing the validity of the point cloud is 1 if and only if a radar point falls within the 3D ground truth box, and 0 otherwise. Represents element-wise product. Represents cross-entropy loss, and This represents an adjustment factor that balances the number of positive and negative point clouds.

[0030] Further, in step S4, the pseudo radar feature map and the corresponding visual feature map are concatenated along the channel dimension, and feature smoothing and fusion are performed through a convolutional sub-network to generate a fused dense feature map, specifically:

[0031] ;

[0032] in, This represents the dense feature map at the i-th scale after fusion. Represents the visual feature map at the i-th scale. This represents the pseudo radar feature map corresponding to the i-th scale. This indicates splicing along the channel dimension. This represents a two-dimensional convolution operation.

[0033] Further, step S5 specifically includes:

[0034] ;

[0035] ;

[0036] in, and Represents the three-dimensional spatial coordinates of radar point clouds and their corresponding features. and This represents the spatial coordinates and characteristics of the radar point cloud of the top K valid targets selected. Indicates the use of effective target probability The operation of selecting the K point clouds with the highest probability. This indicates the sparse fusion characteristics of radar and vision. and This represents the query points and their corresponding features in the visual initialization. This represents the Transformer decoder structure. This indicates splicing along the quantity dimension.

[0037] Further, step S6 specifically includes:

[0038] ;

[0039] ;

[0040] in, Indicates the category of the predicted target. Represents the three-dimensional spatial coordinates of the predicted target. Indicates the length, width, and height dimensions of the predicted target. Indicates the predicted yaw angle of the target. This indicates the predicted velocity information of the target in the x and y directions. This represents a detection head composed of linear layers; This represents the Euclidean distance cost of the detection results and trajectory prediction. and The meanings are the same, only the index representations are different. Indicates the index subscript. Represents the enhanced distance cost of distance threshold gating constraints. Indicates the distance-gated threshold. Represents the penalty bias constant. This is an indicator function; it takes the value 1 when the condition within the parentheses is met, and 0 otherwise. `max` indicates that it takes the maximum value within the set. The maximum value is the number of detected targets. , This represents the matching index mapping with the minimum global cost calculated by the solution. Represents the set of all possible permutations and mappings. This represents the set of valid matching pairs in the final output.

[0041] This invention proposes a heterogeneous sensor perception method based on multi-dimensional physical feature decoupling and two-level interactive fusion. First, a physical feature enhancement coding model is constructed to decouple radar measurement data into geometric state components and physical attribute components. Combined with radar-assisted loss, a physical confidence gating mechanism is used to adaptively suppress environmental clutter, effectively suppressing interference from radar clutter on feature fusion. Based on this, a two-level interactive fusion architecture is designed: at the feature representation layer, a pseudo-radar feature map is constructed and densely stitched with multi-scale visual features to achieve explicit complementarity between semantic information and physical depth; at the logical query layer, high-confidence radar spatial prior initialization is utilized. The query vector of the Transformer decoder is optimized to guide the attention mechanism to lock onto the spatial region where the physical target exists. This not only supplements the deep semantics at the feature level but also provides accurate spatial priors at the query level, significantly solving the depth ambiguity problem in pure vision methods. Finally, by combining the detection results and radar velocity-enhanced track state prediction, multi-target tracking is achieved using distance gating and adaptive penalty matching algorithms. The method of this invention effectively corrects the depth ambiguity of visual perception, significantly improves the robustness of 3D target detection and tracking in complex dynamic scenes, and enhances the reliability of heterogeneous source sensors for environmental perception in advanced driver assistance systems. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a heterogeneous sensor sensing method based on multidimensional physical feature decoupling and two-level interactive fusion provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the three-dimensional target detection results provided in the embodiments of the present invention, wherein (a) represents the detection result of the method of the present invention, and (b) is the detection result of the pure vision method (RayDN);

[0045] Figure 3 This is a schematic diagram of the effective target probability generated based on radar physical enhancement features provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This embodiment provides a heterogeneous sensor sensing method based on multi-dimensional physical feature decoupling and two-level interactive fusion, such as... Figure 1 As shown, the method includes the following steps:

[0048] Step 1, Spatial Alignment and Radar Point Cloud Representation: Acquire image data from multi-view cameras and point cloud data from multi-view millimeter-wave radar. For radar data, analyze its measurement vectors, which include three-dimensional spatial coordinates, relative velocity, radar cross section, and depth information. The specific analysis method is as follows:

[0049] ;

[0050] in, This represents the i-th radar point in the radar point cloud, containing three-dimensional spatial coordinates. velocities in the x and y directions Radar cross section and depth information .

[0051] Then, using a pre-calibrated extrinsic parameter matrix, the distributed visual and radar sensor coordinate systems are transformed to a unified vehicle body coordinate system. A unified radar point cloud is then constructed through homogeneous coordinate transformation, as detailed below:

[0052] ;

[0053] in, express Radars from different perspectives converge at a single point. This represents the radar point cloud from the k-th viewpoint. Represents the extrinsic parameter matrix. The homogeneous coordinates of the radar point cloud are represented by the homogeneous coordinates obtained by adding a one-dimensional unit vector to the three-dimensional coordinates.

[0054] Step 2, Radar Coding with Enhanced Physical Features: To address the sparsity and clutter characteristics of radar point clouds, a dual-stream coding architecture is constructed. The radar measurement vector is decoupled into a geometric state vector (containing coordinates and velocity) and a physical attribute vector (containing radar cross section (RCS), depth, and velocity). A multilayer perceptron is used to map the geometric state vector into high-dimensional motion state features, and the physical attribute vector is encoded into high-dimensional attribute features, as detailed below:

[0055] ;

[0056] ;

[0057] in, Represents the characteristics of high-dimensional motion states. Represents the geometric state vector. Represents high-dimensional attribute features. Represents a physical property vector. Indicates a linear layer. This indicates batch normalization processing. This represents the rectified linear unit function.

[0058] Then, the high-dimensional attribute features are converted into the probability of the point cloud being a valid target by combining the activation function, i.e., the valid target probability. The motion state features are then subjected to element-wise gating modulation to generate physically enhanced radar features, as follows:

[0059] ;

[0060] ;

[0061] in, This represents the Sigmoid activation function. Represents the probability of an effective target. Indicates radar enhancement characteristics, This represents element-wise product.

[0062] Then, based on the truth box constraints, the radar point cloud is divided into positive and negative samples to construct a radar-assisted loss, which is used to supervise the learning of geometric state and physical attribute encoding branches, enabling the model to distinguish clutter from effective targets, as detailed below:

[0063] ;

[0064] ;

[0065] in, Indicates radar-assisted loss, Indicates the predicted probability. The mask vector representing the validity of the point cloud is 1 if and only if a radar point falls within the 3D ground truth box, and 0 otherwise. Represents element-wise product. Represents cross-entropy loss, and This represents an adjustment factor that balances the number of positive and negative point clouds.

[0066] Step 3: Extract multi-scale visual semantic features: Utilize a visual backbone network (such as ResNet or ViT) containing a Feature Pyramid Network (FPN) to extract multi-scale feature maps from multi-view images, denoted as... , where i represents the i-th scale. This is the visual feature map at the i-th scale. This step is a general workflow for visual image feature processing, aiming to extract high-dimensional semantic features from visual images.

[0067] Step 4, Dense Feature-Guided Radar and Visual Fusion: Based on the calibration parameters, the encoded radar point cloud (from Step 2) is projected onto the image plane. The corresponding physically enhanced radar features are filled into the projected pixel locations, and zeros are padded to the remaining locations. The resolution is dynamically adjusted to match the multi-scale feature levels of the visual system, thus constructing a pseudo-radar feature map. The pseudo-radar feature map is then concatenated with the corresponding visual feature map along the channel dimension. A convolutional sub-network is used for feature smoothing and fusion to generate a fused dense feature map, thereby compensating for the lack of explicit depth information in visual features. Specifically:

[0068] ;

[0069] in, This represents the dense feature map at the i-th scale after fusion. Represents the visual feature map at the i-th scale. This represents the pseudo radar feature map corresponding to the i-th scale. This indicates splicing along the channel dimension. This represents a two-dimensional convolution operation.

[0070] Step 5, Sparse Query-Guided Radar and Vision Fusion: Utilizing radar physical enhancement features to generate effective target scores from point clouds, the top K radar point clouds are selected. The coordinates of the selected radar points are encoded as location embedding features, used to initialize partial object queries in the Transformer decoder, as shown below:

[0071] ;

[0072] in, and Represents the three-dimensional spatial coordinates of radar point clouds and their corresponding features. and This represents the spatial coordinates and characteristics of the radar point cloud of the top K valid targets selected. This indicates the use of the effective target probability in step two. The operation of selecting the top K point clouds with the highest probability.

[0073] Then, using the initialized query vector, cross-attention calculation is performed on the fused dense feature map to guide the decoder's attention to the physically existing region, suppressing depth blur of the visual ray. This establishes a radar spatial prior to guide the Transformer decoder's search process, as shown below:

[0074] ;

[0075] in, This indicates the sparse fusion characteristics of radar and vision. and This represents the query points and their corresponding features in the visual initialization. This represents the Transformer decoder structure. This indicates splicing along the quantity dimension.

[0076] Step Six, 3D Target Detection and Multi-Target Tracking: Based on the radar and vision fusion features output by the decoder, the target category and 3D bounding box attributes are predicted by the detection head, and a detection-based tracking module is constructed. Then, based on the radar-enhanced velocity estimate, the position of historical tracks is predicted, and range gating and adaptive penalty are used to establish a match between the detection results and the track predictions. Finally, based on the matching results, a hysteresis-based lifecycle management strategy is adopted to confirm continuously matched tracks and remove tracks that have not matched for a long time, as detailed below:

[0077] ;

[0078] ;

[0079] in, Indicates the category of the predicted target. Represents the three-dimensional spatial coordinates of the predicted target. Indicates the length, width, and height dimensions of the predicted target. Indicates the predicted yaw angle of the target. This indicates the predicted velocity information of the target in the x and y directions. This represents a detection head composed of linear layers; Indicates detection results and trajectory prediction The cost of European distance, Represents the enhanced distance cost of distance threshold gating constraints. Indicates the distance-gated threshold. Represents the penalty bias constant. This is an indicator function; it takes the value 1 when the condition within the parentheses is met, and 0 otherwise. `max` indicates that it takes the maximum value within the set. and They have the same meaning, only the index representation differs, and are used to distinguish the process of iteratively finding the maximum value. Indicates the index subscript. The maximum value is the number of detected targets. , This represents the matching index mapping with the minimum global cost calculated by the solution. Represents the set of all possible permutations and mappings. This represents the set of valid matching pairs in the final output.

[0080] This invention is mainly based on actual measured open source data for verification and evaluation. All steps and results are verified for effectiveness in Python.

[0081] Figure 2 This is a schematic diagram of the three-dimensional target detection results. In the diagram, (a) represents the detection result of the method of the present invention, and (b) represents the detection result of the pure vision method (RayDN). It can be seen from the figure that compared with the pure vision method, the method of the present invention can achieve more comprehensive target detection, and the position of the detection result is closer to the true value. This shows that the method of the present invention can solve the depth ambiguity problem in the pure vision method and improve the detection accuracy.

[0082] Table 1 shows the comparison results of the three-dimensional target detection accuracy of the method of the present invention with other methods, as shown in the table below:

[0083] Table 1 Comparison of the three-dimensional target detection accuracy of the method of the present invention with other methods

[0084]

[0085] NDS and mAP are comprehensive indicators, while mATE, mASE, mAOE, mAVE, and mAAE are position, size, yaw angle, velocity, and angular velocity indicators, respectively. The method column represents the current mainstream methods. It can be seen that the fusion method of the present invention has achieved a significant gain in comprehensive detection accuracy compared with the pure visual baseline method (RayDN) and the radar visual fusion method (RCBEVDet), which shows the reliability of the method of the present invention.

[0086] Figure 3 This is a schematic diagram of the effective target probability generated based on radar physical enhancement features. The darker the color in the diagram, the higher the probability that the point cloud is an effective target. As can be seen from the diagram, step two (radar coding with physical feature enhancement) can effectively suppress clutter point clouds by decoupling the geometric state and physical attribute information of radar point clouds, thereby improving the reliability of fusion.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heterogeneous sensor sensing method based on multi-dimensional physical feature decoupling and two-level interactive fusion, characterized in that, The method includes: Step S1, Spatial Alignment and Radar Point Cloud Representation: Collect image data from multi-view cameras and point cloud data from millimeter-wave radar. For radar data, analyze its measurement vectors. Using a pre-calibrated extrinsic matrix, transform the distributed visual and radar sensor coordinate systems to a unified vehicle body coordinate system. Construct a unified radar point cloud set through homogeneous coordinate transformation. Step S2, Radar coding with enhanced physical features: The radar measurement vector is decoupled into a geometric state vector and a physical attribute vector. The geometric state vector is mapped into a high-dimensional motion state feature using a multilayer perceptron, and the physical attribute vector is encoded into a high-dimensional attribute feature. The high-dimensional attribute feature is converted into the probability of the point cloud being an effective target by combining an activation function. The motion state feature is then subjected to element-wise gating modulation to generate physically enhanced radar features. Finally, the radar point cloud is divided into positive and negative samples based on the ground truth box constraint to construct the radar auxiliary loss. Step S3, extract multi-scale visual semantic features: use a visual backbone network containing a feature pyramid network to extract multi-scale feature maps of multi-view images; Step S4, Dense Feature-Guided Radar and Vision Fusion: Based on the calibration parameters, the encoded radar point cloud is projected onto the image plane, the corresponding physical augmentation radar features are filled at the projected pixel positions, and zeros are padded at the remaining positions. The resolution is dynamically adjusted to match the visual multi-scale feature levels to construct a pseudo radar feature map. The pseudo radar feature map is then concatenated with the corresponding visual feature map in the channel dimension. Feature smoothing and fusion are performed through a convolutional sub-network to generate a fused dense feature map. Step S5, Sparse query-guided radar and vision fusion: The effective target scores of the point cloud generated by the radar physical enhancement features are used to filter the top K radar point clouds. The coordinates of the selected radar points are encoded as position embedding features to initialize the partial object query of the Transformer decoder. The initialized query vector is used to perform cross-attention calculation on the fused dense feature map to guide the attention of the decoder to the physically existing area, thereby establishing the radar spatial prior to guide the search process of the Transformer decoder. Step S6, 3D target detection and multi-target tracking: Based on the radar and visual sparse fusion features output by the decoder, the target category and 3D bounding box attributes are predicted by the detection head, and a detection-based tracking module is constructed. The position of historical tracks is predicted based on the radar-enhanced velocity estimate. The detection results and track predictions are matched using distance gating and adaptive penalty. Based on the matching results, a hysteresis-based lifecycle management strategy is adopted to confirm continuously matched tracks and remove tracks that have not been matched for a long time, thus obtaining a set of effective matching pairs.

2. The method according to claim 1, characterized in that, The step S1 of parsing the radar data measurement vector specifically involves: ； in, This represents the i-th radar point in the radar point cloud, containing three-dimensional spatial coordinates. velocities in the x and y directions Radar cross section and depth information .

3. The method according to claim 1, characterized in that, In step S1, a unified radar point cloud is constructed through homogeneous coordinate transformation, specifically as follows: ； in, express Radars from different perspectives converge at a single point. This represents the radar point cloud from the k-th viewpoint. Represents the extrinsic parameter matrix. Represents the homogeneous coordinates of the radar point cloud.

4. The method according to claim 1, characterized in that, In step S2, a multilayer perceptron is used to map the geometric state vector into high-dimensional motion state features and to encode the physical attribute vector into high-dimensional attribute features. Specifically: ； ； in, Represents the characteristics of high-dimensional motion states. Represents the geometric state vector. Represents high-dimensional attribute features. Represents a physical property vector. Indicates a linear layer. This indicates batch normalization processing. This represents the rectified linear unit function.

5. The method according to claim 4, characterized in that, In step S2, the high-dimensional attribute features are converted into the probability that the point cloud is a valid target, and the motion state features are subjected to element-wise gated modulation to generate physically enhanced radar features, specifically: ； ； in, This represents the Sigmoid activation function. Represents the probability of an effective target. Indicates radar enhancement characteristics, This represents element-wise product.

6. The method according to claim 5, characterized in that, In step S2, the radar point cloud is divided into positive and negative samples based on the truth box constraint to construct the radar auxiliary loss, specifically as follows: ； ； in, Indicates radar-assisted loss, Indicates the predicted probability. The mask vector representing the validity of the point cloud is 1 if and only if a radar point falls within the 3D ground truth box, and 0 otherwise. Represents element-wise product. Represents cross-entropy loss, and This represents an adjustment factor that balances the number of positive and negative point clouds.

7. The method according to claim 1, characterized in that, In step S4, the pseudo radar feature map and the corresponding visual feature map are concatenated along the channel dimension, and feature smoothing and fusion are performed through a convolutional sub-network to generate a fused dense feature map. Specifically: ； in, This represents the dense feature map at the i-th scale after fusion. Represents the visual feature map at the i-th scale. This represents the pseudo radar feature map corresponding to the i-th scale. This indicates splicing along the channel dimension. This represents a two-dimensional convolution operation.

8. The method according to claim 5, characterized in that, Step S5 specifically involves: ； ； in, and Represents the three-dimensional spatial coordinates of radar point clouds and their corresponding features. and This represents the spatial coordinates and characteristics of the radar point cloud of the top K valid targets selected. Indicates the use of effective target probability The operation of selecting the K point clouds with the highest probability. This indicates the sparse fusion characteristics of radar and vision. and This represents the query points and their corresponding features in the visual initialization. This represents the Transformer decoder structure. This indicates splicing along the quantity dimension.

9. The method according to claim 1, characterized in that, Step S6 specifically involves: ； ； in, Indicates the category of the predicted target. Represents the three-dimensional spatial coordinates of the predicted target. Indicates the length, width, and height dimensions of the predicted target. Indicates the predicted yaw angle of the target. This indicates the predicted velocity information of the target in the x and y directions. This represents a detection head composed of linear layers; This represents the Euclidean distance cost of the detection results and trajectory prediction. and The meanings are the same, only the index representations are different. Indicates the index subscript. Represents the enhanced distance cost of distance threshold gating constraints. Indicates the distance-gated threshold. Represents the penalty bias constant. This is an indicator function; it takes the value 1 when the condition within the parentheses is met, and 0 otherwise. `max` indicates that it takes the maximum value within the set. The maximum value is the number of detected targets. , This represents the matching index mapping with the minimum global cost calculated by the solution. Represents the set of all possible permutations and mappings. This represents the set of valid matching pairs in the final output.

Images