3D target detection tracking method based on visual image and radar tensor sparse proposal fusion

Abstract

The invention provides a 3D target detection tracking method based on visual image and radar tensor sparse proposal fusion, which comprises the following steps: firstly, carrying out generalization extraction on color texture information of a visual image, and establishing multi-scale semantic high-dimensional features; secondly, extracting multi-scale space high-dimensional features of a radar tensor by using SCAN, respectively mapping a learnable sensing probe to radar and visual high-dimensional feature spaces, and performing generalization sparseness on different modal features by means of multi-head deformable attention to form radar and visual proposal features; sparse proposal fusion of radar and visual proposal features is carried out, and 3D target detection is completed; and finally, carrying out mixed multi-feature cascade matching and batch track management on a detection result, and feeding back generated track time sequence information to a front-end learnable sensing probe to realize an active target detection and tracking integrated circulating progressive network based on time sequence information guidance. According to the scheme of the invention, an integrated active sensing framework of single-frame target detection and time sequence target tracking is established, and the reliability of environment target sensing by a multi-source sensor in automatic driving is enhanced.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving, deep learning, and multi-sensor fusion technology, and in particular to a 3D target detection and tracking method based on the fusion of visual images and radar tensor sparse proposals. Background Technology

[0002] Perception of environmental targets is crucial in autonomous driving. Millimeter-wave radar (radar) and cameras (also known as visual sensors) are currently highly complementary sensors in autonomous driving. Specifically, radar can operate in all weather conditions and detect key information such as target speed and distance, but it lacks semantic texture features, making it difficult to reliably identify and classify targets. Cameras possess rich semantic texture information, are relatively inexpensive, and are rapidly developing technologically, but they are highly susceptible to extreme environmental conditions such as strong light, rain, snow, and fog, and they lack ranging and speed measurement capabilities. Therefore, conducting research on the fusion of visual and radar sensors (referred to as radar-visual fusion) is extremely important.

[0003] Most current radar-visual fusion methods use radar point clouds and visual images as inputs, extracting high-dimensional features from both separately to achieve target detection. These methods utilize highly processed radar point clouds as input, losing a significant amount of information captured by the radar and making it difficult to obtain reliable fusion features. Using radar tensors—range-azimuth maps generated from radar ADC (Analog-to-Digital Converter) data through FFT (Fast Fourier Transform) and DOA (Direction of Arrival) estimation—and fusing them with visual images can reduce information loss in the radar data processing environment and avoid the difficulty in extracting high-dimensional features from sparse radar point clouds. However, reliable datasets for radar tensors and visual images are currently lacking, and research on the fusion of visual images and radar tensors is extremely rare. The paper "Low-Level Sensor Fusion for 3D Vehicle Detection Using Radar Range-Azimuth Heatmap and Monocular Image[C], ACCV, 2020." uses a two-stage network based on feature map fusion to implement the target detection task. However, this network uses simple stitching to fuse high-dimensional feature maps from different sensors, lacking in-depth exploration of the features of different sensors. Moreover, dense fusion based on feature maps increases memory consumption and reduces running efficiency. The paper "Echoes beyond points: unleashing the power of raw radar data in multi-modality fusion[J]. Advances in Neural Information Processing Systems, 2023" proposes a feature map attention fusion method based on polar coordinate alignment. However, this method uses passively input data to complete the target detection task, without exploring the synergistic promotion between detection and tracking tasks. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a 3D target detection and tracking method based on the fusion of visual images and radar tensor sparse proposals. This method solves the problems of limited operational efficiency and passive processing of single detection tasks in existing methods based on feature map fusion, thereby achieving highly reliable and multi-task integrated proactive radar-visual fusion perception.

[0005] A 3D target detection and tracking method based on the fusion of visual images and radar tensor sparse proposals includes the following steps:

[0006] Step S1, Multi-scale semantic feature extraction and visual proposal feature generation of visual images: Collect image data from vehicle-mounted cameras, use a backbone network composed of convolutions to extract semantic texture features of the images, then use a feature pyramid network (FPN) to generate multi-scale semantic features, generate learnable perception probes in the vehicle coordinate system, combine the projection matrix from vehicle coordinates to image coordinates, and use multi-scale deformable attention to perform feature enhancement and visual proposal feature generation.

[0007] Step S2, Radar Tensor Multi-Scale Spatial Feature Extraction and Radar Proposal Feature Generation: Collect vehicle-side radar data, obtain radar tensor data through preprocessing, extract radar spatial features using the coupled radar echo intensity and spatial orientation perception collaborative enhancement network SCAN, generate multi-scale spatial features using FPN, and then use learnable perception probe positions and corresponding probe features, combined with the projection matrix from vehicle coordinates to radar tensor coordinates, to perform feature enhancement and radar proposal feature generation using multi-scale deformable attention.

[0008] Step S3, sparse gating fusion based on visual proposal features and radar proposal features: the visual proposal features and radar proposal features that have completed association matching are concatenated to generate coarse fused proposal features, and a linear network is used to perform adaptive evidence learning on the coarse fused proposal features. Then, the dynamic confidence differential evaluation of evidence of different modalities is performed through activation function, and proposal-level sparse gating fusion is completed based on the dynamic differential confidence.

[0009] Step S4, 3D target detection based on visual and radar fusion proposal features: After obtaining visual and radar fusion proposal features, the target classification task head and regression task head are used respectively to extract the 3D target category and attribute information.

[0010] Step S5, hybrid multi-feature cascade matching between target detection results and historical tracks: first, consistency matching is performed based on fused proposed features, and then multi-feature local and global cost minimization matching is performed by coupling position, high-dimensional features and size;

[0011] Step S6, Batch Track Management and Track Information Feedback Guidance: The group cognitive track management paradigm is used to process the track inference, association matching and lifecycle management of the entire data batch in parallel. At the same time, the group cognitive track time sequence information is adaptively fed back to the corresponding scenario steps S1 and S2 to expand the perception probe. This allows the perception probe to simultaneously encompass the global statistical prior of the entire training data and the individual historical prior of the current track, so as to achieve integrated active cyclic progression of 3D target detection and tracking.

[0012] Furthermore, a Feature Pyramid Network (FPN) is used to generate multi-scale semantic features. , is represented as:

[0013] ;

[0014] in, Represents the visual high-dimensional semantic features at the s-th scale. Indicates the input visual image;

[0015] A learnable sensing probe is generated in the vehicle coordinate system, and the sensing probe includes the probe position. and corresponding probe characteristics Combined with the projection matrix from vehicle body coordinates to image coordinates generated during sensor deployment calibration Multi-scale deformable attention is used for feature enhancement and visual feature proposal generation, as shown below:

[0016] F v , o = [ ∑ s = 1 S ∑ k = 1 K v A s , k C ( F q , F v , s , H ego 2 image P ′ q + D P q , k ) ] N ;

[0017] in, Indicates visual proposal features. Let S represent the attention weight of the k-th deformable sampling point at the s-th scale. The attention weights are first randomly initialized and then adjusted and optimized through training. S represents the number of multi-scales. The number of visually deformable sampling points. Indicates based on probe coordinates P obtains the feature space. Corresponding features, and with specific features A function that enhances coupling, where N represents the superposition of N identical modules. This indicates an active adaptive correction of the probe coordinates after considering coordinate transformation deviation and time deviation, starting from the initial probe position. The small learnable positional offset is generated by adding a small amount of information, where the positional offset is generated by the initial probe's position-encoded feature (PE) and the probe feature. The sum is generated through a linear network, represented as:

[0018] ;

[0019] This represents the position offset of the deformable attention at the k-th sampling point in the feature space, which is determined by the initial probe position-encoded feature PE and the probe feature. The sum is generated through a linear network, represented as:

[0020] .

[0021] Furthermore, the radar spatial features are extracted using a coupled radar echo intensity and spatial orientation awareness co-enhancement network (SCAN), and then multi-scale spatial features are generated using a multi-scale network (FPN). , is represented as:

[0022] ;

[0023] in, This represents the high-dimensional spatial features of the radar at the s-th scale. Representing the input radar tensor, SCAN is composed of stacked SCA subnetworks. The SCA subnetwork consists of two parts: the first part represents the gain for radar echo intensity, and the second part represents the sensing gain for radar spatial orientation. The two work together to enhance the extraction of radar spatial features, as shown below:

[0024] ;

[0025] Wherein, SCA represents the coupled radar echo intensity and spatial orientation sensing cooperative sub-network. Indicates input features, denoted by Sigmoid activation function, C represents convolution operation, Avg represents averaging the input features along the channel dimension, and Max represents maximizing the input features along the channel dimension. This indicates concatenation along the channel dimension. CRC represents a modular network composed of convolution, ReLU activation function, and convolution. AvgPool represents average pooling in the spatial dimension, and MaxPool represents maximum pooling in the spatial dimension.

[0026] Using the learnable sensing probe position in step S1 and corresponding probe characteristics Combined with the projection matrix from the vehicle body coordinates to the radar tensor coordinates generated during sensor deployment calibration Multi-scale deformable attention is used for feature enhancement and radar-proposed feature generation, as follows:

[0027] F r , o = [ ∑ s = 1 S ∑ k = 1 K r A s , k C ( F q , F r , s , H ego 2 rad P ′ q + D P q , k ) ] N ;

[0028] in, Indicates radar proposal characteristics, This represents the number of deformable sampling points for the radar.

[0029] Further, step S3 is specifically expressed as follows:

[0030] ;

[0031] in, This indicates the features of the vision and radar fusion proposal. This indicates splicing along the channel dimension. This represents the Sigmoid activation function.

[0032] Further, step S4 is specifically expressed as follows:

[0033] { c = Linear ( [ ReLU ( LN ( Linear ( F o ) ) ) ] M ) x , y , z , l , w , h , i = Linear ( [ ReLU ( Linear ( F o ) ) ] M ) ;

[0034] Where c represents the target category information, ReLU represents the activation function, LN represents LayerNorm normalization, M represents M identical modules stacked together, x, y, and z represent the target's 3D coordinates, and l, w, and h represent the target's 3D dimensions, which are length, width, and height, respectively. Indicates the target rotation angle.

[0035] Furthermore, the hybrid multi-feature cascade matching is divided into two stages. The first stage is consistency matching based on the feedback of the fusion proposal features. That is, the fusion proposal features of the previous time are fed back to the target detection process of the current time. If the current target detection result is consistent with the fusion proposal features fed back in the previous time, then the target corresponding to the fusion proposal features of the previous time is matched with the detection result of the current time.

[0036] Secondly, there is the multi-feature local-to-global cost minimization matching that couples position, high-dimensional features, and size. This method uses the position and size information from the target detection results and temporal track information, along with fused proposal features, to construct a local hybrid multi-feature cost matrix. Then, the Hungarian algorithm is used to achieve the local optimal matching between the detected target and the track target, as shown below:

[0037] ;

[0038] Where D represents the local hybrid multi-feature cost matrix, and Let G represent the target position information remaining at the current and previous time steps after consistency matching, respectively. Let G represent the state transition equation, which is usually Newton's equations of motion. and These represent the fusion proposal features corresponding to the current time step and the previous time step, respectively. and These represent the target size information corresponding to the current time and the previous time, respectively. , and These represent the contributions of experience-derived information regarding location, proposed features, and size, respectively.

[0039] This invention provides a 3D target detection and tracking method based on sparse proposal fusion of visual images and radar tensors. First, color and texture information from the visual image is extracted using generalized methods to establish multi-scale semantic high-dimensional features. Second, a coupled radar echo intensity and spatial orientation perception co-enhancement network is used to extract multi-scale spatial high-dimensional features from the radar tensor. Then, learnable perception probes are mapped to the radar and visual high-dimensional feature spaces respectively. A multi-head deformable attention decoupling method is used to generalize and sparsely distribute the features across different modalities, forming radar and visual proposal features. Finally, a gated feature synthesis unit is used to perform sparse proposal fusion of the radar and visual proposal features to complete 3D target detection. Finally, the detection results are mixed. This invention employs multi-feature cascaded matching and batch trajectory management, feeding back the generated trajectory temporal information to a learnable perception probe at the front end. This enables an integrated, progressive, and cyclical model for proactive target detection and tracking guided by temporal information. The method fully considers the differences in characteristics between visual and radar data, extracting visual and radar features through sparse gating fusion. It also establishes an integrated proactive perception framework for 3D target detection and tracking, significantly improving the accuracy of 3D target detection and the stability of target tracking. The method achieves proactive, cyclical, and progressive 3D target detection and tracking through a single network, while sparse proposal fusion maintains low network complexity. This invention can be applied to target perception scenarios in autonomous driving. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a 3D target detection and tracking method based on the fusion of visual images and radar tensor sparse proposals provided in an embodiment of the present invention.

[0042] Figure 2 This is a visualization diagram of the keyframe 3D target detection results provided in the embodiments of the present invention, wherein (a-1) and (a-2) represent the detection results based on the pure radar method, (b-1) and (b-2) represent the detection results based on the pure vision method, (c-1) and (c-2) represent the detection results based on the 3D target detection method based on radar and vision fusion, and (d-1) and (d-2) represent the detection results of the 3D detection and tracking integrated active method provided by the present invention;

[0043] Figure 3 This is the target tracking result of three consecutive frames provided in the embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] This invention proposes a 3D target detection and tracking method based on the fusion of visual images and radar tensor sparse proposals. The method first extracts and establishes high-dimensional semantic features of visual multi-scale. Then, it uses a coupled radar echo intensity and spatial orientation perception collaborative enhancement network to extract high-dimensional spatial features of radar tensors at multiple scales. Next, learnable perception probes are mapped to radar and visual feature spaces respectively. By using a multi-head deformable attention decoupling method, different modal features are generalized and sparsed to form radar and visual proposal features. Then, gated sparse proposal fusion of radar and visual proposal features is carried out to complete 3D target detection. Finally, hybrid multi-feature cascaded matching and batch trajectory management are carried out to feed back trajectory temporal information to the front-end learnable perception probes, thereby realizing an integrated active cyclic progressive model for target detection and tracking based on temporal information guidance.

[0046] like Figure 1 As shown, the method specifically includes the following steps:

[0047] Step 1: Multi-scale semantic feature extraction and visual proposal feature generation of visual images

[0048] Image data is collected from vehicle-mounted cameras. A backbone network composed of convolutional networks (including ResNet50, ResNet101, and V2-99) is used to extract semantic texture features from the images. Then, a Feature Pyramid Network (FPN) is used to generate multi-scale semantic features. The generation method is as follows:

[0049] ;

[0050] in, Represents the visual high-dimensional semantic features at the s-th scale. This represents the input visual image.

[0051] A learnable perception probe is generated in the vehicle coordinate system. The perception probe is generated through random initialization and includes the probe position. and corresponding probe characteristics Combined with the projection matrix from vehicle body coordinates to image coordinates generated during sensor deployment calibration Multi-scale deformable attention is used for feature enhancement and visual feature proposal generation, as follows:

[0052] F v , o = [ ∑ s = 1 S ∑ k = 1 K v A s , k C ( F q , F v , s , H ego 2 image P ′ q + D P q , k ) ] N ;

[0053] in, Indicates visual proposal features. Let S represent the attention weight of the k-th deformable sampling point at the s-th scale. The attention weights are first randomly initialized and then adjusted and optimized through training. S represents the number of multi-scales. The number of visually deformable sampling points. Indicates based on probe coordinates P obtains the feature space. Corresponding features, and with specific features A function that enhances coupling, where N represents the superposition of N identical modules. This indicates an active adaptive correction of the probe coordinates after considering coordinate transformation deviation and time deviation, starting from the initial probe position. The small learnable positional offset is generated by adding a small amount of information, where the positional offset is generated by the initial probe's position-encoded feature (PE) and the probe feature. The sum is generated through a linear network, as shown in the following expression:

[0054] ;

[0055] This represents the position offset of the deformable attention at the k-th sampling point in the feature space, which is determined by the initial probe position-encoded feature PE and the probe feature. The sum is generated through a linear network, as shown in the following expression:

[0056] ;

[0057] Step 2: Radar Tensor Multi-Scale Spatial Feature Extraction and Radar Proposed Feature Generation

[0058] Vehicle-side radar data is collected, and radar tensor data is obtained through preprocessing, including analog-to-digital conversion, mixing, low-pass filtering, and Fourier transform. A coupled radar echo intensity and spatial orientation perception collaborative enhancement network (SCAN) is used to extract radar spatial features, and then a multi-scale spatial feature generator (FPN) is used to generate these features. As shown below:

[0059] ;

[0060] in, This represents the high-dimensional spatial features of the radar at the s-th scale. The input radar tensor is represented by SCAN, which denotes the coupled radar echo intensity and spatial orientation awareness co-enhancement network. It is composed of stacked SCA sub-networks. The SCA sub-network consists of two parts: the first part represents the gain on radar echo intensity, and the second part represents the gain on radar spatial orientation awareness. The two work together to enhance the extraction of radar spatial features, as shown below:

[0061] ;

[0062] Wherein, SCA represents the coupled radar echo intensity and spatial orientation sensing cooperative sub-network. Indicates input features, denoted by Sigmoid activation function, C represents convolution operation, Avg represents averaging the input features along the channel dimension, and Max represents maximizing the input features along the channel dimension. This indicates concatenation along the channel dimension. CRC represents a modular network composed of convolution, ReLU activation function, and convolution. AvgPool represents average pooling in the spatial dimension, and MaxPool represents maximum pooling in the spatial dimension.

[0063] Using the learnable sensor probe position from step one and corresponding probe characteristics Combined with the projection matrix from the vehicle body coordinates to the radar tensor coordinates generated during sensor deployment calibration Multi-scale deformable attention is used for feature enhancement and radar-proposed feature generation, as shown below:

[0064] F r , o = [ ∑ s = 1 S ∑ k = 1 K r A s , k C ( F q , F r , s , H ego 2 rad P ′ q + D P q , k ) ] N ;

[0065] in, Indicates radar proposal characteristics, This represents the number of deformable sampling points for the radar.

[0066] Step 3: Sparse Gated Fusion Based on Visual and Radar Proposed Features

[0067] Visual and radar proposal features are generated by mapping the same perceptual probes to the visual and radar feature spaces, respectively. Using perceptual probes with identical location and feature information, implicit association matching of features from different modalities can be achieved. The visual and radar proposal features that have completed association matching are concatenated to generate coarsely fused features. A linear network is then used to adaptively learn evidence from the coarsely fused proposal features. Then, an activation function is used to dynamically evaluate the differential confidence of evidence from different modalities, and proposal-level sparse gating fusion is completed based on the dynamically differentiated confidence. The details are as follows:

[0068] ;

[0069] in, This indicates the features of the vision and radar fusion proposal. This indicates splicing along the channel dimension. This represents the Sigmoid activation function.

[0070] Step 4: 3D target detection based on proposed features from visual and radar fusion

[0071] After obtaining the proposed features from visual and radar fusion, the target classification task head and regression task head are used respectively to extract the 3D target category and attribute information, as follows:

[0072] { c = Linear ( [ ReLU ( LN ( Linear ( F o ) ) ) ] M ) x , y , z , l , w , h , i = Linear ( [ ReLU ( Linear ( F o ) ) ] M ) ;

[0073] Where c represents the target category information, ReLU represents the activation function, LN represents LayerNorm normalization, M represents M identical modules stacked together, x, y, and z represent the target's 3D coordinates, and l, w, and h represent the target's 3D dimensions, which are length, width, and height, respectively. Indicates the target rotation angle.

[0074] Step 5: Hybrid multi-feature cascade matching between target detection results and historical flight paths

[0075] The matching between 3D target detection results and historical tracks is divided into two stages. The first stage is consistency matching based on the feedback of fusion proposal features, that is, the target detection process of the previous time step feeding back the fusion proposal features to the current time step. If the current target detection result is consistent with the fusion proposal features fed back from the previous time step, then the target corresponding to the fusion proposal features of the previous time step is matched with the detection result of the current time step.

[0076] Secondly, there is the multi-feature local-to-global cost minimization matching that couples position, high-dimensional features, and size. This method uses the position and size information from the target detection results and temporal track information, along with fused proposal features, to construct a local hybrid multi-feature cost matrix. Then, the Hungarian algorithm is used to achieve the local optimal matching between the detected target and the track target, as detailed below:

[0077] ;

[0078] Where D represents the local hybrid multi-feature cost matrix, and Let G represent the target position information remaining at the current and previous time steps after consistency matching, respectively. Let G represent the state transition equation, which is usually Newton's equations of motion. and These represent the fusion proposal features corresponding to the current time step and the previous time step, respectively. and These represent the target size information corresponding to the current time and the previous time, respectively. , and These represent the contributions of experience-derived information regarding location, proposed features, and size, respectively.

[0079] Step Six: Batch Track Management and Track Information Feedback Guidance

[0080] During the deep learning training of the structure of this invention, in response to the situation of multiple data inputs during deep learning training, the group cognitive trajectory management paradigm is used to process the trajectory inference, association matching and life cycle management of the entire data batch in parallel, so as to obtain the group cognitive trajectory time sequence information. The group cognitive trajectory time sequence information is adaptively fed back to the corresponding scenario steps one and two to expand the perception probe, so that the perception probe simultaneously includes the global statistical prior of the entire training data and the individual historical prior of the current trajectory, so as to realize the integrated active cyclic progression of 3D target detection and tracking. The group cognitive trajectory management paradigm performs independent trajectory management for target groups in different times and spaces in the real scene at the same training moment.

[0081] Then, the above detection and tracking method was verified and evaluated based on actual open-source test data. All steps and results were verified for effectiveness in Python. The verification results are shown below:

[0082] Table 1 compares the 3D target detection accuracy of the method of this invention with other methods. RD represents the pure radar-based method, VD represents the pure vision-based method, RVD represents the 3D target detection method based on radar and vision fusion, and RVDT represents the 3D detection and tracking integrated active method provided by this invention. As can be seen from the table, the detection accuracy of the pure radar and pure vision methods is lower than that of the 3D target detection method based on radar and vision fusion. The accuracy of the 3D target detection method based on radar and vision fusion is significantly lower than that of the 3D detection and tracking integrated active method provided by this invention. This indicates that the 3D target detection and tracking integrated active method based on visual image and radar tensor sparse proposal fusion provided by this invention can fully coordinate target detection and target tracking tasks, and achieve synergistic promotion among multiple tasks.

[0083] Table 1. Comparison of 3D target detection accuracy between the method of this invention and other methods.

[0084]

[0085] Figure 2This is a visualization of the 3D target detection results in keyframes. (a-1) and (a-2) represent the detection results based on a pure radar method, (b-1) and (b-2) represent the detection results based on a pure vision method, (c-1) and (c-2) represent the detection results based on a 3D target detection method using radar and vision fusion, and (d-1) and (d-2) represent the detection results of the 3D detection and tracking integrated active method provided by this invention. The left column represents image-based visualization, and the right column represents bird's-eye view-based visualization. As can be seen from the figure, the pure radar method has a significant number of missed detections and false alarms, the pure vision method also has obvious missed detections, and the overlap between the predicted and ground truth positions is poor. The 3D target detection method based on radar and vision fusion reduces missed detections to some extent, while the 3D detection and tracking integrated active method provided by this invention greatly improves the missed detection phenomenon and has a high overlap between the predicted and ground truth positions, indicating that the method of this invention significantly improves and enhances 3D target detection and localization.

[0086] Figure 3 The figure shows the target tracking results for three consecutive frames. As can be seen from the figure, the method provided by this invention can effectively and stably track the ID and achieve reliable target tracking.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D target detection and tracking method based on the fusion of visual images and radar tensor sparse proposals, characterized in that, The method includes the following steps: Step S1, Multi-scale semantic feature extraction and visual proposal feature generation of visual images: Collect image data from vehicle-mounted cameras, use a backbone network composed of convolutions to extract semantic texture features of the images, then use a feature pyramid network (FPN) to generate multi-scale semantic features, generate learnable perception probes in the vehicle coordinate system, combine the projection matrix from vehicle coordinates to image coordinates, and use multi-scale deformable attention to perform feature enhancement and visual proposal feature generation. Step S2, Radar Tensor Multi-Scale Spatial Feature Extraction and Radar Proposal Feature Generation: Collect vehicle-side radar data, obtain radar tensor data through preprocessing, extract radar spatial features using the coupled radar echo intensity and spatial orientation perception collaborative enhancement network SCAN, generate multi-scale spatial features using FPN, and then use learnable perception probe positions and corresponding probe features, combined with the projection matrix from vehicle coordinates to radar tensor coordinates, to perform feature enhancement and radar proposal feature generation using multi-scale deformable attention. Step S3, sparse gating fusion based on visual proposal features and radar proposal features: the visual proposal features and radar proposal features that have completed association matching are concatenated to generate coarse fused proposal features, and a linear network is used to perform adaptive evidence learning on the coarse fused proposal features. Then, the dynamic confidence differential evaluation of evidence of different modalities is performed through activation function, and proposal-level sparse gating fusion is completed based on the dynamic differential confidence. Step S4, 3D target detection based on visual and radar fusion proposal features: After obtaining visual and radar fusion proposal features, the target classification task head and regression task head are used respectively to extract the 3D target category and attribute information. Step S5, hybrid multi-feature cascade matching between target detection results and historical tracks: first, consistency matching is performed based on fused proposed features, and then multi-feature local and global cost minimization matching is performed by coupling position, high-dimensional features and size; Step S6, Batch Track Management and Track Information Feedback Guidance: The group cognitive track management paradigm is used to process the track inference, association matching and lifecycle management of the entire data batch in parallel. At the same time, the group cognitive track time sequence information is adaptively fed back to the corresponding scenario steps S1 and S2 to expand the perception probe. This allows the perception probe to simultaneously encompass the global statistical prior of the entire training data and the individual historical prior of the current track, so as to achieve integrated active cyclic progression of 3D target detection and tracking.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: Multi-scale semantic features are generated using the Feature Pyramid Network (FPN). , is represented as: ； in, Represents the visual high-dimensional semantic features at the s-th scale. Indicates the input visual image; A learnable sensing probe is generated in the vehicle coordinate system, and the sensing probe includes the probe position. and corresponding probe characteristics Combined with the projection matrix from vehicle body coordinates to image coordinates generated during sensor deployment calibration Multi-scale deformable attention is used for feature enhancement and visual feature proposal generation, as shown below: ； in, Indicates visual proposal features. Let S represent the attention weight of the k-th deformable sampling point at the s-th scale. The attention weights are first randomly initialized and then adjusted and optimized through training. S represents the number of multi-scales. The number of visually deformable sampling points. Indicates based on probe coordinates P obtains the feature space. Corresponding features, and with specific features A function that enhances coupling, where N represents the superposition of N identical modules. This indicates an active adaptive correction of the probe coordinates after considering coordinate transformation deviation and time deviation, starting from the initial probe position. The small learnable positional offset is generated by adding a small amount of information, where the positional offset is generated by the initial probe's position-encoded feature (PE) and the probe feature. The sum is generated through a linear network, represented as: ； This represents the position offset of the deformable attention at the k-th sampling point in the feature space, which is determined by the initial probe position-encoded feature PE and the probe feature. The sum is generated through a linear network, represented as: 。 3. The method according to claim 1, characterized in that, Step S2 specifically includes: Radar spatial features are extracted using a coupled radar echo intensity and spatial orientation awareness co-enhancement network (SCAN), and then multi-scale spatial features are generated using a multi-scale process network (FPN). , is represented as: ； in, This represents the high-dimensional spatial features of the radar at the s-th scale. Representing the input radar tensor, SCAN is composed of stacked SCA subnetworks. The SCA subnetwork consists of two parts: the first part represents the gain for radar echo intensity, and the second part represents the sensing gain for radar spatial orientation. The two work together to enhance the extraction of radar spatial features, as shown below: ； Wherein, SCA represents the coupled radar echo intensity and spatial orientation sensing cooperative sub-network. Indicates input features, denoted by Sigmoid activation function, C represents convolution operation, Avg represents averaging the input features along the channel dimension, and Max represents maximizing the input features along the channel dimension. This indicates concatenation along the channel dimension; CRC represents a modular network composed of convolution, ReLU activation function, and convolution; AvgPool represents average pooling in the spatial dimension; and MaxPool represents maximum pooling in the spatial dimension. Using the learnable sensing probe position in step S1 and corresponding probe characteristics Combined with the projection matrix from the vehicle body coordinates to the radar tensor coordinates generated during sensor deployment calibration Multi-scale deformable attention is used for feature enhancement and radar-proposed feature generation, as follows: ； in, Indicates radar proposal characteristics, This represents the number of deformable sampling points for the radar.

4. The method according to claim 1, characterized in that, Step S3 is specifically represented as follows: ； in, This indicates the features of the vision and radar fusion proposal. This indicates splicing along the channel dimension. This represents the Sigmoid activation function.

5. The method according to claim 1, characterized in that, Step S4 is specifically represented as follows: ； Where c represents the target category information, ReLU represents the activation function, LN represents LayerNorm normalization, M represents M identical modules stacked together, x, y, and z represent the target's 3D coordinates, and l, w, and h represent the target's 3D dimensions, which are length, width, and height, respectively. Indicates the target rotation angle.

6. The method according to claim 1, characterized in that, The hybrid multi-feature cascade matching is divided into two stages. The first stage is consistency matching based on the feedback of the fusion proposal features. That is, the fusion proposal features of the previous time are fed back to the target detection process of the current time. If the current target detection result is consistent with the fusion proposal features fed back in the previous time, then the target corresponding to the fusion proposal features of the previous time is matched with the detection result of the current time. Secondly, there is the multi-feature local-to-global cost minimization matching that couples position, high-dimensional features, and size. This method uses the position and size information from the target detection results and temporal track information, along with fused proposal features, to construct a local hybrid multi-feature cost matrix. Then, the Hungarian algorithm is used to achieve the local optimal matching between the detected target and the track target, as shown below: ； Where D represents the local hybrid multi-feature cost matrix, and Let G represent the target position information remaining at the current time and the previous time after consistency matching, respectively, and let G represent the state transition equation. and These represent the fusion proposal features corresponding to the current time step and the previous time step, respectively. and These represent the target size information corresponding to the current time and the previous time, respectively. , and These represent the contributions of experience-derived information regarding location, proposed features, and size, respectively.

Images