3D object detection and tracking method based on fusion of visual image and radar tensor sparse proposal

Abstract

The application provides a 3D target detection and tracking method based on visual image and radar tensor sparse proposal fusion, first, color texture information of the visual image is generalized and extracted, and multi-scale semantic high-dimensional features are established; second, the multi-scale spatial high-dimensional features of the radar tensor are extracted by using SCAN, and then the learnable perception probe is respectively mapped to the radar and visual high-dimensional feature space, and the multi-head deformable attention is used to generalize and sparsify different modal features, so as to form radar and visual proposal features; then, the sparse proposal fusion of the radar and visual proposal features is carried out, and 3D target detection is completed; finally, mixed multi-feature cascade matching and batch track management are carried out on the detection result, and the generated track time sequence information is fed back to the front-end learnable perception probe, so as to realize the active target detection and tracking integrated cycle progressive network based on time sequence information guidance. The application scheme establishes an integrated active perception framework of single-frame target detection and time sequence target tracking, and enhances the reliability of multi-source sensor in the environment target perception of automatic driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of automatic driving, deep learning and multi-sensor fusion, and particularly relates to a 3D target detection and tracking method based on visual image and radar tensor sparse proposal fusion. BACKGROUND

[0002] The perception of environmental targets is crucial in automatic driving, and millimeter wave radar (referred to as radar) and camera (also referred to as visual sensor) are currently the sensors with higher complementarity in automatic driving. Specifically, radar can work all day long and can detect key information such as target speed and distance, but it lacks semantic texture features and is difficult to achieve reliable identification and classification of targets. The camera has rich semantic texture information, and is low in cost and rapid in technological development, but it is extremely susceptible to strong light, rain, snow, fog and other extreme environments, and it does not have the ability to measure distance and speed. Therefore, it is extremely critical to carry out fusion research of visual and radar sensors (referred to as radar-visual fusion).

[0003] Most current radar and vision fusion methods use radar point cloud and visual image as input, and extract high-dimensional features from radar point cloud and visual image respectively, and finally realize target detection task. This kind of method uses radar point cloud with high degree of data processing as input, loses a large amount of information captured by radar, and is difficult to obtain reliable fusion features. Using radar tensor, that is, the range-azimuth map generated by radar ADC (Analog-to-Digital Converter) data through FFT (Fast Fourier Transform) and DOA (Direction of Arrival) estimation, to carry out fusion research with visual image can not only reduce the information loss of radar data processing environment, but also avoid the problem of difficult extraction of high-dimensional features of sparse radar point cloud. However, there is currently a lack of reliable dataset of radar tensor and visual image, and fusion research based on visual image and radar tensor is extremely rare. The literature "Low-Level Sensor Fusion for 3D Vehicle Detection Using Radar Range-Azimuth Heatmap and Monocular Image[C], ACCV, 2020." uses a two-stage network based on feature map fusion to realize target detection task, but the network uses simple splicing to process and fuse high-dimensional feature maps of different sensors, lacks in-depth exploration of features of different sensors, and dense fusion based on feature map will increase memory consumption and reduce running efficiency. The literature "Echoes beyond points: unleashing the power of raw radar data in multi-modality fusion[J]. Advances in Neural Information Processing Systems, 2023" proposes a feature map attention fusion method based on polar coordinate alignment, and at the same time, the method uses passively input data to complete the target detection task, and does not explore the synergistic promotion between detection and tracking multi-task. SUMMARY

[0004] In view of the above problems, the present application provides a 3D target detection and tracking method based on sparse proposal fusion of visual image and radar tensor, to solve the problems of limited running efficiency of existing methods based on feature map fusion and passive processing of single detection task, so as to realize active radar and vision fusion perception with high reliability and multi-task integration.

[0005] A 3D target detection and tracking method based on sparse proposal fusion of visual image and radar tensor, specifically comprising the following steps:

[0006] Step S1, visual image multi-scale semantic feature extraction and visual proposal feature generation: collect vehicle end camera image data, use a backbone network composed of convolution to extract image semantic texture features, then use a feature pyramid network FPN to generate multi-scale semantic features, generate learnable perception probes in the vehicle body coordinate system, use a multi-scale deformable attention for feature enhancement and visual proposal feature generation, combined with the projection matrix from the vehicle body coordinate to the image coordinate;

[0007] Step S2, radar tensor multi-scale spatial feature extraction and radar proposal feature generation: collect vehicle end radar data, obtain radar tensor data through preprocessing, use a sensing collaborative enhancement network SCAN coupled with radar echo intensity and spatial orientation to extract radar spatial features, then use FPN to generate multi-scale spatial features, then use a multi-scale deformable attention for feature enhancement and radar proposal feature generation, combined with the projection matrix from the vehicle body coordinate to the radar tensor coordinate, using the learned perception probe position and the corresponding probe feature;

[0008] Step S3, sparse gating fusion based on visual proposal features and radar proposal features: splicing the visual proposal features and radar proposal features that have completed association matching to generate coarse fusion proposal features, and using a linear network to perform adaptive evidence learning on the coarse fusion proposal features, then through an activation function to dynamically differentiate the confidence of different modal evidences, and based on the dynamic differentiated confidence to complete the proposal level sparse gating fusion;

[0009] Step S4, 3D target detection based on visual and radar fusion proposal features: after obtaining the visual and radar fusion proposal features, respectively using a target classification task head and a regression task head to realize the extraction of 3D target class and attribute information;

[0010] Step S5, mixed multi-feature cascaded matching between target detection results and historical tracks: first based on the fusion proposal features to perform consistency matching, then perform multi-feature local-global cost minimization matching coupled with position, high-dimensional features and size;

[0011] Step S6, batch track management and track information feedback guidance: using a group cognitive track management paradigm to process the track deduction, association matching and life cycle management of the entire data batch in parallel, at the same time, adaptively feeding back the group cognitive track timing information to the corresponding scene step S1 and step S2 to expand the perception probe, so that the perception probe simultaneously includes the global statistical prior of the entire training data and the individual historical prior of the current track, to realize the integrated active cyclic progression of 3D target detection and tracking.

[0012] Further, a feature pyramid network FPN is used to generate multi-scale semantic features , denoted as:

[0013] ;

[0014] wherein, represents the visual high-dimensional semantic feature of the s-th scale, represents the input visual image;

[0015] A learnable perception probe is generated in the vehicle body coordinate system, the perception probe includes probe position and corresponding probe feature , a projection matrix generated by combining sensor deployment and calibration is used to generate a projection matrix from the vehicle body coordinate to the image coordinate , feature enhancement and visual proposal feature generation are performed using multi-scale deformable attention, which is represented as:

[0016] F v , o = [ ∑ s = 1 S ∑ k = 1 K v A s , k I ( F q , F v , s , H ego 2 img P ′ q + Δ P q , k ) ] N ;

[0017] wherein, represents the visual proposal feature, represents the attention weight of the k-th deformable sampling point at the s-th scale, the attention weight is randomly initialized first, and then adjusted and optimized through training, S represents the number of multi-scales, is the number of visual deformable sampling points, represents the corresponding feature in the feature space P obtained according to the probe coordinate , and the function of coupling and enhancing a specific feature , N represents that there are N identical modules stacked, represents the active adaptive correction of the probe coordinate after considering the coordinate transformation deviation and the time deviation, which is generated by adding a small learnable position offset to the initial probe position , wherein the position offset is generated by a linear network Linear from the sum of the position encoding feature PE of the initial probe and the probe feature , which is represented as:

[0018] ;

[0019] represents the position offset of the k-th sampling point of the deformable attention in the feature space, which is generated by a linear network Linear from the sum of the position encoding feature PE of the initial probe and the probe feature , which is represented as:

[0020] .

[0021] Further, radar spatial features are extracted using a coupling radar echo intensity and spatial orientation perception collaborative enhancement network SCAN, and then multi-scale spatial features are generated using FPN, which is represented as:

[0022] ;

[0023] wherein, represents the radar high-dimensional spatial feature of the s-th scale, represents the input radar tensor, SCAN is stacked by SCA subnetworks, the SCA subnetwork includes two parts, the first part represents the gain of radar echo intensity, and the second part represents the perception gain of radar spatial orientation, and the two parts are cooperated to enhance the extraction of radar spatial features, as follows:

[0024] ;

[0025] wherein, SCA represents a coupled radar echo intensity and spatial orientation perception cooperative subnetwork, represents the input feature, represents a Sigmoid activation function, C represents a convolution operation, Avg represents taking an average value of the input feature along the channel dimension, and Max represents taking a maximum value of the input feature along the channel dimension, represents splicing along the channel dimension, CRC represents a module network composed of convolution, ReLU activation function and convolution, AvgPool represents average pooling in the spatial dimension, and MaxPool represents maximum value pooling in the spatial dimension.

[0026] The learnable perception probe position in step S1 is used and the corresponding probe feature , combined with the projection matrix of the vehicle body coordinates to the radar tensor coordinates generated during sensor deployment calibration , feature enhancement and radar proposal feature generation are performed using multi-scale deformable attention, which is represented as:

[0027] F r , o = [ ∑ s = 1 S ∑ k = 1 K r A s , k I ( F q , F r , s , H ego 2 rad P ′ q + Δ P q , k ) ] N ;

[0028] wherein, represents the radar proposal feature, is the number of radar deformable sampling points.

[0029] Further, the step S3 is specifically represented as:

[0030] ;

[0031] wherein, represents the visual and radar fusion proposal feature, represents splicing along the channel dimension, represents a Sigmoid activation function.

[0032] Further, the step S4 is specifically represented as:

[0033] { c = Linear ( [ ReLU ( LN ( Linear ( F o ) ) ) ] M ) x , y , z , l , w , h , θ = Linear ( [ ReLU ( Linear ( F o ) ) ] M )

[0034] wherein c represents target category information, ReLU represents an activation function, LN represents LayerNorm normalization processing, M represents M identical module stacks, x, y and z represent 3D coordinates of the target, and l, w and h represent 3D dimensions of the target, respectively, length, width and height, represents a target rotation angle.

[0035] Further, the hybrid multi-feature cascade matching is divided into two stages. First, consistency matching based on fusion proposal feature feedback, that is, the fusion proposal feature of the previous moment is fed back to the target detection process of the current moment, and the current target detection result keeps consistency with the fusion proposal feature fed back at the previous moment, then the target corresponding to the fusion proposal feature at the previous moment is matched with the detection result at the current moment;

[0036] Second, multi-feature local-global cost minimum matching coupled with position, high-dimensional feature and size. This method uses the position, size information and fusion proposal feature in the target detection result and the time series track information to construct a local hybrid multi-feature cost matrix, and then uses the Hungarian algorithm to realize the local optimal matching of the detected target and the track target, which is represented as follows:

[0037]

[0038] wherein D represents a local hybrid multi-feature cost matrix, and respectively represent the target position information of the current moment and the previous moment after consistency matching, G represents a state transition equation, and the state transition equation is usually a Newton motion equation, and respectively represent the fusion proposal features corresponding to the current moment and the previous moment, and respectively represent the target size information corresponding to the current moment and the previous moment, , and respectively represent the contribution degrees of the position, proposal feature and size information generated by experience.

[0039] ​​The application provides a 3D target detection and tracking method based on visual image and radar tensor sparse proposal fusion. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief introductions will be given to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0041] Figure 1 FIG. 1 is a flowchart of a 3D target detection and tracking method based on visual image and radar tensor sparse proposal fusion provided by an embodiment of the present application;

[0042] Figure 2 FIG. 2 is a visual diagram of key frame 3D target detection results provided by an embodiment of the present application, wherein (a-1) and (a-2) represent detection results of a pure radar-based method, (b-1) and (b-2) represent detection results of a pure vision-based method, (c-1) and (c-2) represent detection results of a 3D target detection method based on radar and vision fusion, and (d-1) and (d-2) represent detection results of a 3D detection and tracking integrated active method provided by the present application;

[0043] Figure 3 FIG. 3 is a target tracking result of consecutive 3 frames provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0045] The present application provides a 3D target detection and tracking method based on visual image and radar tensor sparse proposal fusion. The method first extracts and establishes visual multi-scale semantic high-dimensional features, then uses a coupled radar echo intensity and spatial orientation perception collaborative enhancement network to extract multi-scale spatial high-dimensional features of the radar tensor, and then maps the learnable perception probe to the radar and visual feature space respectively, and uses a multi-head deformable attention decoupling to generalize and sparse different modal features, forms radar and visual proposal features, and then carries out gated sparse proposal fusion of the radar and visual proposal features, and completes 3D target detection, and finally carries out hybrid multi-feature cascade matching and batch track management, and feeds back the track time sequence information to the front-end learnable perception probe, so as to realize a target detection and tracking integrated active cyclic progressive model guided by time sequence information.

[0046] As shown in Figure 1 , the method specifically comprises the following steps:

[0047] Step 1: Visual image multi-scale semantic feature extraction and visual proposal feature generation

[0048] Collect vehicle end camera image data, use a backbone network composed of convolution to extract image semantic texture features, the backbone network includes ResNet50, ResNet101, V2-99, and then use a feature pyramid network FPN to generate multi-scale semantic features , the generation method is as follows:

[0049] ;

[0050] Among them, represents the visual high-dimensional semantic feature of the s-th scale, represents the input visual image.

[0051] Learnable perception probes are generated in the vehicle body coordinate system. The perception probe is generated by random initialization. The perception probe includes probe position and corresponding probe feature , and the projection matrix from the vehicle body coordinate to the image coordinate is generated in combination with the sensor deployment calibration., feature enhancement and visual proposal feature generation using multi-scale deformable attention, in the following way:

[0052] F v , o = [ ∑ s = 1 S ∑ k = 1 K v A s , k I ( F q , F v , s , H ego 2 img P ′ q + Δ P q , k ) ] N

[0053] wherein, represents the visual proposal feature, represents the attention weight of the kth deformable sampling point at the st scale, which is randomly initialized first and then adjusted and optimized through training, and S represents the number of multi-scales, is the number of visual deformable sampling points, represents the corresponding feature in the feature space P obtained according to the probe coordinates P, and the specific feature coupling enhancement function, N represents that there are N identical modules superimposed, represents the active adaptive correction of the probe coordinates after considering the coordinate transformation deviation and the time deviation, which is generated by adding a small learnable position offset to the initial probe position , wherein the position offset is generated by the sum of the position encoding feature PE of the initial probe and the probe feature through a linear network Linear, and the expression is as follows:

[0054]

[0055] represents the position offset of the deformable attention at the kth sampling point in the feature space, which is generated by the sum of the position encoding feature PE of the initial probe and the probe feature through a linear network Linear, and the expression is as follows:

[0056]

[0057] Step 2: Radar tensor multi-scale space feature extraction and radar proposal feature generation

[0058] Collect the radar data at the vehicle end, obtain the radar tensor data through preprocessing, the preprocessing includes digital-to-analog conversion, frequency mixing, low-pass filtering and Fourier transform, etc., use the coupling radar echo intensity and spatial orientation perception cooperative enhancement network SCAN to extract the radar spatial features, then use FPN to generate multi-scale spatial features , as shown below:

[0059]

[0060] wherein, represents the radar high-dimensional spatial feature at the st scale, ​​​​represents the input radar tensor, SCAN represents the coupled radar return intensity and spatial orientation perception collaborative enhancement network, which is stacked by SCA sub-networks, the SCA sub-networks include two parts, the first part represents the gain of radar return intensity, and the second part represents the perception gain of radar spatial orientation, and the two parts are collaboratively enhanced to extract the spatial features of the radar, as follows:

[0061] ;

[0062] wherein SCA represents a coupled radar return intensity and spatial orientation perception collaborative sub-network, represents the input feature, represents a Sigmoid activation function, C represents a convolution operation, Avg represents taking an average value of the input feature along the channel dimension, and Max represents taking a maximum value of the input feature along the channel dimension, represents splicing along the channel dimension, CRC represents a module network composed of convolution, ReLU activation function and convolution, AvgPool represents average pooling in the spatial dimension, and MaxPool represents maximum value pooling in the spatial dimension.

[0063] using the learnable perception probe position in step one and the corresponding probe feature , combining the projection matrix from the vehicle body coordinates to the radar tensor coordinates generated during sensor deployment calibration , using multi-scale deformable attention for feature enhancement and radar proposal feature generation, as follows:

[0064] F r , o = [ ∑ s = 1 S ∑ k = 1 K r A s , k I ( F q , F r , s , H ego 2 rad P ′ q + Δ P q , k ) ] N ;

[0065] wherein, represents the radar proposal feature, is the number of radar deformable sampling points.

[0066] Step three: sparse gating fusion based on visual and radar proposal features

[0067] The visual and radar proposal features are respectively mapped to the visual and radar feature spaces by the same perception probe, and the perception probe with the same position and feature information can realize the invisible correlation matching of different modal features. The visual and radar proposal features that complete the correlation matching are spliced to generate coarse fusion features, and a linear network is used to adaptively learn the evidence of the coarse fusion proposal features. Then, the different modal evidences are dynamically differentiated and evaluated by an activation function, and the proposal level sparse gating fusion is completed based on the dynamic differentiated confidence, as follows:

[0068] ;

[0069] wherein, represents the visual and radar fusion proposal feature, represents splicing along the channel dimension, represents a Sigmoid activation function.

[0070] Step four: 3D target detection based on visual and radar fusion proposal features

[0071] After obtaining the visual and radar fusion proposal features, the extraction of 3D target class and attribute information is realized by using target classification task head and regression task head respectively, as follows:

[0072] { c = Linear ( [ ReLU ( LN ( Linear ( F o ) ) ) ] M ) x , y , z , l , w , h , θ = Linear ( [ ReLU ( Linear ( F o ) ) ] M ) ;

[0073] wherein, c represents target class information, ReLU represents an activation function, LN represents LayerNorm normalization processing, M represents M identical module stacks, x, y and z represent 3D coordinates of the target, l, w and h represent 3D dimensions of the target, respectively, length, width and height, represents the target rotation angle.

[0074] Step five: mixed multi-feature cascade matching between target detection results and historical tracks

[0075] The matching between 3D target detection results and historical tracks is divided into two stages. First, consistent matching based on fusion proposal feature feedback, that is, the fusion proposal feature of the last moment is fed back to the target detection process of the current moment. If the current target detection result is consistent with the fusion proposal feature fed back at the last moment, the target corresponding to the fusion proposal feature at the last moment is matched with the detection result at the current moment.

[0076] Second, multi-feature local-global cost minimum matching coupled with position, high-dimensional features and size. This method uses the position, size information and fusion proposal features in the target detection result and the time series track information to construct a local mixed multi-feature cost matrix, and then uses the Hungarian algorithm to realize the local optimal matching of the detected target and the track target, as follows:

[0077] ;

[0078] wherein, D represents a local mixed multi-feature cost matrix, and respectively represent the target position information of the current moment and the last moment remaining after consistent matching, G represents a state transition equation, which is usually a Newton motion equation, and respectively represent the fusion proposal features corresponding to the current moment and the last moment, and respectively represent target size information corresponding to the current time and the last time, 、 and respectively represent the contribution degree of the experience generated position, proposed feature and size information.

[0079] Step six: batch track management and track information feedback guidance

[0080] In the process of training the structure of the application, for the case of multi-scene data input in the process of deep learning training, the group cognitive track management paradigm is used to process the track deduction, correlation matching and life cycle management of the whole data batch in parallel, to obtain group cognitive track timing information, and the group cognitive track timing information is adaptively fed back to the corresponding scene step one and step two to expand the perception probe, so that the perception probe simultaneously includes the global statistical prior of the whole training data and the individual historical prior of the current track, to realize the integrated active cyclic progressive of 3D target detection and tracking, wherein the group cognitive track management paradigm is independent track management for target groups at different times and different spaces in the real scene at the same training time.

[0081] Then the above detection and tracking method is verified and evaluated based on the measured open source data, all steps and results are verified on Python for effectiveness, and the verification results are as follows:

[0082] Table 1 is the comparison of 3D target detection accuracy of the method of the application and other methods, RD represents the method based on pure radar, VD represents the method based on pure vision, RVD represents the 3D target detection method based on radar and vision fusion, and RVDT represents the 3D detection and tracking integrated active method provided by the application. As can be seen from the table, the detection accuracy of pure radar and pure vision method is lower than that of the 3D target detection method based on radar and vision fusion, and the accuracy of the 3D target detection method based on radar and vision fusion is significantly lower than that of the 3D detection and tracking integrated active method provided by the application, which shows that the 3D target detection and tracking integrated active method based on vision image and radar tensor sparse proposal fusion provided by the application can fully cooperate with the target detection and tracking tasks, and realize the collaborative promotion among multi-tasks.

[0083] Table 1 Comparison of 3D target detection accuracy of the method of the application and other methods

[0084]

[0085] Figure 2is a visualized schematic diagram of key frame 3D target detection result, wherein, (a-1) and (a-2) represent detection results of a pure radar-based method, (b-1) and (b-2) represent detection results of a pure vision-based method, (c-1) and (c-2) represent detection results of a radar and vision fusion-based 3D target detection method, (d-1) and (d-2) represent detection results of the 3D detection and tracking integrated active method provided by the application, the left one represents image-based visualization, and the right one represents bird's-eye view-based visualization. As can be seen from the figure, the pure radar-based method has more missed detection and false alarm, the pure vision-based method also has obvious missed detection, and the position coincidence degree of prediction and true value is poor, the radar and vision fusion-based 3D target detection method reduces missed detection to a certain extent, and the 3D detection and tracking integrated active method provided by the application can greatly improve the missed detection phenomenon, and the position coincidence degree of prediction and true value is high, which indicates that the method of the application has obvious improvement and enhancement on 3D target detection and positioning.

[0086] Figure 3 is target tracking result of continuous 3 frames, as can be seen from the figure, the method provided by the application can effectively stabilize tracking ID and realize reliable target tracking.

[0087] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A 3D target detection and tracking method based on visual image and radar tensor sparse proposal fusion, characterized in that, The method comprises the following steps: Step S1, visual image multi-scale semantic feature extraction and visual proposal feature generation: collecting vehicle end camera image data, using a backbone network composed of convolution to extract image semantic texture features, then using a feature pyramid network FPN to generate multi-scale semantic features, generating a learnable perception probe in the vehicle body coordinate system, combining the projection matrix from the vehicle body coordinate to the image coordinate, using multi-scale deformable attention for feature enhancement and visual proposal feature generation; Step S2, radar tensor multi-scale spatial feature extraction and radar proposal feature generation: collecting vehicle end radar data, obtaining radar tensor data through preprocessing, using a coupled radar echo intensity and spatial orientation perception collaborative enhancement network SCAN to extract radar spatial features, then using FPN to generate multi-scale spatial features, then using the learnable perception probe position and the corresponding probe feature, combining the projection matrix from the vehicle body coordinate to the radar tensor coordinate, using multi-scale deformable attention for feature enhancement and radar proposal feature generation; Step S3, sparse gating fusion based on visual proposal features and radar proposal features: splicing the visual proposal features and the radar proposal features that have completed the association matching to generate coarse fusion proposal features, and using a linear network to perform adaptive evidence learning on the coarse fusion proposal features, then using an activation function to dynamically differentiate the confidence of different modal evidences, and completing proposal-level sparse gating fusion based on the dynamic differentiated confidence; Step S4, 3D target detection based on visual and radar fusion proposal features: after obtaining the visual and radar fusion proposal features, using a target classification task head and a regression task head respectively to realize the extraction of 3D target class and attribute information; Step S5, mixed multi-feature cascade matching between target detection results and historical tracks: first, consistency matching based on fusion proposal features, then multi-feature local-global cost minimum matching coupling position, high-dimensional features and size; Step S6, batch track management and track information feedback guidance: using a group cognitive track management paradigm to process the track deduction, association matching and life cycle management of the entire data batch in parallel, and simultaneously adaptively feeding back the group cognitive track timing information to the corresponding scene steps S1 and S2 to expand the perception probe, so that the perception probe simultaneously includes the global statistical prior of the entire training data and the individual historical prior of the current track, to realize the integrated active cycle progression of 3D target detection and tracking.

2. The method of claim 1, wherein, The step S1 specifically comprises: Generating multi-scale semantic features using a feature pyramid network, FPN is represented as: ； wherein, represents the visual high-dimensional semantic feature of the s-th scale, represents an input visual image; A learnable perception probe is generated in the vehicle body coordinate system, the perception probe contains probe position and corresponding probe features , and a projection matrix generated by calibrating the vehicle body coordinate to the image coordinate when the sensor is deployed , feature enhancement and visual proposal feature generation are performed using multi-scale deformable attention, represented as: ； wherein, denotes the visual proposal feature, denotes the attention weight of the k-th deformable sampling point at the s-th scale, which is randomly initialized at first and then adjusted and optimized through training, and S denotes the number of multi-scales, is the number of visual deformable sampling points, denotes the probe coordinates Pacquired feature space corresponding features in the feature space, and the specific feature is coupled with the enhanced function, and N denotes that there are N identical modules stacked, denotes the active adaptive correction of the probe coordinates after considering the coordinate transformation deviation and the time deviation, which is generated by adding a small learnable position offset to the initial probe position , wherein the position offset is generated by the sum of the position encoding feature PE of the initial probe and the probe feature , which is generated through a linear network Linear, and is denoted as: ； denotes the position offset of the deformable attention at the k-th sample point in the feature space, which is generated by the sum of the position encoding feature PE of the initial probe and the probe feature through a linear network Linear, denoted as: 。 3. The method of claim 1, wherein, The step S2 specifically comprises: Radar spatial features are extracted using coupled radar return intensity and spatial orientation awareness cooperative awareness network (SCAN), and then multi-scale spatial features are generated using FPN , is represented as: ； wherein, represents the radar high-dimensional spatial feature of the s-th scale, represents the input radar tensor, SCAN is stacked by SCA subnetworks, the SCA subnetworks include two parts, the first part represents the gain to the radar echo intensity, and the second part represents the perception gain to the radar spatial direction, and the two parts are cooperated to enhance the extraction of the radar spatial feature together, as shown in the following formula: ； wherein SCA denotes a coupled radar return intensity and spatial orientation awareness subnetwork, denotes an input feature, denotes a sigmoid activation function, C denotes a convolution operation, Avg denotes taking an average value of an input feature along a channel dimension, Max denotes taking a maximum value of an input feature along a channel dimension, denotes concatenation along a channel dimension, CRC denotes a module network composed of a convolution, a ReLU activation function, and a convolution, AvgPool denotes average pooling in a spatial dimension, and MaxPool denotes maximum value pooling in a spatial dimension. Using the learnable perception probe positions in step S1 and corresponding probe features , a projection matrix that maps the body coordinates produced by the sensor deployment to the radar tensor coordinates is calibrated , feature enhancement and radar proposal feature generation using multi-scale deformable attention, denoted as: ； wherein, represents a radar-proposed feature, is the radar deformable sample number.

4. The method of claim 1, wherein, The step S3 is specifically represented as: ； wherein, represents a vision and radar fusion proposal feature, represents concatenation along the channel dimension, represents a sigmoid activation function.

5. The method of claim 1, wherein, The step S4 is specifically represented as: ； wherein c represents target category information, ReLU represents an activation function, LN represents LayerNorm normalization processing, M represents M identical module stacks, x, y, and z represent 3D coordinates of a target, l, w, and h represent 3D dimensions of a target, respectively, length, width, and height, represents a target rotation angle.

6. The method of claim 1, wherein, The mixed multi-feature cascade matching is divided into two stages, first, consistency matching based on fusion proposal feature feedback, that is, the fusion proposal feature of the last moment is fed back to the target detection process of the current moment, and the current target detection result is consistent with the fusion proposal feature fed back from the last moment, then the target corresponding to the fusion proposal feature of the last moment is matched with the detection result of the current moment; Secondly, the multi-feature local-global cost minimum matching method coupled with position, high-dimensional feature and size, which uses the position, size information and fusion proposal features in the target detection results and time-series track information to construct a local hybrid multi-feature cost matrix, and then uses the Hungarian algorithm to realize the local optimal matching of the detected targets and track targets, as follows: ； where D denotes a local mixed multi-feature cost matrix, and respectively denote the target location information of the current time and the last time remaining after consistent matching, G denotes a state transition equation, and respectively denote the fusion proposal features corresponding to the current time and the last time, and respectively denote the target size information corresponding to the current time and the last time, , and respectively denote the contribution degrees of the location, the proposal feature and the size information generated by experience.

Images