Camera and millimeter wave radar fusion three-dimensional target detection method based on time sequence fusion

Abstract

The invention discloses a camera and millimeter wave radar fusion three-dimensional target detection method based on time sequence fusion, and belongs to the technical field of three-dimensional target detection of computer vision. The invention provides a time sequence modeling framework integrating a camera and a millimeter-wave radar for solving the problems that pure visual perception lacks space measurement capability and a traditional time sequence fusion strategy has limitation in the aspects of dynamic target alignment and fusion efficiency. According to the framework, firstly, clustering processing is carried out on radar point clouds, instance features are extracted, and the instance features are used as initialization input of self-adaptive query, so that the number of iterations of a decoder is effectively reduced. By introducing a time sequence transmission mechanism based on a query instance, high calculation overhead caused by global feature alignment is avoided, and the motion state and time-space characteristics of a dynamic target are captured more accurately. A local self-attention mechanism is constructed by introducing a distance penalty term, close-range matching between query instances is realized, and the spatial alignment precision and fusion effect of multi-source data are further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional target detection based on computer vision, and particularly relates to a camera and millimeter wave radar fusion three-dimensional target detection method based on time sequence fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the intelligent wave is deeply changing people's daily life, especially in the field of travel. The intelligent and electric transformation of the automobile industry is accelerating, and driverless cars have become a new direction for future transportation development. At present, L2 level (assisted driving) and L2+ level (advanced assisted driving) technology is the most widely used.

[0003] Traditional automatic driving systems can usually be divided into three main modules, among which the perception system as the first link plays a role similar to the human eye, providing key basic support for a series of subsequent functions such as trajectory prediction, path planning and motion control. The perception system mainly relies on vehicle-mounted sensors to obtain surrounding environment information, and through 3D target detection algorithm to accurately model the environment. At present, the mainstream vehicle-mounted sensors include cameras, laser radars and millimeter wave radars, etc. The multi-vehicle-mounted sensor fusion research for automatic driving mainly focuses on the combination of cameras and laser radars, however, laser radars are difficult to work effectively in bad weather conditions such as rain, snow and hail, and their high cost usually makes them only carried on high-end vehicles, limiting large-scale application. In contrast, millimeter wave radars show important research value and application potential due to their robustness in bad weather and unique advantages in speed measurement and long-distance ranging. Cameras and millimeter wave radars have significant complementary advantages in characteristics, and this complementarity provides a very potential solution for multi-modal perception in complex scenes.

[0004] Currently, 3D object detection based on camera and millimeter-wave radar fusion is still in the development stage. According to the mainstream research scheme, the fusion method can be mainly divided into four types: data-level fusion, feature-level fusion, object-level fusion, and mixed-level fusion. (1) Data-level fusion is to fuse the raw or pre-processed data of millimeter-wave radar and camera at the initial stage of the deep learning model. For example, Nabati proposed a region proposal network, which generates target proposal boxes through radar point cloud. Although this method narrows down the target search range and speeds up the inference, it is prone to missed detection when the quality of radar point cloud is poor (Nabati R., Qi H. Rrpn: Radar region proposal network for object detection in autonomous vehicles [A]. 2019 IEEE International Conference on Image Processing [C]. Piscataway, NJ: IEEE, 2019: 3093-3097).(2) Feature-level fusion maps millimeter-wave radar and image data to the same feature space with high-level semantics through two encoders, and then completes multimodal fusion in this space. For example, Lin et al. use a dual-stream encoder to extract radar features, learn local and global radar features through Injection and Extraction modules, and finally use a deformable cross-attention mechanism to dynamically align and fuse radar and camera BEV features (Lin Z., Liu Z., Xia Z., et al. Rcbevdet: Radar-camera fusion in bird's eye view for 3D object detection [A]. 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition [C]. Piscataway, NJ: IEEE, 2024: 14928-14937).(3) In the object-level fusion method, millimeter-wave radar and visual sensors independently perform target detection, and the matching association strategy between millimeter-wave radar and image detection targets plays a decisive role in the final detection result.(4) Mixed-level fusion is to both symmetrically fuse the two modalities at the same stage and asymmetrically fuse the data of different stages.For example, CenterFusion uses image data to preliminarily detect target positions based on CenterNet, and accurately matches radar targets with image targets through a view cone association strategy, and further enhances image features by using the framed radar features, so as to realize fine adjustment of the visual detection results by using radar data (Nabati R., Qi H. Centerfusion: Center-based radar and camera fusion for 3d object detection [A]. 2021 IEEE Winter Conference on Applications of Computer Vision [C]. Piscat away, NJ: IEEE, 2021: 1526-1535.). The above method borrows the point cloud feature extraction method of lidar in the radar feature extraction stage, and does not make targeted processing for the sparsity of radar data. The conventional voxelization causes a lot of calculation redundancy, and the adaptability is poor. On the other hand, the single-frame model has poor detection accuracy and anti-occlusion ability, and the conventional time sequence modeling method transmits global features, which is low in efficiency and poor in effect. SUMMARY

[0005] In order to at least solve one of the problems existing in the prior art, the present application proposes a time sequence modeling framework for fusing cameras and millimeter wave radars, taking radar instances as adaptive query initialization inputs, and introducing an information transmission mechanism based on query instances to avoid the high computational overhead caused by global feature alignment, and more accurately capture the motion state and spatio-temporal characteristics of dynamic targets. In the feature fusion process, a local self-attention mechanism is introduced by introducing a distance penalty term to realize the close-range matching between query instances, and further improve the spatial alignment accuracy and fusion effect of multi-source data.

[0006] The present application can solve the lack of spatial measurement capability of pure visual perception and the limitations of traditional time sequence fusion strategy in dynamic target alignment and fusion efficiency.

[0007] To achieve the purpose of the present application, the present application provides a camera and millimeter wave radar fusion three-dimensional target detection method based on time sequence fusion, comprising the steps of:

[0008] The high-confidence queries and their related information in the transmission of time sequence information are stored in the queue through the time sequence queue, and are dynamically maintained through the first-in-first-out implementation mechanism;

[0009] Obtain a plurality of two-dimensional images collected by the surround-view cameras, perform feature extraction on the two-dimensional images to obtain a plurality of two-dimensional image feature maps, and fuse the geometric spatial information into the two-dimensional image feature maps in an implicit coding manner;

[0010] Obtain the millimeter wave radar point cloud and cluster, encode the radar point cloud cluster, and obtain the radar instance feature;

[0011] The global query, the radar instance query and the historical query are used as the initialization of the query, and a multi-modal query is obtained;

[0012] The multi-modal query is input into a decoder, the decoder includes multiple decoder layers, each decoder layer includes a local self-attention module, a time cross self-attention module and an image cross self-attention module, the local self-attention module is used to realize information interaction in different time and space and realize multi-modal close-range matching by introducing a distance penalty term, the time cross self-attention module is used to cross attention fusion between the fusion query and the historical feature, and the time sequence information is extracted; the image cross self-attention module is used to cross attention operation between the query and the image feature, and the image feature is fused;

[0013] The output of the decoder is sent to a detection head to generate a three-dimensional target detection box, and the specific position and category of the target are obtained.

[0014] Further, the time sequence queue stores time sequence information, and the queue capacity is determined by two key parameters: the number of stored historical frames N, which determines the length of the time sequence modeling; and the number of targets stored in each frame K; the time sequence queue adopts a first-in first-out update strategy to ensure that the K high-confidence target information in the N frames at the latest time is always retained; each memory unit contains five types of key information: the time information of the target, i.e., the frame sequence, the query feature, the reference point position, the speed information and the ego-vehicle pose. In order to ensure the effectiveness of the information stored in the queue, the K highest-scoring foreground targets are selected and stored each time according to the classification score.

[0015] Further, the geometric space information is fused into the two-dimensional image feature map in an implicit coding manner, specifically including:

[0016] The image features of the two-dimensional image are extracted using a backbone network, and multi-scale features are obtained through a feature pyramid mechanism;

[0017] The viewing cone space of each surround-view camera is discretized into a three-dimensional grid, and the three-dimensional coordinate points of the viewing cone space generated by a linear incremental discretization strategy are adopted;

[0018] The normalized 3D coordinates are encoded to generate 3D position embedding; the multi-scale features of the two-dimensional image extracted by the backbone network are adjusted in dimension through 1x1 convolution, and then added element by element with the 3D position embedding to form 3D position perception features.

[0019] Further, the millimeter wave radar point cloud is obtained and clustered, the feature of the radar point cloud cluster is encoded, and the radar instance feature is obtained, specifically including:

[0020] The radar point cloud is adaptively clustered by a clustering algorithm to obtain radar point cloud clusters;

[0021] Features of each radar point cloud cluster are extracted by a neural network to obtain radar instance features, the neural network comprising a SharedMLP module and a max-pooling layer, each radar point cloud cluster being first fed into a SharedMLP module, sharing the same weight, the SharedMLP module performing feature dimensioning by multiple one-dimensional convolution operations, and finally obtaining radar instance features by max-pooling operation.

[0022] Further, to ensure the clustering effect of the radar point cloud, the clustering effect of the point cloud is ensured by stacking frames, and the clutter noise is filtered by increasing the minimum number of points during DBSCAN clustering.

[0023] Further, the global query, radar instance query and historical query are used as the initialization of the query to obtain a multi-modal query, comprising:

[0024] The spatial affine transformation from the historical query to the current time is learned by a neural network, and then the historical query is transformed to the current coordinate system;

[0025] A predetermined number of reference points and their corresponding queries are randomly initialized in the global coordinate system;

[0026] The global query, radar instance query and historical query are spliced together to realize the initialization of the multi-modal query.

[0027] Further, the neural network uses two MLPs to fit the rotation matrix R and the translation vector T of the historical query. In addition, in order to further improve the adaptive ability of the transformation, an additional MLP is introduced to fit the error term ΔT of the translation vector, and the time sequence query is transformed to the current coordinate system by affine transformation.

[0028] Further, the multi-modal query and its corresponding reference points are fed into a local self-attention module, which includes query distance D in addition to Q, K and V, to calculate distance penalty, and residual connection is performed after self-attention operation, and then layer normalization is performed before being fed into the next layer.

[0029] Further, in the time sequence cross self-attention module, the historical features are respectively mapped as keys K and values V, the similarity between the query Q and the key V is calculated by using dot product attention mechanism, and the attention weight is obtained by normalizing by using softmax function, and these attention weights are used for weighted summation of the value V, that is, finding the most similar historical features to the current query.

[0030] Further, the image cross self-attention module further performs cross-attention operation on the query and the image features to fuse the image features, comprising:

[0031] The two-dimensional image is encoded to obtain a set of feature vectors X. img In this process, the two-dimensional image is divided into P patches, each patch corresponding to a token. In the cross-attention process, the image features are mapped to keys K and values ​​V respectively.

[0032] The similarity between query Q and key V is calculated using the dot product attention mechanism, and the attention weights are obtained by normalization using the softmax function. These attention weights are used to perform a weighted summation on value V, that is, to find the image features most similar to the current query, thereby obtaining the final output features.

[0033] Furthermore, the detection head uses regression neural networks and classification neural networks to generate target bounding boxes and category labels.

[0034] Furthermore, the instance query is sent to the classification head, and the target category is obtained by obtaining the maximum confidence through Softmax. The query is then sent to the regression head, and the target's length, width, height, three-dimensional coordinates, orientation angle, and lateral and longitudinal velocities are obtained through linear regression.

[0035] This invention provides a three-dimensional target detection system based on temporal fusion of camera and millimeter-wave radar to implement the aforementioned method. The system includes the following modules:

[0036] The time-series delivery module is used to queue and store high-confidence queries and related information during the transmission of time-series information through a time-series queue, and to achieve dynamic maintenance through a first-in-first-out (FIFO) mechanism. By maintaining and updating query instances, the time-series delivery module can deeply mine the characteristics and dynamic changes of targets in the time dimension, and can capture key information of foreground targets more precisely, thereby improving the detection capability of dynamic targets.

[0037] The feature extraction module is used to obtain two-dimensional images captured by multiple surround-view cameras, extract features from the two-dimensional images to obtain multiple two-dimensional image feature maps, and integrate geometric spatial information into the two-dimensional image feature maps in an implicit encoding manner.

[0038] The radar instance feature acquisition module is used to obtain millimeter-wave radar point clouds and perform clustering, and to encode the radar point cloud clusters to obtain radar instance features.

[0039] The multi-source query initialization module is used to initialize the query by using global query, radar instance query, and historical query, resulting in a multimodal query.

[0040] The multi-attention mechanism decoder module includes multiple decoder layers. Each decoder layer includes a local self-attention module, a temporal cross-attention module, and an image cross-attention module. The local self-attention module is used to realize information interaction at different times and spaces and to achieve multimodal near-range matching by introducing a distance penalty term. The temporal cross-attention module is used to perform cross-attention fusion of the query and historical features to extract temporal information. The image cross-attention module is used to perform cross-attention operation on the query and image features to fuse image features.

[0041] The detection module is used to send the output of the decoder into the detection head to generate a 3D target detection box and obtain the specific location and category of the target.

[0042] Millimeter-wave radar point clouds are transformed into the vehicle coordinate system and clustered to generate a series of target-level radar data. These 3D radar targets are used as initialization for adaptive queries, providing the model with more intuitive geometric information and reducing the model's inference complexity. Historical query instances from the nearest time step are also used as query initialization to further strengthen the prior.

[0043] The multi-attention mechanism decoder module, based on the Transformer, designs different attention mechanisms at different stages of the decoder to achieve feature fusion. The initial query contains three modalities of data. First, a self-attention operation is performed on these three types of queries to fuse multi-source data. To enhance the spatial alignment between data, a distance penalty term is added during the calculation of attention weights. Then, a cross-attention mechanism is used to interact with historical queries from more distant times to enhance the features at the current time. Finally, a cross-attention operation is also used to interact with image tokens to fuse image features.

[0044] The present invention also provides a device.

[0045] The present invention also provides a computer-readable storage medium.

[0046] Compared with the prior art, the present invention can achieve the following technical effects:

[0047] This invention employs a temporal modeling approach with instance targets as the transmission objects. By maintaining and updating query instances, it deeply mines the characteristics and dynamic changes of targets in the time dimension. Geometric spatial information is implicitly encoded into image features. Millimeter-wave radar point clouds are transformed into a vehicle coordinate system and clustered to generate a series of target-level radar data. A neural network is used to extract point cloud cluster features to obtain radar instances. Radar instance queries, historical queries, and randomly initialized global queries are input into the decoder for iteration. Local self-attention operations are performed on the three modalities of data, and a distance penalty term is introduced on top of conventional self-attention to achieve spatiotemporal alignment. Query is cross-attention with queries in the temporal queue to fuse long-term temporal information. The query is further cross-attention with image tokens to fuse and update image information. Through multiple iterations in the decoder, the query and its corresponding reference point information are continuously updated and optimized, finally being sent to the detection head, and the detection results are output. Simultaneously, high-confidence queries are input into the temporal queue to maintain temporal information. By introducing radar features and temporal information, the detection accuracy of the model is effectively improved.

[0048] This invention uses radar instances as initial input for adaptive queries and avoids the high computational overhead of global feature alignment by introducing an information transmission mechanism based on query instances. It also more accurately captures the motion state and spatiotemporal characteristics of dynamic targets. During feature fusion, a local self-attention mechanism is constructed by introducing a distance penalty term to achieve close-range matching between query instances, further improving the spatial alignment accuracy and fusion effect of multi-source data. This addresses the limitations of pure visual perception in lacking spatial measurement capabilities and the efficiency of traditional temporal fusion strategies in dynamic target alignment and fusion. Attached Figure Description

[0049] Figure 1 This is a flowchart of a three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the overall framework of a three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar in an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the radar clustering feature extraction process provided in an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram of the decoder process provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The present invention provides a three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar, which can solve the problems of low multimodal fusion efficiency caused by the sparsity of radar features in existing methods and the difficulty of handling dynamic targets due to the low efficiency of traditional temporal fusion, thus achieving better detection results.

[0055] Please see Figure 1 and Figure 4 The present invention provides a three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar, comprising the following steps:

[0056] Step 1: Store high-confidence queries and related information during the transmission of time-series information in a time-series queue, and achieve dynamic maintenance through a first-in-first-out (FIFO) mechanism.

[0057] Single-frame object detection has many limitations in practical applications, especially in autonomous driving scenarios where temporal modeling is almost indispensable. Current mainstream temporal fusion methods achieve temporal information fusion by storing temporal BEV feature maps. This mainstream method utilizes the transformation relationship between the vehicle and driver coordinate systems to align historical BEV feature maps with the current frame before stitching them together, and then completes information fusion through a self-attention mechanism. However, this method has a significant problem: when the calibration parameters are accurate, stationary objects in historical frames can be aligned well to the current frame, but for dynamic targets (such as vehicles and pedestrians), which are of most concern in autonomous driving, their positions change over time, and dynamic targets from the past cannot be accurately aligned to the current frame.

[0058] The key to temporal modeling lies in feature alignment, which is essentially the alignment of foreground targets. Therefore, instead of transmitting global historical features, it is better to directly transmit instance-level target information. In attention-based decoders, each query vector corresponds to a potential instance target. By explicitly transmitting the query vector of the instance target and its related information in the temporal sequence, not only can the high computational cost of global feature alignment be avoided, but the motion characteristics of dynamic targets can also be captured more accurately, thereby effectively improving the detection performance and robustness of the model. Step 1 of this embodiment mainly uses an instance queue to explicitly store instance-level target information from past time moments.

[0059] Specifically, step 1 utilizes a temporal queue to store historical features. The design of the temporal queue (instance queue) revolves around instance-level targets, and its capacity is determined by two key parameters: the number of historical frames N stored, which determines the length of the temporal modeling; and the number of targets K stored per frame. The temporal queue employs a first-in, first-out (FIFO) update strategy to ensure that the information of the K high-confidence targets within the most recent N frames is always retained. Each memory unit contains five types of key information: the target's temporal information (frame order), query features, reference point position, velocity information, and vehicle attitude. This information comprehensively describes the spatiotemporal characteristics of the detected targets, providing rich prior knowledge for detecting dynamic changes in the targets. In one embodiment of the invention, to ensure the effectiveness of the information stored in the temporal queue, the K foreground targets with the highest output scores are stored based on the output classification scores.

[0060] Step 2: Obtain two-dimensional images from multiple surround-view cameras, extract features from the two-dimensional images to obtain multiple two-dimensional image feature maps, and integrate geometric spatial information into the two-dimensional image feature maps in an implicit encoding manner.

[0061] Step 2 adopts the processing strategy of the PETR model, integrating geometric spatial information into the two-dimensional image feature map through implicit encoding, specifically including:

[0062] Step 2.1: Use the ResNet50 backbone network to extract image features from the two-dimensional image and obtain multi-scale features through the feature pyramid mechanism.

[0063] Step 2.2: Discretize the view frustum of each surround-view camera into a 3D mesh. The coordinates of these mesh points consist of pixel coordinates and discrete values ​​sampled along the depth axis. To determine the depth sampling points, a linearly increasing discretization strategy (a common existing sampling strategy, which will not be elaborated here) is used to generate 3D coordinate points in the view frustum. The view frustum coordinates are then transformed into a unified 3D world coordinate system using the camera's intrinsic and extrinsic parameter matrix Z.

[0064] Step 2.3: To eliminate scale differences between different viewpoints, the transformed 3D coordinates are normalized to the [0,1] interval. Next, MLP is used to encode the normalized 3D coordinates to generate 3D position embeddings.

[0065] Step 2.4: The multi-scale features of the 2D image extracted by the ResNet50 backbone network are adjusted in dimension by 1×1 convolution, and then added element-wise with the 3D position embedding to form 3D position-aware features, as shown in the following formula:

[0066]

[0067] Among them, F 2dRepresenting multi-scale features, F represents a discrete coordinate point in three-dimensional space. 3d This represents 3D position-aware features, where i, j, and m represent the three axes of an orthogonal coordinate system, and d represents the dimension.

[0068] Step 3: After performing 3D-FFT on the received signal of the millimeter-wave radar to obtain the millimeter-wave radar point cloud, the point cloud density is increased by temporal frame stacking. The millimeter-wave radar point cloud is clustered using a clustering algorithm to obtain radar point cloud clusters. Then, the radar point cloud clusters are feature-encoded using a neural network to obtain radar instance features.

[0069] Step 3 increases the point cloud density by temporal frame stacking, clusters the millimeter-wave radar point cloud using a clustering algorithm, and then encodes the radar point cloud clusters using a neural network to obtain radar instance features, specifically including:

[0070] Step 3.1: Perform visual analysis on different frame stacking scenarios. As the number of frames increases, some occluded targets gradually become visible. When there are more stacked frames, the point cloud clustering effect is better. Select the number of stacked frames based on the clustering effect, and use frame stacking to ensure the clustering effect of the point cloud.

[0071] Step 3.2: Adaptively cluster the millimeter-wave radar point cloud using the DBSCAN clustering algorithm to obtain radar point cloud clusters. In one embodiment of the present invention, the clustering radius is set to 2, and the minimum number of points is set to 3. Clutter noise can be filtered out by increasing the minimum number of points during DBSCAN clustering.

[0072] Step 3.3: Extract the features of each cluster of radar point clouds through a radar feature extraction neural network to obtain radar instance features.

[0073] In one embodiment of the present invention, such as Figure 3 As shown, the radar feature extraction neural network includes a SharedMLP module (multilayer perceptron) and a max pooling layer. Each radar point cloud cluster is first fed into a SharedMLP module, sharing the same weights. The SharedMLP module uses L one-dimensional convolution operations to complete feature upscaling, and then uses the ReLU activation function to increase the nonlinearity of the features. Finally, it is input into the max pooling layer to obtain instance-level feature representations through max pooling operations, thus obtaining radar instance features.

[0074] Step 4: Use the global query, radar instance query, and historical query as initialization for the query to obtain a multimodal query.

[0075] Global queries provide generalization, while historical queries and radar queries provide sufficient prior knowledge, greatly accelerating the inference speed of the model.

[0076] Step 4.1: Use a neural network to learn the spatial affine transformation of the historical query to the current time. Specifically, retrieve the historical query and its related information from the past H frames in the time sequence queue, and then transform the time sequence query to the current coordinate system.

[0077] The neural network employs two MLPs to fit the rotation matrix R and translation vector T of the historical queries, respectively. Furthermore, to further enhance the adaptive capability of the transformation, an additional MLP is introduced to fit the error term ΔT of the translation vector, transforming the time-series query to the current coordinate system through an affine transformation.

[0078] The formula for aligning historical queries to the current coordinate system is as follows:

[0079] query t =γ·LN(query) t-Δt )+β (2) query t and query t-Δt represents instance queries at the current time t and time t-Δt, respectively; γ represents the rotation transformation learned by the neural network; β represents the translation vector learned by the neural network and its total error; LN represents layer normalization; t represents the current time; and Δt represents the interval between the past time and the current time.

[0080] Step 4.2: Randomly initialize some reference points and their corresponding queries in the global BEV space.

[0081] Step 4.3: Combine the global query, radar instance query, and historical query to initialize the multi-source query, and then send it to the decoder for iterative refinement.

[0082] The calculation formula for the multi-source query initialization module is as follows:

[0083] Q init =Concat(Q radar Q historical Q global (3) Q init It is the initial query of the final input decoder, Q radar Q is the radar instance feature extracted after radar clustering. global It uses a randomly initialized global query to ensure the model's generalization ability. Q historical The queries are randomly initialized historical queries, and the Concat operation combines the three types of queries. Queries with sufficient prior knowledge are fed into the decoder for better feature fusion and extraction, effectively reducing the complexity of model inference.

[0084] Step 5: Input the multimodal query into the decoder. The decoder consists of multiple decoder layers. Each decoder layer includes a local self-attention module, a temporal cross-attention module, an image cross-attention module, and a feedforward neural network. The local self-attention module is used for global feature fusion interaction and introduces distance relationship as a penalty weight into the self-attention mechanism to output the fused query. The temporal cross-attention module is used to extract historical features, the image cross-attention module is used to extract image features, and the feedforward neural network is used to further integrate the feature dimensions.

[0085] Step 5.1: Initialize the multimodal query, i.e., query Q. init First, information is fed into a local self-attention module to achieve information interaction at different times and spaces, and multimodal near-field matching is achieved by introducing a distance penalty term. For example... Figure 4 As shown, the multimodal query and its corresponding reference point are fed into the local self-attention module. Besides the query Q, key K, and value V, it also includes the query distance D, used to calculate the distance penalty. After the self-attention operation, residual connections are performed, and after layer normalization, the data is fed into the next layer. The specific operations performed by the local self-attention module include:

[0086] Step 5.1.1: Calculate the Euclidean distance D between each pair of queries. ii，jj Its definition is as follows:

[0087]

[0088] Among them, (x ii ,y ii ) and (x jj ,y jj ) represent the spatial locations of queries ii and jj, respectively.

[0089] Step 5.1.2: Introduce distance relationship as a penalty weight into the self-attention mechanism, wherein a local self-attention mechanism is adopted, and the calculation formula of the local self-attention mechanism is as follows:

[0090]

[0091] Where α is a configurable hyperparameter, D represents the query distance, dm represents the feature dimension, and T represents the transpose of the matrix.

[0092] The query Q, key K, and value V are generated by linear transformation of the input query. Softmax is a normalization function. Compared with the conventional attention mechanism, the embodiment of the present invention introduces a distance penalty term αD, which can increase the fusion strength of the query on the nearest information, thereby achieving close-range matching.

[0093] Step 5.2: The temporal cross-attention module fuses the fusion query output by the local self-attention module with historical features using a cross-attention mechanism to extract temporal information.

[0094] Historical features are mapped to keys Q and values ​​K, respectively. The similarity between query Q and key K is calculated using a dot product attention mechanism, and attention weights are obtained by normalization using a softmax function. These attention weights are then used to perform a weighted summation on values ​​V, finding the historical features most similar to the current query, thus yielding the final output feature Q. out The specific process is as follows:

[0095]

[0096] Q out W represents the output of the temporally interleaved self-attention module. k W v X represents the learnable weights. t d represents the feature vector corresponding to the historical features, and d represents the length of the feature dimension.

[0097] Step 5.3: The image cross-attention module further performs cross-attention operations between the query and image features, fusing the image features.

[0098] Step 5.3.1: The two-dimensional image is encoded to obtain a set of feature vectors X. img The two-dimensional image is divided into P patches, each patch corresponding to a token. In cross-attention, the image features are mapped to keys K and values ​​V respectively.

[0099] Step 5.3.2: Calculate the similarity between query Q and key K using the dot product attention mechanism, and normalize the results using the softmax function to obtain attention weights. These attention weights are used to perform a weighted summation on the value V, i.e., to find the image feature most similar to the current query, thus obtaining the output feature Q. out The specific formula is as follows:

[0100]

[0101] Q out W represents the output of the image cross-attention module. k W v X represents the learnable weights. img dm represents the feature vector corresponding to the image tokens, and dm represents the length of the feature dimension.

[0102] Step 6: Output Q from the query, i.e., the image cross-attention module. out The data is fed into the detection head, which generates a 3D target detection box, thus obtaining the target's specific location and category.

[0103] The detection head uses a regression neural network φ reg and classification neural network φ cls (This is based on existing known networks and will not be elaborated upon here.) Target bounding boxes and class labels are generated. During training, the loss is calculated separately for the output of each decoder layer, while during inference, only the prediction from the last layer is selected as the final output.

[0104] The aforementioned embodiments of this invention propose a temporal fusion-based 3D target detection method combining camera and millimeter-wave radar. Overall, it can be viewed as employing a temporal 3D target detection model based on an attention mechanism for target detection. Its loss function is defined as the sum of classification loss and regression loss.

[0105] L all = λ1L cls +λ2L reg (8)

[0106] Among them, L cls For classification loss, L reg For regression loss, λ1 and λ2 represent the balance coefficients between the two types of losses. Classification loss L cls Focus loss is used to address the imbalance between positive and negative samples. The bounding box regression task uses L1 loss to optimize the coordinate error between the predicted bounding box and the true bounding box.

[0107] In one embodiment of the present invention, a three-dimensional target detection system based on temporal fusion of camera and millimeter-wave radar is provided to implement the method provided in the foregoing embodiments. The system includes:

[0108] The timing delivery module is used to queue and store high-confidence queries and related information during the transmission of timing information through a timing queue, and to achieve dynamic maintenance through a first-in-first-out (FIFO) mechanism.

[0109] The feature extraction module is used to obtain two-dimensional images captured by multiple surround-view cameras, extract features from the two-dimensional images to obtain multiple two-dimensional image feature maps, and integrate geometric spatial information into the two-dimensional image feature maps in an implicit encoding manner.

[0110] The radar instance feature acquisition module is used to obtain millimeter-wave radar point clouds and perform clustering, and to encode the radar point cloud clusters to obtain radar instance features.

[0111] The multi-source query initialization module is used to initialize the query by using global query, radar instance query, and historical query, resulting in a multimodal query.

[0112] The multi-attention mechanism decoder module includes multiple decoder layers. Each decoder layer includes a local self-attention module, a temporal cross-attention module, and an image cross-attention module. The local self-attention module is used to realize information interaction at different times and spaces and to achieve multimodal near-range matching by introducing a distance penalty term. The temporal cross-attention module is used to perform cross-attention fusion of the query and historical features to extract temporal information. The image cross-attention module is used to perform cross-attention operation on the query and image features to fuse image features.

[0113] The detection module is used to send the output of the decoder into the detection head to generate a 3D target detection box and obtain the specific location and category of the target.

[0114] In one embodiment of the present invention, an apparatus is provided, the apparatus including a processor and a memory, the memory for storing instructions or computer programs, and the processor for executing the instructions or computer programs in the memory to cause the apparatus to perform the steps of the method described in the foregoing embodiments.

[0115] In one embodiment of the present invention, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a device, cause the device to perform the steps of the method described in the foregoing embodiments.

[0116] In one embodiment of the present invention, experiments were conducted to verify the technical effects of the embodiments of the present invention.

[0117] This experiment uses the publicly available nuScense dataset. The nuScense dataset was collected using two electric vehicles with identical sensor configurations. The vehicles are equipped with 6 cameras and 1 32-beam LiDAR, providing 360° field of view. The data collection locations were Boston and Singapore, covering various scenarios including urban areas, residential areas, suburbs, and industrial areas, as well as different weather conditions such as daytime, nighttime, sunny days, rainy days, and cloudy days.

[0118] The temporal 3D object detection model of this invention was trained using the Adam optimizer with an initial learning rate of 0.0001. It was trained using four RTX3090 GPUs with a batch size of 2 per GPU for a total of 48 epochs, and the model achieved good convergence.

[0119] For model evaluation, the mAP and NDS metrics proposed by the nuScense dataset were used for evaluation. To fully verify the excellent performance of this method, it was compared with the baseline algorithm and other representative detection models (Centerfusion, RCBEV4d, CRAFT, RCBEVDet). The experimental comparison results are shown in Table 1.

[0120] Table 1 Performance Evaluation Table of the Model of the Present Invention and Existing Models in the Field

[0121] Method / model name mAP (%) NDS (%) Centerfusion 32.6 44.9 RCBEV4d 38.1 49.7 CRAFT 41.1 51.7 RCBEVDet 45.3 56.8 Inventive method (model) 46.9 57.0

[0122] Table 1 presents a comparison of the experimental results of the model proposed in this invention with other representative models in the field. mAP and NDS are two performance metrics. As can be seen from the data in Table 1, the mAP and NDS of the model proposed in this invention are higher than those of other representative models in the field, demonstrating its feasibility and effectiveness.

[0123] The foregoing embodiments of this invention not only achieve multimodal fusion of camera and millimeter-wave radar, but also introduce temporal modeling, significantly improving the model's detection accuracy and robustness. Regarding multimodal fusion, a clustering algorithm is used to process radar point clouds to generate point cloud clusters, and a neural network is used to extract instantiated features of the radar from these clusters. This further generates adaptive queries and their corresponding reference points, effectively reducing the model's inference difficulty by fully utilizing prior radar information. Regarding temporal modeling, using query instances as the core transmission method significantly improves the efficiency of temporal modeling while solving the temporal alignment problem faced by dynamic targets during spatial transformations. In the specific implementation of multimodal information fusion, a local self-attention mechanism is used to align and fuse multimodal features, and a cross-attention mechanism is used to integrate historical features and image features respectively, thereby achieving more efficient feature interaction.

[0124] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar, characterized in that, Includes the following steps: A time-series queue is used to store high-confidence queries and related information during the transmission of time-series information, and dynamic maintenance is achieved through a first-in-first-out (FIFO) mechanism. Multiple 2D images are acquired from surround-view cameras. Feature extraction is performed on the 2D images to obtain multiple 2D image feature maps. Geometric spatial information is incorporated into the 2D image feature maps in an implicit encoding manner. The point cloud of the millimeter-wave radar is obtained and clustered. The radar point cloud clusters are then feature-encoded to obtain the radar instance features. Using global query, radar instance query, and historical query as initialization, a multimodal query is obtained; The multimodal query is input into the decoder, which consists of multiple decoder layers. Each decoder layer includes a local self-attention module, a temporal cross-attention module, and an image cross-attention module. The local self-attention module is used to realize information interaction at different times and spaces and to achieve multimodal near-range matching by introducing a distance penalty term. The temporal cross-attention module is used to perform cross-attention fusion of the fused query and historical features to extract temporal information. The image cross-attention module is used to perform cross-attention operation of the query and image features to fuse image features. The decoder output is fed into the detection head to generate a 3D target detection box, thus obtaining the target's specific location and category.

2. The three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar according to claim 1, characterized in that, The process of implicitly encoding geometric spatial information into a two-dimensional image feature map specifically includes: The backbone network is used to extract image features from two-dimensional images, and multi-scale features are obtained through the feature pyramid mechanism. The view frustum space of each surround-view camera is discretized into a three-dimensional mesh, and the three-dimensional coordinate points of the view frustum space are generated using a linear incremental discretization strategy. The normalized 3D coordinates are encoded to generate 3D position embeddings; the multi-scale features of the 2D image extracted by the backbone network are adjusted in dimension and added element by element to the 3D position embeddings to form 3D position-aware features.

3. The three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar according to claim 1, characterized in that, The process of obtaining millimeter-wave radar point clouds and clustering them, then performing feature encoding on the radar point cloud clusters to obtain radar instance features, specifically includes: Adaptive clustering of millimeter-wave radar point clouds is performed using a clustering algorithm to obtain radar point cloud clusters; Features of each radar point cloud cluster are extracted by a neural network to obtain radar instance features. The neural network includes a SharedMLP module and a max pooling layer. Each radar point cloud cluster is first fed into a SharedMLP module and shares the same weights. The SharedMLP module uses multiple one-dimensional convolution operations to complete feature upscaling. Finally, the radar instance features are obtained through max pooling.

4. The three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar according to claim 1, characterized in that, The process of using global query, radar instance query, and historical query as initialization to obtain a multimodal query includes: The system uses a neural network to learn the spatial affine transformation of historical queries to the current time, and then transforms the historical queries to the current coordinate system. Randomly initialize a preset number of reference points and their corresponding queries in the global coordinate system; The global query, radar instance query, and historical query are concatenated together to initialize the multimodal query.

5. The three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar according to claim 1, characterized in that, In the temporal cross-attention module, historical features are mapped to keys K and values ​​V respectively. The similarity between query Q and key V is calculated using the dot product attention mechanism and normalized to obtain attention weights. These attention weights are used to perform a weighted summation on value V, that is, to find the historical feature most similar to the current query.

6. The three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar according to claim 1, characterized in that, The image cross-attention module further performs cross-attention operations between the query and image features, fusing the image features, including: The two-dimensional image is encoded to obtain a set of feature vectors X. img In this process, the two-dimensional image is divided into P patches, each patch corresponding to a token. In the cross-attention process, the image features are mapped to keys K and values ​​V respectively. The similarity between query Q and key V is calculated using the dot product attention mechanism and normalized to obtain attention weights. These attention weights are used to perform a weighted summation on value V, that is, to find the image feature most similar to the current query, thereby obtaining the final output feature.

7. The three-dimensional target detection method based on temporal fusion of camera and millimeter-wave radar according to any one of claims 1-6, characterized in that, The detection head uses a regression neural network and a classification neural network to generate target bounding boxes and category labels.

8. A three-dimensional target detection system based on temporal fusion of camera and millimeter-wave radar, characterized in that, For implementing the method of any one of claims 1-7, the system comprises the following modules: The timing delivery module is used to queue and store high-confidence queries and related information during the transmission of timing information through a timing queue, and to achieve dynamic maintenance through a first-in-first-out (FIFO) mechanism. The feature extraction module is used to obtain two-dimensional images captured by multiple surround-view cameras, extract features from the two-dimensional images to obtain multiple two-dimensional image feature maps, and integrate geometric spatial information into the two-dimensional image feature maps in an implicit encoding manner. The radar instance feature acquisition module is used to obtain millimeter-wave radar point clouds and perform clustering, and to encode the radar point cloud clusters to obtain radar instance features. The multi-source query initialization module is used to initialize the query by using global query, radar instance query, and historical query, resulting in a multimodal query. The multi-attention mechanism decoder module includes multiple decoder layers. Each decoder layer includes a local self-attention module, a temporal cross-attention module, and an image cross-attention module. The local self-attention module is used to realize information interaction at different times and spaces and to achieve multimodal near-range matching by introducing a distance penalty term. The temporal cross-attention module is used to perform cross-attention fusion of the query and historical features to extract temporal information. The image cross-attention module is used to perform cross-attention operation on the query and image features to fuse image features. The detection module is used to send the output of the decoder into the detection head to generate a 3D target detection box and obtain the specific location and category of the target.

9. A device, characterized in that, The device includes a processor and a memory for storing instructions or computer programs, and the processor for executing the instructions or computer programs in the memory to cause the device to perform the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the device, cause the device to perform the steps of the method according to any one of claims 1-7.

Images