Method, model and equipment for detecting three-dimensional target around view of autonomous vehicle based on parallel time sequence modeling

Abstract

The invention discloses a method, a model and equipment for detecting an all-round three-dimensional target of an automatic driving vehicle based on parallel time sequence modeling. Firstly, a sparse query and BEV combined double-flow sequential network and a perspective stacked sequential network are designed, by means of the characteristic of complementary advantages of the sparse query and BEV combined double-flow sequential network and the perspective stacked sequential network, the sparse query and BEV combined double-flow sequential network and the perspective stacked sequential network have the local-to-global modeling capacity, and accurate position and direction information is obtained; secondly, a parallel attention mechanism is designed, so that the two time sequence features interact with corresponding queries, important information in the image is concerned in a local learning mode, and efficient decoding of the time sequence features is achieved; according to the method, a global channel-space attention mechanism based on cavity convolution is further designed, the global dependency relationship of the features is enhanced from the channel dimension and the space dimension, the representation capacity of the flow type BEV is enhanced, and the receptive field of the flow type BEV is expanded. The problem of detection failure caused by shielding is solved, and the performance of a time sequence sensing algorithm is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving of intelligent vehicles, and relates to a method, model and device for detecting three-dimensional objects in surround view of an autonomous driving vehicle using parallel temporal modeling. Technical Background

[0002] The rapid development of autonomous driving technology has promoted progress in the field of transportation. Three-dimensional target detection technology based on surround view cameras has been widely used in assisted driving systems for passenger cars due to its low cost and rich information. Most previous methods use current moment features for target detection. However, problems such as missed detection and positioning errors often occur in occluded scenes, resulting in detection failures. Recent work has introduced time series features to fuse historical information with current information, which is beneficial to the detection of occluded objects. According to the commonly used time series representation forms, it is divided into BEV-based stacked time series, perspective time series, and sparse query streaming time series strategies.

[0003] The BEV stacking temporal strategy explicitly warps dense BEV features from historical frames to the current frame. BEV temporal features are conducive to predicting spatial position information. The stacking strategy allows stacking specific historical features, which can flexibly obtain information within a long time span and is better at global temporal modeling. However, the highly structured BEV features are not conducive to the modeling of moving objects. This paradigm requires a large receptive field to alleviate this problem. Due to computational efficiency, the number of stacked BEV historical frames is limited. The above factors lead to missed detections in occluded scenes.

[0004] The perspective time series strategy uses sparse queries to interact with all historical perspective features. The design of sparse queries makes it good at time series modeling of moving objects, and perspective time series features are conducive to object posture estimation, such as heading angle. However, the query contains redundant irrelevant information, and it needs to interact with all historical features, resulting in multiple calculations. The streaming time series strategy inherits from the perspective time series strategy, uses transfer queries to interact with current perspective features, and uses sparse queries as hidden states of time series transfer to propagate historical information. It improves computational efficiency while maintaining the advantages of the perspective strategy, and is better at local time series modeling of moving objects. However, the historical information of the long time series hidden state is severely lost during the update process, and its query also contains redundant irrelevant information. The above factors lead to positioning errors and repeated predictions in occluded scenes.

[0005] The above strategies complement each other, which is reflected in the streaming strategy and the stacking strategy as well as the perspective time series features and the BEV time series features. Therefore, from a technical level, they are combined to have the ability to model time series from local to global and more accurate position and posture information. However, how to combine them is a hot topic and a difficult point in current research. Summary of the invention

[0006] Based on the above background, the present invention proposes a method for detecting three-dimensional targets in surround view of autonomous driving vehicles with parallel temporal modeling. In order to realize this method, firstly, the present invention designs a dual-stream temporal network combining sparse query with BEV, and uses hidden state to transmit historical features to enhance the local temporal modeling capability of moving targets, improve computational efficiency and obtain accurate spatial position information; on this basis, a perspective stacked temporal network is established, which together with the dual-stream temporal network constitutes a parallel temporal model, which more flexibly integrates long temporal historical features, enhances the global temporal modeling capability of moving targets, and obtains accurate heading information. Secondly, a parallel attention mechanism is designed, which interacts the temporal features with the corresponding queries based on the parallel aggregation unit, pays attention to the important information in the image through local learning, and realizes efficient decoding of the two temporal features. The present invention also designs a global channel-space attention module based on dilated convolution, which enhances its global dependency from the two dimensions of channel and space, and realizes improving the expression ability of BEV temporal features and expanding its receptive field. The present invention proposes a method for detecting surround 3D objects in an autonomous driving vehicle based on parallel temporal modeling, which solves the problem of detection failure in occluded scenes and improves the performance of surround 3D object detection.

[0007] The present invention proposes a method, model and device for surround-view three-dimensional target detection of an autonomous driving vehicle based on parallel temporal modeling. The present invention is mainly based on a temporal method, and its main components include the following parts: 1. A feature encoding module for acquiring features from a surround-view camera. 2. A global channel-space attention mechanism based on dilated convolution for enhancing the temporal feature representation capability of BEV. 3. A parallel temporal model for temporal modeling, which is composed of a dual-stream temporal network and a perspective stacked temporal network of sparse query combined with BEV. 4. A parallel attention mechanism for efficiently decoding corresponding temporal features using multiple queries.

[0008] The specific steps of the method are as follows:

[0009] Step 1: Prepare a 3D object detection dataset for surround cameras, which mainly includes 1000 scenes, of which 700 scenes are used for training, 150 for verification, and 150 for testing. Each scene lasts about 20 seconds, and key samples (referring to a set of data frames at time points that are annotated and contain complete information, including object category, object location information, and scene environment information) are annotated at a frequency of 2Hz. Each sample consists of RGB images from 6 cameras with a 360° horizontal field of view. The annotation contains up to 1.4M 3D bounding boxes and is evaluated using 10 common categories (cars, trucks, buses, trailers, engineering vehicles, pedestrians, motorcycles, bicycles, obstacles, and traffic cones). The official indicators are used to evaluate the 3D detection results, including average precision, average translation error, average scale error, average direction error, average speed error, average attribute error, and detection score, where the detection score is the weighted average of other indicators.

[0010] Step 2: The feature encoding part is mainly composed of the perspective feature extraction backbone, feature aggregation neck, and BEV feature encoder. The perspective feature extraction backbone and feature aggregation neck are respectively used to extract the sequence surround image I = {I 1 ,...,I n} Perform feature extraction and aggregation to obtain sequential 2D perspective features The BEV encoder improves the sequential 2D perspective features to BEV features

[0011] Step three, build a global channel-space attention mechanism based on dilated convolution. The present invention benefits from the complementary advantages of the two temporal features. The BEV temporal features come from the improvement of the perspective temporal features, which is equivalent to the rearrangement of the perspective features with prior information (self-vehicle posture, position information) in the three-dimensional scene. Therefore, in order to enhance the uniqueness of the BEV features to improve its representation ability and expand its receptive field, the present invention proposes a global channel-space attention mechanism based on dilated convolution, which is mainly composed of channel attention, channel shuffling and spatial attention. In order to enhance the global dependency between channels, multi-layer MLP is used to process features in the channel attention, and the number of channels is reduced to 1 / 4 of the original and then restored to the original dimension. This reduces the redundant information between channels while highlighting important features; then, the ReLU activation function and the Sigmoid activation function are used to introduce nonlinearity and generate a channel attention map. The input features are multiplied element by element by the channel attention map to obtain enhanced features. As shown in formula (1), F channel is the enhanced feature, σ is the Sigmoid function, Permute is the dimension permutation of the feature, ⊙ represents element-by-element multiplication, are input features.

[0012]

[0013] In order to fully mix the information between channels, a channel shuffle is designed. The enhanced feature maps are grouped, and the grouped feature maps are transposed to disrupt the channel order in each group, and then restored to the original shape. This method can more fully mix information between different channels and enhance the feature expression ability. As shown in formula (2), F shuffle It is the feature after shuffle, and ChannelShuffle represents the channel shuffle operation.

[0014] F shuffle =ChannelShuffle(F Channel ) (2)

[0015] Finally, spatial attention is designed to fully capture the local and global spatial characteristics of the features. The channel dimension is first reduced to 1 / 4 of the original dimension and then restored to the original dimension using dilated convolution with various dilation rates; then batch normalization and ReLU activation function are used for nonlinear transformation. This method can better capture the dependency of spatial dimensions and expand the receptive field of streaming BEV; finally, the sigmoid activation function is used to generate the spatial attention map. The shuffled feature map is element-wise multiplied with the spatial attention map to obtain the final output feature. As shown in formula (3), F B′ It is the feature after spatial attention, Conv represents the convolution operation, BN represents batch normalization, and ReLU represents the activation function.

[0016] F B′ =σ(Conv(BN(ReLU(Conv(F shuffle )))))⊙F shuffle (3)

[0017] Step 4. In order to make full use of the complementary advantages of the streaming strategy and the stacking strategy as well as the perspective timing features and the BEV timing features, the present invention combines them from a technical level. First, a dual-stream timing network combining sparse query with BEV is designed, and then a perspective stacking timing network is designed. The two networks constitute a parallel timing model.

[0018] (1) Dual-stream time series network. The present invention integrates the sparse query and BEV that integrate historical time series features into the current frame in a streaming manner. By using the sparse query as the hidden state of time series transmission, the ability to model the local time series of moving objects is improved, and accurate spatial position information is obtained.

[0019] First, the streaming of sparse queries. In the autonomous driving scenario, most target instances are in motion, which indicates that the current instance will persist in subsequent frames, prompting the present invention to pass the K queries with the highest confidence scores to the next frame, providing valuable location priors and preserving the temporal features of historical frames. The present invention defines a memory pool for storing temporal features, including the target query Q corresponding to the BEV temporal features B And the target query Q with perspective time series features P The length of the memory pool is 4, and the target number of frames stored per frame is 256. The first t-2 frame information is and The propagated BEV queries and propagated PV queries are taken out, and the number is 256. Here, the perspective is represented as PV. The propagated BEV / PV queries are first compared with the 644 initialization target queries Q initial (initialobject queries) are added, as shown in formulas (4) and (5),

[0020]

[0021] Next, we perform aggregation to obtain the query Q, key K1, and value V1 of the time series attention, as shown in formula (6). Represents aggregation,

[0022]

[0023] 3 frames remaining in the memory pool for propagated BEV / PV queries and As the key K2 and value V2 of temporal attention, as shown in formula (7),

[0024]

[0025] Then, the key K and value V are aggregated respectively, as shown in formula (8):

[0026]

[0027] Finally, the temporal interaction is performed in the temporal attention to obtain the output at the current moment. and As shown in formula (9),

[0028]

[0029] MHTSA() represents the calculation of the multi-head temporal self-attention mechanism, softmax() represents the activation function here, and D(K) represents the dimension of K.

[0030] After spatial interaction, the query at the current moment is converted into a background query and a foreground query. The top K foreground queries with the highest scores are input into the memory pool. While the information of the t-1 frame is obtained, the information of the t-5 frame is discarded. In this way, the memory pool is updated frame by frame.

[0031] The second is BEV streaming. BEV time series features provide effective spatial information (such as location information), which helps to improve the target detection effect. However, the BEV stacking time series strategy needs to interact with all historical BEV features to obtain historical information, and the time series modeling efficiency is low. Based on this, the present invention designs a streaming BEV time series model, which uses hidden states to transmit historical information. This effectively improves the efficiency of time series modeling while retaining its advantages. The historical information of the streaming BEV is transmitted frame by frame through hidden states, and its hidden states include propagated BEV queries and propagated BEV features. The propagated BEV feature itself is equivalent to a continuously updated memory pool, and the current BEV feature can gain from the fusion of the propagated BEV feature. In this way, historical features are obtained, rather than interacting with all historical frames like the traditional stacking method, to improve the efficiency of time series modeling. The BEV features are updated and transmitted frame by frame, and the propagated BEV features are converted to the current moment based on the posture of the vehicle to achieve spatiotemporal alignment, and then the gated recurrent unit is used to fuse the converted propagated BEV features with the current BEV features. In order to ensure training stability, the present invention uses layer normalization operations. As shown in formula (10) and (11), represents the transfer BEV characteristics before conversion, represents the transformed transfer BEV characteristics, φ represents the conversion function, Represents the coordinate transformation matrix from the previous frame to the current frame, represents the fused BEV features, LN represents layer normalization, GRU represents gated recurrent unit, Represents the BEV characteristics before fusion.

[0032]

[0033]

[0034] (2) Perspective stacked temporal network. Temporal interaction in temporal attention obtains the output at the current moment and After layer normalization, we get the BEV query. and PV query In order to improve the global modeling capability of long time series and enhance the detail information, the present invention designs a perspective-based stacked time series network. The historical perspective features are stacked with the current features. The stacked historical features can be specified to be within the memory pool length or outside the memory pool length, so that historical information can be obtained flexibly. Here, the present invention specifies the stacked historical features outside the memory pool. Assuming that the current time is t-1, the historical 4 frames of information outside the memory pool are stacked and fused with the t-1 frame information, and then interact with the current query containing the most recent 4 frames of historical information in the parallel aggregation unit, as shown in formula (12),

[0035]

[0036] It represents the output of perspective features after passing through the parallel aggregation unit. Fusion represents the fusion operation. represents the perspective feature at time t-1, represents the perspective characteristics from t-9 to t-6, Indicates interaction, represents the target query of perspective features at time t-3, t-4, and t-5, Q initial represents the initial query, Represents the target query of the perspective feature at time t-1.

[0037] The 4 frames of historical information closest to the current moment have been obtained from the memory pool through target query. Therefore, only the 4 frames of historical information outside the memory pool need to be learned, instead of learning the full sequence information of the 8 frames of history as in the traditional stacking method.

[0038] Step 5, the present invention builds a parallel attention mechanism for the spatial interaction between the two temporal features and the corresponding query. After the temporal attention mechanism interaction and layer normalization processing, the BEV query is obtained. and PV query Next, they are used for spatial interaction. Due to the existence of multiple features and queries, the method of using a multi-head attention mechanism to globally learn the entire image is inefficient. Based on this, the present invention designs a parallel attention mechanism, including two parallel aggregation units (Parallel Aggregation Union, PaAgUn) based on a 3D spatial variable attention mechanism, in which the spatial interaction between the query and the corresponding feature is performed respectively, and the important information in the image is paid attention to through local learning, so as to realize efficient decoding of temporal features.

[0039] The stacked PV time series features and enhanced streaming BEV time series features have been obtained in the previous section. Here, a parallel aggregation unit is used to interact the PV / BEV features with the corresponding query. For each 3D spatial reference point, parallel attention is learned for N sampling offsets, and then the reference point P3d Combined sampling offset △P 3d Become a variable reference point in 3D space As shown in formula (13),

[0040]

[0041] According to the transformation matrix T of the mth camera m 3D variable reference point Project it onto the corresponding 2D view to get the corresponding 2D projection point As shown in formula (14),

[0042]

[0043] Use a bilinear interpolation network to sample the historical features of the corresponding view stack to obtain the sampled features As shown in formula (15),

[0044]

[0045] S Bilinear () represents bilinear interpolation, Represents the perspective features from time t-1 to time ti.

[0046] Finally, the target query in 3D space interacts with the sampled features. In this way, the results of the interaction of each perspective are aggregated into the 3D query. The time series PV features at time t-1 are sliced ​​into keys and values ​​after sampling and compared with the corresponding query. interaction, as shown in formula (16),

[0047]

[0048] ParallelAttention (PV) represents the parallel attention calculation of perspective features, MultiHeadPaAgUn() represents the multi-head parallel aggregation unit, Indicates the key corresponding to the perspective feature from time t-1 to time ti, Represents the value corresponding to the perspective feature from time t-1 to time ti.

[0049] Similarly, the time series BEV features at time t-1 are sliced ​​into keys and values ​​after sampling, and the corresponding query interaction, as shown in formula (17),

[0050]

[0051] ParallelAttention(BEV) represents parallel attention calculation of BEV features, MultiHeadPaAgUn() represents multi-head parallel aggregation unit, represents the key corresponding to the BEV feature at time t-1, Indicates the value corresponding to the BEV feature at time t-1.

[0052] After feature extraction and refinement by Parallel Attention, the outputs of ParallelAttention (PV) and ParallelAttention (BEV) are aggregated along the channel dimension to obtain a refined object query, which is fed to the normalization layer and the feedforward network, as shown in Equation (18).

[0053]

[0054] ParallelAttention(Q) represents the output of the two parallel attention mechanisms combined. Represents aggregation.

[0055] After layer-by-layer refinement, the 3D detection head outputs the prediction results of the foreground category and detection box.

[0056] Step 6: Perform training based on the constructed model. The training process is as follows:

[0057] Design the total loss function Loss(x,y):

[0058]

[0059] For classification, the focal loss function L is used 焦点 , represents the loss function between the category and label of the decoded output. The L1 loss function is used for the detection box, which represents the loss function between the predicted box and the label of the decoded output. Since batches of sequence images are input during training, the prediction results and the true values ​​are represented in the form of sets. The prediction set is represented as x = (a, b), and the label set is represented as y = (a′, b′), where a represents the category and b represents the bounding box. a′ represents the category label, b′ represents the bounding box label, and L bbox (b,f H (b′)) represents the calculation of bounding box loss, L class (a,f H (a′)) indicates the calculation of category loss. The Hungarian algorithm is used to assign labels between the label set and the prediction set, f H (b′) and f H (a′) represents the distribution function between the two true values ​​and the two labels, and α and β represent the proportional coefficients of the two losses.

[0060] The basic settings are: based on a single card RTX4090GPU, cuda11.1, and PyTorch1.9.0. Set different image resolutions and set the batch size to 7. The comparative experiment is trained for 60 rounds, and the ablation experiment is trained for 24 rounds. Use the AdamW optimizer, and the weight decay is set to 0.01. The basic learning rate is set to 2e-4, and the cosine annealing strategy is used. Use ResNet50 and ResNet101 as the backbone to conduct comparative experiments on the validation set, and also use VoVNet99 as the backbone to conduct comparative experiments on the test set. Use the improved BEVFormer combined with a single-layer encoder for BEV feature extraction, and use BEV data enhancement. The total number of both queries is set to 900. Use a streaming training strategy to obtain different training sequence lengths by controlling the number of sequence splits, and use the entire sequence for inference.

[0061] Based on the above-mentioned parallel time series modeling method for surround three-dimensional target detection of an autonomous driving vehicle, the present invention also proposes a corresponding inspection model, which is the model built in the above-mentioned steps two to five.

[0062] The present invention also proposes a vehicle device, in which the trained model is arranged.

[0063] The beneficial effects of the present invention are:

[0064] (1) This paper proposes a parallel temporal modeling method for surround 3D object detection in autonomous driving vehicles, creatively integrating perspective and BEV temporal features with stacking and streaming temporal strategies into a 3D object detection framework, thus expanding the new paradigm of temporal object detection.

[0065] (2) The present invention proposes a parallel time series model, which consists of a dual-stream time series network based on sparse query and BEV and a stacked time series network based on perspective. It fully utilizes the complementary advantages of the two time series characteristics and the two time series strategies, and can accurately and efficiently obtain the target's position information and posture information, thereby improving the time series modeling capability and computational efficiency of moving targets from local to global.

[0066] (3) The present invention designs a parallel attention mechanism, which consists of two parallel aggregation units. Based on the parallel aggregation units, the BEV streaming time series features and the perspective stacked time series features are interacted with the corresponding queries, and the important information in the image is paid attention to through local learning, so as to realize efficient decoding of the time series features.

[0067] (4) The present invention designs a global channel-spatial attention module based on dilated convolution, which enhances its global dependency from both channel and spatial dimensions, thereby improving the expressiveness of BEV temporal features and expanding their receptive field. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The overall structure diagram of surround 3D object detection using parallel temporal modeling;

[0069] Figure 2 This is a structural diagram of the global channel-spatial attention mechanism based on dilated convolution;

[0070] Figure 3 It is a dual-stream sequential network structure diagram;

[0071] Figure 4 It is the timing structure diagram of streaming BEV;

[0072] Figure 5 To provide a perspective stacked sequential network structure diagram;

[0073] Figure 6 For parallel attention structure diagram; DETAILED DESCRIPTION

[0074] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present invention is not limited thereto.

[0075] The specific implementation methods of the present invention are as follows:

[0076] (1) Data preparation: The dataset used in this invention is the nuScenes dataset, which is derived from 1,000 driving scenes collected in Boston and Singapore, of which 700 scenes are used for training, 150 for validation, and 150 for testing. Each scene lasts approximately 20 seconds, and key samples are annotated at a frequency of 2 Hz. Each sample consists of RGB images from 6 cameras with a 360° horizontal field of view. The annotation contains up to 1.4M 3D bounding boxes and is evaluated using 10 common categories: cars, trucks, buses, trailers, engineering vehicles, pedestrians, motorcycles, bicycles, roadblocks, and traffic cones. The official nuScenes metrics are used to evaluate the 3D detection results, including average accuracy, average translation error, average scale error, average orientation error, average speed error, average attribute error, and nuScenes detection score (NDS), where NDS is the weighted average of other metrics.

[0077] (2) Build the overall model process: Figure 1The overall structure diagram of surround 3D object detection using parallel temporal modeling, which mainly consists of five parts: 1. Feature encoding module for acquiring features from surround cameras. 2. Global channel-space attention mechanism based on dilated convolution for enhancing BEV representation capability. 3. Parallel temporal model for temporal modeling, consisting of a dual-stream temporal network and a perspective stacked temporal network with sparse query combined with BEV. 4. Parallel attention mechanism for efficiently decoding corresponding temporal features using multiple queries.

[0078] (3) Build a feature encoding module. The feature encoding part is mainly composed of the perspective feature extraction backbone, feature aggregation neck, and BEV feature encoder. The backbone and neck are used to extract the sequence image I = {I 1 ,...,I n} Perform feature extraction and aggregation to obtain sequential 2D perspective features The BEV encoder improves the sequential 2D perspective features to BEV features

[0079] (4) Build a global channel-space attention mechanism based on dilated convolution. The specific structure is shown in the attached figure. Figure 2 As shown. It mainly consists of channel attention, channel shuffling and spatial attention. First, channel attention. In order to enhance the global dependency between channels, a multi-layer MLP is used to process features. The number of channels is reduced to 1 / 4 of the original and then restored to the original dimension. This reduces the redundant information between channels while highlighting important features. Then, the ReLU activation function and the Sigmoid activation function are used to introduce nonlinearity and generate a channel attention map. The input feature is multiplied element by element with the channel attention map to obtain the enhanced feature. As shown in formula (1), F channel is the enhanced feature, σ is the Sigmoid function, ⊙ represents element-by-element multiplication, F B are input features.

[0080]

[0081] The second is channel shuffling. In order to fully mix the information between channels, channel shuffling is introduced. The enhanced feature maps are grouped, and the grouped feature maps are transposed to disrupt the channel order in each group, and then restored to the original shape. This method can more fully mix information between different channels and enhance the feature expression ability. As shown in formula (2), F shuffle is the feature after shuffling.

[0082] F shuffle =ChannelShuffle(F Channel ) (2)

[0083] Finally, spatial attention is used to fully capture the local and global spatial features of the features. A variety of dilated convolutions with different dilated rates are designed to reduce the channel dimension to 1 / 4 of the original dimension and then restore the original dimension; then batch normalization and ReLU activation function are used for nonlinear transformation. This method can better capture the dependency of spatial dimensions and expand the receptive field of streaming BEV; finally, the sigmoid activation function is used to generate a spatial attention map. The shuffled feature map is multiplied element by element with the spatial attention map to obtain the final output feature. As shown in formula (3), F B′ It is the feature after spatial attention.

[0084] F B′ =σ(Conv(BN(ReLU(Conv(F shuffle )))))⊙F shuffle (3)

[0085] (5) Building a parallel time series model. In order to fully utilize the complementary advantages of the streaming strategy and the stacking strategy as well as the perspective time series features and the BEV time series features, the present invention combines them from a technical level. First, a dual-stream time series network combining sparse query with BEV is designed, and then a perspective stacking time series network is designed. The two networks form a parallel time series model.

[0086] 1) Dual-stream time series network. The present invention integrates sparse query and BEV that integrate historical time series features into the current frame in a streaming manner. By using sparse query as the hidden state of time series transmission, the ability to model the local time series of moving objects is improved, and accurate spatial position information is obtained.

[0087] The first is sparse query, as shown in the following Figure 3 As shown. In the autonomous driving scenario, most target instances are in motion, which indicates that the current instance will continue to exist in subsequent frames, prompting the present invention to pass the K queries with the highest confidence scores to the next frame, providing valuable location priors and retaining the temporal features of historical frames. The present invention defines a memory pool for storing temporal features, including target queries Q corresponding to two temporal features B , Q P The length of the memory pool is 4, and the number of targets stored in each frame is 256. The first t-2 frame information is taken out, which is called propagated BEV queries and propagated PV queries. The number is 256, and the perspective is represented as PV here. The propagated BEV / PV queries are first added to the 644 initial object queries, as shown in equations (4) and (5).

[0088]

[0089] Aggregate to get the query, key, and value (Q, K1, V1) of temporal attention, as shown in formula (6).

[0090]

[0091] The remaining 3 frames in the memory pool propagate BEV / PV queries as the key and value (K2, V2) of temporal self attention, as shown in formula (7),

[0092]

[0093] Then, the key and value (K, V) are aggregated respectively, as shown in formula (8).

[0094]

[0095] Finally, the temporal interaction is performed in the temporal attention to obtain the output at the current moment. As shown in formula (9),

[0096]

[0097] After spatial interaction, the query at the current moment is converted into a background query and a foreground query. The top K foreground queries with the highest scores are input into the memory pool. While the information of the t-1 frame is obtained, the information of the t-5 frame is discarded. In this way, the memory pool is updated frame by frame.

[0098] The second is the flow BEV structure, as shown in the attached Figure 4As shown. BEV time series features provide effective spatial information (such as location information), which helps to improve the target detection effect. Based on this, the present invention designs a streaming BEV time series model, which uses hidden states to transmit historical information. This effectively improves the efficiency of time series modeling while retaining its advantages. The historical information of the streaming BEV is transmitted frame by frame through the hidden state, and its hidden state includes propagated BEV queries and propagated BEV features. The propagated BEV feature itself is equivalent to a continuously updated memory pool, and the current BEV feature can gain from the fused propagated BEV feature. In this way, historical features are obtained instead of interacting with all historical frames like the traditional stacking method, which improves the efficiency of time series modeling. The BEV features are updated and propagated frame by frame, and the propagated BEV features are converted to the current moment based on the vehicle posture to achieve spatiotemporal alignment, and then the gated recurrent unit is used to fuse the converted propagated BEV features with the current BEV features. In order to ensure training stability, the present invention uses layer normalization operations. As shown in equations (10) and (11), represents the transfer BEV characteristics before conversion, represents the transformed transfer BEV characteristics, φ represents the conversion function, Represents the coordinate transformation matrix from the previous frame to the current frame, Represents the fused BEV features.

[0099]

[0100] 2) Perspective on stacked sequential networks, as shown in the attached Figure 5 As shown. In the temporal attention, temporal interaction is performed to obtain the output at the current moment. and After layer normalization, we get the BEV query and PV query In order to improve the global modeling capability of long time series and enhance the detailed information, a perspective-based stacking time series strategy is designed. The historical perspective features are stacked with the current features. The stacked historical features can be specified to be within the memory pool length or outside the memory pool length, so that historical information can be obtained flexibly. The present invention specifies the stacked historical features outside the memory pool. Assuming that the current time is t-1, the historical 4 frames of information outside the memory pool are stacked and fused with the t-1 frame information, and then interact with the current query containing the most recent 4 frames of historical information in the parallel aggregation unit, as shown in formula (12),

[0101]

[0102] The 4 frames of historical information closest to the current moment have been obtained from the memory pool through target query. Therefore, only the 4 frames of historical information outside the memory pool need to be learned, instead of learning the full sequence information of the 8 frames of history as in the traditional stacking method.

[0103] (6) Build a parallel attention mechanism, the structure is as shown in the attached Figure 6 In the above steps, the BEV and PV queries are obtained after the temporal attention mechanism interaction and layer normalization. Next, they are used for spatial interaction. Due to the existence of multiple features and queries, the method of using a multi-head attention mechanism to globally learn the entire image is inefficient. Based on this, the present invention designs a parallel attention mechanism, including two parallel aggregation units (Parallel Aggregation Union, PaAgUn), in which the spatial interaction between the query and the corresponding features is performed respectively, and the important information in the image is paid attention to through local learning, so as to realize efficient decoding of temporal features.

[0104] The stacked PV time series features and enhanced streaming BEV time series features have been obtained in the previous section. Here, a parallel aggregation unit is used to interact the PV / BEV time series features with the corresponding query. For each 3D spatial reference point, parallel attention is used to learn N sampling offsets. After that, the reference point is combined with the sampling offset to become a 3D spatial variable reference point, as shown in formula (13):

[0105]

[0106] According to the transformation matrix T of the mth camera m Project the 3D variable reference point onto the corresponding 2D view to obtain the corresponding 2D projection point, as shown in formula (14):

[0107]

[0108] The historical features of the corresponding view stack can be sampled using a bilinear interpolation network to obtain the sampled features, as shown in formula (15):

[0109]

[0110] Finally, the target query in 3D space interacts with the sampled features. In this way, the results of the interaction of each perspective are aggregated into the 3D query. The PV time series features at time t-1 are sampled and sliced ​​into keys and values ​​(key / value) and compared with the corresponding query. interaction, as shown in formula (16),

[0111]

[0112] Similarly, the time series BEV features at time t-1 are sliced ​​into keys and values ​​after sampling, and the corresponding query interaction, as shown in formula (17),

[0113]

[0114] After feature extraction and refinement by Parallel Attention, the outputs of ParallelAttention (PV) and ParallelAttention (BEV) are aggregated along the channel dimension to obtain a refined object query, which is fed to the normalization layer and the feedforward network, as shown in Equation (18).

[0115]

[0116] After layer-by-layer refinement, the 3D detection head outputs the prediction results of the foreground category and detection box.

[0117] (7) For the above model training, the sampling loss function is calculated as follows:

[0118]

[0119] For classification, the focal loss function L is used 焦点 , represents the loss function between the category and label of the decoded output. The L1 loss function is used for the detection box, which represents the loss function between the predicted box and the label of the decoded output. Since batches of sequence images are input during training, the prediction results and the true values ​​are represented in the form of sets. The prediction set is represented as x = (a, b), and the label set is represented as y = (a′, b′), where a represents the category and b represents the bounding box. The Hungarian algorithm is used to assign labels between the label set and the prediction set, f H Represents the distribution function between the true value and the label. α, β represent the proportional coefficients of the two losses respectively.

[0120] The basic settings for training are: based on a single-card RTX4090 GPU, cuda11.1, and PyTorch1.9.0. Set different image resolutions and set the batch size to 7. The comparative experiment was trained for 60 rounds, and the ablation experiment was trained for 24 rounds. The AdamW optimizer was used, and the weight decay was set to 0.01. The basic learning rate was set to 2e-4, and the cosine annealing strategy was used. ResNet50 and ResNet101 were used as the backbone for comparative experiments on the validation set, and VoVNet99 was used as the backbone for comparative experiments on the test set. The improved BEVFormer was used in combination with a single-layer encoder for BEV feature extraction, and BEV data enhancement was used. The total number of both queries was set to 900. A streaming training strategy was used to obtain different training sequence lengths by controlling the number of sequence splits, and the entire sequence was used for inference.

[0121] (8) An embodiment of the present invention further proposes an autonomous driving perception device, which utilizes a trained model in combination with perception devices such as a vehicle-mounted camera to implement the surround three-dimensional target detection method based on parallel time series modeling.

[0122] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting 3D objects in surround view of an autonomous driving vehicle based on parallel temporal modeling, characterized in that: These include: S1, build a dataset for surround 3D object detection; S2, build a surround 3D target detection network model: It includes four parts: A feature encoding module for acquiring features from a surround view camera; A global channel-spatial attention mechanism module based on dilated convolution for enhancing the temporal feature representation capability of BEV; Parallel time series model module for time series modeling, including sparse query combined with BEV dual-stream time series network and perspective stacked time series network; A parallel attention mechanism module that efficiently decodes the corresponding temporal features using multiple queries; S3, trains the model built in S2; S4, uses the trained surround view 3D object detection model for real-time detection.

2. The method for detecting 3D objects around an autonomous driving vehicle using parallel temporal modeling according to claim 1, characterized in that: The S1 specifically includes: setting 1000 scenes, of which 700 scenes are used for training, 150 are used for verification, and 150 are used for testing; setting images of different resolution sizes, including 256×704 and 512×1408, for training and verification; setting the image resolution to 640×1600 for testing; each scene lasts about 20 seconds, and the samples are annotated at a frequency of 2Hz, and the annotations contain a maximum of 1.4M 3D bounding boxes, each sample consists of RGB images from 6 cameras, has a 360° horizontal field of view, and is evaluated using 10 categories; using a streaming training strategy, adjusting the number of sequence segmentations to obtain different sequence lengths for fusion during training, setting the number of segmentations to 2, obtaining a training fusion frame length of 20, 6 images per frame, a total of 120 images, and setting the stacked fusion frame length to 4, a total of 24 frames.

3. The method for detecting 3D objects around an autonomous driving vehicle using parallel temporal modeling according to claim 1, characterized in that: The feature encoding module in S2 includes three parts: a perspective feature extraction backbone, a feature aggregation neck, and a BEV feature encoder; the perspective feature extraction backbone and the feature aggregation neck are used to view the sequence image I={I 1 ,…,I n } Perform feature extraction and aggregation to obtain sequential 2D perspective features The BEV encoder improves the sequential 2D perspective features to BEV features 4. The method for detecting 3D objects around an autonomous driving vehicle using parallel temporal modeling according to claim 1, characterized in that: The global channel-spatial attention mechanism module based on dilated convolution in S2 includes three parts: channel attention, channel shuffling and spatial attention; In order to enhance the global dependency between channels, a multi-layer MLP is used to process features in the channel attention, and the number of channels is reduced to 1 / 4 of the original and then restored to the original dimension. Then, the ReLU activation function and the Sigmoid activation function are used to introduce nonlinearity and generate a channel attention map. The input feature is multiplied element by element with the channel attention map to obtain an enhanced feature map, as shown in formula (1). channel is the enhanced feature, σ is the Sigmoid function, Permute is the dimension permutation of the feature, ⊙ represents element-by-element multiplication, is the input feature; In order to fully mix the information between channels, a channel shuffle is designed to group the enhanced feature maps, transpose the grouped feature maps to disrupt the channel order in each group, and then restore them to the original shape, as shown in formula (2), shuffle It is the feature after shuffling. ChannelShuffle represents the channel shuffle operation. F shuffle =ChannelShuffle(F Channel ) (2) Finally, the spatial attention is designed to fully capture the local and global spatial features of the features. The channel dimension is first reduced to 1 / 4 of the original dimension and then restored to the original dimension using dilated convolution with various dilated rates. Then, batch normalization and ReLU activation function are used for nonlinear transformation. The sigmoid activation function is used to generate the spatial attention map. The shuffled feature map is multiplied element by element with the spatial attention map to obtain the final output feature, as shown in formula (3). B′ It is the feature after spatial attention, Conv represents the convolution operation, BN represents batch normalization, and ReLU represents the activation function; F B′ =σ(Conv(BN(ReLU(Conv(F shuffle )))))⊙F shuffle (3)。 5. The method for detecting 3D objects around an autonomous vehicle using parallel temporal modeling according to claim 1, characterized in that: The implementation of the parallel timing model module in S2 includes: First, we design a dual-stream time series network that combines sparse query with BEV, and then design a transparent stacked time series network. The two networks form a parallel time series model, as follows: (1) The dual-stream temporal network combines the sparse query and BEV of historical temporal features and integrates them into the current frame in a stream manner. By using the sparse query as the hidden state of temporal transmission, the ability to model the local temporal sequence of moving objects is improved and accurate spatial location information is obtained. The first step is to design a streaming method for sparse queries. In the autonomous driving scenario, most target instances are in motion, which indicates that the current instance will continue to exist in the future. The K queries with the highest confidence scores are passed to the next frame, providing valuable location priors and retaining the temporal features of historical frames. A memory pool is defined to store temporal features, including the target query Q corresponding to the BEV temporal features. B And the target query Q with perspective time series features P ,,The length of the memory pool is 4, and the number of targets stored in each frame is 256. The first t-2 frame information is taken out, which is called propagated BEV queries and propagated perspective queries. The number is 256, and the perspective is represented as PV; the propagated BEV / PV query is first added to the 644 initial object queries, as shown in equations (4) and (5). Aggregate to get the query Q, key K1, and value V1 of temporal attention, as shown in formula (6). Represents aggregation, The remaining 3 frames in the memory pool propagate BEV / PV queries as the key K2 and value V2 of temporal self attention, as shown in formula (7), Then, the key and value aggregation are performed separately, as shown in formula (8): Finally, the temporal interaction is performed in the temporal attention to obtain the output at the current moment. and As shown in formula (9), MHTSA() represents the calculation of the multi-head temporal self-attention mechanism, softmax() represents the activation function here, and D(K) represents the dimension of K; After spatial interaction, the query at the current moment is converted into background query and foreground query. The top K foreground queries with the highest scores are input into the memory pool. While the information of frame t-1 is obtained, the information of frame t-5 is discarded. In this way, the memory pool is updated frame by frame. Secondly, the streaming BEV is designed: the historical information of the streaming BEV is transmitted frame by frame through the hidden state. Its hidden state includes propagated BEV queries and propagated BEV features. The propagated BEV features themselves are equivalent to a continuously updated memory pool. The current BEV features can gain from the fused propagated BEV features. In this way, historical features are obtained, and the BEV features are updated and propagated frame by frame. Based on the vehicle posture, the propagated BEV features are converted to the current moment to achieve spatiotemporal alignment. Then, the gated recurrent unit is used to fuse the converted propagated BEV features with the current BEV features. In order to ensure the stability of training, the layer normalization operation is used, as shown in equations (10) and (11). represents the transfer BEV characteristics before conversion, represents the transformed transfer BEV characteristics, φ represents the conversion function, Represents the coordinate transformation matrix from the previous frame to the current frame, represents the fused BEV features, LN represents layer normalization, GRU represents gated recurrent unit, represents the BEV characteristics before fusion; (2) Design a perspective stacking time series network; stack the historical perspective features with the current features. The stacked historical features can be specified to be within the memory pool length or outside the memory pool length, so that historical information can be flexibly obtained. Assuming that the current time is t-1, the historical 4 frames of information outside the memory pool are stacked and fused with the t-1 frame information, and then interact with the current query containing the most recent 4 frames of historical information in the parallel aggregation unit, as shown in formula (12). It represents the output of perspective features after passing through the parallel aggregation unit. Fusion represents the fusion operation. represents the perspective feature at time t-1, represents the perspective characteristics from t-9 to t-6, Indicates interaction, represents the target query of perspective features at time t-3, t-4, and t-5, Q initial represents the initial query, Represents the target query of the perspective feature at time t-1.

6. The method for detecting 3D objects around an autonomous vehicle using parallel temporal modeling according to claim 1, characterized in that: The parallel attention mechanism module in S2 is specifically designed as follows: BEV query obtained by temporal attention mechanism interaction and layer normalization and PV query By using them for spatial interaction, a parallel attention mechanism is designed, including two parallel aggregation units (PaAgUn) based on 3D spatial variable attention mechanism. In the parallel aggregation units, spatial interaction between query and corresponding features is performed respectively, and the important information in the image is paid attention to through local learning, so as to realize efficient decoding of temporal features. For the obtained stacked PV time series features and enhanced streaming BEV time series features, a parallel aggregation unit is used to interact the PV time series features and streaming BEV time series features with the corresponding query. For each 3D spatial reference point, parallel attention is learned for N sampling offsets. After that, the reference point P 3d Combined sampling offset △P 3d Become a variable reference point in 3D space As shown in formula (13), According to the transformation matrix T of the mth camera m Project the 3D variable reference point onto the corresponding 2D view to obtain the corresponding 2D projection point As shown in formula (14), Use a bilinear interpolation network to sample the historical features of the corresponding view stack to obtain the sampled features As shown in formula (15), S Bilinear () represents bilinear interpolation; Finally, the target query in 3D space interacts with the sampled features, and the results of the interaction of each perspective are aggregated into the 3D query. The PV time series features at time t-1 are sampled and sliced ​​into keys and values ​​(key / value) and compared with the corresponding query. interaction, as shown in formula (16), ParallelAttention (PV) represents the parallel attention calculation of perspective features, MultiHeadPaAgUn() represents the multi-head parallel aggregation unit, Indicates the key corresponding to the perspective feature from time t-1 to time ti, Represents the value corresponding to the perspective feature from time t-1 to ti; Similarly, the BEV time series features at time t-1 are sliced ​​into keys and values ​​after sampling, and the corresponding query interaction, as shown in formula (17), ParallelAttention(BEV) represents parallel attention calculation of BEV features, MultiHeadPaAgUn() represents multi-head parallel aggregation unit, represents the key corresponding to the BEV feature at time t-1, represents the value corresponding to the BEV feature at time t-1; After feature extraction and refinement by parallel attention, the outputs of ParallelAttention (PV) and ParallelAttention (BEV) are aggregated along the channel dimension to obtain the refined target query, which is fed into the normalization layer and the feedforward network, as shown in Equation (18). ParallelAttention(Q) represents the output of the two parallel attention mechanisms combined. Indicates aggregation; After layer-by-layer refinement, the 3D detection head outputs the prediction results of the foreground category and detection box.

7. The method for detecting surround 3D objects of an autonomous driving vehicle based on parallel temporal modeling according to claim 1, characterized in that: The model training process of S3 is as follows: Design the total loss function: The focal loss function L is used for needle classification 焦点 , represents the loss function between the category of the decoded output and the label. The L1 loss function is used for the detection box, which represents the loss function between the predicted box of the decoded output and the label; The prediction results and true values ​​are expressed in the form of sets. The prediction set is expressed as x = (a, b), and the label set is expressed as y = (a′, b′). a represents the category, b represents the bounding box, a′ represents the category label, and b′ represents the bounding box label. L bbox (b,f H (b′)) represents the calculation of bounding box loss, L class (a,f H (a′)) represents the calculation of category loss, using the Hungarian algorithm to assign labels between the label set and the prediction set, f H (b′) and f H (a′) represents the distribution function between the two true values ​​and the distribution function between the two labels, and α and β represent the proportional coefficients of the two losses.

8. The method for detecting surround 3D objects of an autonomous driving vehicle using parallel temporal modeling according to claim 1 or 7, characterized in that: The S3 further includes: The software and hardware settings of the training environment are: based on a single-card RTX4090GPU, cuda11.1, and PyTorch1.9.0, different image resolutions are set, the batch size is set to 7, the comparative experiment is trained for 60 rounds, the ablation experiment is trained for 24 rounds, the AdamW optimizer is used, the weight decay is set to 0.01, the basic learning rate is set to 2e-4, and the cosine annealing strategy is used. ResNet50 and ResNet101 are used as the backbone for comparative experiments on the validation set, and VoVNet99 is used as the backbone for comparative experiments on the test set. BEVFormer is used in combination with a single-layer encoder for BEV feature extraction, BEV is used for data enhancement, the total number of the two queries is set to 900, and a streaming training strategy is used. Different training sequence lengths are obtained by controlling the number of sequence segmentations, and the entire sequence is used for inference.

9. A surround 3D target detection model for an autonomous driving vehicle based on parallel temporal modeling, characterized in that: This model is the model constructed in S2 described in any one of claims 1-6.

10. A vehicle device, characterized in that: The device is equipped with a surround three-dimensional target detection model trained according to claim 1.

Images