A 3D target and speed detection method based on time delay correction in V2X scenarios

Abstract

This invention belongs to the field of computer vision technology, specifically to a delay-corrected 3D object and speed detection method in V2X scenarios. The method first converts a point cloud image into a bird's-eye view and feeds it into a point cloud feature extraction module. The current frame features from the infrastructure are then transmitted to the vehicle. The vehicle combines the received current frame with the previously saved k-1 frames of feature data, using a delay compensation module based on motion trend estimation to obtain delay error compensation features. This is then combined using a secondary adaptive feature fusion module. Finally, a 3D object and speed detection head is used to obtain a target frame and speed prediction. This method addresses the fusion detection errors caused by transmission delays between the vehicle and infrastructure, enabling 3D object and speed detection in delay scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically to a 3D target and speed detection method based on time delay correction in a V2X scenario. Background Art

[0002] Collaborative sensing is a multi-agent system in which agents share perception information. Depending on the agent type, it can be categorized into three agent modes: V2V (vehicle-to-vehicle), V2I (vehicle-to-infrastructure), or V2X (vehicle-to-everything). Collaborative sensing is sometimes fused with sensor fusion (or multimodal sensing) because both incorporate the concept of sensor data fusion. A major challenge in collaborative sensing for 3D object detection is the time stamp asynchrony between the data collected by the ego vehicle's sensors and those collected by infrastructure sensors. This asynchrony is caused by different sensor initialization times and communication delays from the infrastructure to the ego vehicle. Previous empirical studies have shown that communication delays typically range from 100 to 500 milliseconds. Due to scene changes and object movement, time asynchrony can lead to significant fusion errors, resulting in inaccuracies in subsequent tasks such as object detection. Therefore, time compensation is necessary to correct fusion errors and ensure accurate object detection.

[0003] There are currently various algorithms for 3D object detection based on cooperative perception driving, such as PointPillars 【1】 、TCLF 【2】 、DiscoNet 【3】 、V2VNet 【4】 and FFNet 【5】Algorithm, PointPillars is an algorithm proposed based on the vehicle perspective. It converts the point cloud into a bird's-eye view, uses a traditional CNN network to extract features, and finally uses SSD as the 3D target detection head. However, PointPillars does not contribute to the fusion of vehicle perspective and infrastructure perspective and the handling of communication delay issues. TCLF is an algorithm proposed for vehicle-road collaboration scenarios to solve problems such as communication delay. After performing 3D target detection on the infrastructure side and the vehicle side respectively, the 3D target detection results of the two parts are combined as the final result. However, TCLF does not associate infrastructure data with vehicle-side data at the feature level, and its accuracy is lower than other methods. DiscoNet is an algorithm proposed for vehicle-road collaboration scenarios. It uses a teacher-student framework to train DiscoGraph through knowledge distillation, which enhances the stability of training. However, the accuracy under communication delay conditions needs to be improved. V2VNet is an algorithm proposed for vehicle-road collaboration scenarios. It collects sensory data from surrounding vehicles or infrastructure, enhances the vehicle's own sensory field of view, and transmits compressed deep feature maps (a middle fusion approach). This activation method achieves high accuracy while meeting communication bandwidth requirements, but it also lacks an effective solution for fusion errors caused by communication latency. FFNet transmits feature streams between infrastructure features and feature information within the current frame (a middle fusion approach), and uses this feature stream information to compensate for data delays. However, FFNet requires a large transmission bandwidth and is still lacking in target object motion estimation. Therefore, improvements to existing algorithms are needed.

[0004] References:

[0005] 【1】Lang AH, Vora S, Caesar H, et al. Pointpillars: Fast encoders for object detection from point clouds[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019:12697-12705;

[0006] 【2】Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J., & Nie, Z. (nd). DAIR-V2X: A Large-Scale Dataset for Vehicle-InfrastructureCooperative 3D Object Detection. https: / / thudair.baai.ac.cn / index ;

[0007] 【3】Li,Y.,Ren,S.,Wu,P.,Chen,S.,Feng,C.,&Zhang,W.(2021).LearningDistilled Collaboration Graph for Multi-Agent Perception. http: / / arxiv.org / abs / 2111.00643 ;

[0008] 【4】Wang, T.-H., Manivasagam, S., Liang, M., Yang, B., Zeng, W., Tu, J., & Urtasun, R. (2020). V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction. http: / / arxiv.org / abs / 2008.07519 ;

[0009] 【5】Yu, H., Tang, Y., Mao, J., Xie, E., Yuan, J., Luo, P., & Nie, Z. VEHICLE-INFRASTRUCTURE COOPERATIVE 3D DETECTION VIA FEATURE FLOW PREDICTION. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide a 3D target and speed detection method based on delay correction in V2X scenarios, so as to solve the fusion detection error caused by the transmission delay between the vehicle end and the infrastructure end, and realize 3D target detection and speed detection in the delay scenario.

[0011] To address the above technical issues, the present invention provides a 3D object and speed detection method based on delay correction in V2X scenarios. The delay compensation network includes a point cloud feature extraction module, a delay compensation module based on motion trend estimation, an adaptive feature fusion module, and a 3D object and speed detection head. The process of performing 3D object and speed detection using the delay compensation network is as follows:

[0012] S1. Collect point cloud data from the infrastructure and vehicle ends at the same time, convert the point cloud data from the infrastructure and vehicle ends into bird's-eye view images, and then send them to the point cloud feature extraction module to obtain the current frame features of the infrastructure. and the vehicle's current frame features Among them, t is the current time, τ is the delay time;

[0013] S2. The current frame features of the infrastructure Sent to the vehicle side; the vehicle side will receive the current frame features of the infrastructure and the retained feature data of the previous frame The motion trend feature M between two frames is calculated by the motion trend estimation module t-1-τ , and combined with the historically saved inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ}Use ConvGRU recurrent neural network to infer the current frame features of the infrastructure The delay compensation characteristic M t-τ , where k is the previous k frames of data and tk-τ is the time;

[0014] S3, delay compensation feature M t-τ Current frame features with infrastructure The adaptive feature fusion module is sent to perform adaptive fusion to obtain the infrastructure features after delay compensation correction. Then the infrastructure characteristics after delay compensation correction are and the vehicle's current frame features Then send it to the adaptive feature fusion module for adaptive fusion to obtain the fusion feature

[0015] S4. Fusion Features The 3D object detection regression frame and speed prediction are obtained through the 3D object and speed detection head.

[0016] As an improvement to the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0017] The specific process of converting the point cloud data into a bird's-eye view is as follows:

[0018] The point cloud data is voxelized into P voxels from a bird's-eye view perspective. Assuming that there are P non-empty voxels in each sample, each voxel is fixedly sampled or padded with 0 to N point cloud data, which is represented as a tensor of (D, P, N). Then, the dimension D of the point cloud data is extracted into dimension C. After the Max Pooling operation, a feature map of (C, P) dimension is obtained. Finally, the dimension P is converted into H×W dimension, and the features (C, H, W) based on the bird's-eye view perspective are obtained.

[0019] As a further improvement of the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0020] The feature extraction process of the point cloud feature extraction module is to extract the features (C, H, W) under the input bird's-eye view perspective through the CNN network combined with CBAM attention to obtain multi-scale features, and then upsample them to the same feature size for splicing.

[0021] As a further improvement of the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0022] The specific process of step S2 is:

[0023] At the current time t, the vehicle receives the current frame features of the infrastructure hour,

[0024] (1) Calculate infrastructure characteristics through the motion trend estimation module and features The movement trend characteristics M t-1-τ ;

[0025] (2) The historical inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ} and motion trend characteristics M t-1-τ Sequentially feed it into k-1 ConvGRU recurrent neural networks to perform compensation for the delay feature M that needs to be compensated and corrected at time t t-τ predictions.

[0026] As a further improvement of the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0027] The motion trend estimation module assumes that the neighborhood motion direction is consistent, and uses the position derivative and the least square method to calculate the motion trend between two infrastructure feature frames.

[0028] As a further improvement of the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0029] Infrastructure characteristics after delay compensation correction in step S3 The calculation process is:

[0030]

[0031] Among them, M(X) represents the calculation function for obtaining the adaptive distribution of features for a single feature:

[0032] M(X)=B(PWConv2(δ(B(PWConv1(X))))) (12)

[0033] Among them, PWConv1 is a 1×1 point convolution that reduces the number of channels of the input feature to The operation of BatchNorm layer, δ represents the Relu activation function, and PWConv2 represents the operation of 1×1 point convolution to scale the number of channels back to the original channel size;

[0034] The fusion features The calculation process is:

[0035]

[0036] As a further improvement of the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0037] The 3D object and speed detection head includes an SSD network and a speed detection head. In the regression branch of the SSD network, two convolution branches are added to estimate the regression box height and elevation angle respectively. The speed detection head includes two convolution branches to predict the speed in the x and y directions respectively.

[0038] The convolution branches are composed of two convolutional layers and one fully connected layer.

[0039] As a further improvement of the 3D target and speed detection method based on delay correction in the V2X scenario of the present invention:

[0040] The training and testing process of the delay compensation network is as follows: collecting point cloud frame data and image data from the vehicle-side perspective and the infrastructure-side perspective, generating paired vehicle-side and infrastructure-side point cloud frame data based on timestamps, and then constructing a delay dataset and dividing it into a training set, a validation set, and a test set for training and testing the delay compensation network;

[0041] The m-th frame data of the vehicle end at time t in the time-delay data set corresponds to the m-1-th frame data in the original infrastructure data.

[0042] The beneficial effects of the present invention are mainly reflected in:

[0043] 1. This paper creates a new collaborative perception dataset VOX-V2I by collecting and constructing point cloud sensing data streams of vehicles and road infrastructure at the same time.

[0044] 2. This paper proposes a 3D object and velocity detection network based on a transmission delay compensation module, referred to as the Delay Compensation Network (SyncFixNet). This delay compensation module, based on motion trend estimation, utilizes infrastructure features from multiple historical frames to calculate inter-frame motion trend sequences. This effectively addresses the fusion error caused by communication delays in vehicle-infrastructure collaborative environments. Compared to other algorithms, it demonstrates higher accuracy on the collaborative perception dataset VOX-V2I while minimizing transmission costs.

[0045] 3. Compared with the feature fusion method of directly adding features, the present invention uses a secondary adaptive feature fusion module to better fuse the unique information contained in different features, so that the unique semantic information contained in different features can be fully expressed;

[0046] 4. In addition to the ability to detect 3D objects, the network of the present invention also has the ability to detect speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0048] Figure 1 Schematic diagram of the delay compensation network (SyncFixNet) of the present invention;

[0049] Figure 2 Schematic diagram of the adaptive fusion module (FFusion Part). DETAILED DESCRIPTION

[0050] The present invention is further described below with reference to specific embodiments, but the protection scope of the present invention is not limited thereto:

[0051] Example 1: A 3D target and speed detection method based on delay correction in a V2X scenario, such as Figure 1-2 The specific steps are as follows:

[0052] S1. Collection and construction of point cloud sensor data streams of vehicles and road infrastructure at the same time

[0053] The acquisition equipment used is Haojie's LIVOX HAP laser radar, which has a detection distance of 150 meters, a field angle of 120 degrees horizontally and 25 degrees vertically. It is a 144-line laser radar with an angular resolution of 0.18° to 50.23°, a data rate of 450,000 points per second, and a frame rate of 10Hz. It is a mass-produced product for commercial and passenger vehicles such as Xiaopeng Motors and FAW Jiefang.

[0054] For the data collection scenario, the vehicle was driven through a preset route, and a lidar and camera were deployed on the top of the vehicle to collect point cloud frame data and image data from the vehicle's perspective. LiDAR and cameras were deployed at traffic intersections with overpasses to collect point cloud frame data and image data from the infrastructure's perspective. Paired vehicle-side and infrastructure-side data frames, including picture frames and point cloud frames, were generated based on the same timestamp less than 10ms. The above steps were repeated in 10 selected traffic intersections to form 3,000 frames of paired frame data. The 3D target boxes were annotated using the annotate tool, and the dataset was named the VOX-V2I dataset.

[0055] The specific data set information is shown in Table 1:

[0056] Table 1

[0057] Dataset Collaborative Perception Driving Dataset VOX-V2I Number of categories 1 Category Details Car Number of picture frames 3000 Number of point cloud frames 3000 Number of target boxes 2w+

[0058] S2. Building a Delay Compensation Network (SyncFixNet)

[0059] The delay compensation network (SyncFixNet) includes a point cloud feature extraction module, a delay compensation module based on motion trend estimation (Synchronous Fix Module), an adaptive feature fusion module and a 3D target and speed detection head. Figure 2 As shown in the figure, the point cloud image is first converted into a bird's-eye view and sent to the point cloud feature extraction modules on the infrastructure and vehicle sides. The current frame features of the infrastructure side are then sent to the vehicle side. The vehicle side combines the received current frame (delayed frame) with the previously saved k-1 frames of feature data through the Synchronous Fix Module based on motion trend estimation to obtain the delay error compensation features. The adaptive feature fusion module is then used to obtain the corrected infrastructure current frame features. The adaptive feature fusion module is then used to fuse the corrected infrastructure current frame features with the vehicle current frame data. Finally, the 3D object detection regression box and speed prediction are obtained through the 3D object and speed detection head.

[0060] 2.1 Constructing a point cloud feature extraction network based on bird’s-eye view

[0061] The bird's-eye view-based point cloud feature extraction network is used to extract vehicle-side features and infrastructure-side features, and then transfer the extracted infrastructure features to the vehicle side. The specific operation details are as follows:

[0062] First, the point cloud frame data is converted to a bird's-eye view, and the classic method of Pillar Feature Network is used to voxelize it into P voxels under the bird's-eye view; the point cloud in each voxel is represented by a D = 9-dimensional vector (x, y, z, r, x c ,y c ,z c ,x p ,y p ), where x, y, z, and r are the real coordinate information and reflection intensity of the point cloud, x c ,y c ,z c is the geometric center of all points in the voxel where the point cloud is located, x p ,y p is the difference between the true coordinates of the point cloud and the geometric center in the x, y scale; assuming that there are P non-empty voxels in each sample, and each voxel is fixedly sampled or padded with 0 to N point cloud data, the sample can be represented as a tensor of (D, P, N), and then the dimension D of the point cloud data is extracted into dimension C. At this time, the sample is represented as (C, P, N). After the Max Pooling operation, the feature map of the (C, P) dimension is obtained, and finally the dimension P is converted into H×W dimension, and finally the feature (C, H, W) based on the bird's-eye view is obtained.

[0063] Then, in further feature extraction operations, a CNN network combined with CBAM attention was used to extract and retain features of multiple scales. Finally, they were upsampled to the same feature size for splicing, so that the target detection head can maintain good detection accuracy for targets of different sizes.

[0064] After both the vehicle-side features and the infrastructure-side features are extracted, the infrastructure-side features are sent to the vehicle-side. Compared with transmitting the original point cloud data, sending the extracted features can effectively reduce the transmission cost and, to a certain extent, alleviate the impact of transmission delay.

[0065] S2.2. Constructing a Synchronous Fix Module Based on Motion Trend Estimation

[0066] The overall idea of ​​the Synchronous Fix Module is to use the historically preserved infrastructure frame feature sequence and infrastructure current frame characteristics Using the motion trend estimation module, the inter-frame motion trend feature sequence of the target object is obtained {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-1-τ}, the inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-1-τ}Sequentially fed into the ConvGRU recurrent neural network to predict the current frame features of the infrastructure The delay compensation characteristic M t-τ .

[0067] At the current time t, the vehicle receives the current frame (delayed frame) feature of the infrastructure hour,

[0068] (1) Calculate infrastructure characteristics through the motion trend estimation module and features The movement trend characteristics M t-1-τ ;

[0069] (2) The historical inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ} and motion trend characteristics M t-1-τ Sequentially fed into the ConvGRU recurrent neural network, each ConvGRU recurrent neural network correspondingly calculates a motion trend feature; wherein, the historical inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ} is the feature frame including the k-1 frames from time tk-τ (k is the previous k frames of data, τ is the delay time) to time t-1-τ;

[0070] The motion trend estimation module is a module that obtains inter-frame motion trend features between sequential feature frames (every two frames). Assuming that the neighborhood motion direction is consistent, the motion trend between each two frames of infrastructure feature frames is calculated using the position derivative and the least squares method. Specifically, assuming that the infrastructure feature F of the kth frame at time t is k and infrastructure features F of the k+1th frame at time t+Δt k+1 , for a certain position (x, y) in the feature map at time t, assuming that it should be located at (x+u, y+v) in the feature map at time t+Δt, we can list formula (1):

[0071] I(x,y,t)=I(x+u,y+v,t+Δt) (1)

[0072] Performing a first-order Taylor expansion on the right side of the equation yields formula (2):

[0073] I(x+u,y+v,t+Δt)=I(x,y,t)+I′ x u+I′ y u+I′ t Δt (2)

[0074] We can get formula (3):

[0075] I′ x u+I′ y v+I′ t Δt=0 (3)

[0076] It can be expressed in matrix form as formula (4):

[0077]

[0078] As shown in formula (4), I′ x and I′ y Represents the gradient (partial derivative) of the feature point (x, y) in the x and y directions, I′ t is the time derivative of the feature point (x, y) at time t, I′ t Δt is expressed as feature F k and feature F k+1 The difference between the feature points (x, y). So, given two feature maps, we can simply calculate the known I′ x , I′ y , I′ t Δt, then the motion trends u and v in the x and y directions are the quantities to be solved. Since there are two quantities to be solved, another constraint equation is required for joint solution.

[0079] Since the local motion features of the same object are consistent, it is assumed here that the motion direction of the neighborhood of the feature point (x, y) is consistent. Then, the following formula (5) can be listed for all areas of the feature point (x, y):

[0080]

[0081] Since formula (5) is in the form of Ax=b, it can be solved by the least squares method:

[0082] (u,v)=(A T A) -1 A T b (6)

[0083] Where A is b is

[0084] According to the above operation steps, we can obtain the feature maps representing the movement trend in the x and y directions respectively. The two feature maps are cat-operated at the channel level to obtain the infrastructure point cloud features F at the kth frame at time t and the k+1th frame at time t+Δt. k and F k+1 The movement trend characteristics M k .

[0085] (3) Model a series of motion trend information and estimate the current state, and finally estimate the delay compensation feature M that needs to be compensated for the delay t-τ .

[0086] Infrastructure features received this time and the previous frame infrastructure features Use the motion trend extraction module to calculate the motion trend feature M t-1-τ , combined with the retained inter-frame motion trend feature sequence of the infrastructure extracted by the motion trend estimation module (specifically, the inter-frame motion trend feature sequence including the k-1 frame features from time tk-τ (k is the previous k frames of data, τ is the delay time) to time t-1-τ {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ}), and obtain the inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-1-τ}, sequentially fed into the ConvGRU recurrent neural network, to compensate for the delay feature M that needs to be compensated at time t t-τ predictions.

[0087] The purpose of the ConvGRU (C-GRU) recurrent neural network is to obtain the movement information of the target object between the historical frame sequences, so the change information of the local space is particularly important. Therefore, the matrix multiplication step of the ordinary GRU is replaced by the feature extraction of the convolution, and then upsampled to the original size for splicing; the delay of the current frame received is often unknown, but the relationship between its historical data is known and accurate. It is physically meaningful to infer the delay compensation correction of the current frame based on the known historical relationship to alleviate the time synchronization problem caused by communication delay. The ConvGRU recurrent neural network models a series of theoretical cooperation information and estimates the delay compensation feature M of the current frame t-τ Specifically, multiple C-GRU units are used to calculate the features of multiple historical infrastructure frames. The calculation method of each C-GRU unit is:

[0088] In ConvGRU, the gate r that controls the reset gate and the gate z that controls the update gate, the hidden state h′ at the intermediate moment, and the hidden state h of the current node are controlled. t They are expressed by formulas (7)-(10) respectively:

[0089] r=σ(W r *x t +U r *h t-1 ) (7)

[0090] z=σ(W z *x t +U z *h t-1 ) (8)

[0091] h′=tanh(W*x t +U*(r⊙h t-1 )) (9)

[0092] h t =(1-z)⊙h t-1 +zh′ (10)

[0093] In the above formula, * represents the convolution operation, x t Represents the current input (i.e., the inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-1-τ}), W r , W z , W and U r , U z , U is a 2D convolution kernel, which preserves the spatial topology in the hidden recursive representation, h t =(h t (i,j)), h t (i, j) represents the feature vector at position (i, j), and zero padding is used in the recursive convolution to ensure that the size of the hidden representation space remains fixed over time. In the convolution process, W r , W z , the size of the convolution W is k1×k2×O x ×O h , where k1×k2 represents the convolution kernel size (e.g. 3×3), and the selection criteria is that it needs to be significantly lower than the convolution map size N1×N2. The hidden state h′(i, j), reset gate r(i, j) and update gate z(i, j) at the intermediate moment are based on the input data x t and the hidden state h of the previous nodet-1 The local neighboring convolution kernel size at position (i, j) is defined as k1×k2. In addition, the hidden state h of the current node t The receptive field size associated with (i, j) in the historical hidden state h t-1 , h t-2 ...increases over time. Therefore, the model is capable of describing spatiotemporal series predictions with high spatial variation.

[0094] S2. Adaptive feature fusion module (FFusion Part)

[0095] (1) The delay compensation feature M t-τ Current frame features with infrastructure Feed it into the adaptive feature fusion module (i.e. Figure 2 The adaptive fusion of FFusion Part 1 in the delay compensation correction is performed to obtain the infrastructure characteristics. The calculation process is:

[0096]

[0097] Among them, M(X) represents the calculation function for obtaining the adaptive distribution of features for a single feature:

[0098] M(X)=B(PWConv2(δ(B(PWConv1(x))))) (12)

[0099] Among them, PWConv1 is a 1×1 point convolution that reduces the number of channels of the input feature to The operation of B represents the BatchNorm layer, δ represents the Relu activation function, and FWConv2 represents the operation of 1×1 point convolution to scale the number of channels back to the original channel size;

[0100] (2) Infrastructure characteristics after delay compensation correction and the vehicle's current frame features Send to the adaptive feature fusion module (attached Figure 2 FFusion Part2) in the adaptive fusion to obtain the fusion feature (FFusion Part1 and FFusion Part2 are the same module, both representing adaptive feature fusion modules, but are used to indicate that they are used twice in succession). The calculation process is:

[0101]

[0102] Among the three modules, the point cloud feature extraction module, the synchronous fix module, and the adaptive feature fusion module, the point cloud feature extraction network based on the bird's-eye view can maintain the accuracy of feature extraction while improving the speed of feature extraction; the delay compensation module based on motion trend estimation innovatively uses the infrastructure features of multiple historical frames to calculate the motion trend sequence between frames, and then sequentially feeds the motion trend sequence into the ConvGRU network to predict the delay compensation feature M for the current infrastructure frame. t-τ ; and use the adaptive feature fusion module to make the delay compensation feature M t-τ and infrastructure current frame characteristics Adaptive fusion to obtain infrastructure characteristics after delay compensation correction Then the infrastructure characteristics after delay compensation correction and the features of the vehicle’s current frame Perform adaptive feature fusion to obtain fusion features Compared with the feature fusion method of directly adding features, the adaptive feature fusion module can better fuse the unique information contained in different features, so that the unique semantic information contained in different features can be fully expressed; the composition of the above three network modules enables the overall network to accurately and reasonably estimate the infrastructure characteristics after delay compensation correction. And on the basis of being able to fully express the unique semantic information of different features, the above-mentioned features are fused, and finally, the fused feature is obtained. Send the 3D target and speed detection head for result detection.

[0103] S2.5, 3D target and speed detection head

[0104] (1) In the regression branch of the SSD network, two convolution branches are added to estimate the regression box height and elevation angle respectively. The fusion feature obtained in step 2.4 is Send it to the SSD target detection network branch to output the 3D target detection box.

[0105] (2) The fusion features obtained in step 2.4 The velocity detection head is composed of two convolution branches, which predict the velocity in the x and y directions respectively.

[0106] The four convolution branches mentioned above are composed of two convolutional layers and one fully connected layer.

[0107] S3. Training and testing of the delay compensation network (SyncFixNet)

[0108] (1) Creation of time-delay dataset

[0109] The VOX-V2I dataset collected in step 1 is divided into the asynchronous paired dataset VOX-Async-1. This is done by replacing the previous frame of the infrastructure data with the current frame to form the VOX-Async-1 delay dataset. In the VOX-Async-1 delay dataset, the mth frame of the vehicle data at time t corresponds to the m-1th frame of the original infrastructure data. The VOX-Async-1 dataset described above is then divided into training, validation, and test sets in a 5:2:3 ratio.

[0110] (2) Experimental parameter setting

[0111] In terms of point cloud gridding operations, the detection range was set according to the point cloud distribution range after fusion of the VOX-Async-1 dataset: [0, -39.68] m on the X axis, [-3, 92.16] m on the Y axis, and [39.68, 1] m on the Z axis, and the minimum voxel resolution was set to [0.16, 0.16, 4] m.

[0112] In terms of training parameters, training is set on a single NVIDIA GeForce RTX 3090 GPU, with a training cycle of 40 cycles, the optimizer is Adam, the initial learning rate is set to 0.001, the weight decay is set to 0.01, the momentum size is set between 0.895 and 1, and the maximum gradient norm of the gradient truncation is set to 35.

[0113] (3) Training and testing the network

[0114] The training set is input into the delay compensation network (SyncFixNet) for training and validation on the validation set. The trained delay compensation network (SyncFixNet) is saved after 40 rounds of training.

[0115] The test set was then input into the trained delay compensation network (SyncFixNet), and the mAP@3D IoU0.5, mAP@3D IoU0.7, mAP@BEV IoU0.5, mAP@BEV IoU0.7 accuracy and average transmission consumption AB (Byte) indicators were used for evaluation. The final accuracy reached 56.73, 33.6, 63.79, and 55.34 respectively, and the average transmission consumption was 1.1×10^5 Byte, thus obtaining a delay compensation network (SyncFixNet) that can be used in practice.

[0116] S4. The actual use process of the trained network

[0117] Collect the point cloud data of the infrastructure end and the vehicle end at the same time, convert the point cloud data of the infrastructure end and the vehicle end into the form of a bird's-eye view, and then send them to the point cloud feature extraction module to extract features. Sent to the vehicle side; at the current time t, the vehicle side receives the current frame (delayed frame) feature of the infrastructure When the motion trend estimation module is used to extract the infrastructure feature inter-frame motion trend feature sequence, the retained feature inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ}, and, the infrastructure characteristics calculated by the movement trend estimation module and features The movement trend characteristics M t-1-τ , sequentially fed into the ConvGRU recurrent neural network, which models a series of motion trend information and estimates the current state, and finally estimates the delay compensation feature M that needs to be compensated and corrected due to the delay t-τ Then M t-τ Current frame features with infrastructure Send it to the adaptive feature fusion module (FFusionPart1) for adaptive fusion to obtain the features after delay compensation correction The characteristics after delay compensation correction and the features of the vehicle’s current frame Then send it to the adaptive feature fusion module (FFusion Part2) to get the fusion feature Finally, the fusion features Send it to the 3D target detection head to output the 3D target detection box information (3D box) and fuse the features The data is sent to the velocity detection head to estimate the velocity (Velocity) of the target object in the x and y directions.

[0118] S5. Experiment

[0119] The asynchronous pairing dataset VOX-Async-1 constructed in step S3 of Example 1 can be used to simulate the communication delay situation very well. In order to verify the effectiveness of the proposed method in solving the problem of inaccurate fusion caused by communication delay in the vehicle-road collaboration scenario, the algorithm proposed in this paper is compared with PointPillars in the constructed asynchronous pairing dataset VOX-Async-1. 【1】 、TCLF 【2】 、DiscoNet 【3】 、V2VNet【4】 and FFNet 【5】 We compared the algorithms and used our own data to construct a latency dataset to evaluate the effectiveness of 3D object detection and transmission loss. The training was conducted on a CentOS server, accelerated by a Tesla P4 GPU, and developed in the Pytorch deep learning framework. The specific hardware and software configurations are shown in Table 2, and the evaluation results are shown in Table 3.

[0120] Table 2

[0121] name Environment Configuration operating system Ubuntu 18.04.6 Graphics card GeForce 24GB Memory 256GB Development Environment Python 3.8 PyTorch 1.13.1

[0122] Table 3

[0123]

[0124] From the experiments in Table 3 above, we can know the effectiveness of the present invention in solving the problem of inaccurate fusion caused by communication delay in the vehicle-road collaboration scenario. Its 3D target detection accuracy in the delay data set has reached the highest level compared with other algorithms, and the transmission cost has been reduced to the lowest level, which can also alleviate the problems caused by communication delay to a certain extent. The delay compensation module (Synchronous Fix Module) can well solve the problem of fusion detection error caused by transmission delay, and its accuracy has reached the highest level among all the networks that solve transmission delay errors in Table 2, while reducing transmission loss to the lowest level.

[0125] In summary, the present invention provides a 3D target detection solution for the constructed collaborative driving dataset, and solves the problem of inaccurate multi-terminal data fusion caused by communication delays, etc., and the accuracy of 3D target detection in delay scenarios reaches the optimal level, providing an effective, accurate and sustainable solution for autonomous driving scenarios under collaborative driving conditions.

[0126] Finally, it should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples and is subject to numerous variations. All variations that can be directly derived or conceived by a person of ordinary skill in the art from the disclosure of the present invention are considered to be within the scope of protection of the present invention.

Claims

1. A 3D target and speed detection method based on time delay correction in V2X scenarios, characterized by: The delay compensation network includes a point cloud feature extraction module, a delay compensation module based on motion trend estimation, an adaptive feature fusion module, and a 3D object and speed detection head. The process of 3D object and speed detection using the delay compensation network is as follows: S1. Collect point cloud data from the infrastructure and vehicle ends at the same time, convert the point cloud data from the infrastructure and vehicle ends into bird's-eye view images, and then send them to the point cloud feature extraction module to obtain the current frame features of the infrastructure. and the vehicle's current frame features Among them, t is the current time, τ is the delay time; S2. The current frame features of the infrastructure Sent to the vehicle side; the vehicle side will receive the current frame features of the infrastructure With the retained feature data of the previous frame The motion trend feature M between two frames is calculated by the motion trend estimation module t-1-τ , and combined with the historically saved inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ }Use ConvGRU recurrent neural network to infer the current frame features of the infrastructure The delay compensation characteristic M t-τ , where k is the previous k frames of data and tk-τ is the time; S3, delay compensation feature M t-τ Current frame features with infrastructure The adaptive feature fusion module is sent to perform adaptive fusion to obtain the infrastructure features after delay compensation correction. Then the infrastructure characteristics after delay compensation correction are and the vehicle's current frame features Then send it to the adaptive feature fusion module for adaptive fusion to obtain the fusion feature S4. Fusion Features The 3D object detection regression frame and speed prediction are obtained through the 3D object and speed detection head.

2. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 1, characterized in that: The specific process of converting the point cloud data into a bird's-eye view is as follows: The point cloud data is voxelized into P voxels from a bird's-eye view perspective. Assuming that there are P non-empty voxels in each sample, each voxel is fixedly sampled or padded with 0 to N point cloud data, which is represented as a tensor of (D, P, N). Then, the dimension D of the point cloud data is extracted into dimension C. After the Max Pooling operation, a feature map of (C, P) dimension is obtained. Finally, the dimension P is converted into H×W dimension, and the features (C, H, W) based on the bird's-eye view perspective are obtained.

3. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 2, characterized in that: The feature extraction process of the point cloud feature extraction module is to extract the features (C, H, W) under the input bird's-eye view perspective through the CNN network combined with CBAM attention to obtain multi-scale features, and then upsample them to the same feature size for splicing.

4. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 3 is characterized by: The specific process of step S2 is: At the current time t, the vehicle receives the current frame features of the infrastructure hour, (1) Calculate infrastructure characteristics through the motion trend estimation module and features The movement trend characteristics M t-1-τ ; (2) The historical inter-frame motion trend feature sequence {M t-k-τ ,M t-k+1-τ ,M t-k+2-τ ,...,M t-2-τ } and motion trend characteristics M t-1-τ Sequentially feed it into k-1 ConvGRU recurrent neural networks to perform compensation for the delay feature M that needs to be compensated and corrected at time t t-τ predictions.

5. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 4 is characterized in that: The motion trend estimation module assumes that the neighborhood motion direction is consistent, and uses the position derivative and the least square method to calculate the motion trend between two infrastructure feature frames.

6. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 5, characterized in that: Infrastructure characteristics after delay compensation correction in step S3 The calculation process is: Among them, M(X) represents the calculation function for obtaining the adaptive distribution of features for a single feature: Among them, PWConv1 is a 1×1 point convolution that reduces the number of channels of the input feature to The operation of BatchNorm layer, δ represents the Relu activation function, and PWConv2 represents the operation of 1×1 point convolution to scale the number of channels back to the original channel size; The fusion features The calculation process is:

7. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 6, characterized in that: The 3D object and speed detection head includes an SSD network and a speed detection head. In the regression branch of the SSD network, two convolution branches are added to estimate the regression box height and elevation angle respectively. The speed detection head includes two convolution branches to predict the speed in the x and y directions respectively. The convolution branches are composed of two convolutional layers and one fully connected layer.

8. The method for 3D target and speed detection based on delay correction in a V2X scenario according to claim 7, characterized in that: The training and testing process of the delay compensation network is as follows: collecting point cloud frame data and image data from the vehicle-side perspective and the infrastructure-side perspective, generating paired vehicle-side and infrastructure-side point cloud frame data based on timestamps, and then constructing a delay dataset and dividing it into a training set, a validation set, and a test set for training and testing the delay compensation network; The m-th frame data of the vehicle end at time t in the time-delay data set corresponds to the m-1-th frame data in the original infrastructure data.

Images