Multi-vehicle cooperative fusion perception method based on spatiotemporal context

Abstract

This invention discloses a multi-vehicle cooperative fusion perception method based on spatiotemporal context. The method first acquires a large-scale open cooperative perception dataset. Next, the metadata from the dataset is input into a metadata sharing and feature extraction module to obtain the feature representation of each vehicle. Then, the feature representation of the autonomous vehicle and the feature representation of each cooperating vehicle are input into their respective neighborhood spatiotemporal modules, outputting two hidden states. These hidden states are then processed through a location embedding input space decoding and communication module, outputting the autonomous vehicle features, cooperating vehicle features, cooperating vehicle features in the shared space, and autonomous vehicle features in the shared space. Finally, a spatial adaptive fusion mechanism is used to align and fuse the obtained features, outputting fused features. These fused features are input into a regression head and a classification head, outputting bounding box prediction information and classification prediction information. This invention enables autonomous vehicles to achieve more reliable and timely environmental perception in complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and intelligent transportation, and in particular to a multi-vehicle cooperative fusion perception method based on spatiotemporal context. Background Technology

[0002] Multi-vehicle cooperative perception technology aims to improve the accuracy and real-time performance of overall environmental perception by sharing perception information among multiple autonomous vehicles, and has received widespread attention in the field of autonomous driving in recent years. Currently, research on multi-vehicle cooperative perception is mainly applied to scenarios such as safe driving, traffic management, and intelligent connected vehicles in complex traffic environments, building a global perception model in real time by sharing sensor data from each vehicle. However, effectively addressing issues such as transmission latency, communication bandwidth limitations, and heterogeneous data fusion remains a technical challenge.

[0003] In multi-vehicle cooperative perception research, most existing work focuses on improving perception accuracy and real-time performance, employing complex algorithms for data fusion and optimization. However, overly complex models and computationally intensive operations can significantly increase the system burden. In environments with limited hardware resources (such as edge devices or in-vehicle computing platforms), this can easily lead to problems such as inference latency and memory overflow, severely limiting the system's scalability and practical application capabilities. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a multi-vehicle cooperative fusion perception method based on spatiotemporal context. This method solves the problems of transmission delay, communication bandwidth limitations, and data fusion in multi-vehicle cooperative perception by utilizing spatiotemporal context perception and efficient communication technologies. This method introduces spatiotemporal alignment and lightweight communication strategies to ensure that the perception data of different vehicles are accurately aligned in the spatiotemporal dimension, and prioritizes the transmission of the most valuable information under limited bandwidth conditions. By designing a robust feature fusion method, this method can effectively integrate the perception data of different vehicles, improve the accuracy and real-time performance of the multi-vehicle cooperative perception system, thereby enhancing the autonomous driving system's ability to cope with complex traffic scenarios and effectively reducing safety risks caused by perception errors.

[0005] The technical problem solved by this invention includes the following steps:

[0006] S1. Obtain large-scale open collaborative sensing simulation datasets and real datasets, mix them, and divide them into training, validation, and test data.

[0007] S2. The metadata (such as pose and extrinsic parameters) from the mixed dataset is input into the metadata sharing and feature extraction module. The autonomous vehicle constructs a collaborative graph and broadcasts the metadata. After receiving the metadata, the collaborating vehicles project their point cloud data onto the autonomous vehicle's coordinate system and align it with the autonomous vehicle's current LiDAR point cloud data. The point cloud of the autonomous vehicle, aligned with each collaborating vehicle, is encoded into bird's-eye view features by a point cloud encoder, outputting the feature of each vehicle in the bird's-eye view. Time-time feature representation The feature dimension is ,in , and These represent the height, width, and number of channels, respectively. Indicates the index of the vehicle.

[0008] S3. The vehicle (indexed as) Feature representation and each cooperating vehicle (indexed as) Feature representation The inputs are fed into the corresponding neighborhood spatiotemporal LSTM modules, and the outputs are the hidden states refined with spatiotemporal information. The specific steps of the neighborhood spatiotemporal attention LSTM module are as follows:

[0009] S31. Input the previous hidden state. and current moment features First, the data is spliced ​​along the channel dimension, and then normalized.

[0010] S32. Input the normalized result output from S31 into the neighborhood attention module, calculate attention only within the local window at each location, extract local spatiotemporal context features, and obtain the neighborhood attention. .

[0011] S33. Apply the neighborhood attention obtained in step S32 Apply the Sigmoid activation function element by element Obtain the gating weights It is used to control information retention and updating.

[0012] S34. Calculation Cellular state at any given moment : Obtaining the cell state at the previous moment ( Cellular state at any given moment ),current Neighborhood attention features at time step and gating weights Next, first set the gating weights Cell state compared to the previous moment Element-by-element multiplication. Activated by the tanh function, and then... Element-wise multiplication is performed to extract the new features that need to be added. Finally, the results of the two element-wise multiplications are added together to obtain the updated cell state. .

[0013] S35. Input Cell Status First, activate it with tanh to obtain... The feature map within the interval is then used to calculate the activation result and the gating weights. Element-wise multiplication yields the hidden state after refining the spatiotemporal information. As a spatiotemporal characteristic.

[0014] S4. Hide the output from step S3. After location embedding, the input spatial decoding and communication module outputs vehicle-specific features. , unique regional characteristics of collaborative vehicles Characteristics of collaborative vehicles in shared spaces and the characteristics of vehicles in shared spaces The specific steps for the spatial decoding and communication module are as follows:

[0015] S41. Input the spatiotemporal characteristics of each vehicle. Encoding the perceptual space involves adding a learnable location embedding to the spatiotemporal features to obtain features containing location information. .

[0016] S42. Input features containing location information Average pooling operations are applied to both the width and height dimensions to encode the width and height dimensional information into the channel dimension, thus obtaining spatial features. and .

[0017] S43. Spatial features and After aligning dimensions through broadcasting, the components are stitched together and then subjected to a nonlinear transformation. The process involves processing to obtain intermediate vectors. Then, spatial attention score vectors are calculated in the width and height dimensions, and a spatial confidence map is generated. .

[0018] Obtain the spatial confidence map Next, a filtering function is used to filter important locations based on a set threshold. If the confidence level of a location is greater than the threshold, it is marked as 1; otherwise, it is marked as 0. This yields the important spatial locations. Then, the corresponding key features are obtained. .

[0019] S44. Collaborative vehicles will occupy important spatial locations and corresponding important features The data is sent to the vehicle. After receiving the data, the vehicle calculates the features of its unique sensing area. Characteristics of the unique sensing area of ​​collaborative vehicles and characteristics of shared sensory regions , .

[0020] S5. Input common sensing region features and Alignment and fusion are performed through a spatial adaptive fusion mechanism to output unified common perception region features. Then, the unique sensing area characteristics of the vehicle will be used. Characteristics of the unique sensing area of ​​collaborative vehicles and unified shared sensory region features The input is fed into a vehicle-centric collaborative feature fusion module for robust feature fusion, and the final output is the fused feature. The specific steps of the vehicle-centric collaborative feature fusion module are as follows:

[0021] S51. First, identify the shared perception area features of the autonomous vehicle and the cooperative vehicles. and The features are concatenated to form a joint feature tensor containing information from both sources. Next, max pooling and average pooling are performed on this concatenated tensor, and the results are concatenated along the channel direction to obtain a composite feature that simultaneously possesses strong local responses and a global average response. Finally, the composite feature is fused and reorganized spatially using a sliding convolution operation to output a unified common perceptual region feature. .

[0022] S52. Spatiotemporal characteristics of the vehicle Each position in First, extract the query vector. , It is a learnable weight matrix; and its neighborhood matrix is ​​defined. Then based on the neighborhood matrix The corresponding neighborhood matrix in the cooperative space is calculated through coordinate alignment operations. and to , and neighborhood matrix Multiply by the corresponding learnable matrix respectively and This yields the corresponding three sets of key vectors. Sum value vector Then, the attention weights are calculated to obtain a unified representation of the features of the vehicle-specific perception region from the vehicle's perspective. A unified representation of the characteristics of the cooperative vehicle's unique perception area from the driver's perspective. Unified representation of features of the unified shared perception area from the vehicle's perspective .

[0023] S53. Integrate the vehicle-specific perception area features The unique sensing area characteristics of the integrated collaborative vehicle and the unified shared sensory region features after fusion The values ​​are input into the same importance generator, where features at each pixel location are convolved to generate the corresponding original weight distribution map. Then, a Sigmoid activation function is applied to the generated weight map, mapping all values ​​to... Within the range, a spatial importance map of the vehicle's unique perception area is obtained. Map showing the spatial importance of the unique perception area of ​​collaborative vehicles And a unified shared perception area spatial importance map .

[0024] S54. Combining three spatial importance diagrams , and The standardized attention map between different features is obtained by using the softmax function. Similarly, the formula for calculating the vehicle-specific spatial attention map is: The formula for calculating the collaborative vehicle-specific perception space attention map is as follows: .

[0025] S55. Spatial Attention Map , and Each with its corresponding feature map , and Element-wise multiplication along the spatial dimension yields three weighted feature maps. These three weighted feature maps are then directly added together along the channel and spatial dimensions to obtain the final fused feature.

[0026] S6. During the inference process, the final fused features output from step S5 are input into the regression head, and the bounding box prediction information is output, including the position, size, and yaw angle of the bounding box at each location.

[0027] The final fused features are input into the classification head, which outputs classification prediction information, representing the confidence value of each predefined box as either a target or background.

[0028] S7. During training, this method uses smoothed L1 loss for regression and focal loss for classification.

[0029] The beneficial effects of this invention are as follows: The proposed multi-vehicle collaborative fusion perception method based on spatiotemporal context achieves high efficiency and high accuracy in multi-vehicle collaborative perception in complex dynamic traffic environments through the overall technical path of "prediction compensation – lightweight communication – robust fusion". Overall, this method can significantly reduce spatiotemporal asynchronous errors caused by communication delays, break through the information transmission bottleneck of bandwidth limitation, and improve perception robustness through cross-vehicle fusion strategy, enabling autonomous vehicles to achieve more reliable and timely environmental perception in complex scenarios. Specifically, the metadata sharing and feature extraction module uses point cloud projection in a unified coordinate system and shared encoder extraction to align the perception information of each vehicle in the same semantic space, improving the prerequisite for multi-source data fusion. The neighborhood spatiotemporal attention LSTM module uses a local spatiotemporal attention mechanism to predict and compensate for features in historical frames, effectively alleviating the perception offset problem caused by transmission delay and improving feature prediction accuracy while ensuring real-time performance. The spatial decoding and communication module uses position embedding and spatial attention filtering to select only potential key area features for sparse transmission, significantly reducing the amount of communication data and ensuring efficient information exchange under limited bandwidth conditions. The vehicle-centric collaborative feature fusion module uses cross-neighborhood attention and a spatial importance generator to selectively map the features of the vehicle and collaborating vehicles to the vehicle's perspective and suppresses noise interference based on a weighted fusion strategy, achieving robust fusion of multi-source features. The detection head module accurately decodes the fused global features and outputs high-confidence 3D detection results, further improving accuracy and reliability in the final target detection task. Attached Figure Description

[0030] Figure 1 A diagram illustrating a multi-vehicle collaborative fusion perception method based on spatiotemporal context;

[0031] Figure 2 For neighborhood attention graphs;

[0032] Figure 3 For neighborhood LSTM graphs;

[0033] Figure 4 This is a diagram of a collaborative feature fusion module centered on the vehicle. Detailed Implementation

[0034] The following attached diagrams further illustrate the multi-vehicle cooperative fusion perception method based on spatiotemporal context, such as... Figure 1 As shown, it includes the following steps:

[0035] S1. The simulation dataset OPV2V (73 scenarios, an average of 2.89 cooperative vehicles, and a total of 11,464 frames) and the real dataset V2V4Real (410 km of real vehicle road testing, 20,000 frames) are used as the two V2V cooperative perception benchmark datasets for training, validation, and testing.

[0036] S2. Metadata Sharing and Feature Extraction Module: The metadata (such as pose and extrinsic parameters) from the dataset obtained in step S1 is input into the metadata sharing and feature extraction module. A connected vehicle is identified as the autonomous vehicle to construct a cooperative graph, and other vehicles t within the communication range will provide information as collaborators. The autonomous vehicle broadcasts metadata, and after receiving the metadata, the collaborating vehicles project their original point clouds onto the autonomous vehicle's coordinate system. Similarly, each vehicle's historical frame is aligned with the autonomous vehicle's current frame. Then, the point clouds of the autonomous vehicles and each collaborating vehicle, aligned together, are encoded into bird's-eye view features by a point cloud encoder. The transformed point cloud of the k-th vehicle at time t... The extracted features are ,in This indicates a PointPillar point cloud encoder shared by all vehicles. t represents the timestamp H, W, and C represent height, width, and number of channels, respectively.

[0037] S3. Neighborhood Spatiotemporal Attention LSTM Module: This module retrieves the vehicle (as output from step S2) from the system. Feature representation and each cooperating vehicle (retrieved as) Feature representation The hidden states are fed into the corresponding neighborhood spatiotemporal attention LSTM modules, and the output is the hidden state refined with spatiotemporal information. By integrating "neighborhood attention" into the hidden state update of traditional LSTM, this module efficiently captures the dynamic features of target objects within a local spatiotemporal range. This enables faster and more accurate prediction and alignment of temporal features in multi-vehicle cooperative perception, effectively mitigating the impact of temporal asynchrony. The specific steps of the neighborhood spatiotemporal attention LSTM module are as follows:

[0038] S31. Input the hidden state of the previous time step (i.e., time step t-1). Features of the current time (i.e., time t) First, the components are spliced ​​along the channel dimension to obtain... Then, normalization is performed using LayerNorm, and the normalization result is input into the neighborhood attention module, such as... Figure 2 As shown, the top image represents conventional self-attention, while the bottom image represents neighborhood attention.

[0039] S32. Perform neighborhood attention calculation on the normalized output of S31. Compared to the widely used self-attention mechanism, the neighborhood attention mechanism is an attention pattern similar to convolution. In neighborhood attention, the focus is restricted to a local window centered on a specific query location, thereby extracting relevant keys and values ​​for calculation within that window. For example... Figure 2 As shown, the self-attention mechanism involves the interaction of global features, taking into account the influence of all pixels in the input feature map on the current pixel. Conversely, as... Figure 2 As shown, neighborhood attention limits the scope of attention operations to the neighborhood of each pixel. This approach simulates the "sliding window" effect in convolution operations, focusing only on the feature interactions of local regions.

[0040] This design not only significantly reduces computational overhead but also better reflects the characteristics of target object state changes in real-world traffic scenarios. In real-world environments, the motion state of a target object (such as position and velocity) typically changes more significantly within a local range, while changes over a global range are often sparse and difficult to capture. Therefore, by limiting the attention range, the model can more efficiently capture the local dynamic features of the target object, thereby improving the accuracy and real-time performance of perception.

[0041] Attention is computed only within a local window at each location, thereby extracting local spatiotemporal context features and obtaining neighborhood attention. .

[0042] S33. Although neighborhood attention reduces the computational cost of global attention, the introduction of the attention mechanism inevitably increases the computational burden. To further reduce computational cost and network latency, this paper merges the forget gate, input gate, and output gate from the original LSTM. For example... Figure 3 As shown, the neighborhood attention obtained in step S32 Apply the Sigmoid activation function element by element Obtain the gating weights This gating is used to control information retention and updates.

[0043] S34. Calculation Cellular state at any given moment : Obtaining the cell state at the previous moment ( Cellular state at any given moment ),current Neighborhood attention features at time step and gating weights Next, first of all and Element-wise multiplication is performed to retain only the historical information that is allowed to continue being passed through the gating mechanism. Then, for... First, perform tanh activation and map it to... Interval, and then with Element-wise multiplication is performed to extract the new features that need to be added. Finally, the two results are added together to obtain the updated cell state. This allows historical and current information to be integrated into a single tensor. The formula is as follows:

[0044]

[0045] Among them, the first item This indicates that the cell information from the previous time step is retained; the second item... This indicates that new information is added using the current features. The updated cell state is obtained. .

[0046] S35. Obtain the updated cell state. Next, activate it with tanh to obtain... The feature map within the interval is then used to calculate the activation result and the gating weights. Element-wise multiplication ensures that only information permitted by the gating system can continue to be transmitted, resulting in the hidden state refined with spatiotemporal information. This serves as a spatiotemporal characteristic. The formula is as follows:

[0047]

[0048] S4. Spatial Decoding and Communication Module: Hidden State Output in Step S3 After location embedding, the data is sent to the spatial decoding and communication module to filter out the vehicle (indexed as...). Unique characteristics Collaborative vehicles (indexed as) The unique regional characteristics of ) Characteristics of collaborative vehicles in shared spaces and the characteristics of vehicles in shared spaces The spatial decoding and communication module decodes the perception space of the collaborative vehicles into vehicle-specific regions, collaborative vehicle-specific regions, and shared regions. It calculates the importance confidence map for each region through position encoding and spatial attention, filters out high-value sparse features, and packages and transmits them. This allows for the exchange of only key perception information within limited bandwidth, significantly reducing communication overhead while ensuring the vehicle can acquire key spatial cues from different perspectives, laying the foundation for subsequent collaborative feature fusion. The specific steps of the spatial decoding and communication module are as follows:

[0049] S41. Input the spatiotemporal features of each vehicle calculated in step S3. Then the perceptual space is encoded by embedding a learnable location. Add to this spatiotemporal feature ( (As learnable parameters), this enhances the vehicle's sensitivity to spatial location, thereby obtaining features containing location information. .

[0050] S42. Input the feature containing location information obtained in step S41. , respectively in width and high Applying average pooling across dimensions encodes the width and height information into the channel dimension, thus obtaining spatial features. and .

[0051] S43. The spatial features obtained in step S42 and After aligning dimensions through broadcasting, the components are stitched together and then subjected to a nonlinear transformation. Processing is performed to obtain the intermediate vector. .

[0052] Then, spatial attention scores are calculated in the width and height dimensions, and a spatial confidence map is generated. The formula is as follows.

[0053]

[0054] in, and This represents a segmentation operation in the width and height dimensions. This represents the Sigmoid activation function. The spatial confidence plot is obtained. Then, the filtering function is used to filter the data based on the set threshold. This is used to filter important locations. If the confidence level of a location is greater than a threshold, it is marked as 1; otherwise, it is marked as 0. This yields the important spatial locations. Then, the corresponding key features are obtained. .

[0055] S44. Collaborative vehicles will occupy important spatial locations and corresponding important features The data is sent to the autonomous vehicle. Upon receiving the data, the autonomous vehicle calculates the characteristics of its own unique sensing area, the collaborative vehicle's unique sensing area, and the shared sensing area, respectively.

[0056] in This means removing the mask from the area covered by the collaborating vehicles and retaining only the mask for the area exclusively occupied by the vehicle itself, combined with the sparse position of the vehicle itself. and weighted features Obtain the unique and effective features of the vehicle The formula is as follows:

[0057]

[0058] in This indicates element-wise multiplication. This means removing the mask from the area covered by the autonomous vehicle and retaining only the mask for the area exclusively occupied by the collaborating vehicle, combined with the sparse mask of the collaborating vehicle. and weighted features Obtain the unique effective features of the collaborative vehicle The formula is as follows:

[0059]

[0060] The area shared by both autonomous and collaborative vehicles is divided into two parts. This represents the location that both vehicles can perceive, with shared features using the updated features of each vehicle. Features of the updated cooperative vehicle Weighting is applied to ensure that the perceived information about the shared area is preserved by both. The formula is as follows:

[0061]

[0062]

[0063] Ultimately, the unique characteristics of the vehicle were obtained. , unique regional characteristics of collaborative vehicles Characteristics of collaborative vehicles in shared spaces and the characteristics of vehicles in shared spaces This provides complete and hierarchical spatial information for subsequent fusion and detection modules.

[0064] S5. Vehicle-centric collaborative feature fusion module: Input the features of the common area output from step S4. and Alignment and fusion are performed through a spatial adaptive fusion mechanism to generate unified shared region features. Then, the unique sensing area characteristics of the vehicle will be used. Characteristics of the unique sensing area of ​​collaborative vehicles and unified shared sensory region features The input is fed into a vehicle-centric collaborative feature fusion module for robust feature fusion, ultimately yielding the fused features. The vehicle-centric collaborative feature fusion module maps and fuses perceived features from surrounding vehicles or infrastructure into the vehicle's coordinate system to improve the accuracy and robustness of environmental perception. The specific steps of the vehicle-centric collaborative feature fusion module are as follows:

[0065] S51. Input feature maps of the autonomous vehicle and cooperative vehicles in the shared perception area. , This involves aggregating the shared perception area features of autonomous vehicles and collaborative vehicles. First, the feature maps of autonomous vehicles and collaborative vehicles within the shared perception area are... and The features are concatenated to form a joint feature tensor containing information from both sources. Then, max pooling is performed on this concatenated tensor. and average pooling Max pooling is used to extract saliency information from the feature map (i.e., capture the most prominent activations), while average pooling is used to preserve the overall distribution of features. The results of these two pooling methods are concatenated along the channel direction to obtain a composite feature that simultaneously possesses strong local responses and a global average response. Finally, a 3×3 convolution kernel is applied to the concatenated tensor. By using sliding convolution operations to fuse and recombine this information in the spatial dimension, a unified common perceptual region fusion feature is output. The formula is as follows:

[0066]

[0067] S52. This method implements cross-neighborhood attention, unifying the perception regions of all viewpoints into a representation of the vehicle's viewpoint. For example... Figure 4 As shown, the spatiotemporal characteristics of the vehicle Each position in First, extract the query vector. , It is a learnable weight matrix; and its neighborhood matrix is ​​defined. Then based on the neighborhood matrix The corresponding neighborhood matrix in the cooperative space is calculated through coordinate alignment operations. and to , and neighborhood matrix Multiply by the corresponding learnable matrix respectively and This yields the corresponding three sets of key vectors. Sum value vector Then, the attention weights are calculated; next, the attention weights are normalized using the softmax function, and the value vector is then processed. By performing weighted summation, a unified representation of the characteristics of the vehicle-specific perception region from the vehicle's perspective is obtained. A unified representation of the characteristics of the cooperative vehicle's unique perception area from the driver's perspective. Unified representation of features of the unified shared perception area from the vehicle's perspective The formula is as follows:

[0068]

[0069] in, Indicates different sensory regions.

[0070] S53. Use an importance generator to calculate the enhanced spatial importance maps of different perception spatial regions. First, use the fused vehicle-specific perception region features obtained in step S52. The unique sensing area characteristics of the integrated collaborative vehicle and the unified shared sensory region features after fusion Each input is fed into the same importance generator The importance generator processes the features at each pixel location to generate a corresponding original weight distribution map. Then, a sigmoid activation function is applied to the generated weight map, mapping all values ​​to... Within the range, a spatial importance map of the vehicle's unique perception area is obtained. Map showing the spatial importance of the unique perception area of ​​collaborative vehicles And a unified shared perception area spatial importance map These three importance maps use numerical values ​​closer to 1 to indicate a greater contribution of a spatial location to the final perception result, and closer to 0 to indicate a weaker contribution, thus achieving a quantitative labeling of the contribution of different spatial perception regions.

[0071] S54. Three spatial importance maps obtained by stitching together S53. , and Through the softmax function Obtain standardized attention maps between different features The formula is as follows:

[0072]

[0073] in, This represents exponential operations. Similarly, the formula for calculating the vehicle-specific perception spatial attention map is: The formula for calculating the collaborative vehicle-specific perception space attention map is as follows: .

[0074] S55. Apply the spatial attention map obtained in step S54. , and Each with its corresponding feature map , and Element-wise multiplication along the spatial dimension yields three weighted feature maps. This step amplifies or suppresses the feature response at each location based on its importance in the perceptual space. Then, these three weighted feature maps are directly added along the channel and spatial dimensions to obtain the final fused feature. The purpose of this approach is to dynamically weight the information from three parts—public perception, vehicle-specific perception, and cooperative vehicle-specific perception—using an attention map. This allows for stronger retention of more important features at the same location, thereby generating a comprehensive feature map that takes into account the key information from all three perception spaces. The formula is as follows:

[0075]

[0076] in, This indicates element-wise multiplication of spatial dimensions.

[0077] S6. During the inference process, the final fused features output in step S5 are... Input to the regression head, output bounding box prediction information This includes the position, size, and yaw angle of the bounding box at each location.

[0078] Final fusion features Input is sent to the classification head, and classification prediction information is output. This represents the confidence value of each predefined box as either a target or a background.

[0079] S7. During training, this method uses smoothed L1 loss for regression and focal loss for classification. The total training loss is calculated by summing all positive and negative samples within a batch:

[0080]

[0081] The specific training details are as follows:

[0082] S71. The formula for smoothing L1 loss is as follows:

[0083]

[0084] in, It is the true value of the nth sample. This is the predicted value for the nth sample. The formula is as follows:

[0085]

[0086] in, This is a parameter that defines the smooth region and is used to control the range of smooth transitions.

[0087] The formula for focus classification loss is as follows:

[0088]

[0089] in Let be the predicted probability of the nth sample for its true label; Used to balance positive and negative samples Used to focus on difficult samples.

[0090] S72. Training begins with "single-vehicle warm-up": the collaborative feature fusion module is frozen, and the regression-classification head is trained on independent vehicle data for several epochs using only the aforementioned loss to stabilize it. The attention layer is then unfrozen for "collaborative fine-tuning," and communication packet loss, occlusion, and temporal jitter are randomly applied in each batch to improve robustness to dynamic topology and bandwidth fluctuations. The optimizer uses AdamW (initial learning rate...). Weight decay (cosine annealing), batch normalization and gradient clipping are used together for stable training; hyperparameters are .

[0091] Example:

[0092] The steps in this embodiment are the same as those in the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0093] To comprehensively evaluate model performance, this paper sets up two benchmark comparison groups: No Fusion (single-vehicle perception baseline without cooperative communication) and Late Fusion (traditional late-stage fusion scheme). Also included in the comparison are five advanced cooperative perception models: F-Cooper: 3D point cloud-based vehicle-to-everything (V2VNet) collaborative perception for autonomous driving; V2VNet: vehicle-to-vehicle communication for joint perception and prediction; V2X-ViT: visual Transformer-based V2V4 Real collaborative perception; Where2comm: collaborative perception with efficient communication via spatial confidence maps; and ERMVP: multi-vehicle perception with efficient communication and robust cooperation in complex environments. ERMVP is the baseline scheme in the V2V4Real dataset.

[0094] Table 1 shows the performance comparison results of the 3D collaborative detection method at different distances on the V2V4Real dataset. Experimental results show that on the V2V4Real dataset, the proposed spatiotemporal context-based multi-vehicle collaborative fusion perception method achieves an overall performance of 66.1% and 45.2% on AP@0.5 and AP@0.7, respectively. Overall is the average of all detected targets on AP@0.5 and AP@0.7. Compared with No Fusion, this method improves performance by 26.9% and 23.2% on AP@0.5 and AP@0.7, respectively; compared with Late Fusion, this method improves performance by 11.1% and 18.5%, respectively. Notably, compared with the state-of-the-art intermediate fusion scheme ERMVP, this method achieves improvements of 1.2% and 1.9% on AP@0.5 and AP@0.7, respectively.

[0095] Table 1. Performance comparison with baseline methods on the V2V4Real test set.

[0096]

Claims

1. A multi-vehicle collaborative fusion perception method based on spatiotemporal context, characterized in that, Includes the following steps: S1. Obtain large-scale open collaborative sensing simulation datasets and real datasets, and then mix them; S2. Input the metadata from the mixed dataset into the metadata sharing and feature extraction module to obtain the feature representation of each vehicle; S3. Input the feature representation of the autonomous vehicle and the feature representation of each cooperating vehicle into the corresponding neighborhood spatiotemporal LSTM module, and output two hidden states refined by spatiotemporal information. The specific implementation process is as follows: S31. Input the previous hidden state. and current moment features The data is spliced ​​along the channel dimension and then normalized. S32. Input the normalization result into the neighborhood attention module, calculate the attention only within the local window of each location, extract the local spatiotemporal context features, and obtain the neighborhood attention; S33. Apply the Sigmoid activation function element-wise to the neighborhood attention to obtain the gating weights. ; S34. Multiply the gating weights element-wise with the cell state from the previous time step; apply the neighborhood attention through the tanh activation function, and then multiply... Multiply each element by itself; finally, add the results of these two element-by-element multiplications to obtain the updated cell state. S35. Input the cell state, first perform tanh activation on it, then combine the activation result with the gating weights. Element-wise multiplication yields the hidden state after refining the spatiotemporal information. As a spatiotemporal characteristic; S4. After embedding the hidden state into the location, input it into the spatial decoding and communication module, and output the features of the autonomous vehicle, the features of the cooperative vehicle, the features of the cooperative vehicle in the shared space, and the features of the autonomous vehicle in the shared space. S5. Align and fuse the features obtained in S4 using a spatial adaptive fusion mechanism, and output the fused features; S6. During inference, the features are fused into the regression head and the classification head, and the bounding box prediction information and classification prediction information are output. During training, smooth L1 loss is used for regression and focal loss is used for classification.

2. The multi-vehicle cooperative fusion perception method based on spatiotemporal context according to claim 1, characterized in that, The specific implementation of step S2 is as follows: input the metadata in the mixed dataset into the metadata sharing and feature extraction module, the autonomous vehicle constructs a collaborative graph and broadcasts the metadata, and after receiving the metadata, the collaborative vehicle projects its point cloud data onto the autonomous vehicle's coordinate system and aligns it with the autonomous vehicle's current LiDAR point cloud data. The point cloud of each vehicle, after being aligned with the point cloud of each collaborating vehicle, is encoded into bird's-eye view features by a point cloud encoder, outputting the feature of each vehicle in the bird's-eye view. Time-time feature representation The feature dimension is ,in , and These represent the height, width, and number of channels, respectively. Indicates the index of the vehicle.

3. The multi-vehicle cooperative fusion perception method based on spatiotemporal context according to claim 2, characterized in that, The specific implementation process of step S4 is as follows: S41. Input the spatiotemporal characteristics of each vehicle. Encoding the perceptual space involves adding a learnable location embedding to the spatiotemporal features to obtain features containing location information. ; S42. Input features containing location information Average pooling is applied in both the width and height dimensions to obtain spatial features. and ; S43. Spatial features and After aligning dimensions through broadcasting, the vectors are concatenated and processed through nonlinear transformation to obtain intermediate vectors; then, spatial attention score vectors in the width and height dimensions are calculated, and a spatial confidence map is generated. After obtaining the spatial confidence map, important locations are filtered according to a set threshold using a filtering function. If the confidence level of a location is greater than the threshold, it is marked as 1; otherwise, it is marked as 0. Important spatial locations... Multiplying the features containing location information by a dot product yields the corresponding important features. ; S44. Collaborative vehicles will occupy important spatial locations and corresponding important features The data is sent to the vehicle; after receiving the data, the vehicle calculates the features of its sensing area. Features of the sensing area of ​​collaborative vehicles and features of shared sensory regions , Specifically: and Each with the corresponding and Perform element-wise multiplication to obtain and ; and respectively with Perform element-wise multiplication to obtain and , This indicates element-wise multiplication.

4. The multi-vehicle cooperative fusion perception method based on spatiotemporal context according to claim 3, characterized in that, The specific process for generating the spatial confidence map is as follows: the intermediate vector is divided into wide-dimensional and high-dimensional segments, and then subjected to nonlinear transformation and sigmoid activation function. The two results are multiplied together to obtain the spatial confidence map.

5. The multi-vehicle cooperative fusion perception method based on spatiotemporal context according to claim 4, characterized in that, The specific implementation process of step S5 is as follows: S51. Features of the shared perception area for autonomous vehicles and cooperative vehicles. and The splicing tensor is then subjected to max pooling and average pooling respectively. The results of these two pooling methods are then spliced ​​along the channel direction to obtain a composite feature that simultaneously has strong local response and global average response. Composite features fuse and recombine information in the spatial dimension through sliding convolution operations, outputting unified common perceptual region features. ; S52. For each location in the spatiotemporal characteristics of the vehicle. The query vector is extracted by multiplying spatiotemporal features by a weight matrix; the definition... Neighborhood matrix Then based on the neighborhood matrix The corresponding neighborhood matrix in the cooperative space is calculated through coordinate alignment operations. and to , and neighborhood matrix Multiply by the corresponding learnable matrix respectively and We obtain the corresponding three sets of key vectors and value vectors; then we calculate the attention weights and obtain the unified feature representation of the autonomous vehicle's perception area from the autonomous vehicle's perspective, the unified feature representation of the cooperative vehicle's perception area from the autonomous vehicle's perspective, and the unified feature representation of the unified shared perception area from the autonomous vehicle's perspective. S53. The three fused features are uniformly represented and input into the same importance generator. Convolution is performed on the features at each pixel location to generate the corresponding original weight distribution map. Then, the Sigmoid activation function is applied to the generated weight map to obtain the spatial importance map of the autonomous vehicle's perception area, the spatial importance map of the cooperative vehicle's perception area, and the unified shared perception area spatial importance map. S54. Stitch together the three spatial importance maps and obtain the standardized spatial attention map between different features through the softmax function; similarly, obtain the spatial attention map of the vehicle's perception and the spatial attention map of the cooperating vehicle's perception. S55. Connect the three spatial attention maps with their corresponding feature maps respectively. , and Multiplying each element in the spatial dimension yields three weighted feature maps; Then, these three weighted feature maps are directly added together in both the channel and spatial dimensions to obtain the final fused feature.

Images