A spatiotemporal error robust multi-agent collaborative perception method

Abstract

The application discloses a kind of multi-agent collaborative perception methods for space-time error robustness, it is related to robot perception technical field, effectively solve the information misalignment problem caused by time delay and positioning error.Design time enhancement method, by using time cross attention to different historical time enhancement feature, to realize the feature enhancement of current processing time, to effectively handle random timing interference.Spatial enhancement method is proposed, features are learned from multiple scales, global context information and local semantics are captured simultaneously, and the adverse effects of positioning errors are reduced.The method of the application has carried out experimental verification on virtual simulation dataset and measured dataset, and in the case of no interference and the existence of space-time multi-dimensional interference, the optimal detection effect and robustness are highlighted.

Description

Technical Field

[0001] This invention relates to the field of robot perception technology, specifically to a multi-agent cooperative perception method robust to spatiotemporal errors. Background Technology

[0002] As robotics technology matures, it is widely applied in military and civilian fields such as logistics and transportation, intelligent warfare, and wilderness rescue, providing convenience for people's lives and safety guarantees for special scenarios. Among these applications, information perception technology, by acquiring information from different sensors to detect various moving and stationary obstacles (vehicles, pedestrians, buildings, etc.) in the surrounding environment and collecting various information in the road (lane lines, traffic lights, drivable areas, etc.), is an important foundation for achieving downstream tasks such as autonomous positioning and navigation planning, and a crucial prerequisite for autonomous robot operation.

[0003] Furthermore, individual robots, due to the limited effectiveness of their sensors, suffer from limited sensing range and frequent random failures, failing to meet the demands of large-scale, high-communication, and highly stable applications. Multi-robot collaborative operation technology, by establishing communication groups, enables each robot to share and fully utilize information from other robots, reducing the impact of individual sensors on its own sensing performance and improving the stability of each agent's sensing capabilities. Therefore, collaborative sensing technology is a key technology for advancing the intelligence and systematization of mobile robots.

[0004] To achieve reliable collaborative perception, an ideal prerequisite is the precise localization of each agent within a unified coordinate system and the real-time transmission of perception information from multiple agents. However, in real-world scenarios, interference and noise are unavoidable, manifesting as temporal delays and spatial localization errors. This noise introduces challenges of temporal asynchrony and feature misalignment, significantly impacting detection performance. In the realm of autonomous driving, these errors can lead to collision risks and traffic congestion.

[0005] The existing SyncNet network can align different timestamps and eliminate the adverse effects of time asynchrony; in the spatial dimension, the Coalign network can correct spatial pose.

[0006] The above methods all perform error compensation separately in the time and space dimensions. Currently, there is no solution that can simultaneously achieve error compensation in both the time and space dimensions. Summary of the Invention

[0007] In view of this, the present invention provides a multi-agent collaborative perception method robust to spatiotemporal errors, which mines spatiotemporal correlation features to address the time delay (time) and positioning error (space) problems in the spatiotemporal dimension, mitigate the impact of feature misalignment, and ensure the detection performance and robustness of the algorithm in real-world scenarios.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0009] Using the sensory source data of each agent as input, a data transmission channel is established between multiple agents to achieve data sharing; a self-agent is set, and other agents project their respective point clouds onto their own coordinate system through a coordinate transfer matrix; feature extraction is performed on the point clouds of all agents to obtain the features at each time step.

[0010] The features extracted by different agents at various time points are compressed and fused to obtain collaborative features.

[0011] The collaborative features are iteratively enhanced in both the temporal and spatial dimensions to obtain spatiotemporally enhanced features.

[0012] The spatiotemporally enhanced features are decoded to obtain classification and regression outputs.

[0013] Furthermore, the features extracted by different agents at each time step are compressed and fused to obtain collaborative features, specifically:

[0014] First, calculate the similarity between the features of other agents and the features of the self agent.

[0015] Next, the calculated attention scores are normalized using the Softmax function to obtain the attention weights corresponding to each other agent feature.

[0016] Finally, a weighted summation is performed, multiplying each feature by the attention weight and adding the results to obtain the weighted feature vector corresponding to the current self-agent after the update, which is the collaborative feature.

[0017] Furthermore, the collaborative features are iteratively enhanced in both the temporal and spatial dimensions to obtain spatiotemporally enhanced features. The specific steps are as follows:

[0018] Step 31: Analyze the synergistic features after fusion. To enhance the time dimension, we first obtain the features of the current moment and the reference features of each historical moment. The weight λ of each cross-attention result is adaptively determined based on the correlation between the features of each historical moment and the features of the current moment. τ The updated features are then weighted and summed, and then passed through a feedforward network to obtain the final output. This is a feature that enhances the time dimension.

[0019] Step 32: Features enhanced based on the time dimension The design space pays attention to the algorithm for obtaining enhanced features of the current frame. And save to the historical feature set First, global and local feature information is captured through a multi-scale sliding window. Then, multi-head attention learning is performed on each branch. Finally, the updated features of each branch are fused together with attention in the channel dimension.

[0020] Furthermore, historical moment reference features Obtained through the following methods:

[0021] The features in the first four frames are not spatially enhanced and are directly used as historical moment reference features. Starting from the fifth frame, spatially enhanced historical moment reference features are introduced.

[0022] Furthermore, regarding the synergistic characteristics after fusion Enhancement of the time dimension includes the following steps:

[0023] Step (311): Using the fused collaborative features at time t As a reference input, and passed through the linear mapping matrix W q Calculation output: For the query deformable embedding.

[0024] The application consists of a lightweight subnetwork θ, composed of two non-linearly activated convolutional modules. offset (·), calculate adaptive offset to simulate the three-dimensional motion trend of the object.

[0025] Step (312): Considering the temporal correlation between different historical frames and the current frame, based on the sampling point p with displacement deviation, use bilinear interpolation to sample features from the feature map for each historical feature used in the preceding frame, and perform key / value projection to obtain deformable embeddings corresponding to keys and values. and

[0026]

[0027] Where τ = 1, ..., n; W k W v Let represent the mapping matrices for keys and values, respectively, where subscripts k and v represent the key and value, respectively, and φ is the bilinear interpolation function. This represents the historical moment reference feature of the preceding τ-th frame.

[0028] Step (313): Perform multi-head attention on the mapped query / key / value respectively, and introduce a relative position offset R to obtain the cross-attention features of each historical frame and the current frame features. Where τ∈{1,…,n}, d is the output dimension of each head, δ(*) is the softmax function, and the temporal influence of each historical frame feature on the current frame is used to generate adaptive attention weights λ through an attention mechanism. τ The features are represented, then weighted and summed. Each feature is multiplied by the attention weight, and the results are summed to obtain the current frame features after historical frame feature enhancement.

[0029] Step (314): Finally, following the basic backbone network of the attention mechanism, the updated features are processed through the feedforward network and the final temporal augmentation features are output. Where AN(·) and FFN(·) represent the summation batch normalization network and the feedforward neural network, respectively, and θ dr and θ dr2 These represent the information loss rate, respectively.

[0030] Furthermore, the temporal impact of each historical frame feature on the current frame is used to generate adaptive attention weights λ through an attention mechanism. τ The representation is as follows:

[0031] First, calculate the similarity Siml between each cross-attention feature and the feature at the current time step. τ :

[0032] ⊙ is the dot product function;

[0033] Next, the calculated attention scores are normalized using the Softmax function to obtain the weight λ corresponding to each cross-attention feature. τ =Softmax(Siml) τ ).

[0034] Furthermore, step 32 specifically includes the following steps:

[0035] Step (321) Based on enhanced temporal feature input B, L, H, W, and C represent batch size, number of agents, feature height, feature width, and number of channels, respectively. R B×L×H×W×C Let m be a real matrix of dimension B×L×H×W×C, with m distinct branches, each based on a different scale [s1, s2, ..., s]. m Transform it into window features of different shapes, with the scale of the j-th branch being s. j=j×s1, where each window feature of the j-th branch is in h is the number of heads required by the multi-head attention mechanism;

[0036] Step (322) for each window feature M of each branch u Multi-head attention learning is performed using a multi-head attention function, and position embedding P is introduced to improve the accuracy of position awareness. The features of each window in each branch are integrated to obtain the final output. The specific implementation is as follows:

[0037]

[0038] Y j =[Y1,Y2,…,Y h ]

[0039] Among them, multi-head attention function W q W k W v These represent the mapping matrices for queries, keys, and values, respectively, with D representing the query / key dimension.

[0040] Step (323) is based on the output feature Y of each branch. j The attention weight ω corresponding to each channel of the feature is obtained through global average pooling. o ∈Ω=AP(Y j ), o=[1,2,…,q]. where AP(·) represents the global average pooling function, and q represents each active feature Y. j The number of channels;

[0041] Based on the generated weights, each active feature is weighted and summed according to the channel dimension, resulting in the updated features. Finally, the updated features of each branch are concatenated to obtain the final output of that part. Here, Convat(·) represents the concatenation function.

[0042] Beneficial effects:

[0043] 1. This invention proposes a multi-agent cooperative perception method robust to spatiotemporal errors, effectively solving the information misalignment problem caused by time delays and localization errors. A temporal augmentation method is designed, which uses temporal cross-attention to enhance features at different historical moments to enhance features at the current processing moment, effectively handling random temporal interference. A spatial augmentation method is proposed, which learns features from multiple scales to simultaneously capture global contextual information and local semantics, reducing the adverse effects of localization errors.

[0044] 2. This invention provides a multi-agent cooperative perception method robust to spatiotemporal errors, constructing a novel framework for interactive updating of spatiotemporal features. This effectively addresses the information misalignment problem caused by time delays and positioning errors. Modules targeting these two types of errors iteratively reinforce each other, jointly promoting performance improvement.

[0045] 3. This invention provides a multi-agent cooperative perception method robust to spatiotemporal errors. The temporal enhancement module utilizes historical features that have undergone spatial enhancement to iteratively guide the features at the current moment. By applying temporal cross-attention to features from different historical moments, feature enhancement for the current processing moment is achieved, effectively handling random interference.

[0046] 4. This invention provides a multi-agent cooperative perception method robust to spatiotemporal errors, proposing a spatial enhancement module to capture local semantics at multiple scales and enhance global contextual learning to address a range of localization errors. Specifically, a large-scale window captures global contextual information, reducing localization errors; a small-scale window captures subtle local features, amplifying local semantics.

[0047] 5. This invention provides a multi-agent cooperative perception method robust to spatiotemporal errors, which not only outperforms previous methods in the absence of interference, but also maintains optimal detection performance even under conditions of multi-dimensional spatiotemporal interference. We conducted experimental verification on virtual simulation datasets (e.g., OPV2V, V2XSet) and real vehicle datasets (DAIR-V2X). Furthermore, related ablation experiments also demonstrated the advantages of each module. Attached Figure Description

[0048] Figure 1 This is a diagram illustrating the overall framework of the spatiotemporal error-robust multi-agent cooperative perception method of the present invention.

[0049] Figure 2 This is a schematic diagram illustrating the temporal dimension feature enhancement learning in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] This invention provides a multi-agent cooperative sensing method robust to spatiotemporal errors, the process framework of which is as follows: Figure 1 As shown, it includes:

[0052] Step 1: Using the sensory source data of each agent as input, establish a data transmission channel between multiple agents to achieve data sharing. Figure 1The metadata sharing section involves setting up a self-agent, where one agent from a multi-agent group is selected as the self-agent. Other agents project their point clouds onto the self-agent coordinate system using a coordinate transition matrix (obtained from each agent's pose). The self-coordinate system refers to the point cloud coordinate system of the self-intelligent agent. Then, feature extraction is performed on all point clouds to obtain the features at time t. correspond Figure 1 Feature extraction section.

[0053] Step 2: Extract features from different agents at time t Data compression and intermediate fusion are performed to obtain collaborative features. ψ fuse This is an attention fusion method based on graph neural networks, where N is the total number of agents, corresponding to... Figure 1 The compression and fusion part.

[0054] Specifically, the similarity between the features of other agents and the features of the self agent is first calculated. ⊙ is the dot product function. Next, the calculated attention score is normalized using the Softmax function to obtain the attention weight β corresponding to each other agent feature. j =Softmax(Siml) j ), Finally, a weighted summation is performed, multiplying each feature by the attention weight and summing the results to obtain the weighted feature vector corresponding to the updated self-agent.

[0055] Step 3: Analyze the synergistic features after fusion. Iterative enhancements are performed separately in the temporal and spatial dimensions. Features enhanced temporally... Spatial global learning is performed at each time step, and the features at different time steps are spatially augmented. Establish a set of historical features K represents the total number of historical frames. Specifically, when performing spatiotemporal dimension compensation enhancement at a particular moment, the n frames preceding that moment are selected from the historical feature set as the historical sequence input. The specific steps include:

[0056] Step 31: The features of the first 4 frames are not spatially enhanced and are directly used as historical reference features. Starting from the fifth frame, the spatially enhanced historical reference features are introduced to enhance the temporal dimension of the fused collaborative features.

[0057] Synergistic features after fusion Enhancement of the time dimension, corresponding to Figure 1 The time augmentation method based on DAT in the text first obtains the features of the current time step and the reference features of each historical time step. The cross-attention features are adaptively determined by considering the correlation between features from different historical moments and features from the current moment, with the weight λ for each cross-attention feature being calculated. τ The updated features are then weighted and summed. Next, the updated features are passed through a feedforward network to obtain the final output. The specific process of performing time-dimensional feature enhancement learning is as follows: Figure 2 As shown, it specifically includes:

[0058] Step (311): Use the collaborative features at time t as reference input. And pass it through the linear mapping matrix W q Calculation output: For the deformable embedding of the query, a lightweight subnetwork θ consisting of two non-linearly activated convolutional modules is applied. offset (·), calculate adaptive offset to simulate the three-dimensional motion trend of the object. W q The training parameters are learned through backpropagation during training, enabling the model to better capture the relationships and features in the input sequence. In this embodiment, W... k W v similar.

[0059] Step (312): Considering the temporal correlation between different historical frames and the current frame, based on the sampling point p with displacement deviation, use bilinear interpolation to sample features from the feature map for each historical feature used in the preceding frame, and perform key / value projection to obtain deformable embeddings corresponding to keys and values. and Where τ=1,…,nW k W v Let represent the mapping matrices for keys and values, respectively, where subscripts k and v represent the key and value, respectively, and φ is the bilinear interpolation function. This represents the historical moment reference feature of the preceding τ-th frame.

[0060] Step (313) is similar to existing methods. Multi-head attention is performed on the mapped query / key / value respectively, and a relative position offset R is introduced to obtain the cross-attention features of each historical frame and the current frame features. Where τ∈{1,…,n}, d is the output dimension of each head, which can be set to 4, and δ(*) is the Softmax function, and The temporal impact of each historical frame feature on the current frame is used to generate an adaptive attention weight λ through an attention mechanism. τ To express.

[0061] Specifically, the adaptive attention weight λ is calculated as follows: τ :

[0062] First, calculate the similarity between each cross-attention feature and the feature at the current time step. ⊙ is the dot product function. Next, the calculated attention scores are normalized using the Softmax function to obtain the weight λ corresponding to each cross-attention feature. τ =Softmax(Siml) τ Finally, a weighted sum is performed, multiplying each feature by its attention weight and adding the results to obtain the current frame's enhanced features after historical frame feature enhancement.

[0063] Step (314): Finally, following the basic backbone network of the attention mechanism, the updated features are processed through the feedforward network and the final temporal augmentation features are output. Where AN(·) and FFN(·) represent the summation batch normalization network and the feedforward neural network, respectively, and θ dr and θ dr2 θ represents the information loss rate, respectively, in this embodiment of the invention. dr and θ dr2 It can be set to 0.3.

[0064] Step 32: Output of the enhancement module based on a specific frame's temporal dimension The design space pays attention to the algorithm for obtaining enhanced features of the current frame. And save them to a historical feature set. First, a multi-scale sliding window is used to capture global and local feature information. Then, multi-head attention learning is applied to each branch. Finally, the updated features of each branch are fused with attention along the channel dimension. Figure 1 The spatial pyramid method section in [the text]. Specific steps include:

[0065] Step (321) Based on enhanced temporal feature input B, L, H, W, and C represent batch size, number of agents, feature height, feature width, and number of channels, respectively. R B×L×H×W×C Let m be a real matrix of dimension B×L×H×W×C, with m distinct branches, each based on a different scale [s1, s2, ..., s]. m This is converted into window features of different shapes. In this embodiment, m is set to 3 and s1 is set to 4. The scale of the j-th branch is s.j =j×s1, where each window feature of the j-th branch is in h represents the number of heads required by the multi-head attention mechanism. In this embodiment, the number of heads h corresponding to each branch is set to 16, 8, and 4, respectively. The captured window features contain rich local semantics.

[0066] Step (322) for each window feature M of each branch u Multi-head attention learning is performed using a multi-head attention function, and a position embedding P is introduced to improve the accuracy of position awareness. Features from each window in each branch are integrated to obtain the final output. The specific implementation is as follows:

[0067]

[0068] Y j =[Y1,Y2,…,Y h ]

[0069] Among them, multi-head attention function W q W k W v These represent the mapping matrices for queries, keys, and values, respectively. D represents the query / key dimension.

[0070] Step (323) is based on the output feature Y of each branch. j The attention weight ω corresponding to each channel of the feature is obtained through global average pooling. o ∈Ω=AP(Y j ), o=[1,2,…,q]. where AP(·) represents the global average pooling function, and q represents each active feature Y. j The number of channels. Based on the generated weights, each active feature is weighted and summed according to the channel dimension, and the updated feature is... Finally, the updated features of each branch are concatenated to obtain the final output of that part. Here, Concat(·) represents the concatenation function.

[0071] Step (4) collects the spatial augmentation features from all time points to form a historical feature set. This set serves as the historical time input for the temporal augmentation algorithm, compensating for the damaged features of the current time. This iterative, spatiotemporally nested augmentation ensures that the reference historical features at each time step encompass almost all historical features. Furthermore, the effectiveness of each historical feature on the current feature is inversely proportional to its temporal distance.

[0072] Step 4, Decoder. This step involves processing the spatiotemporally enhanced features at each time step. Decode the data to obtain classification and regression outputs.

[0073]

[0074] In this context, the classification output is the confidence score of the object or background used as each anchor box. The regression output consists of bounding boxes, which can be represented as (x, y, z, w, l, h, α), referring to the predefined position, size, and angle of the bounding box, respectively.

[0075] Example 1

[0076] Table 1 shows the performance of each method on OPV2V, V2XSet, and DAIR-V2X.

[0077]

[0078] 1. Detection performance on the dataset

[0079] Datasets: To demonstrate the performance of our algorithm in 3D object detection across different domains, experiments were conducted on the virtual simulation datasets OPV2V and V2XSet, as well as the real vehicle dataset DAIR-V2X.

[0080] OPV2V is a vehicle-to-vehicle cooperative perception dataset jointly simulated by CARLA and OpenCDA, with an average of approximately three connected vehicles per frame. Specifically, the training set and validation set consist of 6765 and 1980 frames, respectively. The test set comprises 2170 frames of DefaultTown and 550 frames of Culver City segmentation. The entire dataset contains 11464 frames of 3D point clouds and 232913 annotated 3D bounding boxes.

[0081] V2XSet is a large-scale V2X perception simulation dataset, with 2 to 5 connected vehicles per frame. It consists of 73 representative scenarios, each lasting 25 seconds. The entire dataset contains 11,447 frames, with 58% / 18% / 24% divided into training / validation / test sets, respectively.

[0082] DAIR-V2X is the only publicly available real-world collaborative perception dataset. It includes over nine thousand collaborative frames, each consisting of a vehicle and an infrastructure unit with 3D annotations. Augmentation is achieved by relabeling all objects, enabling a 360-degree detection range.

[0083] Comparison Methods: This method presents a multi-agent cooperative perception framework. To highlight the advantages of cooperative perception, it is compared with non-cooperative fusion methods. Since this method is an intermediate fusion cooperative approach, it is compared with state-of-the-art intermediate fusion methods Where2comm and CoBEVT, as well as some representative methods such as F-Cooper, When2com, V2VNet, V2X-ViT, and AttFuse. Furthermore, it is also compared with later fusion methods.

[0084] Evaluation Method: The performance of each method was evaluated using the following commonly used evaluation metrics in object detection. AP@0.5 / 0.7: Average Precision (AP) at an Intersection over Union (IoU) threshold of 0.5 / 0.7. The IoU is a metric used to measure the degree of overlap between two bounding boxes, with a value between 0 and 1, where 0 indicates no overlap and 1 indicates complete overlap.

[0085] Comparison Results Analysis: The comparison results are shown in Table 1. The results demonstrate that our proposed method exhibits superior performance on all three datasets, achieving high accuracy in target detection across different intersection-over-union (IoU) benchmarks. Specifically, our method achieves accuracies of 0.765, 0.760, and 0.468 on the three datasets with AP@0.7, representing improvements of at least 6.8%, 5.7%, and 6.7% respectively compared to other methods, demonstrating our superiority.

[0086] Example 2

[0087] Table 2 shows the robustness test performance of each method on V2XSet.

[0088]

[0089] Comparison Method: This method is a robust architecture against noise interference in both time and space dimensions. To highlight the robust performance of this method, its performance is tested under different noise environments and compared with the latest existing networks AttFuse, Where2comm, and CoBEVT under the same experimental conditions.

[0090] Evaluation method: The detection performance of each method is evaluated using the evaluation index commonly used in target detection. AP@0.5 / 0.7: Average Precision (AP) at an Intersection over Union (IoU) threshold of 0.5 / 0.7. Among them, the Intersection over Union is a metric used to measure the degree of overlap between two bounding boxes, and the value ranges from 0 to 1, where 0 indicates no overlap and 1 indicates complete overlap. In addition, the noise settings are as follows: (1) The time delay is 200ms, (2) The position and attitude errors are simulated by Gaussian noise, where the standard deviation of position noise is set to 0.2m and the standard deviation of attitude noise is set to 0.2°. (3) The noise containing two or more types is superimposed according to the noise settings in (1) and (2) above.

[0091] Comparison Results Analysis: The comparison results are shown in Table 2. The results demonstrate that our proposed method exhibits excellent robustness on the V2XSet dataset, maintaining relatively high detection accuracy across various spatiotemporal error configurations. Specifically, time delay has a greater impact on detection performance compared to spatial errors. All reference methods show a decrease in accuracy under noise interference, while our proposed method consistently maintains the best detection performance compared to the comparison methods. Overall, due to the addition of historical features, our temporal augmentation method captures more dynamic trend changes and compensates for time biases when time delays occur. Furthermore, the spatial augmentation method performs global-local context learning on the temporally augmented features, mitigating error accumulation and reducing the impact of potential localization errors. Therefore, our proposed method effectively addresses spatiotemporal noise in real-world scenarios, ensuring stable and relatively accurate detection results, demonstrating excellent precision and robustness.

[0092] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-agent cooperative sensing method robust to spatiotemporal errors, characterized in that, The steps include the following: Using the sensory source data of each agent as input, a data transmission channel is established between multiple agents to achieve data sharing; a self-agent is set, and other agents project their respective point clouds onto their own coordinate system through a coordinate transfer matrix; feature extraction is performed on the point clouds of all agents to obtain the features at each time step; The collaborative features are obtained by data compression and intermediate fusion of the features extracted by different agents at each time step. Specifically, the similarity between the features of other agents and the features of the self agent is calculated first. Then, the Softmax function is applied to normalize the calculated attention scores to obtain the attention weights corresponding to each feature of other agents. Finally, a weighted summation is performed, multiplying each feature by the attention weight and adding the results to obtain the weighted feature vector corresponding to the current self-agent after the update, which is the collaborative feature; The collaborative features are iteratively enhanced in both the temporal and spatial dimensions to obtain spatiotemporally enhanced features, specifically through the following steps: Step 31: Analyze the synergistic features after fusion. To enhance the time dimension, we first obtain the features of the current moment and the reference features of each historical moment. The weight of each cross-attention result is adaptively determined based on the correlation between the features of each historical moment and the features of the current moment. The updated features are then weighted and summed, and then passed through a feedforward network to obtain the final output. This refers to features enhanced in the time dimension; The synergistic features after fusion Enhancement of the time dimension includes the following steps: Step (311): Using the fused collaborative features at time t... As a reference input, and passed through a linear mapping matrix Calculation output: , For query deformable embeddings; The application uses a lightweight subnetwork consisting of two non-linearly activated convolutional modules. Calculate adaptive offsets to simulate the three-dimensional motion trend of objects. ; Step (312): Considering the temporal correlation between different historical frames and the current frame, based on the sampling point p with displacement deviation, use bilinear interpolation to sample features from the feature map for each historical feature used in the preceding frame, and perform key / value projection to obtain deformable embeddings corresponding to keys and values. and : ； in These represent mapping matrices for keys and values, respectively, with subscripts k and v indicating the key and value, respectively. It is a bilinear interpolation function. This represents the historical moment reference feature of the preceding τ-th frame; Step (313): Perform multi-head attention on the mapped query / key / value respectively, and introduce a relative position offset R to obtain the cross-attention features of each historical frame and the current frame features. ,in ., where d is the output dimension of each head. It is a softmax function. The temporal impact of each historical frame feature on the current frame is used to generate adaptive attention weights through an attention mechanism. The features are represented, then weighted and summed. Each feature is multiplied by the attention weight, and the results are summed to obtain the current frame features after historical frame feature enhancement. ; Step (314): Finally, following the basic backbone network of the attention mechanism, the updated features are processed through the feedforward network and the final temporally enhanced features are output. ,in and FFN These represent the summation batch normalization network and the feedforward neural network, respectively. and These represent the information loss rate, respectively. Step 32: Features enhanced based on the time dimension The design spatial attention algorithm is used to obtain the enhanced features of the current frame. And will be saved to the historical feature set. First, global and local feature information is captured through a multi-scale sliding window. Then, multi-head attention learning is performed on each branch. Finally, the updated features of each branch are fused with attention in the channel dimension.

2. The multi-agent cooperative sensing method robust to spatiotemporal errors as described in claim 1, characterized in that, The historical moment reference features It can be obtained in the following way: The features in the first four frames are not spatially enhanced and are directly used as historical moment reference features. Starting from the fifth frame, spatially enhanced historical moment reference features are introduced.

3. The multi-agent cooperative perception method robust to spatiotemporal errors as described in claim 2, characterized in that, The temporal impact of each historical frame feature on the current frame is used to generate adaptive attention weights through an attention mechanism. The representation is as follows: First, calculate the similarity between each cross-attention feature and the feature at the current time step. : , It is a dot product function; Next, the calculated attention scores are normalized using the Softmax function to obtain the weights corresponding to each cross-attention feature. .

4. The multi-agent cooperative sensing method robust to spatiotemporal errors as described in claim 3, characterized in that, Step 32 specifically includes the following steps: Step (321) Based on enhanced temporal feature input , These represent batch size, number of agents, feature height, feature width, and number of channels, respectively. The dimension is Given a real matrix, set up m different branches, each branch based on a different scale. This is transformed into window features of different shapes, with the scale of the j-th branch being... Each window feature of the j-th branch is .in h is the number of heads required by the multi-head attention mechanism; Step (322) involves each window feature of each branch. Multi-head attention learning is performed using a multi-head attention function, and position embedding P is introduced to improve the accuracy of position awareness. The features of each window in each branch are integrated to obtain the final output. The specific implementation is as follows: Among them, multi-head attention function , These represent the mapping matrices for queries, keys, and values, respectively, with D representing the query / key dimension. Step (323) is based on the output features of each branch. Attention weights for each channel corresponding to the feature are obtained through global average pooling. . in, Let q represent the global average pooling function, and q represent each active feature. The number of channels; Based on the generated weights, each active feature is weighted and summed according to the channel dimension, resulting in the updated features. Finally, the updated features of each branch are concatenated to obtain the final output of that part. . in, This represents the concatenation function.

Images