Asynchronous point cloud fusion method for air-ground cooperative sensing system

Abstract

The invention particularly relates to an asynchronous point cloud fusion method for an air-ground cooperative sensing system, and the method comprises the steps: collecting point cloud data corresponding to a current environment through a current intelligent agent, and carrying out the coding processing of the point cloud data, so as to obtain a real-time feature tensor; receiving an asynchronous feature tensor corresponding to k frames of historical point clouds sent by the cooperative agent; wherein the cooperative intelligent agent and the current intelligent agent belong to a ground-air cooperative system; performing feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor by using a time delay compensation network to obtain a synchronous feature tensor; performing multi-scale feature weighted fusion on the synchronous feature tensors corresponding to the plurality of cooperative agents to obtain a fused feature tensor; and performing decoding processing on the fusion feature tensor to display fusion perception data corresponding to the fusion feature tensor. According to the method, the real-time performance, the accuracy and the safety of a sensing system can be improved, and the integrity and the reliability of environment sensing are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an asynchronous point cloud fusion method for an air-ground collaborative perception system. Background Art

[0002] Among related technologies, collaborative sensing is a method that utilizes collaboration between multiple agents to perceive the environment. Under ideal communication conditions, collaborative sensing has demonstrated great potential, significantly improving perception capabilities compared to single-agent sensing. However, in practical applications, communication systems inevitably face latency issues, which can lead to potential performance degradation and pose high-risk issues in safety-critical applications. The rapid development of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) has injected unprecedented vitality into numerous application areas. Among them, air-ground collaborative sensing technology, with its unique advantages, achieves comprehensive capture of environmental information through close collaboration between UAVs and ground sensors, significantly improving perception performance and decision-making accuracy. For example, in smart city scenarios, the combination of UAVs and UGVs forms an air-ground collaborative sensing system, providing more comprehensive and efficient support for smart city perception. However, in practical applications, due to data acquisition delays between UAVs and ground sensors, as well as limited communication resources, it is often difficult to synchronize data captured by different sensors. This latency issue severely impairs collaborative sensing systems, resulting in performance even worse than single-agent sensing. Therefore, effectively compensating for this latency and ensuring the quality of data fusion have become pressing challenges. Furthermore, in complex environments, such as urban canyons or areas with poor signal quality, the GPS positioning capabilities of drones and autonomous vehicles may fail, further limiting storage and computing resources. This undoubtedly poses a greater challenge to air-ground collaborative perception. Efficiently fusion point clouds in resource-constrained environments to ensure the accuracy and completeness of information is also crucial.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0004] The present invention provides an asynchronous point cloud fusion method and device for an air-ground collaborative perception system, a storage medium, a computer program product, and an electronic device, which improve the perception accuracy under data asynchrony conditions and can overcome the defects in the existing technology to a certain extent.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to a first aspect of the present invention, an asynchronous point cloud fusion method for an air-ground collaborative perception system is provided, the method comprising:

[0007] The current agent collects point cloud data corresponding to the current environment and encodes the point cloud data to obtain real-time feature tensors; and receives asynchronous feature tensors corresponding to k frames of historical point clouds sent by the collaborative agent; wherein the collaborative agent and the current agent belong to the ground-air collaborative system;

[0008] Performing feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor using a time delay compensation network to obtain a synchronized feature tensor;

[0009] Perform weighted fusion of multi-scale features on the synchronized feature tensors corresponding to several collaborative agents to obtain a fused feature tensor;

[0010] The fused feature tensor is decoded to display the fused perception data corresponding to the fused feature tensor.

[0011] In some exemplary embodiments, performing feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor using a delay compensation network to obtain a synchronized feature tensor includes:

[0012] Performing feature estimation on the current feature tensor and the asynchronous feature tensor using a feature estimation branch of a time delay compensation network to obtain collaborative feature estimation data based on multi-scale global features corresponding to the current feature tensor based on the delay time; and

[0013] Performing attention estimation on the current feature tensor and the asynchronous feature tensor using the attention estimation branch of the delay compensation network to obtain collaborative attention weight estimation data corresponding to the current feature tensor based on the delay time; wherein the feature estimation branch and the attention estimation branch are constructed based on the P-LSTM network;

[0014] A time modulation network is used to respectively estimate the confidence of features and attention weights of the collaborative feature estimation data and the asynchronous feature tensor, and the estimation results are combined to obtain the synchronous feature tensor.

[0015] In some exemplary embodiments, using a temporal modulation network to perform confidence estimation of features and attention weights on the collaborative feature estimation data and the asynchronous feature tensor, and combining the estimation results to obtain the synchronous feature tensor, includes:

[0016] Using a time modulation network, the collaborative feature estimation data, the asynchronous feature tensor, and the delay tensor are spliced ​​together to obtain spliced ​​feature data; and feature confidence estimation is performed on the spliced ​​feature data to obtain collaborative feature confidence estimation data; and

[0017] Performing splicing processing on the collaborative attention weight estimation data, the attention weight corresponding to the asynchronous feature tensor, and the delay tensor to obtain spliced ​​attention weight data; and performing attention confidence estimation on the spliced ​​attention weight data to obtain collaborative attention weight estimation data;

[0018] Based on the collaborative attention weight estimation data, the time domain attention weight is allocated to the collaborative feature confidence estimation data to obtain the synchronized feature tensor after feature fusion.

[0019] In some exemplary embodiments, performing weighted fusion of multi-scale features on synchronized feature tensors corresponding to a plurality of collaborative agents to obtain a fused feature tensor includes:

[0020] Perform convolution processing on each synchronized feature tensor to obtain feature data after dimensionality reduction;

[0021] Perform Affinity operation on the reduced feature data in the spatial dimension to generate the corresponding attention weight map;

[0022] The synchronized feature tensors are weightedly fused based on the attention weight map to obtain a fused feature tensor.

[0023] In some exemplary embodiments, the method further comprises:

[0024] The optimization goal of feature fusion based on cross attention is defined as: maximizing the difference between the system perception accuracy after fusion optimization and the system accuracy before optimization.

[0025] max i mize(A after -A before )

[0026] Among them, A before To optimize the accuracy of feature fusion, A after The perception accuracy of the optimized cross-attention-based perception system;

[0027] According to the loss configuration corresponding to the system accuracy improvement constraint:

[0028] (A after -A before )<=ΔA max

[0029] Where ΔA max The maximum acceptable loss in system accuracy.

[0030] In some exemplary embodiments, the real-time feature tensor and the asynchronous feature tensor are respectively configured with a timestamp parameter.

[0031] According to a second aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the asynchronous point cloud fusion method for the air-ground collaborative perception system is implemented.

[0032] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, the asynchronous point cloud fusion method for the air-ground collaborative perception system is implemented.

[0033] According to a fourth aspect of the present invention, there is provided an electronic device, comprising:

[0034] processor; and

[0035] a memory for storing executable instructions of the processor;

[0036] Wherein, the processor is configured to implement the above-mentioned asynchronous point cloud fusion method for the air-ground collaborative perception system by executing the executable instructions.

[0037] The asynchronous point cloud fusion method for air-ground collaborative perception systems provided by the embodiments of the present invention performs feature-level time synchronization processing on the real-time feature tensors corresponding to the currently collected point cloud data and the received asynchronous features to obtain synchronized feature tensors, thereby synchronizing the asynchronous features and achieving delay compensation based on collaborative estimation of feature attention, effectively solving the delay problem caused by different air-ground sensors. By performing weighted fusion of multi-scale features on the synchronized feature tensors corresponding to multiple collaborative intelligent agents to obtain a fused feature tensor, feature fusion based on cross-attention is achieved. In a resource-constrained environment, complete, accurate and reliable point cloud data can be fused without relying on GPS signals, significantly improving the collaborative operation capabilities of drones and unmanned vehicles, improving the real-time, accuracy and safety of the perception system, and enhancing the integrity and reliability of environmental perception, thereby meeting the needs in complex scenarios.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0040] Figure 1A schematic diagram schematically illustrates an asynchronous point cloud fusion method for an air-ground collaborative perception system according to an exemplary embodiment of the present invention;

[0041] Figure 2 A schematic diagram schematically illustrates the working principle of the pyramid LSTM convolution of each branch in a feature-attention collaborative estimation module according to an exemplary embodiment of the present invention;

[0042] Figure 3 A schematic diagram schematically illustrates a feature-attention collaborative estimation mechanism according to an exemplary embodiment of the present invention;

[0043] Figure 4 A schematic diagram schematically illustrating the principle of a time modulation method according to an exemplary embodiment of the present invention;

[0044] Figure 5 A schematic diagram schematically illustrates the process of an asynchronous point cloud fusion method for an air-ground collaborative perception system according to an exemplary embodiment of the present invention;

[0045] Figure 6 The figure schematically shows the composition of an electronic device in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0047] In addition, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] In related technologies, in air-ground collaborative perception systems, actual communication systems can cause data asynchrony due to factors such as sensor data acquisition delays, data transmission delays, data processing delays, and data synchronization delays. This asynchrony can lead to inaccurate data fusion and affect the performance of downstream perception tasks, even resulting in performance worse than that of single-agent perception. In collaborative perception, current collaborative perception technologies are currently limited by storage and computing resources in extreme resource-constrained environments. Therefore, how to efficiently integrate features has become a key issue in air-ground collaborative perception in this context. Effectively integrating information from multiple sensors can improve the accuracy and completeness of environmental perception.

[0049] Currently, air-ground collaborative perception models have been widely applied and studied in practical applications. Previous collaborative perception methods have achieved remarkable success in various perception tasks, such as 2D / 3D object detection and semantic segmentation. These methods have been applied across diverse domains and scenarios. Some studies have focused on semantic segmentation for drones, achieving fine-grained segmentation and understanding of ground scenes through collaborative perception. These methods leverage the visual sensors and communication capabilities of drones to achieve high-precision perception and segmentation of ground objects. Furthermore, some studies have explored the task of 3D object detection for autonomous driving assisted by vehicle-to-vehicle communication. These methods leverage inter-vehicle communication to share and integrate perception information, thereby improving the autonomous driving system's environmental awareness and safety. To achieve collaborative perception, recent research has provided high-quality datasets, which provide a foundation for the development and evaluation of collaborative perception algorithms. Furthermore, researchers have proposed several effective collaborative methods for sharing, fusing, and making decisions based on perception data. These methods cover aspects such as data transmission, feature fusion, and model integration, aiming to improve the performance and efficiency of collaborative perception systems.

[0050] Collaborative sensing is a method that utilizes the collaboration of multiple agents to perceive the environment. Under ideal communication conditions, collaborative sensing has demonstrated great potential, significantly improving perception capabilities compared to single-agent sensing. However, in practical applications, communication systems inevitably face latency issues, which can lead to potential performance degradation and pose high-risk issues in safety-critical applications. To address the challenges posed by communication latency, researchers have proposed a number of methods and technologies. For example, one common approach is to reduce communication latency by optimizing communication protocols and network architecture.

[0051] However, most of these studies focus on collaborative strategy learning in ideal situations, without considering the communication delay and incomplete communication of collaborative perception systems in reality.

[0052] In view of the shortcomings and deficiencies of the existing technology, this example embodiment provides an asynchronous point cloud fusion method for an air-ground collaborative perception system. Figure 1 As shown, the following steps may be specifically included:

[0053] In step S11, the current agent collects point cloud data corresponding to the current environment and encodes the point cloud data to obtain real-time feature tensors; and receives asynchronous feature tensors corresponding to k frames of historical point clouds sent by the collaborative agent; wherein the collaborative agent and the current agent belong to the ground-air collaborative system;

[0054] Step S12, performing feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor using a time delay compensation network to obtain a synchronized feature tensor;

[0055] Step S13, performing weighted fusion of multi-scale features on the synchronized feature tensors corresponding to the multiple collaborative agents to obtain a fused feature tensor;

[0056] Step S14: decoding the fused feature tensor to display the fused perception data corresponding to the fused feature tensor.

[0057] Below, the various steps of the asynchronous point cloud fusion method for the air-ground collaborative perception system in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0058] Exemplarily, the execution entity of the above method can be any agent in a ground-air collaborative system. For one agent, it can act as the current agent, receiving perception data sent by other agents in the collaborative system. Similarly, with respect to other agents in the collaborative system, it can also act as a collaborative agent, sending local perception data to other agents. The ground-air collaborative system can include multiple drones and unmanned ground vehicles. For example, the unmanned ground vehicle can be an unmanned vehicle.

[0059] In step S11, the current agent collects point cloud data corresponding to the current environment, and encodes the point cloud data to obtain real-time feature tensors; and receives asynchronous feature tensors corresponding to k frames of historical point clouds sent by the collaborative agent; wherein the collaborative agent and the current agent belong to the ground-air collaborative system.

[0060] For example, the current intelligent agent can be a drone or an unmanned vehicle. For the current intelligent agent, the assembled sensor components can be used to collect real-time data about the current environment. For example, the assembled sensor components can include a lidar, a camera, etc. The raw point cloud data collected in real time can be encoded and processed. For example, an encoding module can be provided to facilitate a neural network-based encoder to perform convolution, pooling, and other operations on the raw point cloud data to capture feature representations at different levels and generate corresponding real-time feature tensors. In addition, corresponding timestamp parameters can be configured for the real-time feature tensor.

[0061] The agent can also be equipped with a communication module for data transmission between other collaborative agents within the collaborative system. Specifically, the current agent can simultaneously receive asynchronous feature tensors sent by one or more collaborative agents. The asynchronous feature tensors can be obtained by encoding k frames of historical point cloud data. Furthermore, the asynchronous feature tensors can include timestamp parameters corresponding to each frame of historical point cloud data.

[0062] In step S12, a time delay compensation network is used to perform feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor to obtain a synchronized feature tensor.

[0063] For example, the above step S12 may specifically include:

[0064] Step S121, performing feature estimation on the current feature tensor and the asynchronous feature tensor using a feature estimation branch of a time delay compensation network to obtain collaborative feature estimation data based on multi-scale global features corresponding to the current feature tensor based on the delay time; and

[0065] Step S122: performing attention estimation on the current feature tensor and the asynchronous feature tensor using the attention estimation branch of the delay compensation network to obtain collaborative attention weight estimation data corresponding to the current feature tensor based on the delay time; wherein the feature estimation branch and the attention estimation branch are constructed based on the P-LSTM network;

[0066] Step S123: Use a time modulation network to perform confidence estimation of features and attention weights on the collaborative feature estimation data and the asynchronous feature tensor respectively, and combine the estimation results to obtain the synchronous feature tensor.

[0067] Specifically, in actual communication processes, communication system latency can negatively impact collaborative perception systems, degrading performance and increasing risk in safety-critical applications. In the presence of severe latency, collaborative perception can even be worse than that of a single agent. To mitigate the impact of latency, this method proposes a delay-compensating network that proactively adapts to the asynchronous perception features of multiple agents and synchronizes them to the same timestamp, thereby improving the robustness and effectiveness of collaborative perception. To achieve feature-level synchronization, we propose a delay-compensating network that leverages feature-attention collaborative estimation and temporal modulation techniques to adjust asynchronous perception features, aligning the perception features of different agents in time. This achieves feature-level synchronization between agents in the collaborative perception system, improving the performance of collaborative perception in real-time applications. This will help overcome the challenges of communication latency and provide more reliable and efficient collaborative perception capabilities in safety-critical applications.

[0068] Specifically, to compensate for asynchronous delays based on collaborative feature-attention estimation, a delay compensation network can be provided, comprising a collaborative feature-attention estimation module and a temporal modulation module. The collaborative feature-attention estimation module is a two-branch pyramid LSTM network sharing the same input, which aggregates features and attention weights. The temporal modulation module distributes temporal attention between the estimated features and the received asynchronous features.

[0069] Feature-attention collaborative estimation adopts a dual-branch architecture, including a feature estimation branch and an attention estimation branch, which simultaneously estimates features and their corresponding collaborative attention. The two branches of the dual LSTM network share the same input, and the input parameters can include the real-time features perceived by the current agent i and the k-frame historical features perceived by the collaborator agent j. Figure 2 As shown, each branch is implemented by a pyramid LSTM, which models a series of historical collaboration information and estimates the current state. The pyramid LSTM is specifically designed to capture spatially related collaboration features. The pyramid LSTM in this paper captures local to global features at multiple scales.

[0070] refer to Figure 2 As shown in the figure, the feature estimation branch and attention weight estimation branch use pyramid LSTM (P-LSTM). The feature estimation branch obtains the most informative collaborative features at the current moment, and the attention weight estimation branch finds the most informative region in the current collaboration.

[0071] P-LSTM uses the estimated feature map of the previous timestamp and collaborative attention as the input of the next timestamp of the two branches. To capture the spatial correlation of collaborative features, P-LSTM modifies the matrix multiplication in LSTM into a multi-scale convolution structure.

[0072] The goal of the attention weight estimation branch is to find the collaborative region with the largest current information content while suppressing the region with large estimation error. To achieve this, the attention weight estimation branch is feature-aware. Figure 3 As shown, to allow the feature estimation and the corresponding attention to be mutually aware, we recurrently utilize the estimated feature map and collaborative attention from the previous timestamp as the input of the next timestamp of any branch.

[0073] For example, when performing feature-attention collaborative estimation, assuming t0 is the current time, τ is the delay time, and the number of historical point cloud frames received by the current agent from the collaborative agent is k, the hidden states and cell states in the two P-LSTM branches are initialized; the two branches share the same input, including the real-time features perceived by the current agent at the current moment (encoded as a real-time feature tensor) and the k-frame historical features perceived by the collaborative agent (encoded as an asynchronous feature tensor).

[0074] in, and denote the attention weight and features of collaborative agent j at timestamp t respectively; and denote the estimation of attention weight and feature at timestamp t respectively; e (t) is the input of the pyramid LSTM at timestamp, are the hidden state and cell state of the pyramid LSTM in each branch, respectively.

[0075] For the feature estimation branch and the weighted attention estimation branch, the specific calculation process may include the following steps:

[0076] 1) Initialization: Let the current time be t0, the sliding step size be k, and τ be the delay time; for the time window t, we have:

[0077] t=t0-τ-k+1,t0-τ-k+2,...,t0-τ-1

[0078] 2) Calculate the attention weight matrix within the time window. The formula includes:

[0079]

[0080] Update the state of the feature LSTM. The formula includes:

[0081]

[0082] 3) Update of collaborative force estimation, for t=t0-τ+1,t0-τ+2,...,t0-1,

[0083] According to the feature LSTM hidden state, calculate the feature output:

[0084]

[0085] According to the characteristic LSTM hidden state, calculate the weight output:

[0086]

[0087] Calculate the feature weight collaborative attention value:

[0088]

[0089] Update the features and weights LSTM state:

[0090]

[0091] 4) The final output is at time t0-1, and the feature output is calculated based on the feature LSTM hidden state:

[0092]

[0093] Calculate the weight output based on the weighted LSTM hidden state:

[0094]

[0095] Exemplarily, using a temporal modulation network to perform confidence estimation of features and attention weights on the collaborative feature estimation data and the asynchronous feature tensor, respectively, and combining the estimation results to obtain the synchronous feature tensor, including:

[0096] Using a time modulation network, the collaborative feature estimation data, the asynchronous feature tensor, and the delay tensor are spliced ​​together to obtain spliced ​​feature data; and feature confidence estimation is performed on the spliced ​​feature data to obtain collaborative feature confidence estimation data; and

[0097] Performing splicing processing on the collaborative attention weight estimation data, the attention weight corresponding to the asynchronous feature tensor, and the delay tensor to obtain spliced ​​attention weight data; and performing attention confidence estimation on the spliced ​​attention weight data to obtain collaborative attention weight estimation data;

[0098] Based on the collaborative attention weight estimation data, the time domain attention weight is allocated to the collaborative feature confidence estimation data to obtain the synchronized feature tensor after feature fusion.

[0099] Specifically, to address the issue of low latency, where the performance degradation caused by latency is smaller than the estimation noise caused by FASE, this method proposes a temporal modulation network that uses the delay time to allocate temporal attention between the estimated features and the received asynchronous features to adaptively adjust the final estimate of the collaborative features. To cope with different delay conditions, it is necessary to assign weights to the original features and the estimated features based on the delay time. At low latency, the original features may be more reliable, while at high latency, the estimated features may be more reliable. The two features can then be fused using the temporal attention weights to produce a more comprehensive and reliable estimation result.

[0100] Specifically, feature concatenation can be used to concatenate the estimated collaborative features output by the feature estimation branch, the received asynchronous features, and the delay tensor. Simultaneously, the collaborative attention weights estimated by the attention weight branch, the received attention weights, and the delay tensor are concatenated. By concatenating the features and attention weights, we can estimate the features and the confidence level of the collaborative attention for each spatial region. The delay tensor is a feature tensor configured based on the delay time τ.

[0101] Specifically, two lightweight convolutional neural networks (m F and m W ), process the spliced ​​features. F The network is used to estimate the confidence of the feature, and m W The network is used to estimate the confidence of the collaborative attention. These confidences reflect the level of uncertainty in the estimation of each spatial region, that is, a measure of the reliability of the estimation. F (·) and m W (·) are all lightweight convolutional neural networks with sigmoid activation function, 1∈R H×W is a matrix whose elements are all 1.

[0102] Processing the confidence matrix can be divided into two steps. First, the estimated features / attention and the latest asynchronous features / attention are combined. This step aims to use the weights of the confidence matrix to determine the contribution of the estimated features / attention to the final estimate. When the latency is higher, the confidence matrix weights will be higher, which means that the estimated features / attention will have a greater impact on the final estimate.

[0103] Next, the system performs further processing and analysis based on the combined results of the estimated features / attention and the latest asynchronous features / attention to generate the final perception output. This perception output includes the recognition, detection, and tracking of objects, behaviors, or events in the environment. By considering the weights of the confidence matrix, the system can more accurately utilize the perception capabilities of multiple agents in collaborative perception tasks and make appropriate feature trade-offs based on latency, thereby improving the effectiveness and robustness of collaborative perception tasks.

[0104] In summary, through the combination of concatenation, confidence matrices, and features / attention, the latency-aware collaborative sensing system comprehensively considers the estimation uncertainty and latency of each spatial region, thereby improving the performance and reliability of collaborative sensing. This design enables the system to adaptively adjust feature trade-offs based on latency, better adapting to different collaborative sensing scenarios and requirements.

[0105] For example, T is the confidence matrix reflecting the uncertainty of the estimation of each spatial region. F ∈R H×W×C ,T W ∈R H×W×C The delay tensor obtained by expanding the delay time τ∈R is in the form of The weight allocation algorithm based on time modulation may specifically include the following steps:

[0106] 1) Initialize the feature allocation matrix and calculate the feature allocation matrix based on the collaborative feature estimation data, asynchronous feature tensor, and delay tensor:

[0107]

[0108] in, Estimation data for collaborative features, F (t-τ) is the received asynchronous feature tensor based on historical features; T W is a delayed tensor.

[0109] 2) Initialize the feature weight distribution matrix, concatenate the collaborative attention weight estimation data, the attention weight corresponding to the asynchronous feature tensor, and the delay tensor, and calculate the weight distribution matrix:

[0110]

[0111] in, Estimating data for collaborative attention weights, W (t-τ) The historical weights of the received historical features.

[0112] 3) Update the feature vector. Calculate the updated feature vector based on the collaborative feature estimation data, the asynchronous feature tensor, and the feature allocation matrix:

[0113]

[0114] 4) Update the weight vector. Calculate the updated weight vector based on the collaborative attention weight estimation data, the attention weight corresponding to the asynchronous feature tensor, and the weight distribution matrix:

[0115]

[0116] 5) The final updated feature vector and weight vector are used as input for the next step of calculation.

[0117] This method provides a delay compensation approach that improves the robustness and effectiveness of collaborative sensing by proactively adapting to the asynchronous perception features of multiple agents and synchronizing them to the same timestamp. To achieve feature-level synchronization, this method proposes a novel delay compensation network. This network utilizes feature-attention collaborative estimation and temporal modulation techniques to adjust the asynchronous perception features, aligning the perception features of different agents in time. This achieves feature-level synchronization between agents in the collaborative sensing system, improving the performance of collaborative sensing in real-time applications. This will help overcome the challenges of communication delays and provide more reliable and efficient collaborative sensing capabilities in safety-critical applications.

[0118] In step S13, the synchronized feature tensors corresponding to the multiple collaborative agents are subjected to weighted fusion of multi-scale features to obtain a fused feature tensor.

[0119] Exemplarily, the above step S13 may specifically include:

[0120] Step S131, performing convolution processing on each synchronized feature tensor to obtain feature data after dimensionality reduction;

[0121] Step S132: perform an Affinity operation on the reduced feature data in the spatial dimension to generate a corresponding attention weight map;

[0122] Step S133: Perform weighted fusion on the synchronized feature tensors based on the attention weight map to obtain a fused feature tensor.

[0123] Exemplarily, the method further includes:

[0124] The optimization goal of feature fusion based on cross attention is defined as: maximizing the difference between the system perception accuracy after fusion optimization and the system accuracy before optimization.

[0125] max i mize(A after -A before )

[0126] Among them, Abefore To optimize the accuracy of feature fusion, A after The perception accuracy of the optimized cross-attention-based perception system;

[0127] According to the loss configuration corresponding to the system accuracy improvement constraint:

[0128] (A after -A before )<=ΔA max

[0129] Where ΔA max The maximum acceptable loss in system accuracy.

[0130] Specifically, this method uses an additive mechanism to fuse the synchronous features from different collaborative intelligences. The accuracy of the additive mechanism is defined as A before , the perception accuracy of the optimized cross-attention-based perception system is A after The maximum acceptable loss of system accuracy improvement is ΔA max .

[0131] The optimization goal of feature fusion based on cross attention is to maximize the difference between the system perception accuracy after fusion module optimization and the system accuracy before optimization, that is:

[0132] max i mize(A after -A before )

[0133] Constraints: The constraints that need to be met at the same time are that the loss of system accuracy improvement is within an acceptable range, that is:

[0134] (A after -A before )<=ΔA max

[0135] By optimizing the feature fusion strategy, this paper aims to find the appropriate feature fusion mechanism and parameter settings to maximize the difference between the optimized and pre-optimized system accuracy, while ensuring that the loss of accuracy improvement is within an acceptable range. This optimization process can further improve the performance of the collaborative perception system and enhance its feature fusion capabilities while maintaining system reliability and stability.

[0136] Exemplarily, the feature fusion method based on the cross-attention mechanism may specifically include the following steps:

[0137] 1: Use 1×1 convolution to reduce the dimension to get Q, K∈R C′×W×H .

[0138] 2: Generate attention map A∈R through Affinity operation(H+W-1)×(H×W) .

[0139] 3: For each position u in the Q space dimension, get a vector Q u ∈R C′ .

[0140] 4: At the same time, we get a set Ω on K u ∈R (H+W-1)×C , which represents the same row or column of position u.

[0141]

[0142] Used to express the correlation between the two, we can finally get D∈R (H×W-1)×(H×W) .

[0143] 6: Finally, Softmax is applied to D in the channel dimension to obtain the attention map A. It should be noted that the channel dimension here represents the dimension H+W-1, which represents the correlation between a pixel at a certain position and its vertical and horizontal pixels.

[0144] 7: Aggregation into the final feature map:

[0145]

[0146] Among them, H′ u ∈R C is the eigenvector at a certain position.

[0147] The affinity operation in the Transformer is essentially a calculation process to establish the strength of the association between tokens. In neural networks, Softmax is an activation function used for multi-category classification, also known as the normalized exponential function. Given a vector containing raw scores (unprocessed model output), Softmax converts these scores into a vector representing a probability distribution. Specifically, for a given raw score vector (z), the Softmax function is calculated as follows:

[0148]

[0149] Among them, z i is the (i)th element in the vector, and C is the number of output nodes.

[0150] This method provides an improved feature fusion method based on the cross-attention mechanism, reducing the computational workload and storage requirements during the feature fusion process. It adaptively learns the weights and correlations between different features, achieving more efficient and accurate feature fusion, thereby reducing the pressure on computing resources. This network optimizes computing resource utilization, improves computational efficiency, and reduces memory usage. This optimizes the efficiency and resource utilization of feature fusion, providing a reliable solution for collaborative perception systems in resource-limited environments.

[0151] In step S14, the fused feature tensor is decoded to display the fused perception data corresponding to the fused feature tensor.

[0152] For example, after obtaining the fused feature tensor, it can be decoded and processed to restore the fused perception data, and the restored fused feature data can be displayed on the current intelligent body; the final perception output is obtained, and finally a visualization result is obtained.

[0153] The method provided by the embodiment of the present invention is referred to Figure 5 As shown, the raw data passes through the encoding module to generate feature tensors, which are then transmitted to other agents through the communication module, potentially subject to communication delays. Next, the asynchronous feature tensors enter the delay compensation module, where a delay compensation network performs feature-level synchronization processing to produce synchronized feature tensors. Finally, the synchronized feature tensors enter the fusion module, where a neural network performs weighted fusion of multi-scale features to produce a fused feature tensor. Finally, the fused feature tensors pass through the decoding module to generate the perception output.

[0154] The beneficial effects of the present invention are as follows:

[0155] 1. This paper proposes a lightweight deployment method for deep models with algorithm-system cross-layer adaptive optimization. It constructs a search space by comprehensively considering the inter-layer dependencies and performance requirements of the model, and uses a two-layer collaborative evolutionary optimization algorithm to intelligently select combination strategies and configuration parameters, thereby achieving joint optimization of model structure and system acceleration.

[0156] 2. This invention more comprehensively considers optimization requirements at the model and system levels. Compared to single-layer optimization methods, it significantly reduces model memory usage and computational latency, while barely compromising model accuracy. Furthermore, it can be flexibly configured based on the hardware resources and performance requirements of different mobile devices, demonstrating strong versatility and adaptability.

[0157] 3. The method of the present invention can effectively solve the deployment problem of deep learning models on resource-constrained mobile devices, promote the widespread application of deep learning technology in the field of intelligent Internet of Things, and improve user experience and data security.

[0158] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0159] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0160] Figure 6 A schematic diagram of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0161] It should be noted that Figure 6 The electronic device 1000 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present invention. For example, the electronic device can be an electronic device mounted on an intelligent body, used to implement the asynchronous point cloud fusion method for the air-ground collaborative perception system.

[0162] like Figure 6 As shown, electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 1002 or the program loaded from storage portion 1008 into random access memory (RAM) 1003. Various programs and data required for system operation are also stored in RAM 1003. CPU 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0163] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0164] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a storage medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009 and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the various functions defined in the system of the present application are performed.

[0165] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] The units involved in the embodiments of the present invention may be implemented in software or hardware, and the units described may also be provided in a processor. In some cases, the names of these units do not limit the units themselves.

[0168] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device; or it can exist independently without being installed in the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement the following Figure 1 The individual steps of the method are shown.

[0169] In one embodiment, the present application provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0170] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0171] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the claims.

[0172] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof, which is limited only by the appended claims.

Claims

1. An asynchronous point cloud fusion method for air-ground collaborative perception system, characterized by: The method comprises: The current agent collects point cloud data corresponding to the current environment and encodes the point cloud data to obtain real-time feature tensors; and receives asynchronous feature tensors corresponding to k frames of historical point clouds sent by the collaborative agent; wherein the collaborative agent and the current agent belong to the ground-air collaborative system; Performing feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor using a time delay compensation network to obtain a synchronized feature tensor; Perform weighted fusion of multi-scale features on the synchronized feature tensors corresponding to several collaborative agents to obtain a fused feature tensor; The fused feature tensor is decoded to display the fused perception data corresponding to the fused feature tensor.

2. The method according to claim 1, characterized in that The method of performing feature-level time synchronization processing on the current feature tensor and the asynchronous feature tensor using a time delay compensation network to obtain a synchronized feature tensor includes: Performing feature estimation on the current feature tensor and the asynchronous feature tensor using a feature estimation branch of a time delay compensation network to obtain collaborative feature estimation data based on multi-scale global features corresponding to the current feature tensor based on the delay time; and Performing attention estimation on the current feature tensor and the asynchronous feature tensor using the attention estimation branch of the delay compensation network to obtain collaborative attention weight estimation data corresponding to the current feature tensor based on the delay time; wherein the feature estimation branch and the attention estimation branch are constructed based on the P-LSTM network; A time modulation network is used to respectively estimate the confidence of features and attention weights of the collaborative feature estimation data and the asynchronous feature tensor, and the estimation results are combined to obtain the synchronous feature tensor.

3. The method according to claim 2, characterized in that Using a time modulation network to respectively estimate the confidence of features and attention weights of the collaborative feature estimation data and the asynchronous feature tensor, and combining the estimation results to obtain the synchronous feature tensor, including: Using a time modulation network, the collaborative feature estimation data, the asynchronous feature tensor, and the delay tensor are spliced ​​together to obtain spliced ​​feature data; and feature confidence estimation is performed on the spliced ​​feature data to obtain collaborative feature confidence estimation data; and Performing splicing processing on the collaborative attention weight estimation data, the attention weight corresponding to the asynchronous feature tensor, and the delay tensor to obtain spliced ​​attention weight data; and performing attention confidence estimation on the spliced ​​attention weight data to obtain collaborative attention weight estimation data; Based on the collaborative attention weight estimation data, the time domain attention weight is allocated to the collaborative feature confidence estimation data to obtain the synchronized feature tensor after feature fusion.

4. The method according to claim 1, wherein The synchronized feature tensors corresponding to several collaborative agents are weightedly fused with multi-scale features to obtain a fused feature tensor, including: Perform convolution processing on each synchronized feature tensor to obtain feature data after dimensionality reduction; Perform Affinity operation on the reduced feature data in the spatial dimension to generate the corresponding attention weight map; The synchronized feature tensors are weightedly fused based on the attention weight map to obtain a fused feature tensor.

5. The method according to claim 4, characterized in that The method further comprises: The optimization goal of feature fusion based on cross attention is defined as: maximizing the difference between the system perception accuracy after fusion optimization and the system accuracy before optimization. max i mize(A after -A before ) Among them, A before To optimize the accuracy of feature fusion, A after The perception accuracy of the optimized cross-attention-based perception system; According to the loss configuration corresponding to the system accuracy improvement constraint: (IN after -IN before )<=ΔA max Where ΔA max The maximum acceptable loss in system accuracy.

6. The method according to claim 1, wherein The real-time feature tensor and the asynchronous feature tensor are respectively configured with a timestamp parameter.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the asynchronous point cloud fusion method for an air-ground collaborative perception system according to any one of claims 1 to 6 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the asynchronous point cloud fusion method for an air-ground collaborative perception system according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the asynchronous point cloud fusion method for an air-ground collaborative perception system according to any one of claims 1 to 6 by executing the executable instructions.

Images