Multi-wheel multi-modal multi-agent cooperative perception method based on spatial confidence map

Abstract

The application provides a multi-round multi-modal multi-agent cooperative perception method and system based on a spatial confidence map, a terminal and a medium, comprising: extracting features of input data using a deep convolutional neural network; generating a spatial confidence map using a spatial confidence generator for the features; using a communication module based on the spatial confidence map to pack compact information packets, construct a sparse communication graph, and transmit the information packets between agents through the communication graph; using an information fusion module based on the spatial confidence map to fuse the information packets harvested by the communication module with the local features of each agent; and decoding the fused features to obtain detection results. The application focuses on key perception areas, uses less communication consumption to achieve higher perception performance, and is suitable for various communication bandwidths and rounds.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a multi-round, multimodal, and multi-agent collaborative perception method, system, terminal, and medium based on spatial confidence maps. Background Technology

[0002] Multi-agent collaborative perception enables multiple agents to share complementary perception capabilities, providing a new direction for promoting more comprehensive perception and fundamentally solving the limitations of single-agent perception, such as occlusion and long-distance problems. Related methods have wide-ranging applications, such as autonomous driving in multi-vehicle networks, automated warehouse systems with multiple robots, and search and rescue operations using multiple drones.

[0003] In this emerging field, the biggest challenge currently lies in balancing sensing performance and communication bandwidth. Real-world communication systems are always constrained and struggle to withstand significant communication overhead, such as transmitting complete raw observation data and large amounts of features. Therefore, improving sensing performance requires considering communication costs.

[0004] To achieve a better trade-off between performance and bandwidth, previous work has proposed solutions from several angles. For example, handshake communication mechanisms can be used to select the most relevant collaborators, thereby reducing unnecessary communication between agents; appropriate feature selection can be used to transmit more compact information packets between agents; and information compression can be achieved by using existing image compression techniques or learning an autoencoder. This further reduces bandwidth consumption. However, these works all make a reasonable but unnecessary assumption: once two agents cooperate, they will indiscriminately share perceptual information across all spatial regions. Because a large portion of the spatial region contains only information irrelevant to the perception task, this unnecessary assumption leads to a significant waste of bandwidth.

[0005] Communication strategies in multi-agent systems have been extensively studied, with early work often employing predefined protocols or heuristics to determine how agents communicate. However, it is difficult to generalize these methods to more complex tasks. Therefore, recent research has explored learning-based approaches for complex scenarios, such as employing attention mechanisms in multi-agent systems to help agents selectively integrate information from others. However, due to a lack of explicit supervision, most of these works only consider decision-making tasks and employ reinforcement learning techniques.

[0006] Collaborative perception, as a recent application of multi-agent communication systems in perception, remains immature. To support research in this area, several high-quality datasets have emerged. These collaborative methods aim to achieve better performance-bandwidth tradeoffs, such as using handshake communication mechanisms to determine when to communicate and create sparse communication patterns, achieving better perception on graph neural networks based on multi-turn message passing, leveraging the advantages of early and mid-stage cooperation through information extraction, proposing graph-based intermediate fusion to improve perception, or introducing novel heterogeneous multi-agent modules to fuse information. However, they have limitations, namely, they only handle a predefined communication bandwidth and a fixed number of rounds. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multi-round, multimodal, and multi-agent collaborative perception method, system, medium, and terminal based on spatial confidence maps.

[0008] According to one aspect of the present invention, a multi-round, multi-modal, multi-agent cooperative perception method based on spatial confidence maps is provided, comprising:

[0009] Use deep convolutional neural networks to extract features from the input data;

[0010] A spatial confidence map is generated using a spatial confidence generator on the aforementioned features;

[0011] Using the communication module based on the spatial confidence graph, compact information packets are packaged to construct a sparse communication graph, and the information packets are transmitted between agents through the communication graph;

[0012] The information fusion module based on spatial confidence maps is used to fuse the information packets obtained by the communication module with the local features of each agent.

[0013] The detection results are obtained by decoding the fused features.

[0014] Preferably, the extraction of features from the input data using a deep convolutional neural network includes:

[0015] Use a backbone network to process the input data from the i-th agent. Extracting features for

[0016]

[0017] Where Φ enc (·) is the encoding function, and the superscript 0 indicates that the feature was obtained before communication. H, W, and D are the length, width, and channel dimension of the feature map, respectively.

[0018] Preferably, the step of generating a spatial confidence map using a spatial confidence generator for the features includes:

[0019] Using a network with the same structure as the detector decoder, a spatial confidence map is generated based on the extracted features at the k-th round of communication:

[0020] Where, Φ generator (·) is the detection decoder. This is the spatial confidence graph at the k-th round of communication.

[0021] Preferably, the method of using a communication module based on spatial confidence graphs to package compact information packets, construct a sparse communication graph, and transmit information packets between agents through the communication graph includes:

[0022] Information packets are packaged based on spatial confidence maps. Each information packet consists of two parts: one part is feature information used to help other intelligent agents, and the other part is the information request map needed by each intelligent agent.

[0023]

[0024] Where k represents the communication round, The graph represents the information request of agent i. This represents the features sent from agent i to agent j;

[0025] Information request graph of agent i in the k-th round of communication From the spatial confidence map Decide,

[0026]

[0027] Feature information sent from agent i to agent j in the k-th round of communication From the binary selection matrix and extracted features Decide,

[0028]

[0029] Where the binary choice matrix Using spatial confidence maps And information request diagram To select the spatial region with the most information among the features.

[0030]

[0031] In this context, ⊙ represents element-wise multiplication. This is the information request graph of the j-th agent in the previous round. The spatial confidence graph of the i-th agent, function Φ select (·) Select the most critical communication area based on the input matrix;

[0032] Based on the selection matrix Construct a communication graph, represented as an adjacency matrix.

[0033]

[0034] in In the k-th round of communication, the directed communication edge between agent i and agent j, spatial dimension, and features are... Figure 1 In the expression, h and w represent the length and width of the space, respectively, where i is the sender, j is the receiver, 1 indicates communication is established, and 0 indicates no communication is established.

[0035] Preferably, the information fusion module using spatial confidence maps fuses the information packets received by the communication module with the local features of each agent, including:

[0036] For the i-th agent, upon receiving a message from the j-th agent... Then, the features are obtained through decoding. Spatial confidence plot

[0037] The fusion weight of the features of the i-th agent and the j-th agent during the k-th round of communication. for

[0038]

[0039] MHA W (·,·,·) is a multi-head attention network applied individually to each spatial location, with the input being the features of agent i. and the feature information of the j-th agent obtained by decoding from the received information packet. The output is the attention weights calculated based on feature similarity;

[0040] Feature fusion and updating are performed based on the attention weights, and the feature map of the i-th agent after fusing information in the k-th round of communication is shown. for

[0041]

[0042] Here, FFN(·) is a feedforward network, implemented by a multilayer perceptron. It is communication diagram A (k) The neighbors of the i-th agent are defined as the set of agents that send information to agent i, and the fused features. Let be the features of the i-th agent in the (k+1)-th round of communication.

[0043] Preferably, the step of decoding the fused features to obtain the detection result includes:

[0044] Based on the assumption of the characteristics of the k-th round of communication Decoder Φ dec (·) Obtain the object bounding box:

[0045]

[0046] in This represents a rotating bounding box containing (c,x,y,h,w,cosα,sinα), representing an object with coordinates (x,y), width w, length l, rotation angle α, and object type c.

[0047] Preferably, each round of communication uses a detection loss function to supervise object detection, with the loss being:

[0048]

[0049] in, L is the real object bounding box of the i-th agent. det It is the detection loss function.

[0050] According to a second invention of the present invention, a multi-round, multi-modal, multi-agent collaborative perception system based on a spatial confidence communication strategy is provided. The method described in the present invention includes:

[0051] The feature extraction module uses a deep convolutional neural network to extract features from the input data.

[0052] A spatial confidence generator that uses the data features to generate a spatial confidence map;

[0053] The communication module based on spatial confidence graph packs compact information packets, constructs a sparse communication graph, and transmits the information packets between agents through the communication graph;

[0054] The information fusion module based on spatial confidence maps is used to fuse the information packets obtained by the communication module with the local features of each agent.

[0055] The detection box decoding module decodes the fused features to obtain the detection results.

[0056] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described herein, or to run the system described herein.

[0057] According to a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described herein, or to perform the system described herein.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The multi-round, multimodal, and multi-agent collaborative perception method and system based on spatial confidence maps in this invention achieve higher perception performance by focusing on key perception areas and using less communication consumption; it adapts to various communication bandwidths and rounds.

[0060] The core idea of ​​the multi-round, multimodal, multi-agent collaborative perception method and system based on spatial confidence graphs in this invention is to establish a spatial confidence graph for each agent so that they can decide where and with whom to communicate and whose information to aggregate. Each agent provides spatially sparse but critical features to support other agents, while requesting supplementary information from other agents in multi-round communication, improving bandwidth efficiency and achieving a superior performance and bandwidth tradeoff. Attached Figure Description

[0061] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0062] Figure 1 This is a flowchart of a multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence graphs according to an embodiment of the present invention;

[0063] Figure 2 This is a block diagram of a multi-round, multimodal, multi-agent collaborative perception system based on spatial confidence graphs, according to another embodiment of the present invention. Detailed Implementation

[0064] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0065] Each agent in this invention has an encoded perceptron composed of a deep convolutional neural network; each agent also has a spatial confidence map generator, a communication module, and an information fusion module. Based on this, the main technical concept of this invention lies in its three key modules: i) a spatial confidence map generator, which generates a spatial confidence map to represent important perceptual regions; ii) a communication module based on the spatial confidence map, which uses the spatial confidence map to determine which spatial regions' information to package and construct a communication graph to determine which agents communicate with; and iii) an information fusion module based on the spatial confidence map, which fuses information received from other agents to upgrade the features of each agent.

[0066] Based on the above technical concept, see Figure 1 The flowchart below shows a multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence graphs, according to an embodiment of the present invention. The method is as follows:

[0067] S100 uses a deep convolutional neural network to extract features from the input data;

[0068] S200, use the spatial confidence generator to generate a spatial confidence map for the features in S100;

[0069] S300 uses a communication module based on the spatial confidence graph in S200 to package compact information packets, construct a sparse communication graph, and transmit information packets between agents through the communication graph;

[0070] S400 uses an information fusion module based on the spatial confidence map in S200 to fuse the information packets transmitted in S300 obtained by the communication module with the local features of each agent.

[0071] S500 decodes the fused features obtained from S400 to obtain the detection results.

[0072] In this embodiment, a compact information packet refers to the maximum amount of information that can be contained within the same storage capacity, representing a high information density. A sparse communication graph refers to a graph that does not require fully connected communication but only constructs a few necessary communications to achieve a sparse effect, i.e., low communication consumption. In this embodiment, each agent provides spatially sparse but perception-critical features to other agents through communication, focusing on perceiving key areas and achieving higher perception performance with less communication overhead.

[0073] In a preferred embodiment of the invention, S100 is implemented. First, input data is extracted from sensor data. This embodiment accepts single or multi-mode input data, such as RGB images or 3D point cloud images. The input image undergoes a geometric transformation, converting features into a bird's-eye view. Using features from the bird's-eye view means that all agents project their respective perceptual information onto the same full-time coordinate system, avoiding complex coordinate transformations and supporting better cross-agent sharing and collaboration. Specifically: for the i-th agent, assuming the input is X... i ,

[0074] Where, Φ enc (·) represents the encoding function, with the superscript 0 indicating that the feature was obtained before communication. H, W, and D are the height, weight, and channels of the feature map, respectively. All agents share a common bird's-eye view coordinate system. For RGB images, the encoding function extracts features, and the viewpoint is transformed based on camera intrinsic and extrinsic parameters to convert the frontal view into bird's-eye view features. For 3D point cloud input, the 3D points are discretized into a bird's-eye view, Φ enc The (·) function extracts features from the bird's-eye view, and the extracted feature map is output to the spatial confidence generator and information fusion module.

[0075] In a preferred embodiment of the present invention, based on the feature map obtained in S100 above, S200 is performed to generate a spatial confidence map using a spatial confidence generator.

[0076] The spatial confidence generator produces a spatial confidence map from the feature map of each agent. This spatial confidence map reflects the perception criticality level of each spatial region. Intuitively, for object detection tasks, the spatial confidence map represents the regions containing objects, which are more critical than background regions. During collaboration, due to limited field of view, regions containing objects can help recover missed objects, while background regions can be omitted to save valuable bandwidth. Therefore, the spatial confidence map is represented by a detection confidence map, meaning it reuses common detection confidence maps already defined in the detection system. Regions with high perception criticality levels are those containing objects with high confidence scores. To implement the spatial confidence generator, a detection decoder structure is used to generate the feature map at the k-th round of communication during detection execution. Specifically, as follows...

[0077]

[0078] Where, Φ generator (·) is the detection decoder. This is the spatial confidence graph at the k-th round of communication.

[0079] Because it involves multi-round collaboration, this embodiment iteratively updates the feature map. Once the feature map is updated, the spatial confidence generator Φ... generatorThe (·) signal will be triggered, updating the spatial confidence map and thus reflecting the perception criticality level of each spatial location in a timely manner. This answers a key question that has been overlooked in previous work: which spatial regions of information are worth sharing for each agent, and provides a solid foundation for efficient communication and effective information fusion.

[0080] In a preferred embodiment of the present invention, based on the feature map obtained in S100 and the spatial confidence map obtained in S200, S300 is implemented. Specifically, the communication module based on the spatial confidence map packages compact information packets as follows:

[0081] By packaging information, it is determined which information should be included in the information to be sent. The suggested information includes an information request graph indicating the spatial regions that the agent needs to know more about, and a spatially sparse but perceptually critical feature graph, which consists of two parts: one part is feature information to help other agents, and the other part is the information request graph that each agent needs.

[0082] Specifically, the information request graph of the i-th agent is as follows: It shows a negative correlation with the spatial confidence map. This is because for locations with low confidence, it's difficult for the agent to determine whether there are truly no objects or if the issue is simply due to limited information (such as occlusion). Therefore, lower confidence indicates potential information gaps at those locations. Requesting information from other agents at these locations can improve the current agent's detection accuracy. If there are indeed no objects or only limited information, requesting information from other agents at these locations can further improve the current agent's detection accuracy.

[0083] Specifically, spatially sparse feature maps The selection is based on each agent's spatial confidence graph and information request graph received from others. Furthermore, the binary choice matrix This is used to represent whether each position is selected or not, where 1 indicates selection and 0 indicates not selection. The binary selection matrix for agent i to agent j in the k-th round of communication is as follows:

[0084]

[0085] In this context, ⊙ represents element-wise multiplication. This is the demand graph of the j-th agent in the previous round, Φ selectThe (·) function selects the most critical region within the given input matrix, representing the critical level at a specific spatial location. This function is implemented by selecting the position of the largest element in the given input matrix, conditioned on bandwidth limitations. Alternatively, a Gaussian filter can be applied to remove outliers and introduce some background information. In the initial communication rounds, each agent selects the most critical region from its own perspective, as the requested maps from other agents are not yet available. In subsequent minutes, each agent also considers the requests from its partners, allowing them to work more effectively. The selected feature map is then processed... Obtain feature information It provides spatially sparse but perceptually crucial information.

[0086] In this embodiment, in the k-th round of communication, the information sent by the i-th agent to the j-th agent is

[0087]

[0088] Here, Providing spatial priors and requiring supplementary information feature maps to provide knowledge information for the j-th agent in this round of demand, they jointly achieve mutually beneficial cooperation; feature information It is sparse, transmitting only non-zero features and their corresponding exponents, ensuring low communication costs. Feature information The sparsity is determined by the binary selection matrix, which dynamically allocates the communication budget according to the perception critical level of different spaces and regions to adapt to various communication conditions.

[0089] When implementing S300, the communication module based on the spatial confidence graph constructs a sparse communication graph and transmits message packets between agents through the constructed communication graph, specifically as follows:

[0090] The goal of communication graph construction is to determine when and with whom to communicate, avoiding a large amount of unnecessary communication. This embodiment proposes a handshake mechanism that uses similar global characteristics to match partners. This is because two agents with similar global characteristics may not necessarily need each other's information. This embodiment provides an explicit design principle to determine the necessity of communication between agent i and agent j, measured by the overlap between the information possessed by agent i and agent j. With the help of a spatial trust graph and a demand graph, this embodiment constructs a more easily interpretable communication graph.

[0091] For each agent in the initial communication round, to activate cooperation, a fully connected communication graph is constructed, and each agent broadcasts information to the rest of the system. For subsequent communication rounds, according to... The maximum value is used to check whether communication between agents is necessary. If at least one patch is activated, then communication is necessary. Specifically, assume A... {(k)} Let be the adjacency matrix of the communication graph in the k-th round of communication.

[0092]

[0093] in It is the directed communication edge between agent i and agent j in the k-th round of communication, possessing the characteristics of and . Figure 1 The spatial dimensions are as follows: h and w represent the length and width of the space, respectively. i is the sender, j is the receiver, 1 indicates communication is established, and 0 indicates no communication is established.

[0094] This embodiment proposes a communication module using a spatially sparse feature map to package compact information and transmits the information through a sparsely connected communication graph. In contrast, most existing collaborative sensing systems consider both complete feature maps and fully connected communication graphs within the message. This embodiment reduces communication bandwidth without compromising sensing capabilities, utilizes the spatial confidence map to select the spatial region with the highest information content in the feature map, and determines the most advantageous collaborating partner.

[0095] In a preferred embodiment of the present invention, in step S400, a communication module based on the spatial confidence map is used to fuse the information packets received by the communication module in step S300 with the local features of each agent in step S100, thereby improving the perception effect of each agent on the scene.

[0096] The goal of spatial confidence-based information fusion is to augment information received from other agents by aggregating the features of each agent. To achieve this, this embodiment employs a converter structure that leverages multi-head attention to fuse corresponding features from multiple agents. At each individual spatial location, the fusion process is performed independently, fusing the spatial confidence maps of all agents to facilitate attentional learning. The spatial confidence map clearly reflects the perceptual threshold, providing a useful prior for attentional learning. Specifically, for the i-th agent, upon receiving a message from the j-th agent… Then, unpack and retrieve the feature map. Spatial confidence plot To integrate the features of the j-th agent during the k-th round of communication, the cross-agent weights of the i-th agent are...

[0097]

[0098] MHA W(·,·,·) represents multi-head attention applied to each individual spatial location, outputting scaled dot-product attention weights. Note that: 1. Spatial confidence maps influence attention weights because features with higher perceptual levels are favored in feature aggregation; 2. Cross-agent attention weights model the cooperation strength, possessing a spatial resolution that allows for more flexible information fusion across different spatial regions. The feature map of the i-th agent after information fusion in the k-th round of communication. for

[0099]

[0100] Here, FFN(·) is a feedforward network, implemented by a multilayer perceptron. It is communication diagram A (k) The neighbors of the i-th agent are defined as the set of agents that send information to agent i. The fused features. The characteristics of the i-th agent in the (k+1)-th round of communication.

[0101] The sensor's location encoding represents the physical distance between the sensor and the perception of each agent. It employs a standard location encoding function, conditional on sensing distance and feature dimension. Between the input and the transformer, the features are added to the location encoding for each location. Each location emphasizes features specific to that location, making feature fusion more targeted. Compared to methods that also use a fusion module based on per-location attention, this fusion module utilizes multi-head attention with two additional prior factors: a spatial confidence map and sensing distance. Both contribute to attention learning, prioritizing high-quality and critical features.

[0102] In a preferred embodiment of the present invention, S500 is implemented, in which a spatial confidence-aware message fusion module decodes the features fused in the above embodiment S400 to obtain a detection decoder for the detection result.

[0103] The detector decoder decodes features into objects, including categories and regression outputs. Assume the feature map from the k-th round of communication... Detector decoder Φ dec (·) Generate the detection result of the i-th agent.

[0104]

[0105] The object frame The bounding box represents a rotation of an agent (c,x,y,h,w,cosα,sinα), and represents an object with coordinates (x,y), width w, length l, rotation angle α, and object type c. This is also the final output of the proposed cooperative perception system.

[0106] In other embodiments of the present invention, in order to train the entire system, the task is supervised by setting the loss function in multiple rounds as follows:

[0107] By supervising both spatial confidence generation and object detection in each round, as mentioned earlier, the spatial confidence generator functions similarly to the classification function of the detection decoder. To improve parameter efficiency, the spatial confidence generator reuses the parameters of the detection decoder. For multi-round setups, each round is performed under the supervision of a detection loss, with the overall loss being...

[0108]

[0109] in, L is the true value of the i-th agent. det It's about detecting loss.

[0110] To adapt to multi-round communication and dynamic bandwidth, a multi-round training strategy is employed, using a course-based learning approach to train the model under various communication settings. First, the communication bandwidth and number of rounds are gradually increased, then bandwidth and rounds are randomly sampled to improve robustness. Through this training strategy, a single model can perform well under diverse communication conditions.

[0111] Based on the same inventive concept, this invention also provides a multi-round, multi-modal, multi-agent collaborative perception system based on a spatial confidence communication strategy, wherein the method described in the present invention includes:

[0112] The feature extraction module uses a deep convolutional neural network to extract features from the input data.

[0113] A spatial confidence generator that uses the data features to generate a spatial confidence map;

[0114] The communication module based on spatial confidence graph packs compact information packets, constructs a sparse communication graph, and transmits the information packets between agents through the communication graph;

[0115] The information fusion module based on spatial confidence maps is used to fuse the information packets obtained by the communication module with the local features of each agent.

[0116] The detection box decoding module decodes the fused features to obtain the detection results.

[0117] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the multi-round multimodal multi-agent collaborative perception method based on spatial confidence graph in the above embodiments, and will not be repeated here.

[0118] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, can be used to execute any of the methods described above, or to run the system described above.

[0119] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used for any of the methods described above, or for executing the systems described above.

[0120] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding modules, devices, units, etc. in the system. Those skilled in the art can implement the steps of the method by referring to the technical solution of the system. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0121] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0122] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence maps, characterized in that, include: Use deep convolutional neural networks to extract features from the input data; A spatial confidence map is generated using a spatial confidence generator on the aforementioned features; Using the communication module based on the spatial confidence graph, compact information packets are packaged to construct a sparse communication graph, and the information packets are transmitted between agents through the communication graph; The information fusion module based on spatial confidence graphs is used to fuse the information packets received by the communication module with the local features of each agent. The detection results are obtained by decoding the fused features; The input data includes RGB images or 3D point cloud images; The step of generating a spatial confidence map using a spatial confidence generator for the features includes: Using a network with the same structure as the detector decoder, a spatial confidence map is generated based on the extracted features at the k-th round of communication: Where, Φ generator (·) is the detection decoder. This is the spatial confidence graph at the k-th round of communication. These are the features of the i-th agent during the k-th round of communication; The method of using a communication module based on spatial confidence graphs to package compact information packets, construct a sparse communication graph, and transmit information packets between agents through the communication graph includes: Information packets are packaged based on spatial confidence maps. Each information packet consists of two parts: one part is feature information used to help other intelligent agents, and the other part is the information request map needed by each intelligent agent. Where k represents the communication round, The graph represents the information request of agent i. This represents the features sent from agent i to agent j. This represents the information packet sent from agent i to agent j; Information request graph of agent i in the k-th round of communication From the spatial confidence map Decide, Feature information sent from agent i to agent j in the k-th round of communication From the binary selection matrix and extracted features Decide, Where the binary choice matrix Using spatial confidence maps And information request diagram To select the spatial region with the most information among the features. In this context, ⊙ represents element-wise multiplication. This is the information request graph of the j-th agent in the previous round. The spatial confidence graph of the i-th agent, function Φ select (·) Select the most critical communication area based on the input matrix; Based on the selection matrix Construct a communication graph, represented as an adjacency matrix. in It is the directed communication edge between agent i and agent j in the k-th round of communication. The spatial dimension is consistent with the feature graph. h and w represent the length and width of the space, respectively. i is the sender and j is the receiver. 1 indicates that communication is established and 0 indicates that communication is not established.

2. The multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence graphs according to claim 1, characterized in that, The extraction of features from input data using a deep convolutional neural network includes: Use a deep convolutional neural network to process the input data from the i-th agent. Extracting features for Where Φ enc (·) is an encoding function implemented using a deep convolutional neural network. The superscript 0 indicates that the feature was obtained before communication. H, W, and D are the length, width, and channel dimension of the feature map, respectively.

3. The multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence graphs according to claim 1, characterized in that, The information fusion module using spatial confidence maps fuses the information packets received by the communication module with the local features of each agent, including: For the i-th agent, upon receiving a message from the j-th agent... Then, the features are obtained through decoding. Spatial confidence plot The fusion weight of the features of the i-th agent and the j-th agent during the k-th round of communication. for MHA W (·,·,·) is a multi-head attention network applied individually to each spatial location, with the input being the features of agent i. and the feature information of the j-th agent obtained by decoding from the received information packet. The output is the attention weights calculated based on feature similarity; Feature fusion and updating are performed based on the attention weights, and the feature map of the i-th agent after fusing information in the k-th round of communication is shown. for Here, FFN(·) is a feedforward network, implemented by a multilayer perceptron. It is communication diagram A (k) The neighbors of the i-th agent are defined as the set of agents that send information to agent i, and the fused features. Let be the features of the i-th agent in the (k+1)-th round of communication.

4. The multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence graphs according to claim 3, characterized in that, The step of decoding the fused features to obtain the detection result includes: Based on the assumption of the characteristics of the k-th round of communication Decoder Φ dec (·) Obtain the object frame: in This represents a rotating bounding box containing (c,x,y,h,w,cosα,sinα), representing an object with coordinates (x,y), width w, length l, rotation angle α, and object type c.

5. A multi-round, multi-modal, multi-agent collaborative perception method based on spatial confidence maps according to claim 4, characterized in that, Each round of communication uses a detection loss function to supervise object detection, which is: in, L is the real object bounding box of the i-th agent. det Here, k is the detection loss function, k is the number of rounds, and N is the number of agents.

6. A multi-round, multi-modal, multi-agent collaborative perception system based on a spatial confidence communication strategy, comprising the method described in any one of claims 1-5, characterized in that, include: The feature extraction module uses a deep convolutional neural network to extract features from the input data. A spatial confidence generator that uses the data features to generate a spatial confidence map; The communication module based on spatial confidence graph packs compact information packets, constructs a sparse communication graph, and transmits the information packets between agents through the communication graph; The information fusion module based on spatial confidence maps is used to fuse the information packets obtained by the communication module with the local features of each agent. The detection box decoding module decodes the fused features to obtain the detection results.

7. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-5, or to run the system of claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-5, or to perform the system of claim 6.

Images