Multi-agent cooperative sensing method and system based on vector quantization and entropy coding

Abstract

The invention provides a multi-agent collaborative perception method and system based on vector quantization and entropy coding, relates to the technical field of collaborative perception, introduces entropy coding technology, and after vector quantization is completed, applies entropy coding compression to a codeword index sequence generated by vector quantization, eliminates statistical redundancy in data, and further reduces communication data volume. A high-dimensional shared feature is greatly dimensionally reduced to a compact code word index sequence through vector quantization, and entropy coding is further used for performing deep lossless compression on code word indexes, so that an efficient dual compression mechanism is constructed. Therefore, on the premise that the same sensing precision is guaranteed, the required communication data volume is sharply reduced, and the communication bandwidth can be further reduced; or under the same bandwidth limitation, more accurate multi-agent cooperation can be supported, the communication efficiency and the sensing accuracy under the condition of the limited communication bandwidth are improved, and the communication overhead of the multi-agent cooperative sensing system is reduced under the condition that the performance is not lost as far as possible.

Description

Technical Field

[0001] This invention relates to the field of collaborative sensing technology, and in particular to a multi-agent collaborative sensing method and system based on vector quantization and entropy coding. Background Technology

[0002] The goal of multi-agent cooperative perception methods is to overcome the perception limitations of single agents in complex scenes and improve the comprehensiveness and accuracy of environmental understanding by sharing and complementing information among multiple agents. The sensors carried by a single agent have insufficient field of view and limited coverage, making it difficult to detect occluded or distant objects, easily leading to information omissions and even security risks. Multi-agent systems, by fusing observation data from various agents from different perspectives, can overcome the limitations of single-agent perception and recognition difficulties.

[0003] While multi-agent cooperative sensing methods offer significant advantages in improving the comprehensiveness and accuracy of environmental perception, they still face a series of technical challenges in practical applications, particularly the problem of efficient cooperative sensing under limited communication bandwidth. Multi-agent cooperative sensing methods rely on information exchange and fusion among multiple agents. However, this structure leads to a sharp increase in communication bandwidth requirements as the number of agents increases, which can easily cause communication delays and even network congestion under limited communication bandwidth conditions, thus affecting the system's real-time performance and stability.

[0004] Based on the type of information transmitted between multiple agents, cooperative sensing can be divided into three modes: early cooperation, mid-term cooperation, and late cooperation. Early cooperation involves sharing and fusing raw sensor data at the level of raw data transmission. While it transmits the most comprehensive information, it requires extremely high communication bandwidth. Mid-term cooperation involves exchanging data at the feature level, balancing the comprehensiveness of transmitted information with the communication bandwidth requirement. Late cooperation fuses the perception results independently completed by each agent. It has the lowest communication bandwidth requirement, but the transmitted information is limited and the fusion accuracy is constrained. Among these, the mid-term cooperation method not only reduces the volume of raw sensor data but also retains relatively rich semantic information, balancing the performance and communication bandwidth of multi-agent cooperative sensing systems, and has been widely used.

[0005] Recent research has further optimized the mid-term collaboration model by introducing a communication strategy learning module to achieve dynamic selection of communication content and objects. For example, the When2com communication framework proposes a learnable handshake mechanism, where agents determine when to communicate with other agents by assessing the uncertainty of their own perception, thereby dynamically constructing a communication graph; the Where2comm collaborative perception framework introduces a spatial confidence graph, where agents selectively share perception information based on the spatial importance of the perception task, focusing on key perception regions; and the CodeFilling collaborative perception framework proposes an information-filling-driven message selection strategy and introduces a codebook-based message representation method, enabling agents to transmit integer codes instead of high-dimensional feature maps.

[0006] While current cooperative sensing methods have improved communication efficiency, some shortcomings remain, failing to meet the demands for efficient information transmission under limited communication bandwidth. For example, the When2com and Where2comm cooperative sensing frameworks transmit continuous numerical features without effective discretization, resulting in significant communication bandwidth requirements. Although the CodeFilling cooperative sensing framework discretizes the codebook, it still exhibits high communication bandwidth demands, leading to high communication overhead. Summary of the Invention

[0007] This invention provides a multi-agent collaborative sensing method and system based on vector quantization and entropy coding to address the deficiencies in related technologies.

[0008] This invention provides a multi-agent cooperative perception method based on vector quantization and entropy coding, applied to a central agent, comprising: Obtain the raw observation data of the central intelligent agent; Based on the feature extraction module, shared features of the original observation data are extracted; Based on the vector quantization module, the shared features are vector quantized using the target codebook to obtain the codeword index sequence. Based on the entropy coding module, the codeword index sequence is entropy encoded to obtain the first binary data, and the first binary data is sent to the cooperative intelligent agent; The system receives the second binary data sent by the collaborative agent, and performs entropy decoding and dequantization on the second binary data sequentially based on the entropy decoding and dequantization module to obtain the reconstructed features. Based on the collaborative perception module, the shared features and the reconstructed features are fused to obtain a fusion result, and target detection is performed based on the fusion result.

[0009] According to the multi-agent cooperative perception method based on vector quantization and entropy coding provided by the present invention, the training steps of the feature extraction module and the cooperative perception module include: Obtain observation data samples from the central intelligent agent; Based on the initial feature extraction module, the first shared feature sample of the observed data sample is extracted; The system receives a second shared feature sample sent by the collaborative intelligent agent, performs feature fusion on the first shared feature sample and the second shared feature sample based on the initial collaborative perception module to obtain a first fused sample, and performs target detection based on the first fused sample to obtain a first detection result. Based on the first detection result and the target annotation information in the observation data sample, the initial feature extraction module and the initial collaborative perception module are jointly trained to obtain the feature extraction module and the collaborative perception module.

[0010] According to the multi-agent cooperative perception method based on vector quantization and entropy coding provided by the present invention, the training steps of the vector quantization module include: The observation data samples of the central agent are obtained, and the third shared feature samples of the observation data samples are extracted based on the feature extraction module. Based on the initial vector quantization module, the third shared feature sample is vector quantized using the initial codebook to obtain codeword index sequence samples; Based on the entropy coding module, the codeword index sequence sample is entropy encoded to obtain a first binary data sample, and the first binary data sample is sent to the cooperative intelligent agent; The system receives a second binary data sample sent by the collaborative agent, and performs entropy decoding and dequantization on the second binary data sample sequentially based on the entropy decoding and dequantization module to obtain a reconstructed feature sample. Based on the collaborative perception module, feature fusion is performed on the third shared feature sample and the reconstructed feature sample to obtain a second fused sample, and target detection is performed based on the second fused sample to obtain a second detection result; Based on the second detection result, the target annotation information in the observation data sample, the third shared feature sample, and the reconstructed feature sample, the initial vector quantization module is trained using a gradient pass-through estimation strategy, and the initial codebook is updated to obtain the vector quantization module and the target codebook.

[0011] According to the present invention, a multi-agent cooperative perception method based on vector quantization and entropy coding is provided, wherein the initial vector quantization module is trained and the initial codebook is updated using a gradient pass-through estimation strategy based on the second detection result, the target annotation information in the observed data sample, the third shared feature sample, and the reconstructed feature sample, to obtain the vector quantization module and the target codebook, including: The gradient stopping operation is performed on the third shared feature sample and the reconstructed feature sample respectively to obtain the first operation result and the second operation result; Based on the result of the first operation and the reconstructed feature samples, calculate the codebook update loss; Based on the result of the second operation and the third shared feature sample, the commitment loss is calculated; Based on the second detection result and the target annotation information in the observation data sample, the main task loss is calculated; Based on the codebook update loss, the commitment loss, and the main task loss, the total training loss is calculated. Based on the total training loss, the initial vector quantization module is trained, and the initial codebook is updated to obtain the vector quantization module and the target codebook.

[0012] According to the present invention, a multi-agent collaborative sensing method based on vector quantization and entropy coding is provided, wherein the feature extraction module extracts shared features from the original observation data, including: Based on the feature extraction module, bird's-eye view features of the original observation data are extracted; Based on the target space confidence map, the features of the bird's-eye view are filtered to determine the shared features.

[0013] According to the present invention, a multi-agent cooperative perception method based on vector quantization and entropy coding is provided, wherein the target space confidence map includes a space confidence map generated by the target detection task of the central agent and a space confidence map generated by the target detection task of the cooperative agents and transmitted to the central agent.

[0014] The multi-agent cooperative sensing method based on vector quantization and entropy coding provided by the present invention further includes: The agent information of the central agent is broadcast to the collaborative agent so that the collaborative agent can transform its original observation data into the coordinate system of the central agent based on the agent information.

[0015] This invention also provides a multi-agent cooperative perception system based on vector quantization and entropy coding, applied to a central agent, comprising: The data acquisition unit is used to acquire the raw observation data of the central intelligent agent; The feature extraction unit is used to extract shared features from the original observation data based on the feature extraction module. The vector quantization unit is used to perform vector quantization on the shared features based on the vector quantization module and the target codebook to obtain the codeword index sequence. An entropy coding unit is used to entropy code the codeword index sequence based on the entropy coding module to obtain first binary data, and then send the first binary data to the cooperative intelligent agent. The inverse operation unit is used to receive the second binary data sent by the cooperative agent, and based on the entropy decoding and inverse quantization module, sequentially perform entropy decoding and inverse quantization on the second binary data to obtain the reconstructed features. The collaborative perception unit is used to perform feature fusion on the shared features and the reconstructed features based on the collaborative perception module to obtain a fusion result, and to perform target detection based on the fusion result.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-agent cooperative perception method based on vector quantization and entropy coding as described above.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-agent cooperative perception method based on vector quantization and entropy coding as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-agent cooperative perception method based on vector quantization and entropy coding as described above.

[0019] This invention provides a multi-agent collaborative sensing method and system based on vector quantization and entropy coding. It introduces entropy coding technology, and after vector quantization, further applies entropy coding compression to the codeword index sequence generated by vector quantization, eliminating statistical redundancy in the data and further reducing the amount of communication data. Vector quantization significantly reduces the dimensionality of high-dimensional shared features to a compact codeword index sequence, and entropy coding is further used to perform deep lossless compression on these codeword indices, constructing an efficient dual compression mechanism. This allows for a sharp reduction in the amount of communication data required while maintaining the same sensing accuracy, further reducing communication bandwidth; or, under the same bandwidth constraints, it can support more accurate multi-agent collaboration, improving communication efficiency and sensing accuracy under limited communication bandwidth conditions, and reducing the communication overhead of the multi-agent collaborative sensing system without sacrificing performance as much as possible. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the multi-agent collaborative perception method based on vector quantization and entropy coding provided by the present invention.

[0022] Figure 2 This is the second flowchart of the multi-agent collaborative perception method based on vector quantization and entropy coding provided by the present invention.

[0023] Figure 3 This is a flowchart illustrating the phased training strategy in the multi-agent collaborative perception method based on vector quantization and entropy coding provided by this invention.

[0024] Figure 4 This is a schematic diagram of the structure of the multi-agent collaborative perception system based on vector quantization and entropy coding provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] Figure 1 This is a flowchart illustrating a multi-agent collaborative perception method based on vector quantization and entropy coding provided in an embodiment of the present invention. This method is applied to a central agent. Figure 1 As shown, the method includes: S1, Obtain the raw observation data of the central intelligent agent; S2, Based on the feature extraction module, extract the shared features of the original observation data; S3, based on the vector quantization module, using the target codebook, the shared features are vector quantized to obtain the codeword index sequence; S4, based on the entropy coding module, the codeword index sequence is entropy encoded to obtain the first binary data, and the first binary data is sent to the cooperative intelligent agent; S5, receive the second binary data sent by the collaborative intelligent agent, and perform entropy decoding and dequantization on the second binary data in sequence based on the entropy decoding and dequantization module to obtain the reconstructed features; S6. Based on the collaborative perception module, feature fusion is performed on the shared features and the reconstructed features to obtain a fusion result, and target detection is performed based on the fusion result.

[0028] Specifically, the multi-agent collaborative perception method based on vector quantization and entropy coding provided in this embodiment of the invention is executed by a multi-agent collaborative perception system based on vector quantization and entropy coding. This system can be configured within each agent in the multi-agent system. Here, the agent can be a drone, vehicle, industrial robot, environmental monitoring equipment, etc., without specific limitations.

[0029] It is understood that the central agent is each agent in the multi-agent system, meaning that each agent in the multi-agent system acts as the central agent to execute the multi-agent collaborative perception method based on vector quantization and entropy coding provided in this embodiment of the invention. In this case, the other agents in the multi-agent system act as collaborating agents with the central agent. Each agent can be equipped with a sensor, such as a camera or radar, to collect raw observation data such as image data and point cloud data.

[0030] First, step S1 is executed to obtain the raw observation data of the central agent. This raw observation data may be image data and / or point cloud data collected by the sensors carried by the central agent.

[0031] Then, step S2 is executed. A collaborative perception model can be configured on the central agent. This collaborative perception model may include a feature extraction module, a vector quantization module, an entropy encoding module, an entropy decoding and dequantization module, and a collaborative perception module. Among them, the feature extraction module can extract shared features from the original observation data of the central agent. Shared features refer to the data features extracted from the original observation data of the central agent that need to be sent to the collaborative agent. These features can be obtained by extracting and filtering features from the original observation data.

[0032] Next, step S3 is executed, using the vector quantization module and the target codebook to perform vector quantization on the shared features, obtaining a codeword index sequence. Here, the target codebook is a codebook shared by all agents and may include multiple codewords. For example, the target codebook can be represented as... , Let C be the k-th codeword in the target codebook, C be the dimension of each codeword in the target codebook, and K be the total number of codewords in the target codebook.

[0033] In the vector quantization stage, each feature vector in the shared features is mapped to the codeword in the target codebook with the smallest Euclidean distance. Here, the vector quantization process can be represented as: ; in, This represents the i-th feature vector among the shared features of the central agent a. H and W represent the width and height of the shared feature. This is the codeword index corresponding to the i-th feature vector in the shared features.

[0034] Through the vector quantization process of the vector quantization module, high-dimensional shared features in continuous space can be compressed and discretized to achieve efficient encoding, significantly reducing the amount of data transmission while ensuring key perceptual information.

[0035] Next, step S4 is executed, using the entropy coding module to entropy code the codeword index sequence to obtain the first binary data. Here, based on the vector-quantized codeword index sequence, an entropy coding mechanism is further introduced to exploit statistical redundancy in the data and achieve more efficient compression. By probabilistically modeling the codeword index sequence, the frequency of occurrence of each codeword in the codeword index sequence is statistically analyzed, and then the entropy value of the codeword index sequence is calculated. ; in, Let N be the codeword index sequence, N be the total number of codeword indices in the codeword index sequence, and H(Q) be the entropy value of the codeword index sequence. is the frequency of the k-th codeword in the codeword index sequence.

[0036] The entropy value of a codeword index sequence can measure the theoretical minimum code length of the index sequence. Using entropy-based coding methods such as Huffman coding or arithmetic coding, short codes can be assigned to frequently occurring codewords, and long codes can be assigned to infrequently occurring codewords, achieving lossless compression. This makes the average code length as close as possible to the entropy value, thereby further reducing communication costs.

[0037] Subsequently, the central agent can send the first binary data obtained by entropy encoding to the collaborative agent, so that the collaborative agent can act as the central agent and perform target detection based on the received binary data to achieve collaborative perception.

[0038] Then, step S5 is executed to receive the second binary data sent by the collaborative agent. The process by which the collaborative agent determines the second binary data is the same as the process by which the central agent determines the first binary data. See the above process for details, which will not be repeated here.

[0039] Furthermore, the central agent, as the data receiver, receives second binary data that has undergone vector quantization and entropy encoding operations in sequence. In order to recover the original shared features from the second binary data, the entropy decoding and dequantization module can be used to perform entropy decoding and dequantization on the second binary data in sequence to obtain the reconstructed features.

[0040] During entropy decoding, a probabilistic model identical to that used for the encoding of the second binary data is employed for parsing. This inverse process allows for lossless reconstruction of the original codeword index sequence, ensuring the integrity of the codeword index during compression and transmission.

[0041] During the dequantization operation, the target codebook shared with the sending end needs to be accessed. For each index in entropy decoding, its corresponding codeword is searched and extracted from the target codebook. By rearranging these codewords according to their corresponding spatial positions in the shared features, the reconstructed features can be obtained.

[0042] Finally, step S6 is executed, which uses the collaborative perception module to perform feature fusion on shared features and reconstructed features to enhance perception capabilities and obtain fusion results.

[0043] The feature fusion process can be represented as: ; in, The fusion result obtained by the central agent a Represents the fusion function. Let A be the set of neighbors of the central agent A, that is, the set of cooperating agents. For the shared characteristics of the raw observation data of the central agent a, The reconstructed features are the second binary data sent from the b-th collaborative agent to the central agent a.

[0044] Here, the feature fusion strategy can employ simple weighted averaging, concatenation, or more complex attention mechanisms, without any specific limitations.

[0045] After feature fusion is completed, the process moves to the object detection stage. The main task of this stage is to identify and locate objects in the environment, providing support for subsequent decision-making. Object detection yields the class probability and bounding box parameters for each object in the environment.

[0046] like Figure 2As shown, the raw data (i.e., raw observation data) of each agent are processed by each agent to extract and filter features, thereby obtaining the shared features corresponding to each agent; Each agent performs vector quantization and entropy encoding on its shared features to obtain the binary data of each agent, and exchanges information, i.e., data sharing, so that each agent can master the binary data of the collaborating agents. After receiving binary data from the collaborating agents, each agent performs entropy decoding and dequantization to obtain reconstructed features, and then fuses these features with its own shared features to obtain a fusion result. The fusion result is then used for target detection to obtain the detection result.

[0047] The multi-agent collaborative perception method based on vector quantization and entropy coding provided in this embodiment of the invention is applied to a central agent. First, the original observation data of the central agent is acquired. Then, based on a feature extraction module, shared features of the original observation data are extracted. Based on a vector quantization module, using a target codebook, the shared features are vector-quantized to obtain a codeword index sequence. Based on an entropy coding module, the codeword index sequence is entropy-coded to obtain first binary data, which is then sent to the collaborative agent. The second binary data sent by the collaborative agent is received, and based on an entropy decoding and dequantization module, the second binary data is sequentially entropy-decoded and dequantized to obtain reconstructed features. Finally, based on a collaborative perception module, the shared features and reconstructed features are fused to obtain a fusion result, and target detection is performed based on the fusion result. This method introduces entropy coding technology. After completing vector quantization, entropy coding compression is applied to the codeword index sequence generated by vector quantization, eliminating statistical redundancy in the data and further reducing the amount of communication data. By significantly reducing the dimensionality of high-dimensional shared features to a compact codeword index sequence through vector quantization, and further using entropy coding for deep lossless compression of these codeword indices, an efficient dual compression mechanism is constructed. This allows for a sharp reduction in the amount of communication data required while maintaining the same perception accuracy, thereby further reducing communication bandwidth; or, under the same bandwidth constraints, it enables more precise multi-agent collaboration, improving communication efficiency and perception accuracy under limited communication bandwidth conditions, and reducing the communication overhead of multi-agent collaborative perception systems without sacrificing performance as much as possible.

[0048] Based on the above embodiments, the training steps for the feature extraction module and the collaborative perception module include: Obtain observation data samples from the central intelligent agent; Based on the initial feature extraction module, the first shared feature sample of the observed data sample is extracted; The system receives a second shared feature sample sent by the collaborative intelligent agent, performs feature fusion on the first shared feature sample and the second shared feature sample based on the initial collaborative perception module to obtain a first fused sample, and performs target detection based on the first fused sample to obtain a first detection result. Based on the first detection result and the target annotation information in the observation data sample, the initial feature extraction module and the initial collaborative perception module are jointly trained to obtain the feature extraction module and the collaborative perception module.

[0049] Specifically, in this embodiment of the invention, the entropy encoding module and the entropy decoding and inverse quantization module in the collaborative perception model do not need to be trained, while the feature extraction module, the vector quantization module, and the collaborative perception module can be trained in two stages.

[0050] In the first training phase, the feature extraction module and the collaborative perception module are trained first. During the training of the feature extraction module and the collaborative perception module, it is not necessary to perform vector quantization and dequantization of features; instead, shared feature samples are directly transmitted between the agents.

[0051] First, the observation data samples of the central agent are obtained. The observation data samples may correspond to target annotation information, which may include the target's category label and bounding box label.

[0052] The initial feature extraction module extracts the first shared feature sample of the observation data sample and can send the first shared feature sample to the collaborative agent so that the collaborative agent can use the received first shared feature sample in combination with its own second shared feature sample to perform target detection and achieve collaborative perception.

[0053] The central agent receives the second shared feature sample sent by the collaborative agent, uses the initial collaborative perception module to fuse the first shared feature sample and the second shared feature sample to obtain the first fused sample, and uses the first fused sample to perform target detection to obtain the first detection result.

[0054] Using the first detection result and the target annotation information in the observation data sample, the training loss is calculated. The initial feature extraction module and the initial collaborative perception module are jointly trained by the training loss. The structural parameters of the initial feature extraction module and the initial collaborative perception module are iteratively updated. When the training loss converges or reaches the preset number of iterations, the feature extraction module and the collaborative perception module are obtained.

[0055] In this embodiment of the invention, by first training the feature extraction module and the collaborative perception module, the stability and efficiency of discrete target codebook learning can be significantly improved, ensuring that the training process of the subsequent vector quantization module is more stable, thereby obtaining a stable discrete codeword index sequence.

[0056] Based on the above embodiments, the training steps of the vector quantization module include: The observation data samples of the central agent are obtained, and the third shared feature samples of the observation data samples are extracted based on the feature extraction module. Based on the initial vector quantization module, the third shared feature sample is vector quantized using the initial codebook to obtain codeword index sequence samples; Based on the entropy coding module, the codeword index sequence sample is entropy encoded to obtain a first binary data sample, and the first binary data sample is sent to the cooperative intelligent agent; The system receives a second binary data sample sent by the collaborative agent, and performs entropy decoding and dequantization on the second binary data sample sequentially based on the entropy decoding and dequantization module to obtain a reconstructed feature sample. Based on the collaborative perception module, feature fusion is performed on the third shared feature sample and the reconstructed feature sample to obtain a second fused sample, and target detection is performed based on the second fused sample to obtain a second detection result; Based on the second detection result, the target annotation information in the observation data sample, the third shared feature sample, and the reconstructed feature sample, the initial vector quantization module is trained using a gradient pass-through estimation strategy, and the initial codebook is updated to obtain the vector quantization module and the target codebook.

[0057] Specifically, the training steps for the vector quantization module may include: Obtain observation data samples from the central intelligent agent.

[0058] The feature extraction module extracts the third shared feature samples from the observed data samples. Then, the initial vector quantization module, using an initial codebook, performs vector quantization on these third shared feature samples to obtain codeword index sequence samples. Here, the initial codebook can be obtained through random initialization.

[0059] Entropy coding module is used to entropy code the codeword index sequence sample to obtain the first binary data sample. The first binary data sample is then sent to the cooperative agent so that the cooperative agent can use the received first binary data sample in combination with its own second binary data sample to perform target detection and achieve cooperative perception.

[0060] The central agent receives the second binary data sample sent by the collaborative agent, and uses the entropy decoding and dequantization module to sequentially perform entropy decoding and dequantization on the second binary data sample to obtain the reconstructed feature sample.

[0061] Subsequently, the collaborative perception module is used to fuse the third shared feature sample and the reconstructed feature sample to obtain the second fused sample. The second fused sample is then used for target detection to obtain the second detection result.

[0062] Because vector quantization is discrete, its gradient cannot be directly calculated, and the collaborative sensing model cannot directly perform backpropagation to update the gradient. Therefore, in this embodiment of the invention, a Straight-Through Estimator (STE) strategy is adopted during training, enabling the vector quantization process to be embedded into the neural network for end-to-end optimization. During the backpropagation stage, the gradient of the quantization operation is treated as an identity mapping, and the gradient is directly copied from the reconstructed feature sample to the third shared feature sample, ignoring the discontinuity of the quantization operation. This allows the entire collaborative sensing model to be trained using the standard backpropagation algorithm.

[0063] Finally, using the second detection result, the target annotation information in the observation data sample, the third shared feature sample, and the reconstructed feature sample, a gradient pass-through estimation strategy is adopted to calculate the training loss. The training loss is then used to train the initial vector quantization module and update the initial codebook, resulting in the vector quantization module and the target codebook.

[0064] In this embodiment of the invention, gradient pass-through estimation technology and a phased training strategy are employed to effectively alleviate the gradient propagation and optimization challenges brought about by vector quantization operations in end-to-end training. This not only ensures the stable convergence of the entire collaborative sensing model training process but also helps to learn a high-quality quantized codebook, thereby guaranteeing the quality of the deeply compressed sensing information and the overall sensing performance of the system.

[0065] Based on the above embodiments, the step of training the initial vector quantization module and updating the initial codebook using a gradient pass-through estimation strategy based on the second detection result, the target annotation information in the observation data sample, the third shared feature sample, and the reconstructed feature sample to obtain the vector quantization module and the target codebook includes: The gradient stopping operation is performed on the third shared feature sample and the reconstructed feature sample respectively to obtain the first operation result and the second operation result; Based on the result of the first operation and the reconstructed feature samples, calculate the codebook update loss; Based on the result of the second operation and the third shared feature sample, the commitment loss is calculated; Based on the second detection result and the target annotation information in the observation data sample, the main task loss is calculated; Based on the codebook update loss, the commitment loss, and the main task loss, the total training loss is calculated. Based on the total training loss, the initial vector quantization module is trained, and the initial codebook is updated to obtain the vector quantization module and the target codebook.

[0066] Specifically, during the training of the initial vector quantization module and the updating of the initial codebook, the total training loss is a weighted sum of the codebook update loss, commitment loss, and main task loss based on the VQ-VAE loss function, which can be expressed as: ; in, For total training losses, The main task loss includes the losses caused by the feature extraction module and the collaborative perception module. This indicates stopping the gradient operation (stop-gradient). and These are all hyperparameters, representing the weights of the codebook update loss and the commitment loss, respectively. This is the result of the first operation. To reconstruct feature samples, This is the result of the second operation. Due to codebook update losses, To commit to losses.

[0067] In this embodiment of the invention, the total training loss ensures both effective codebook updates and avoids rapid drift between features and codewords, which is beneficial for stable model convergence.

[0068] Based on the above embodiments, the step of extracting shared features from the original observation data using the feature extraction module includes: Based on the feature extraction module, bird's-eye view features of the original observation data are extracted; Based on the target space confidence map, the features of the bird's-eye view are filtered to determine the shared features.

[0069] Specifically, in this embodiment of the invention, the feature extraction module can extract bird's-eye view features from the original observation data, which are the original features of the original observation data. If the original observation data of the central agent a is X, its bird's-eye view feature is F.

[0070] Subsequently, using the target space confidence map, a binary selection matrix M is calculated. Then, by multiplying the binary selection matrix M by F pixel by pixel, the bird's-eye view features F are filtered to obtain the shared features of the central agent. ,in This indicates element-wise multiplication.

[0071] Optionally, the binary selection matrix M can be calculated using some existing methods. For example, the information filtering method proposed by Where2comm can be used: the central agent a generates a spatial confidence map. and request diagram and utilize feature selection functions Generate selection matrix , Let be the request graph for the b-th collaborative agent. The feature selection function can... Locations with a spatial confidence level higher than a preset threshold are assigned a value of 1, while other locations are assigned a value of 0. This preset threshold can be set as needed, and no specific limit is set here.

[0072] The binary selection matrix M can also employ a code-filling information-driven strategy. The central agent collects the spatial confidence maps of the collaborating agents, and then, using a method similar to spatial jigsaw puzzles, selectively fuses the spatial confidence scores of the collaborating agents at each discrete spatial unit. The fusion process involves accumulating the spatial confidence scores at corresponding spatial units until a preset information requirement threshold is met. Based on the result of this selective fusion, a binary selection matrix is ​​generated.

[0073] At this point, the target space confidence map used includes the space confidence map generated by the target detection task of the central agent and the space confidence map generated by the target detection task of the cooperative agent and transmitted to the central agent.

[0074] Based on the above embodiments, it also includes: The agent information of the central agent is broadcast to the collaborative agent so that the collaborative agent can transform its original observation data into the coordinate system of the central agent based on the agent information.

[0075] Specifically, the agent information may include the location information and identification information of the central agent. After receiving the agent information from the central agent, the collaborating agent can transform its own raw observation data into the coordinate system of the central agent to ensure the feasibility of subsequent feature fusion.

[0076] like Figure 3 As shown, in the phased training strategy, the central agent initiates communication first, broadcasting agent information to the collaborating agents, enabling the collaborating agents to perform coordinate transformation on their own observation data samples.

[0077] Subsequently, each agent, including the central agent and the collaborating agents, extracts shared feature samples from its own observation data samples.

[0078] First, the feature extraction module and the collaborative perception module are trained. At this time, the vector quantization module has not yet been trained. The coordinating agent directly sends the second shared feature sample extracted by the initial feature extraction module to the central agent.

[0079] The central agent fuses the first shared feature sample extracted by the initial feature extraction module with the second shared feature sample to obtain the first fused sample. Based on the first fused sample, it performs target detection to obtain the first detection result. While the training process is not yet complete, it calculates the training loss using the first detection result and the target annotation information in the observation data sample, and backpropagates it to jointly train the initial feature extraction module and the initial collaborative perception module until the feature extraction module and the collaborative perception module are obtained.

[0080] Then the vector quantization module is trained. Each agent needs to use the initial vector quantization module to perform vector quantization on the shared feature samples extracted by the feature extraction module and entropy encoding to obtain binary data samples. The central agent sends the first binary data sample to the coordinating agent, and the coordinating agent sends the second binary data sample to the central agent.

[0081] The central agent uses the entropy decoding and dequantization module to sequentially perform entropy decoding and dequantization on the second binary data sample to obtain reconstructed feature samples. Then, using the collaborative perception module, it performs feature fusion on the third shared feature sample extracted by the feature extraction module and the reconstructed feature sample to obtain a second fused sample. Based on the second fused sample, it performs target detection to obtain a second detection result.

[0082] While the training process is not yet complete, the total training loss is calculated using the second detection result, target annotation information in the observed data samples, third shared feature samples, and reconstructed feature samples, employing a gradient pass-through estimation strategy. Backpropagation is then used to train the initial vector quantization module and update the initial codebook until the vector quantization module and target codebook are obtained. At this point, the training process ends, and the collaborative perception model is obtained.

[0083] In summary, by introducing an efficient feature compression mechanism and a stable phased training strategy, the embodiments of the present invention significantly improve the communication efficiency and perception accuracy of the multi-agent collaborative perception system under limited communication bandwidth conditions, providing strong support for real-time perception and decision-making in practical applications.

[0084] like Figure 4 As shown, based on the above embodiments, this embodiment of the invention provides a multi-agent collaborative perception system based on vector quantization and entropy coding, applied to a central agent, comprising: Data acquisition unit 41 is used to acquire the raw observation data of the central intelligent agent; Feature extraction unit 42 is used to extract shared features from the original observation data based on the feature extraction module; Vector quantization unit 43 is used to perform vector quantization on the shared features based on the vector quantization module and using the target codebook to obtain a codeword index sequence. Entropy coding unit 44 is used to entropy code the codeword index sequence based on the entropy coding module to obtain first binary data, and send the first binary data to the cooperative intelligent agent; The reverse operation unit 45 is used to receive the second binary data sent by the cooperative agent, and based on the entropy decoding and dequantization module, sequentially perform entropy decoding and dequantization on the second binary data to obtain the reconstructed features; The collaborative perception unit 46 is used to perform feature fusion on the shared features and the reconstructed features based on the collaborative perception module to obtain a fusion result, and to perform target detection based on the fusion result.

[0085] Based on the above embodiments, this invention provides a multi-agent cooperative perception system based on vector quantization and entropy coding, which further includes a training unit for: Obtain observation data samples from the central intelligent agent; Based on the initial feature extraction module, the first shared feature sample of the observed data sample is extracted; The system receives a second shared feature sample sent by the collaborative intelligent agent, performs feature fusion on the first shared feature sample and the second shared feature sample based on the initial collaborative perception module to obtain a first fused sample, and performs target detection based on the first fused sample to obtain a first detection result. Based on the first detection result and the target annotation information in the observation data sample, the initial feature extraction module and the initial collaborative perception module are jointly trained to obtain the feature extraction module and the collaborative perception module.

[0086] Based on the above embodiments, this invention provides a multi-agent cooperative perception system based on vector quantization and entropy coding, wherein the training unit is further used for: The observation data samples of the central agent are obtained, and the third shared feature samples of the observation data samples are extracted based on the feature extraction module. Based on the initial vector quantization module, the third shared feature sample is vector quantized using the initial codebook to obtain codeword index sequence samples; Based on the entropy coding module, the codeword index sequence sample is entropy encoded to obtain a first binary data sample, and the first binary data sample is sent to the cooperative intelligent agent; The system receives a second binary data sample sent by the collaborative agent, and performs entropy decoding and dequantization on the second binary data sample sequentially based on the entropy decoding and dequantization module to obtain a reconstructed feature sample. Based on the collaborative perception module, feature fusion is performed on the third shared feature sample and the reconstructed feature sample to obtain a second fused sample, and target detection is performed based on the second fused sample to obtain a second detection result; Based on the second detection result, the target annotation information in the observation data sample, the third shared feature sample, and the reconstructed feature sample, the initial vector quantization module is trained using a gradient pass-through estimation strategy, and the initial codebook is updated to obtain the vector quantization module and the target codebook.

[0087] Based on the above embodiments, this invention provides a multi-agent cooperative perception system based on vector quantization and entropy coding, wherein the training unit is specifically used for: The gradient stopping operation is performed on the third shared feature sample and the reconstructed feature sample respectively to obtain the first operation result and the second operation result; Based on the result of the first operation and the reconstructed feature samples, calculate the codebook update loss; Based on the result of the second operation and the third shared feature sample, the commitment loss is calculated; Based on the second detection result and the target annotation information in the observation data sample, the main task loss is calculated; Based on the codebook update loss, the commitment loss, and the main task loss, the total training loss is calculated. Based on the total training loss, the initial vector quantization module is trained, and the initial codebook is updated to obtain the vector quantization module and the target codebook.

[0088] Based on the above embodiments, this invention provides a multi-agent collaborative perception system based on vector quantization and entropy coding, wherein the feature extraction unit is specifically used for: Based on the feature extraction module, bird's-eye view features of the original observation data are extracted; Based on the target space confidence map, the features of the bird's-eye view are filtered to determine the shared features.

[0089] Based on the above embodiments, this invention provides a multi-agent collaborative perception system based on vector quantization and entropy coding. The target space confidence map includes a space confidence map generated by the target detection task of the central agent and a space confidence map generated by the target detection task of the collaborative agents and transmitted to the central agent.

[0090] Based on the above embodiments, this invention provides a multi-agent cooperative perception system based on vector quantization and entropy coding, which also includes a broadcast module for: The agent information of the central agent is broadcast to the collaborative agent so that the collaborative agent can transform its original observation data into the coordinate system of the central agent based on the agent information.

[0091] Specifically, the functions of each module in the multi-agent collaborative perception system based on vector quantization and entropy coding provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-mentioned method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0092] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the multi-agent cooperative perception method based on vector quantization and entropy coding provided in the above embodiments.

[0093] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multi-agent cooperative perception method based on vector quantization and entropy coding provided in the above embodiments.

[0095] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the multi-agent cooperative perception method based on vector quantization and entropy coding provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent collaborative perception method based on vector quantization and entropy coding, characterized in that, The application is applied to a central agent, comprising: obtaining original observation data of the central agent; extracting shared features of the original observation data based on a feature extraction module; based on a vector quantization module, using a target codebook to perform vector quantization on the shared features to obtain a code word index sequence; based on an entropy encoding module, performing entropy encoding on the code word index sequence to obtain first binary data, and sending the first binary data to a collaborative agent; receiving second binary data sent by the collaborative agent, and sequentially performing entropy decoding and inverse quantization on the second binary data based on an entropy decoding and inverse quantization module to obtain reconstructed features; based on a collaborative perception module, performing feature fusion on the shared features and the reconstructed features to obtain a fusion result, and based on the fusion result, performing target detection.

2. The multi-agent collaborative perception method based on vector quantization and entropy coding according to claim 1, characterized in that, The training steps of the feature extraction module and the collaborative perception module include: obtaining observation data samples of the central agent; extracting first shared feature samples of the observation data samples based on an initial feature extraction module; receiving second shared feature samples sent by the collaborative agent, performing feature fusion on the first shared feature samples and the second shared feature samples based on an initial collaborative perception module to obtain first fusion samples, and based on the first fusion samples, performing target detection to obtain a first detection result; based on the first detection result and target annotation information in the observation data samples, jointly training the initial feature extraction module and the initial collaborative perception module to obtain the feature extraction module and the collaborative perception module.

3. The multi-agent collaborative perception method based on vector quantization and entropy coding according to claim 1, characterized in that, The training steps of the vector quantization module include: obtaining observation data samples of the central agent, and extracting third shared feature samples of the observation data samples based on the feature extraction module; based on an initial vector quantization module, using an initial codebook to perform vector quantization on the third shared feature samples to obtain code word index sequence samples; based on the entropy encoding module, performing entropy encoding on the code word index sequence samples to obtain first binary data samples, and sending the first binary data samples to a collaborative agent; receiving second binary data samples sent by the collaborative agent, and sequentially performing entropy decoding and inverse quantization on the second binary data samples based on the entropy decoding and inverse quantization module to obtain reconstructed feature samples; based on the collaborative perception module, performing feature fusion on the third shared feature samples and the reconstructed feature samples to obtain second fusion samples, and based on the second fusion samples, performing target detection to obtain a second detection result; based on the second detection result, target annotation information in the observation data samples, the third shared feature samples and the reconstructed feature samples, using a gradient pass estimation strategy to train the initial vector quantization module, and updating the initial codebook to obtain the vector quantization module and the target codebook.

4. The multi-agent collaborative perception method based on vector quantization and entropy coding according to claim 3, characterized in that, The gradient pass estimation strategy is adopted based on the second detection result, target label information in the observation data sample, the third shared feature sample, and the reconstructed feature sample to train the initial vector quantization module and update the initial codebook to obtain the vector quantization module and the target codebook, including: The stop gradient operation is performed on the third shared feature sample and the reconstructed feature sample respectively to obtain first operation result and second operation result; The codebook update loss is calculated based on the first operation result and the reconstructed feature sample; The commitment loss is calculated based on the second operation result and the third shared feature sample; The main task loss is calculated based on the second detection result and the target label information in the observation data sample; The total training loss is calculated based on the codebook update loss, the commitment loss, and the main task loss, and the initial vector quantization module is trained and the initial codebook is updated based on the total training loss to obtain the vector quantization module and the target codebook.

5. The multi-agent collaborative perception method based on vector quantization and entropy coding according to claim 1, characterized in that, The shared feature of the original observation data is extracted based on the feature extraction module, including: The bird's eye view feature of the original observation data is extracted based on the feature extraction module; The shared feature is determined by screening the bird's eye view feature based on the target spatial confidence map.

6. The multi-agent collaborative perception method based on vector quantization and entropy coding according to claim 5, characterized in that, The target spatial confidence map includes a spatial confidence map generated by the target detection task of the center agent and a spatial confidence map generated by the target detection task of the collaborative agent and transmitted to the center agent.

7. The vector quantization and entropy coding based multi-agent collaborative perception method according to any one of claims 1-6, characterized in that, Further comprising: The agent information of the center agent is broadcasted to the collaborative agent, so that the collaborative agent converts the original observation data of the collaborative agent to the coordinate system of the center agent based on the agent information.

8. A multi-agent collaborative perception system based on vector quantization and entropy coding, characterized in that, Applied to the center agent, including: The data acquisition unit is configured to acquire the original observation data of the center agent; The feature extraction unit is configured to extract the shared feature of the original observation data based on the feature extraction module; The vector quantization unit is configured to perform vector quantization on the shared feature based on the vector quantization module and the target codebook to obtain a code word index sequence; The entropy encoding unit is configured to perform entropy encoding on the code word index sequence based on the entropy encoding module to obtain first binary data, and send the first binary data to the collaborative agent; The inverse operation unit is configured to receive second binary data sent by the collaborative agent, and perform entropy decoding and inverse quantization on the second binary data in sequence based on the entropy decoding inverse quantization module to obtain reconstructed features; The collaborative perception unit is configured to perform feature fusion on the shared feature and the reconstructed feature based on the collaborative perception module to obtain a fusion result, and perform target detection based on the fusion result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the multi-agent collaborative perception method based on vector quantization and entropy coding according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the multi-agent collaborative perception method based on vector quantization and entropy coding according to any one of claims 1-7.

Images