An instance-level semantic interaction method for intelligent collaborative perception

Abstract

The application provides an instance-level semantic interaction method for intelligent collaborative perception, and is used in the technical field of automatic driving. The method is characterized in that a perception model, a decoupling module, a fusion module and a detection module are arranged on each agent in an application scenario; sensor data collected by each agent is converted into object-level masks and mid-term features through the perception model; the decoupling module decouples the mid-term features into two parts of private features and public features, and the agent only transmits the private features after processing by the object-level mask; the agent fuses the historical features, the public features and the private features of the agent, and the object-level private features of the collaborative agent through the fusion module, and inputs the fused features into the detection module for target detection. The application realizes instance-level feature interaction, reduces communication resource consumption, expands the perception field of view of the agent, and further enhances the fusion effect by using time sequence data, so that efficient feature fusion in a complex scene is realized.

Description

Technical Field

[0001] This invention relates to the fields of multi-agent collaboration technology and autonomous driving technology, specifically to an instance-level semantic simplified interaction method for intelligent collaborative perception. Background Technology

[0002] Perception capabilities are crucial for intelligent vehicles because they directly affect their ability to navigate safely and effectively in complex road environments. Intelligent vehicles utilize a range of sensors, such as cameras, radar, LiDAR, and ultrasonic sensors, to acquire detailed information about their surroundings, including but not limited to the position and speed of other vehicles, pedestrians, traffic light status, road signs, and obstacles. However, real-world road conditions are highly complex, with numerous obstructions and blind spots. A single agent struggles to perceive these areas using its own data, necessitating collaborative perception through inter-agent communication. Compared to traditional single-vehicle perception, multi-agent collaborative perception expands the perception range, improves environmental understanding accuracy, and enhances system robustness, providing a significant opportunity for the development of intelligent vehicles and autonomous driving.

[0003] Multi-agent cooperative perception has proven to be an effective way to improve the perception capabilities of intelligent systems. Given the communication limitations in real-world applications, information exchange between multi-agent systems urgently requires more intelligent and efficient design strategies. Current technologies largely focus on compressing global representations to reduce data volume and improve transmission efficiency, but these methods often neglect the value and role of instance-level semantic information. Furthermore, efficient feature fusion in complex scenarios is also a crucial issue that cannot be ignored in cooperative perception systems, as it relates to the safe operation of autonomous vehicles.

[0004] Chinese invention patent application CN202411109995.5, published on November 26, 2024, discloses a collaborative confidence fusion perception method for autonomous driving. This method utilizes the time-aligned 3D target detection results of different agents to evaluate the agent's environmental perception capability, corrects the detection results, and constructs a bird's-eye view BEV spatial confidence map using the corrected detection results, confidence scores, offline maps, and agent coordinates. However, this method does not employ mid-stage feature fusion in multi-vehicle collaboration; instead, it uses late-stage fusion, which is weaker and more susceptible to interference. To achieve multi-vehicle collaborative perception based on late-stage fusion, the method designs a mechanism to correct the detection results of a single agent, performing local and global confidence score corrections. While this design reduces communication volume during collaboration, in practical applications, correcting and merging only the detection results is easily affected by agent positioning errors, which is detrimental to the system's robustness. At the same time, because it relies too much on the detection results of a single agent, this technical solution is not good at detecting difficult targets, performs poorly in complex scenarios, and poses certain safety hazards in autonomous driving.

[0005] Chinese invention patent application CN202411155073.8, published on December 6, 2024, discloses a vehicle-road cooperative joint perception method and system under non-ideal conditions. This method constructs a V2X graph around selected self-nodes, shares metadata, extracts and shares visual feature maps at each node, and progressively compresses these visual feature maps along the channel dimension using a series of 1×1 convolutions. At the self-nodes, V2X attention fusion and latency compensation are performed on the received feature maps, and the aggregated and corrected feature maps are used for detection. The drawback of this approach is that it does not consider how to handle instance-level semantic information, which is crucial for understanding complex scenes. The lack of this component may limit the system's application scope in complex environments. Furthermore, although the method considers communication constraints and progressively compresses mid-range features along the channel dimension using a series of 1×1 convolutions, this compression method may lose some important spatial information, affecting the accuracy of the final detection and classification.

[0006] Chinese invention patent application CN202111495732.9, published on April 12, 2022, discloses a communication-sensitive multi-agent cooperative method. In this method, a vehicle agent encodes locally observed messages into hidden state vectors, scores the messages before sending them, and edge nodes send acknowledgment messages to agents with top-K message value. Upon receiving the acknowledgment, the agent sends its own message. The edge nodes receive all messages, extract valid information, and then aggregate and distribute them. The drawback of this scheme is that it does not consider the differences in confidence levels between agents' models, directly performing uniform sorting and filtering on messages generated by all agents. Furthermore, since acknowledgment messages are only sent to agents with top-K message value, it may cause uneven resource allocation, especially when some agents frequently gain sending privileges while others remain in a waiting state for extended periods.

[0007] Chinese invention patent application CN202410771385.5, published on November 1, 2024, discloses a method and device for cooperative perception in autonomous driving based on a multimodal large model. The method processes point cloud data from a master vehicle using a multimodal large model to obtain text information, extracts text features from the text information, extracts image features from the master vehicle's image data, and extracts depth map features from the depth map corresponding to the point cloud data. The depth map features and image features are fused based on the text features to obtain a first fused feature. The first fused feature is then fused with the features of the object to be detected sent by the target end to obtain a second fused feature. The target end includes at least one of the master vehicle's cooperative end and the road end. A multi-end cooperative perception visual task is performed based on the second fused feature. The drawback of this solution is that it does not consider the constraints of communication conditions and the computing power of onboard units in practical applications. The vehicle needs to process multiple modal data such as text, images, and point clouds, and generate the final perception result by collecting multimodal fused features from itself and other vehicles. This places certain demands on the computing and communication capabilities of the vehicle initiating the cooperative driving request, resulting in a high vehicle cost. Summary of the Invention

[0008] Existing multi-vehicle intelligent agent collaborative perception technologies suffer from the following shortcomings: they neglect the value and role of instance-level semantic information and fail to consider the impact of temporal data on feature fusion; while compressing global representations to reduce data volume and improve transmission efficiency can reduce communication overhead, it also results in the loss of some foreground information, which is detrimental to the final perception effect. This invention addresses these deficiencies by providing an instance-level semantic simplified interaction method for intelligent collaborative perception. This method enables instance-level feature interaction, reduces communication resource overhead, expands the agent's perceptual field of view, and utilizes temporal data to enhance fusion effects, achieving efficient feature fusion in complex scenarios.

[0009] This invention provides an instance-level semantically simplified interaction method for intelligent collaborative perception, which sets up a perception model, decoupling module, fusion module, and detection module on each intelligent agent in autonomous driving application scenarios. This invention includes the following steps:

[0010] Step 1: Each agent inputs data collected by its own sensors into the perception model. The perception model transforms the input data into an object-level mask and intermediate features. The perception model consists of a feature extractor and a detection head. The feature extractor transforms the input data into bird's-eye view BEV features represented by multi-scale features. The detection head aggregates the BEV features using an attention mechanism, generates candidate regions and confidence scores for the detected objects through a convolutional network, and filters them based on confidence and IoU thresholds to generate an object-level mask. The intermediate features are bird's-eye view BEV features represented by multi-scale features. Each agent processes its private features using the object-level mask before transmitting them out, achieving distributed instance-level feature interaction among agents.

[0011] Step 2: The decoupling module of the agent decouples its intermediate features into private and public features. The agent only processes its private features using object-level masking before transmitting them. Each agent decouples its intermediate features into private features and transmits only the private features, achieving simplified data representation driven by collaborative tasks among agents. Agents first transmit information about their current position and perception range to each other, determining collaborating partners and the overall collaborative perception range. Then, collaborating agents transmit object-level private features to each other.

[0012] Step 3: After receiving the object-level private features transmitted by the collaborating agent, the agent inputs its own historical features, its own public and private features, and the object-level private features of the collaborating agent into the fusion module. The fusion module uses a feature fusion method based on time-series information to obtain fused features and inputs them into the detection module. The detection module sends the fused features into a multi-layer convolutional network for target detection.

[0013] The fusion module uses a feature fusion method based on temporal information to obtain fused features, including: first, inputting the object-level private features of the collaborative agent and the private features of the current agent into the attention module to obtain fused private features; then, fusing the public features of the current agent to obtain preliminary fused features; concatenating the preliminary fused features with the fused private features of the current agent from the previous frame on the channel; and then sending them to the self-attention module for processing to obtain the fused features at the current moment.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0015] (1) Existing technologies ignore the value and role of instance-level semantic information and do not consider the impact of time-series data on feature fusion. However, the method of this invention, through collaborative task-driven data simplification representation, distributed instance-level feature interaction, and time-series information-based feature fusion, can maintain the model perception effect while reducing communication overhead and enhance the ability of intelligent collaborative perception models to perceive objects in complex scenarios.

[0016] (2) The method of this invention uses decoupling to compress features, achieving simplified data representation driven by collaborative tasks, and only transmits the private features of the current agent, thereby reducing communication overhead while achieving feature complementarity among multiple agents. The simplified data representation method driven by collaborative tasks of this invention overcomes to some extent the drawback of traditional compression methods that rely too much on single-task driving, and can reduce communication overhead without affecting the overall performance of the model.

[0017] (3) The method of this invention adopts a distributed instance-level feature interaction approach, overcoming the system risks brought about by centralized processing. Simultaneously, it performs instance-level masking processing on private features, transmitting only instance-level information and reducing communication overhead. This invention, through its distributed instance-level feature interaction approach, can also reduce the impact of background information on downstream tasks.

[0018] (4) The present invention also designs a novel feature fusion method that incorporates temporal information, which can improve the feature fusion effect of multi-agent systems and also improve the robustness of the system to a certain extent. Attached Figure Description

[0019] Figure 1 This is a block diagram of the overall implementation of the instance-level semantic simplified interaction method for intelligent collaborative perception of the present invention.

[0020] Figure 2 This is a flowchart illustrating the implementation of distributed instance-level feature interaction in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram illustrating the implementation of data simplification representation in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram illustrating the implementation of feature fusion based on time-series information in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0024] This invention designs an instance-level semantic simplification interaction method for intelligent collaborative perception. The method enables vehicles to make autonomous decisions, process single-vehicle perception data, and transmit foreground information, thereby achieving instance-level feature interaction and reducing communication resource overhead. Furthermore, a time-series-based data fusion module is designed to enhance the fusion effect using time-series data, achieving efficient feature fusion in complex scenarios and improving the overall perception capability of the vehicle collaborative perception system. This invention addresses the following three issues and provides corresponding solutions: (1) designing a collaborative task-driven data simplification representation method; (2) designing a distributed instance-level feature interaction method; and (3) designing a time-series-based feature fusion method.

[0025] Firstly, existing research on feature compression mostly focuses on compression along two dimensions: real space and channels. Spatial confidence selection primarily focuses on reducing the spatial resolution of feature maps while preserving important information. This method typically evaluates the importance or confidence of features at each location for the task, such as by calculating the impact of different regions on the final prediction result. Regions with high confidence mean that the features in these parts are more critical to the task and should therefore be retained; while regions with low confidence can be compressed or ignored. On the other hand, channel feature downsampling focuses on reducing the number of channels to reduce the number of model parameters and computational cost. It generally adopts a learnable approach, such as using convolution, fully connected, or attention mechanisms, allowing the network to adaptively learn which channels are more important and automatically adjust resource allocation. These algorithms compress the amount of data transmitted during the collaborative process, but they still have two drawbacks: First, it is difficult to accurately measure the impact on the overall performance of the model when compressing data, which may lead to the loss of key information. Second, it may be too dependent on supervised compression methods, meaning that spatial confidence generation and channel selection algorithms driven by single-vehicle tasks cannot take into account the needs of the entire collaborative perception task. To address these issues, this invention presents a collaborative task-driven data simplification representation method that compresses features using a decoupling approach. Specifically, a decoupling module separates the features that an agent needs to transmit into private and public parts, transmitting only the private features of the current agent. This reduces communication overhead while achieving feature complementarity among multiple agents. Since the decoupling module measures the feature relationships between agents, it overcomes, to some extent, the drawback of traditional compression methods that rely too heavily on single-task-driven approaches. Furthermore, by distinguishing between public and private features and reducing the transmission of redundant features, communication overhead can be reduced without affecting the overall model performance.

[0026] Secondly, existing multi-agent feature interaction methods generally directly transmit scene-level features. These methods can typically be categorized into message-passing-based and centralized-processing-based approaches. In message-passing-based methods, each agent extracts scene-level features based on its local observations and exchanges these features with other agents via a communication protocol. However, this approach faces the challenge that excessive message passing can lead to communication bottlenecks, especially when there are many agents or limited bandwidth. In centralized-processing-based methods, all agents' local observations are collected by a central node, which processes and generates global scene-level features before distributing them back to the agents. This approach has the advantage of optimizing the decision-making process from a global perspective. However, it increases the overall system risk because the system's performance depends on the central node's processing power. Furthermore, directly transmitting scene-level features can cause background features to influence the learning of downstream tasks. Considering the limitations of communication bandwidth and the overall robustness of the system in practical applications, this invention designs a distributed instance-level feature interaction method. Specifically, by using distributed information transmission, the system risks associated with centralized processing are overcome. At the same time, instance-level masking is performed on features to transmit only instance-level information, reducing communication overhead and minimizing the impact of background information on downstream tasks.

[0027] Thirdly, most existing multi-agent mid-term feature fusion methods directly fuse features. Non-learnable fusion methods, such as direct concatenation and weighted averaging, simply process the feature vectors of different agents. This approach is easy to implement and requires no additional learning parameters, but it does not consider the complex dependencies between different features. Neural network-based methods introduce adaptive weights to dynamically adjust the importance of features for each agent. By learning the mutual influence between features of different agents, these methods can focus on the most relevant feature parts, improving the fusion effect. However, these methods do not consider the improvement of feature fusion effect by temporal information. This invention designs a feature fusion method based on temporal information, which can utilize temporal information to improve the fusion effect of multi-agent features, and also improves the system robustness to a certain extent.

[0028] like Figure 1 As shown in the embodiment of the present invention, the instance-level semantic simplified interaction method for intelligent collaborative perception is applied in a scenario that includes two types of intelligent agents: intelligent vehicles and roadside equipment. Both the vehicles and the roadside equipment are equipped with a perception model, a decoupling module, a fusion module, and a detection module.

[0029] The vehicle perception model is responsible for transforming data acquired by vehicle sensors into object-level masks and intermediate features. The model consists of a feature extractor and a detection head. Specifically, the feature extractor transforms the input data into multi-scale feature representations of bird's-eye view BEV features through multiple convolutional and fully connected layers; these are highly refined versions of the original data. The detection head aggregates these BEV features using an attention mechanism and generates detection objects through a convolutional network. These detection objects are then filtered (using confidence score thresholds and IoU thresholds) to generate object-level masks based on their physical information. In this embodiment of the invention, the intermediate features are the multi-scale feature representations of bird's-eye view BEV features.

[0030] The roadside perception model and the vehicle perception model have similar structures. The roadside perception model is responsible for converting the data acquired by the sensors of the roadside equipment into object-level masks and intermediate features.

[0031] The decoupling module decouples the intermediate features that the agent needs to transmit into two parts: private features and public features. Each agent transmits its private features. At this point, the private features are still scene-level features, and object-level masks are needed to process them, further reducing communication overhead and minimizing the impact of background features on downstream tasks.

[0032] After receiving object-level private features from other agents, the fusion module sends historical features (time series information), its own public and private features, and the private features of neighboring agents into the fusion module, ultimately obtaining fused features rich in semantic information.

[0033] The detection module feeds the fused features into a multi-layer convolutional network to obtain the final detection result.

[0034] For the application scenario studied, the method of this invention models multi-agent cooperative perception as an optimization problem, as follows:

[0035]

[0036] Where R represents the optimization objective, B represents the communication bandwidth constraint, and N represents the number of agents. Let i be the feature observed by agent i at time t. Let represent the features transmitted by agent j to agent i at time t, and Decoder() represent the encoder. This optimization problem can be formulated as finding the optimal model that maximizes the collaborative perception effect among multiple agents, given a fixed communication bandwidth. Therefore, the core of this optimization problem is to simplify the representation of the features to be transmitted without affecting the model's accuracy.

[0037] In this invention, each agent first inputs data collected by its own sensors into a perception model, which then transforms the input data into an object-level mask and intermediate features. A decoupling module decouples the intermediate features of each agent into private and public features. Each agent only transmits its own private features, processing them using an object-level mask before transmitting them. This invention relates to an improved collaborative task-driven data simplification representation method and a distributed instance-level feature interaction method.

[0038] Considering the limited communication bandwidth and stringent requirements for overall system robustness in practical applications, this invention proposes a distributed instance-level feature interaction method to optimize data transmission efficiency and enhance system reliability. In today's complex and ever-changing data processing environment, especially for applications requiring real-time response and high concurrency, such as intelligent traffic management, industrial automation, and large-scale IoT deployments, the effective utilization of communication bandwidth and system stability have become critical considerations. Traditionally, many data processing tasks rely on centralized architectures, where all data is collected and processed uniformly at a central node. However, this model not only increases the risk of single points of failure but can also lead to network bottlenecks, especially when facing massive data streams, potentially significantly reducing system response speed and processing efficiency. Furthermore, centralized processing often requires the transmission of a large amount of redundant background information, further exacerbating the communication burden. In addition, directly transmitting scene-level features introduces a large number of background features, which can negatively impact the learning of downstream tasks. To address these issues, this invention overcomes the system risks associated with centralized processing by distributing computational tasks across multiple nodes and reduces the interference of background features on downstream tasks through object-level instance interaction. Specifically, this invention provides a distributed instance-level feature interaction method, introducing instance-level masking technology to transmit only feature information relevant to a specific instance, rather than all attributes of the entire dataset or object. For example, in video surveillance applications, the system can transmit only the key features of detected pedestrians or vehicles, instead of complete image frames, thereby significantly reducing communication overhead. This process ensures efficient data transmission even under poor network conditions and minimizes the impact of background information (such as irrelevant environmental details) on downstream tasks. In this way, the model can focus more on the important characteristics of the target object, improving the accuracy and efficiency of task execution. Figure 2As shown, the distributed instance-level feature interaction method consists of five parts: (1) First, the BEV features are fed into the detection head to generate a large number of candidate regions and their corresponding confidence scores; (2) Then, the candidate regions are screened and deduplicated by NMS (non-maximum suppression) according to the confidence threshold and IoU threshold, and an object-level mask that can be applied to the features is generated according to the spatial location information of the screened candidate regions. The IoU threshold (Intersection over Union threshold) is used to measure the degree of overlap between candidate regions. In this embodiment of the invention, the spatial feature values ​​corresponding to these candidate regions after screening are marked as 1, and the spatial feature values ​​of other regions are marked as 0, thus obtaining the object-level mask. The object-level mask will change according to the observed data at each time step. The object-level mask is a 0-1 matrix, and the size of the matrix is ​​consistent with the size of the private features. (3) In order to further reduce the communication volume, the intermediate features can be divided into public features and private features, and the private features of the current agent can be masked to obtain the object-level private features. (4) Communication between multiple agents begins, and simple information is sent first, such as the position and perception range of the agent. (5) After determining the collaborating partners and the overall collaborative perception range, the object-level private features are selectively sent to other intelligent agents.

[0039] This invention innovatively implements a collaborative task-driven data simplification representation method, aiming to overcome common limitations in traditional data compression techniques. In traditional data processing and transmission schemes, compression algorithms are often designed with the needs of a single agent at the core. While this single-agent task-driven approach can meet requirements in specific scenarios, it may lead to significant data redundancy or information loss in complex scenarios, thus affecting the system's real-time performance and effectiveness. To address these challenges, this invention introduces the concept of collaborative task-driven methods, achieving more efficient data compression and transmission. It not only considers the needs of a single agent but also pays special attention to the complementarity and redundancy of features among multiple agents, thereby providing more optimized data processing capabilities in complex multi-task environments. Specifically, this invention's collaborative task-driven data simplification representation method distinguishes between public and private features in the agent's observation data. Public features refer to the background information observed by the agent, as well as redundant information in the features of common objects that can be observed by multiple agents. Private features refer to features observed by an agent that differ from those of other agents, such as blind spots of other agents or special information about a common object from the agent's perspective. By clearly distinguishing between these two types of features, this invention can intelligently reduce the transmission of redundant information. For example... Figure 3As shown, in this data simplification and representation process, agents share the parameters of private and public feature encoders. Their purpose is to compress features at the channel level, thereby reducing communication between agents. To capture deeper features implied in the intermediate features, the method of this invention uses a combination of convolutional and fully connected layers to implement the encoder for further feature extraction. In design, the private and public feature encoders can have similar architectures, but their parameters are trained independently. Agents A and B use their own public and private feature encoders to decouple the intermediate features obtained from their own perceptual models into public and private features, respectively. After obtaining the private and public features, consistency constraints need to be added to the public features among multiple agents to ensure consistency of public feature information. To ensure that the private and public features retain the original feature information as losslessly as possible, we need to design a decoder to implement reconstruction loss supervision, so that the decoded features are as consistent as possible with the original features. This decoder is labeled as decoder A. Specifically, after feeding private and public features into the parameter-sharing decoder A, the reconstructed features are output. Reconstruction loss supervision is added between the input features (i.e., intermediate features) and the reconstructed features, thus enabling the decoupling module to compress features while minimizing the loss of key information. Furthermore, decoder A only serves the reconstruction loss during training and can be removed from the decoupling module during inference to accelerate inference.

[0040] In this invention, after the collaborating agents transmit object-level private features to each other, the agents need to perform feature fusion. In multi-agent systems, feature fusion is a crucial step in achieving efficient collaboration and decision-making. Existing multi-agent mid-term feature fusion methods typically employ direct fusion strategies, such as direct concatenation and weighted averaging—methods that are not learnable. While these methods are easy to implement and require no additional learning parameters, they fail to fully consider the complex dependencies between different features, resulting in fused information that may not accurately reflect the actual situation. Neural network-based methods introduce adaptive weights to dynamically adjust the importance of features for each agent. This method can learn the mutual influence between features of different agents, allowing the system to focus more on the most relevant features, thereby improving the fusion effect. However, these methods primarily focus on the fusion of static features, neglecting the important role of temporal information in the feature fusion effect. To overcome the shortcomings of existing methods, this invention proposes a feature fusion method based on temporal information, which not only considers the complex dependencies between features of different agents but also emphasizes the role of temporal information. This method not only improves the effect of multi-agent feature fusion but also enhances the robustness and adaptability of the system to a certain extent, enabling it to cope with more complex dynamic environments. Figure 4As shown, the corresponding multi-agent feature fusion module, after receiving object-level private features from other agents, sends these features along with the agent's own private features to the attention module. The agent's own private features are used as the query feature Q, and the other agents' object-level private features are used as the key K and value V. The attention module outputs the fused private features, which contain key information about the current agent and collaborating agents. The attention weight F calculated in this process is expressed as follows:

[0041]

[0042] Among them, W Q W K and W V These are the weight matrices for the linear transformations applied to Q, K, and V, respectively, d k This is the scale factor in attention, usually equal to the dimension of K, with the superscript T indicating transpose. At this point, the fused private features still lack background information and some redundant information about common objects. The fused private features are then fused with the common features of the current agent through decoder B to obtain preliminary fused features. After obtaining the preliminary fused features, it is necessary to fuse the key information of the fused features from the previous frame of the current agent to achieve better perception results. To fuse temporal information, the method of this invention first concatenates the preliminary fused features of the current frame of the current agent with the fused private features from the previous frame on a channel-wise basis, and then sends them to the self-attention module for processing. This yields the fused features at the current moment, which are then sent to the detection head of the final detection module to obtain more comprehensive and accurate perception results.

[0043] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessary limitation. The embodiments described above do not represent all embodiments consistent with this application. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this invention are still within the protection scope of this invention.

Claims

1. An instance-level semantically simplified interaction method for intelligent collaborative perception, characterized in that, In autonomous driving application scenarios, a perception model, a decoupling module, a fusion module, and a detection module are set on each intelligent agent; the method includes the following steps: Step 1: Each agent inputs the data collected by its own sensors into the perception model, which then converts the input data into object-level masks and intermediate features; The perception model consists of a feature extractor and a detection head. The feature extractor transforms the input data into bird's-eye view BEV features with multi-scale feature representation. The detection head uses an attention mechanism to aggregate the BEV features, generates candidate regions and confidence scores for the detection objects through a convolutional network, and filters them according to confidence thresholds and IoU thresholds to generate object-level masks. Intermediate features are BEV features represented by multi-scale features; Step 2: The decoupling module of the agent decouples its own mid-term features into two parts: private features and public features. The agent only transmits its own private features and processes the private features using an object-level mask to obtain object-level private features before transmitting them out. The agents first transmit information about their current position and perception range to each other to determine the cooperative partners and the overall cooperative perception range. After that, the cooperative agents transmit object-level private features to each other. Step 3: After receiving the object-level private features transmitted by the collaborating agent, the agent inputs its own historical features, its own public and private features, and the object-level private features of the collaborating agent into the fusion module. The fusion module uses a feature fusion method based on time-series information to obtain fused features and inputs them into the detection module. The detection module sends the fused features into a multi-layer convolutional network for target detection. The fusion module uses a feature fusion method based on temporal information to obtain fused features, including: first, inputting the object-level private features of the collaborative agent and the private features of the current agent into the attention module to obtain fused private features; then, fusing the public features of the current agent to obtain preliminary fused features; concatenating the preliminary fused features with the fused private features of the current agent from the previous frame on the channel; and then sending them to the self-attention module for processing to obtain the fused features at the current moment.

2. The method according to claim 1, characterized in that, In step 1, the generation of the object-level mask includes: the detection head of the perceptual model generates candidate regions of the detected object, then filters the candidate regions according to the confidence threshold and the IoU threshold, and performs non-maximum suppression (NMS) to remove duplicates. The values ​​of the finally filtered candidate regions are marked as 1, and the values ​​of other regions are marked as 0, thus generating an object-level mask. The size of the mask matrix is ​​consistent with the size of the private feature.

3. The method according to claim 1, characterized in that, In step 2, each agent decouples the intermediate features into private features and transmits only the private features, thereby achieving simplified data representation driven by collaborative tasks among agents. In the decoupling module, a private feature encoder and a public feature encoder are designed by combining convolutional layers and fully connected layers. The intermediate features are input into the private feature encoder and the public feature encoder respectively to obtain the private features and the public features. Consistency constraints are added to the common features among agents to ensure the consistency of information in the common features; the parameters of the private feature encoder and the public feature encoder are shared among agents. During training, the parameters of the private feature encoder and the public feature encoder are trained independently, and a decoder with shared parameters is set up among the agents. The agents feed their own private and public features into the decoder to output the reconstructed features. Reconstruction loss supervision is added between the input intermediate features and the reconstructed features. The parameters in the decoupling module are trained to make the reconstructed features as consistent as possible with the input intermediate features.

4. The method according to claim 1, characterized in that, In step 3, the private features of the current agent are used as query features. The received object-level private features of other collaborative agents are used as keys. Sum The fusion is performed through an attention module.

Images