Robust collaborative perception fusion system and method suitable for lossy communication scenarios in workshops

Abstract

The present invention discloses a robust collaborative perception fusion system and method suitable for lossy communication scenarios in workshops. The system includes a spatiotemporal feature enhancement module, a feature aggregation module, and a dynamic block prioritizer module. By processing received bird's-eye view features on spatial and temporal channels, the self-attention and cross-attention mechanisms are used to reduce data feature degradation caused by transmission damage during the communication process. The received features are dynamically segmented, and feature maps from the vehicle itself and other connected vehicles are fused using multi-scale dynamic routing cross-attention to expand its own perception range and enhance its own perception capabilities. In feature aggregation, a deep routing learning strategy is combined to select the feature blocks with the highest weight for priority processing, and the selection and learning of blocks are dynamically controlled at each layer. Compared with the existing technology, the present invention can help collaborative perception 3D target detection tasks in lossy communication scenarios achieve better detection accuracy and perception effects.

Description

Technical Field

[0001] The present invention relates to the field of collaborative perception of intelligent connected vehicles, and in particular to a collaborative perception fusion system and method for lossy communication scenarios between vehicles. Background Art

[0002] Accurate environmental perception is crucial for the full implementation of autonomous driving technology. However, the perception capabilities of a single vehicle are often limited by sensor coverage, blind spots, and environmental complexity, making it difficult to fully meet the requirements for efficient and safe autonomous driving. To address this issue, cooperative perception, as an emerging perception paradigm, has attracted widespread attention in recent years. By sharing and fusing multi-perspective sensor data between vehicles, cooperative perception overcomes the limitations of single-vehicle sensors and provides more comprehensive scene perception capabilities. The core of this technology lies in the real-time sharing of multi-vehicle perception information, enabling each vehicle to obtain multi-dimensional, full-perspective perception data, thereby improving the perception accuracy and safety of autonomous driving systems in complex traffic environments. During the lossy communication process and feature integration process, two communication losses—wireless channel degradation and network performance degradation—occur simultaneously during feature transmission, while the ego vehicle carries its own BEV features without loss.

[0003] Currently, research on collaborative perception primarily focuses on three data fusion methods: early fusion, intermediate fusion, and late fusion. Early fusion involves initially processing sensor data locally before transmitting it to other vehicles for further joint processing. While this approach offers advantages in terms of data integrity, it places extremely high demands on communication bandwidth, which can easily lead to network congestion. Late fusion involves sharing the final perception results after each vehicle independently completes the perception processing. This approach effectively reduces communication overhead, but due to the lack of raw data sharing, the perception results are less accurate, making it difficult to cope with complex and changing traffic environments.

[0004] In contrast, cooperative perception algorithms based on mid-term fusion offer a solution that balances communication overhead and data accuracy. In this approach, vehicles partially process perception data before sharing it, reducing the amount of data transmitted while retaining key information to ensure fused perception accuracy. Mid-term fusion not only effectively reduces communication bandwidth requirements but also ensures comprehensive and accurate perception, making it one of the most advanced cooperative perception technologies with broad application prospects.

[0005] The technical issues that this invention urgently addresses are: idealized communication assumptions are difficult to implement in reality, resulting in degraded received data quality; while compressed data transmission strategies alleviate the communication burden, they increase computational overhead and perform poorly in harsh environments. In contrast, self-recovery strategies with redundant transmissions are superior in addressing data loss and reducing system load. However, multiple communication loss factors, such as wireless channel fading, time delay, and packet loss, still severely impact the overall perceived quality. Summary of the Invention

[0006] In response to the technical defect of poor coping ability of collaborative perception fusion algorithms caused by lossy communication in real communication scenarios, the present invention aims to propose a collaborative perception fusion system and method suitable for lossy communication scenarios in workshops, which utilizes the multi-dimensional correlation of features and dynamic regional selective cross-learning to perform multi-perspective feature loss recovery and fusion.

[0007] The present invention is achieved by utilizing the following technical solutions:

[0008] In the first aspect, the present invention proposes a robust collaborative perception fusion system suitable for lossy communication scenarios in workshops. The system includes a spatiotemporal feature enhancement module, a feature aggregation module, and a dynamic block prioritizer module:

[0009] The spatiotemporal feature enhancement module is used to process the bird's-eye view features of other vehicles received in the spatial channel and the temporal channel, and utilize the spatial self-attention mechanism submodule to realize spatial channel enhancement of the original features. Then, the temporal cross-attention mechanism submodule is used to combine the original features enhanced by the spatial channel with the time-related historical features to realize further temporal channel enhancement of the original features enhanced by the spatial channel, thereby reducing the data feature degradation caused by transmission damage during the communication process. The obtained spatiotemporal enhanced features are transmitted to the feature aggregation module;

[0010] The feature aggregation module is used to dynamically segment the received spatiotemporal enhancement features, perform multi-scale pyramid segmentation on the self-vehicle features and other connected vehicle features in the H and W dimensions to obtain multi-scale blocks; after preferentially selecting the feature block with the highest weight as the dynamic block, cross-attention learning is performed between the self-vehicle and the other vehicles to obtain a multi-head combined feature block, the multi-head combined feature block is fused with the original feature through a residual connection, and combined with the unselected feature block at the original spatial position to obtain an aggregated feature; the aggregated feature is processed using a self-attention mechanism to integrate features from different scale levels, realizing a selective feature block learning process and cross-block interaction, and ultimately obtaining an aggregated self-vehicle feature for target perception and detection;

[0011] The feature aggregation module further calls the dynamic block prioritizer module; the dynamic block prioritizer module includes multiple weight allocation experts composed of linear layers and their corresponding multiplication units, a gating unit and an adding unit, which is used to adopt a deep routing learning strategy to select the feature block with the highest weight for priority cross-attention learning, and dynamically control the selection and learning of the block in each linear layer; each weight allocation expert performs weighted assignment of the feature block, and uses the output weight of each linear layer through the gating unit, and outputs the normalized weight contribution ratio finally assigned to the weight allocation expert as the final control result, and uses the multiplication unit to realize the weight × the ratio of each feature block being preferentially selected to obtain the weight allocated to each linear layer, and the final feature block priority weight allocation result is obtained by summing up through the adding unit.

[0012] In some embodiments, the spatiotemporal feature enhancement module further comprises:

[0013] Through metadata sharing and sensor feature encoding, bird's-eye view features are obtained from other connected vehicles. The self-attention mechanism is used to enhance the spatial channel information of the received features. By calculating the channel attention weight matrix, the spatial channel dependency ambiguity caused by noise interference during the transmission process is corrected, thereby enhancing the spatial channel information.

[0014] By leveraging the feature similarity of time channels, the cross-attention mechanism is used to repair the features of historical data. A learnable scaling factor is set to dynamically control the residual mechanism to balance the influence of historical frame features and current received features. The fusion of these features is adaptively adjusted under communication degradation conditions to cope with frame loss and data missing.

[0015] In some embodiments, the feature aggregation module further includes:

[0016] Multi-scale pyramid segmentation is performed on BEV feature maps of multiple vehicles. The feature maps are divided into regional blocks of different scales. Coarse-grained blocks capture large-scale contextual information, while fine-grained blocks capture local details, thereby enhancing robustness to noise interference and local data loss. A learned arbiter is set to dynamically select the optimal segmentation scale to adapt to vehicle distribution and traffic flow characteristics in different scenarios, optimize computational efficiency, and reduce processing load.

[0017] For each segmentation scale, the feature maps of the own vehicle and other connected vehicles are reshaped, and the top k significant feature blocks with the highest weights are selected using the dynamic block prioritizer module;

[0018] Through the multi-head cross-attention method, the vehicle can fully learn the sensor feature information of other vehicles to expand its own perception area, fill its own perception blind spots, and hash the learned features with the unextracted low-weight features at their original positions;

[0019] SplitAttention is used to aggregate features at different scales and multi-scale interaction information at different granularities. After this step is completed, it returns to multi-scale pyramid segmentation and executes multiple rounds to finally complete the feature fusion of multiple vehicles.

[0020] In some embodiments, the dynamic block prioritizer module further comprises:

[0021] A deep routing learning strategy is constructed. A block prioritizer is created through multiple weight allocation experts composed of linear layers and gating units. Different feature blocks are assigned different weights, their importance is calculated based on their spatial information and temporal correlation, and the top k most representative salient feature blocks are selected for priority processing. The block prioritizer is continuously trained to prioritize regions of interest containing missing features and target objects, allocating additional computing resources to these key areas to improve perception, while minimizing the learning process in non-interest areas such as static areas to improve overall resource utilization efficiency.

[0022] In a second aspect, the present invention proposes a robust collaborative perception fusion method applicable to lossy workshop communication scenarios, comprising:

[0023] Performing spatiotemporal feature enhancement further includes processing bird's-eye view features of the ego vehicle and other vehicles received in the spatial and temporal channels, utilizing a spatial self-attention mechanism submodule to perform spatial channel enhancement of the original features, and then utilizing a temporal cross-attention mechanism submodule to combine the spatially enhanced original features with time-related historical features to further perform temporal channel enhancement on the spatially enhanced original features, thereby obtaining spatiotemporal enhanced features to reduce data feature degradation caused by transmission damage during the communication process;

[0024] Performing feature aggregation further includes dynamically segmenting the received spatiotemporal enhanced features, performing multi-scale pyramid segmentation on the own vehicle features and other connected vehicle features in the H and W dimensions to obtain multi-scale blocks; after preferentially selecting the feature block with the highest weight as the dynamic block, performing cross-attention learning between the own vehicle and the other vehicles to obtain a multi-head combined feature block, fusing the multi-head combined feature block with the original feature through a residual connection, and combining it with the unselected feature block at the original spatial position to obtain an aggregated feature; processing the aggregated feature using a self-attention mechanism to integrate features from different scale levels, realizing a selective feature block learning process and cross-block interaction, and finally obtaining an aggregated own vehicle feature for target perception detection;

[0025] The step of performing feature aggregation further includes dynamic block priority processing, adopting a deep routing learning strategy to select the feature block with the highest weight for priority cross-attention learning, and dynamically controlling the selection and learning of the block in each linear layer; each weight allocation expert performs weighted assignment of the feature block, and uses the output weights of each linear layer through the gating unit, and outputs the normalized weight contribution ratio finally assigned to the weight allocation expert as the final control result, and uses the multiplication unit to realize the weight × the ratio of each feature block being prioritized to obtain the weight assigned to each linear layer, and the final feature block priority weight allocation result is obtained by summing up through the addition unit.

[0026] In some embodiments, the spatiotemporal feature enhancement module further comprises:

[0027] Step 1.1: Obtain bird's-eye view features from other connected vehicles through metadata sharing and sensor feature encoding. Use the self-attention mechanism to enhance the spatial channel information of the received features. Compute the channel attention weight matrix to correct for the spatial channel dependency ambiguity caused by noise interference during transmission, thereby enhancing the spatial channel information.

[0028] Step 1.2: Leverage the feature similarity of time channels to repair the features of historical data through the cross-attention mechanism. Set a learnable scaling factor to dynamically control the residual mechanism to balance the influence of historical frame features and current received features. Adaptively adjust the fusion of these features under communication degradation conditions to cope with frame loss and data loss.

[0029] In some embodiments, the feature aggregation module further includes:

[0030] Step 2.1: Perform multi-scale pyramid segmentation on the bird's-eye view feature map of multiple vehicles. This segmentation divides the feature map into blocks of different scales. Coarse-grained blocks capture broad contextual information, while fine-grained blocks capture local details, thereby enhancing robustness to noise interference and local data loss. A learned arbiter is set to dynamically select the optimal segmentation scale to adapt to vehicle distribution and traffic flow characteristics in different scenarios, optimizing computational efficiency and reducing processing load.

[0031] Step 2.2: For each segmentation scale, reshape the feature maps of the own vehicle and other connected vehicles, and use the dynamic block prioritizer module to filter out the top k significant feature blocks with the highest weights;

[0032] Step 2.3: Use the multi-head cross-attention method to allow the vehicle to fully learn the sensor feature information of other vehicles to expand its own perception area and fill its own perception blind spots. The learned features are then hashed with the unextracted low-weight features at their original positions.

[0033] Step 2.4: Use SplitAttention to aggregate features at different scales and aggregate multi-scale interaction information at different granularities. After this step is completed, return to step 2.1 and execute multiple rounds to finally complete the feature fusion of multiple vehicles.

[0034] In some embodiments, the dynamic block prioritizer module further comprises:

[0035] A deep routing learning strategy is constructed. A block prioritizer is created through multiple weight allocation experts composed of linear layers and gating units. Different feature blocks are assigned different weights, their importance is calculated based on their spatial information and temporal correlation, and the top k most representative salient feature blocks are selected for priority processing. The block prioritizer is continuously trained to prioritize regions of interest containing missing features and target objects, allocating additional computing resources to these key areas to improve perception, while minimizing the learning process in non-interest areas such as static areas to improve overall resource utilization efficiency.

[0036] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for robust collaborative perception fusion applicable to lossy communication scenarios in workshops according to any one of claims 5 to 8 is implemented.

[0037] In a fourth aspect, a non-transitory computer-readable storage medium is provided, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, the robust collaborative perception fusion method suitable for lossy communication scenarios in workshops according to any one of claims 5 to 8 is implemented.

[0038] Compared with the prior art, the present invention has the following positive technical effects:

[0039] 1) The fusion method designed by the feature enhancement module and the feature aggregation module combined with the dynamic block optimizer can fully consider the negative impact of lossy communication on features. The collaborative perception 3D object detection task implemented by this invention in lossy communication scenarios achieves better detection accuracy and perception effect;

[0040] 2) Using a feature aggregation module to refine and extract fusion representations to adaptively emphasize important features; using multi-scale dynamic routing to perform a cross-attention mechanism sub-module process to fuse bird's-eye view BEV feature maps from the own vehicle and other vehicles, thereby enhancing its own perception capabilities by expanding its own perception range;

[0041] 3) Further combined with the dynamic block prioritizer module, a deep routing learning strategy is adopted to select the feature blocks with the highest weight for priority processing, and the selection and learning of blocks are dynamically controlled at each layer. Multi-scale segmentation feature blocks are selected according to the dynamic weight distribution, realizing the deep routing learning strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a module diagram of a collaborative perception fusion system suitable for lossy communication scenarios in workshops according to the present invention;

[0043] Figure 2 This is a schematic diagram of the overall process of a collaborative perception fusion method applicable to lossy communication scenarios in workshops according to the present invention;

[0044] Figure 3 This is a data flow chart of collaborative sensing data fusion of the present invention;

[0045] Figure 4 This is a flowchart of the multi-scale dynamic routing cross-attention data processing of the feature aggregation module of the present invention;

[0046] Figure 5 A structural diagram of the dynamic block prioritizer module and a schematic diagram of the deep routing strategy of the present invention;

[0047] Figure 6 Specific implementation block diagram of the feature aggregation module; (a) spatial channel feature aggregation, (b) temporal channel feature aggregation.

[0048] Reference numerals:

[0049] 100. Spatiotemporal feature enhancement module, 200. Feature aggregation module, 300. Dynamic block prioritizer module. DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0051] like Figure 1 As shown, the collaborative perception fusion system of the present invention, which is applicable to lossy communication scenarios in workshops, includes a spatiotemporal feature enhancement module 100, a feature aggregation module 200, and a dynamic block prioritizer module 300.

[0052] The spatiotemporal feature enhancement module 100 is used to process the bird's-eye view (BEV) feature tensors received in the spatial channel and the temporal channel, and utilize the spatial self-attention mechanism submodule to achieve spatial channel enhancement of the original features. Then, the temporal cross-attention mechanism submodule is used to combine the original features enhanced by the spatial channel with the time-related historical features to further enhance the temporal channel of the original features enhanced by the spatial channel, thereby reducing the data feature degradation caused by transmission damage during the communication process. The obtained spatiotemporal enhanced features are transmitted to the feature aggregation module 200.

[0053] The feature aggregation module 200 is used to dynamically segment the received spatiotemporally enhanced features and utilize a multi-scale dynamic routing cross-attention mechanism sub-module process to fuse bird's-eye view BEV feature maps from the vehicle itself and other vehicles, thereby enhancing its own perception capability by expanding its own perception range;

[0054] The dynamic block prioritizer module 300 adopts a deep routing learning strategy to select the feature blocks with the highest weight for priority processing, and dynamically controls the selection and learning of blocks at each layer.

[0055] This module specifically includes the following steps:

[0056] Step 1: The ego vehicle (Ego) obtains the bird's-eye view (BEV) features of other connected vehicles (CAVs) in the network during lossy communication as raw features. At the same time, it obtains the bird's-eye view features of other connected vehicles (CAVs) from a known database as historical features. By processing the received bird's-eye view features in spatial and temporal channels, the self-attention and cross-attention mechanisms are used to reduce the data feature degradation caused by transmission loss during the communication process.

[0057] Step 1-1: Process the original features in the spatial channel to semantically enhance the spatial channel information using the self-attention mechanism, thereby correcting the spatial channel dependency ambiguity caused by noise interference during transmission. Furthermore, the residual connection network is used to reduce the gap in the output self-attention value, and the scaling factor is combined to control the degree of influence of the self-attention value. The specific processing includes: Figure 2As shown in the figure, through metadata sharing and sensor feature encoding, the ego vehicle (Ego) obtains the bird's-eye view (BEV) features of other connected vehicles (CAVs) in the network during lossy communication as the original features F others ∈RL ×H×W×C , the formal definition includes: L represents the number of bird's-eye view (BEV) feature maps of connected vehicles (CAV), where H, W and C are the height, width and number of channels of the BEV feature map tensor respectively, and an independent convolution layer is set to receive the original features of other connected vehicles. The number of convolution layers is set to 1,...n2 concatenated convolution layers, and the number of outputs of the convolution layer is 3. The outputs of the convolution layer are defined as the query vector matrix query, key vector matrix key and value vector matrix value of the spatial self-attention mechanism submodule, and the input of the temporal self-attention mechanism submodule is defined as F others ;

[0058] The shape of each vector matrix representation is defined as the number of other connected vehicles, height * width and number of channels of each BEV feature map tensor; the query vector matrix query and the key vector matrix key are dot-producted to calculate the attention score of each element in the input sequence with other elements in the sequence, and then the softmax function is applied to normalize the score. The weight of the spatial channel is determined by the attention score, and a spatial channel attention weight matrix is ​​formed. The weight of the spatial channel of each element is multiplied by the value vector matrix, and then the spatial channel attention weight matrix is ​​used to calculate the weighted sum of each element in the sequence to the value vector of other elements to obtain the final spatial self-attention value as the final output of the temporal self-attention mechanism submodule; the output spatial self-attention value is combined with the original feature through a residual connection and a learnable scaling factor of 1. The purpose of setting the residual connection network in this step is to avoid a large gap with the original feature. The output spatial self-attention value is used as the input of the residual connection network. The expression of H(x) of the residual connection network is as follows:

[0059] H(x)=gamma*F others +(1-gamma)*x

[0060] Where x represents the spatial attention value obtained in the above steps, gamma represents the scaling factor, H(x) represents the output of the residual network, * is the multiplication sign, and F othersare the original features of other vehicles; H(x) = F(x) + x, where H(x) represents the output of the residual network, F(x) represents the nonlinear transformation of the residual network, and x represents the input of the residual network. The learnable scaling factor 1 is used to control the influence of the spatial self-attention value. The normalized data is multiplied by a learnable first scaling factor (gamma) and a learnable offset (beta) to maintain the expressive power of the model.

[0061] The self-attention mechanism semantically enhances spatial channel information, thereby correcting the spatial channel dependency ambiguity caused by noise interference during transmission. The spatial channel dimension enhancement 1-2 is;

[0062] Step 1-2, using the feature similarity between adjacent frames in time, repair the features of the current spatial channel enhancement obtained in step 1 through historical features to enhance the features of the temporal channel. The specific processing includes: Figure 2 As shown, a continuously updated historical frame feature is created, and the spatial channel enhanced feature after step 1-1 is received at time, and a temporal cross-attention Cross-attention mechanism sub-module is constructed. The spatial channel enhanced feature and the historical frame feature are used as the input sequence of the temporal self-attention mechanism sub-module, and the temporal cross-attention Cross-attention mechanism sub-module is used to learn the historical frame feature for self-repair: the feature enhanced by the spatial channel is converted into the query vector matrix query, and the historical frame feature is converted into the key vector matrix key and the value vector matrix value. The query vector matrix query and the key vector matrix key are dot-producted to calculate the attention score of each element in the input sequence with other elements in the sequence, and then the softmax function is applied to normalize the score. The weight of the temporal channel is determined by the attention score, and the temporal channel attention weight matrix W is formed. ca ∈R L×(H*W)×(H*W), multiply the weight of each element's spatial channel by the value vector matrix, and then use the time channel attention weight matrix to calculate the weighted sum of each element in the sequence to the value vectors of other elements to obtain the final time self-attention value as the final output of the time self-attention mechanism submodule; use the time cross-attention mechanism to learn the characteristics of historical frame features for self-repair. Only the query vector is obtained from the original features of the result of step 1-1 corresponding to time t. The key and value are obtained by projection. These two are combined to output the channel attention weight matrix 2. The feature similarity between adjacent frames in time is used to repair the current feature through historical data. In this step, a learnable second scaling factor is set to dynamically control the residual mechanism, thereby adjusting the influence of the time cross-attention mechanism submodule on the original feature under different communication degradation conditions; at the same time, the second scaling factor is used to update the historical features. The specific expression is as follows:

[0063] At the same time, update the historical feature F h , the specific expression is as follows:

[0064]

[0065] in, Represents the features after cross attention, τ represents the scaling factor, and τ plays an important role in balancing the contribution of subsequent historical features and the current received features in the repair process;

[0066] Therefore, steps 1-2 utilize the feature similarity between temporally adjacent frames to cope with partial data loss and inter-frame loss caused by the reconnection problem;

[0067] Step 2: Use the multi-scale dynamic routing cross-attention mechanism sub-module process to fuse the bird's-eye view BEV feature map of other vehicles, such as Figure 3 As shown, it includes multi-scale feature pyramid segmentation, selective feature block learning and cross-block interaction and block self-attention learning; specifically, it includes the following steps: Multi-layer perceptron and Attention both belong to the Transformer architecture, as well as Figure 4 As shown in the figure, each multi-layer perceptron is matched with a normalization layer, and each cross-attention mechanism submodule is also matched with a normalization layer. The normalization layer (Layer Normalization) normalizes the mean and variance of each feature channel of the input in the sample dimension, making the data distribution more stable and helping to accelerate training.

[0068] Step 2-1: Multi-scale pyramid segmentation is performed on the vehicle's own features and other connected vehicle features in the H and W dimensions, obtaining pyramid-shaped cutting results of 2*2, 4*4, and 8*8 feature blocks. The feature map is divided into multi-scale blocks, capturing the information of each segmented area of ​​the feature map from coarse to fine according to the scale, thereby enhancing the robustness to noise and local data loss, and facilitating cross-scale target detection; in order to adaptively select the optimal segmentation scale in different scenarios and reduce the computational load, two linear layers are used to create a learnable arbiter to dynamically determine the appropriate scale representation window size for the th scale segmentation; the arbiter learns to balance between coarse-grained and fine-grained representations, thereby effectively handling diverse traffic conditions and object scales.

[0069] Step 2-2, Selective feature block learning process and cross-block interaction: For a given partition scale S i , the feature map is reshaped into and by For example, the feature map can be divided into feature blocks, each block has the shape of (S i ,S i ,C), prioritize extracting the top k most significant features, i.e., important dynamic blocks, with the following formula:

[0070]

[0071] Among them, k represents the number of dynamic blocks selected, φ represents the weight calculation of each feature block, and Top k It means extracting and serializing the k feature blocks with the highest weight from the feature map, and generating the selected self and connection features as output features. k The shape of the feature obtained is described in detail in the dynamic block prioritizer module 300. and At the same time, set the scale factor By adjusting the scaling factor p k To control the ratio of selected blocks to the total number of blocks;

[0072] Step 2-3: After collecting high-weight, significant feature blocks, cross-attention learning is performed between the ego vehicle and other vehicles (learning what is worth learning from other vehicles). The attention operator defined in three-dimensional space is as follows:

[0073]

[0074] Among them, Q, K, V represent query, key and value matrices, B represents the learnable relative position deviation between different feature blocks, and d k‘Denotes the dimensions of Q and K. Further proceeding:

[0075]

[0076] Among them, Proj ego and Proj others Represents the projection function, which respectively transforms the learnable weight matrix W Q 、W K 、W V Applied to the input feature tensor and To obtain the corresponding query matrix, key matrix and value matrix. In order to capture multi-level interactive information, multi-head attention is used to extract complementary information from various representation subspaces of the input features in parallel. The features (the output features above) are divided into h heads (an operation embedded in the attention mechanism), where the channel dimension of each head is d h = C / h. For the jth head, get and Matrix, whose shape is (k,S i ,S i ,d h ) and (L,k,S i ,S i ,d h ) corresponds to self-features and other features. Next, cross-attention calculation is performed on the selected features of each head:

[0077]

[0078]

[0079] in, and Connect them separately to obtain multi-head combination features and The processed feature blocks are ultimately fused with the original features through residual connections, scattering them back to their original spatial locations and combining them with unselected feature blocks to produce aggregated features. This output is a concentrated and refined feature result. Furthermore, a self-attention submodule is applied to the aggregated features to refine and extract the fused representation, enabling the model to adaptively emphasize important features.

[0080] The input here is and The two processes are respectively processed by Attention, that is, they are also projected to query, key, and value, and perform the same operation as step 1-1. But note and They are performed independently, without intersection, and do not share the parameters of the neural network (that is, the operations of the projection layer are not shared). They are expressed by refining and extracting fusion, so that the model can adaptively emphasize important features.

[0081] Steps 2-4: The results obtained in the self-attention submodule are assimilated through the SplitAttention mechanism to integrate feature knowledge from different scale levels:

[0082]

[0083] Here, p still represents the process between the self-vehicle characteristics and the characteristics of other vehicles, and n scale The above overall design process can be combined into a single process. The feature fusion module will perform multiple rounds of this single process, ultimately obtaining the ego features of the vehicle for subsequent classification and regression to complete the target detection task.

[0084] The step 2-1 of prioritizing the extraction of the first k significant feature blocks (i.e., the most important dynamic blocks) is implemented by the dynamic block prioritizer module 300, using a deep routing learning strategy, specifically described as follows: Figure 4 As shown, the feature aggregation module performs multiple rounds of cross-attention learning. At each cross-attention layer, the block prioritizer calculates the weight of each feature block using a function and selects the top k most significant feature blocks with the highest weights for cross-attention learning. Unselected feature blocks bypass the cross-attention layer. The dynamic block prioritizer consists of multiple weight assignment experts consisting of linear layers, their corresponding multiplication units, a gating unit, and an addition unit. Each expert assigns weights to the blocks, while the gating unit controls the weight contribution of each expert. Specifically, the gating unit consists of a neural network linear layer, and its input is the weights assigned by each expert. More specifically, the gating unit is controlled by the neural network parameters, requiring continuous training and adjustment to achieve better and more reasonable control capabilities. Each linear layer (acting as an expert) takes as input the features obtained above and outputs its weights. The gating unit takes as input the output weights of each linear layer and outputs the final normalized weight ratio as the final control result (each weight multiplied by its respective ratio, and the final result is summed up), thus completing the function design. At each cross-attention layer, as a learning layer, the following process occurs, corresponding to steps 2-3:

[0085]

[0086] in and Represent the data of the lth layer and the lth block and the assigned weight, W lrepresents the feature weight set of all blocks in the lth layer, CA represents the cross attention function, Representing the vehicle's own features and the features of other vehicles, respectively. The weight values ​​of the top k significant feature blocks are calculated based on the scaling coefficient, and the th weight is extracted as the threshold. Each block is processed according to its weight relative to the threshold. The block prioritizer is continuously trained to prioritize regions of interest containing missing features and target objects, allocating additional computing resources to these key areas to improve perception, while minimizing the learning process in non-interest areas such as static areas to improve overall resource utilization efficiency.

[0087] In summary, the present invention proposes a collaborative perception fusion system suitable for lossy communication scenarios in workshops, including a spatiotemporal feature enhancement module, a feature aggregation module, and a dynamic block prioritizer module;

[0088] The spatiotemporal feature enhancement module addresses semantic loss and inter-frame loss caused by invalid signal noise and reconnection issues in real-world communication scenarios. By leveraging semantic relevance in the feature space and the similarity of features between adjacent frames in the temporal channel, it uses self-attention and cross-attention methods to initially restore received feature degradation caused by communication impairments.

[0089] The feature aggregation module uses a multi-scale dynamic routing cross-attention method to further recover feature losses caused by wireless signal loss and network performance degradation, and perform multi-vehicle multi-view feature fusion. Specifically, multiple connected autonomous vehicles (CAVs) encode the point cloud data generated by their own lidar to generate a bird's eye view (BEV) feature map. The own vehicle divides the BEV feature map collected from other vehicles into multiple scales through vehicle-to-vehicle communication, and the BEV features are further divided into feature blocks of different scales for feature interaction. At the same time, a neural network arbiter is designed to dynamically select the scale resolution to be processed and adaptively capture long-range dependencies and local context details between vehicles. By utilizing the cross-attention process between the own vehicle and other vehicles, the module performs multi-granularity data recovery and feature fusion across vehicles, and finally integrates multi-granularity regional information through multi-scale feature aggregation.

[0090] The dynamic block prioritizer module is used to further optimize the interaction area, reduce the fusion interference of the noise area, and improve the computational efficiency. Specifically, before the cross-learning process of the feature aggregation module, the BEV features are further divided into multiple regional blocks at the corresponding scale. The dynamic block prioritizer dynamically assigns the weight of each regional block, sets the threshold, and determines the cross-learning area composed of multiple regional blocks through a deep routing learning strategy. During the iterative dynamic region selection learning process, the vehicle's own features continuously extract valuable information from the focus area, while the features of other vehicles continuously recover the lost area and share information.

[0091] In a second aspect, the present invention proposes a collaborative perception fusion method applicable to lossy communication scenarios in workshops, comprising:

[0092] Step 1: The ego vehicle establishes a connection with other vehicles. Each vehicle performs PointPillars-based feature encoding on the point cloud data generated by its own lidar sensor to generate BEV features. Through lossy vehicle communication in real-world scenarios, the ego vehicle obtains the lossy BEV features of other connected vehicles. The spatiotemporal feature enhancement module first performs preliminary loss recovery on each vehicle. Self-attention learning is performed on the BEV feature spatial channel, and cross-attention is used based on the short-term feature similarity of adjacent frames to recover features in the temporal channel.

[0093] Step 2: The initially restored BEV features of each vehicle are further processed by a specialized aggregation module. Multiple scales are set, and dynamic feature segmentation is performed at different scales. At each scale, the BEV features are horizontally segmented into multiple feature blocks. To further capture features of interest (such as the vehicle's blind spot), each block is prioritized. A neural network is designed to assign weights to each block, and a specified threshold is set. Blocks exceeding the threshold are further processed, while feature blocks with low weights are not selected and skip the subsequent learning process.

[0094] Step 3: After selecting the feature blocks, the selected high-priority feature blocks form an irregular region of interest. Multi-scale cross-attention learning is performed on the selected features of the vehicle itself and the selected feature blocks of other vehicles. The resulting features are merged with the unselected features and scattered back to their original locations. The multi-scale learning results are further merged to integrate interactive information at different granularities. Through multiple rounds of learning, the information of the own features is continuously enriched, completing the feature aggregation process. Finally, a simple classification regressor is used to process the features for object detection and complete vehicle position perception.

[0095] In addition, based on similar inventive concepts, an embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0096] In addition, based on similar inventive concepts, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program implements the above method when executed by a processor.

[0097] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] Those skilled in the art can make improvements and changes based on the above invention. Any modifications, improvements and changes made on the basis of the present invention should fall within the scope of protection of the present invention.

Claims

1. A robust collaborative perception fusion system suitable for lossy communication scenarios in workshops, characterized by: The system includes a spatiotemporal feature enhancement module, a feature aggregation module, and a dynamic block prioritizer module: The spatiotemporal feature enhancement module is used to process the bird's-eye view features of other vehicles received in the spatial channel and the temporal channel, and utilize the spatial self-attention mechanism submodule to realize spatial channel enhancement of the original features. Then, the temporal cross-attention mechanism submodule is used to combine the original features enhanced by the spatial channel with the time-related historical features to realize further temporal channel enhancement of the original features enhanced by the spatial channel, thereby reducing the data feature degradation caused by transmission damage during the communication process. The obtained spatiotemporal enhanced features are transmitted to the feature aggregation module; The feature aggregation module is used to dynamically segment the received spatiotemporal enhancement features, perform multi-scale pyramid segmentation on the self-vehicle features and other connected vehicle features in the H and W dimensions to obtain multi-scale blocks; after preferentially selecting the feature block with the highest weight as the dynamic block, cross-attention learning is performed between the self-vehicle and the other vehicles to obtain a multi-head combined feature block, the multi-head combined feature block is fused with the original feature through a residual connection, and combined with the unselected feature block at the original spatial position to obtain an aggregated feature; the aggregated feature is processed using a self-attention mechanism to integrate features from different scale levels, realizing a selective feature block learning process and cross-block interaction, and ultimately obtaining an aggregated self-vehicle feature for target perception and detection; The feature aggregation module further calls the dynamic block prioritizer module; the dynamic block prioritizer module includes multiple weight allocation experts composed of linear layers and their corresponding multiplication units, a gating unit and an adding unit, which is used to adopt a deep routing learning strategy to select the feature block with the highest weight for priority cross-attention learning, and dynamically control the selection and learning of the block in each linear layer; each weight allocation expert performs weighted assignment of the feature block, and uses the output weight of each linear layer through the gating unit, and outputs the normalized weight contribution ratio finally assigned to the weight allocation expert as the final control result, and uses the multiplication unit to realize the weight × the ratio of each feature block being preferentially selected to obtain the weight allocated to each linear layer, and the final feature block priority weight allocation result is obtained by summing up through the adding unit.

2. A robust collaborative perception fusion system suitable for lossy workshop communication scenarios according to claim 1, wherein: The spatiotemporal feature enhancement module further comprises: Through metadata sharing and sensor feature encoding, bird's-eye view features are obtained from other connected vehicles. The self-attention mechanism is used to enhance the spatial channel information of the received features. By calculating the channel attention weight matrix, the spatial channel dependency ambiguity caused by noise interference during the transmission process is corrected, thereby enhancing the spatial channel information. By leveraging the feature similarity of time channels, the cross-attention mechanism is used to repair the features of historical data. A learnable scaling factor is set to dynamically control the residual mechanism to balance the influence of historical frame features and current received features. The fusion of these features is adaptively adjusted under communication degradation conditions to cope with frame loss and data missing.

3. A robust collaborative perception fusion system method applicable to lossy workshop communication scenarios according to claim 1, wherein: The feature aggregation module further includes: Multi-scale pyramid segmentation is performed on BEV feature maps of multiple vehicles. The feature maps are divided into regional blocks of different scales. Coarse-grained blocks capture large-scale contextual information, while fine-grained blocks capture local details, thereby enhancing robustness to noise interference and local data loss. A learned arbiter is set to dynamically select the optimal segmentation scale to adapt to vehicle distribution and traffic flow characteristics in different scenarios, optimize computational efficiency, and reduce processing load. For each segmentation scale, the feature maps of the own vehicle and other connected vehicles are reshaped, and the top k significant feature blocks with the highest weights are selected using the dynamic block prioritizer module; Through the multi-head cross-attention method, the vehicle can fully learn the sensor feature information of other vehicles to expand its own perception area, fill its own perception blind spots, and hash the learned features with the unextracted low-weight features at their original positions; SplitAttention is used to aggregate features at different scales and multi-scale interaction information at different granularities. After this step, it returns to multi-scale pyramid segmentation and executes multiple rounds to finally complete the feature fusion of multiple vehicles.

4. A robust collaborative perception fusion system suitable for lossy workshop communication scenarios according to claim 1, wherein: The dynamic block prioritizer module further comprises: A deep routing learning strategy is constructed. A block prioritizer is created through multiple weight allocation experts composed of linear layers and gating units. Different feature blocks are assigned different weights, their importance is calculated based on their spatial information and temporal correlation, and the top k most representative salient feature blocks are selected for priority processing. The block prioritizer is continuously trained to prioritize regions of interest containing missing features and target objects, allocating additional computing resources to these key areas to improve perception, while minimizing the learning process in non-interest areas such as static areas to improve overall resource utilization efficiency.

5. A robust collaborative perception fusion method suitable for lossy communication scenarios in workshops, characterized by: include: Performing spatiotemporal feature enhancement processing further includes processing bird's-eye view features of the self-vehicle and other vehicles received in the spatial channel and the temporal channel, utilizing a spatial self-attention mechanism submodule to perform spatial channel enhancement of the original features, and then utilizing a temporal cross-attention mechanism submodule to combine the spatial channel-enhanced original features with time-related historical features to further perform temporal channel enhancement on the spatial channel-enhanced original features, thereby obtaining spatiotemporal enhanced features to reduce data feature degradation caused by transmission damage during the communication process; Performing spatiotemporal enhanced feature aggregation processing further includes dynamically segmenting the received spatiotemporal enhanced features, performing multi-scale pyramid segmentation on the own vehicle features and other connected vehicle features in the H and W dimensions to obtain multi-scale blocks; after preferentially selecting the feature block with the highest weight as the dynamic block, performing cross-attention learning between the own vehicle and the other vehicles to obtain a multi-head combined feature block, fusing the multi-head combined feature block with the original feature through a residual connection, and combining it with the unselected feature block at the original spatial position to obtain an aggregated feature; processing the aggregated feature using a self-attention mechanism to integrate features from different scale levels, realizing a selective feature block learning process and cross-block interaction, and ultimately obtaining an aggregated own vehicle feature for target perception detection; The step of performing feature aggregation further includes dynamic block priority weight allocation processing, adopting a deep routing learning strategy to select the feature block with the highest weight for priority cross-attention learning, and dynamically controlling the selection and learning of the block in each linear layer; each weight allocation expert performs weighted assignment of the feature block, and uses the output weights of each linear layer through the gating unit, and outputs the normalized weight contribution ratio finally assigned to the weight allocation expert as the final control result, and uses the multiplication unit to realize the weight × the ratio of each feature block being preferentially selected to obtain the weight allocated to each linear layer, and the final feature block priority weight allocation result is obtained by summing up through the addition unit.

6. A robust collaborative sensing fusion method applicable to lossy workshop communication scenarios according to claim 5, wherein: The spatiotemporal feature enhancement module further comprises: Step 1.1: Obtain bird's-eye view features from other connected vehicles through metadata sharing and sensor feature encoding. Use the self-attention mechanism to enhance the spatial channel information of the received features. Compute the channel attention weight matrix to correct for the spatial channel dependency ambiguity caused by noise interference during transmission, thereby enhancing the spatial channel information. Step 1.2: Leverage the feature similarity of time channels to repair the features of historical data through the cross-attention mechanism. Set a learnable scaling factor to dynamically control the residual mechanism to balance the influence of historical frame features and current received features. Adaptively adjust the fusion of these features under communication degradation conditions to cope with frame loss and data loss.

7. The robust collaborative sensing fusion method for lossy vehicle communication scenarios according to claim 5, wherein: The feature aggregation module further includes: Step 2.1: Perform multi-scale pyramid segmentation on the bird's-eye view feature map of multiple vehicles. This segmentation divides the feature map into blocks of different scales. Coarse-grained blocks capture broad contextual information, while fine-grained blocks capture local details, thereby enhancing robustness to noise interference and local data loss. A learned arbiter is set to dynamically select the optimal segmentation scale to adapt to vehicle distribution and traffic flow characteristics in different scenarios, optimizing computational efficiency and reducing processing load. Step 2.2: For each segmentation scale, reshape the feature maps of the own vehicle and other connected vehicles, and use the dynamic block prioritizer module to filter out the top k significant feature blocks with the highest weights; Step 2.3: Use the multi-head cross-attention method to allow the vehicle to fully learn the sensor feature information of other vehicles to expand its own perception area and fill its own perception blind spots. The learned features are then hashed with the unextracted low-weight features at their original positions. Step 2.4: Use SplitAttention to aggregate features at different scales and aggregate multi-scale interaction information at different granularities. After this step is completed, return to step 2.1 and execute multiple rounds to finally complete the feature fusion of multiple vehicles.

8. The robust collaborative sensing fusion method for lossy workshop communication scenarios according to claim 1, wherein: The dynamic block prioritizer module further comprises: A deep routing learning strategy is constructed. A block prioritizer is created through multiple weight allocation experts composed of linear layers and gating units. Different feature blocks are assigned different weights, their importance is calculated based on their spatial information and temporal correlation, and the top k most representative salient feature blocks are selected for priority processing. The block prioritizer is continuously trained to prioritize regions of interest containing missing features and target objects, allocating additional computing resources to these key areas to improve perception, while minimizing the learning process in non-interest areas such as static areas to improve overall resource utilization efficiency.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robust collaborative perception fusion method for lossy communication scenarios in a workshop according to any one of claims 5 to 8 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the robust collaborative perception fusion method applicable to lossy communication scenarios in workshops according to any one of claims 5 to 8 is implemented.

Images