Multi-vehicle cooperative perception method

Abstract

This invention relates to a multi-vehicle cooperative perception method, comprising the following steps: Step 1, acquiring multi-view images captured by the driver vehicle and cooperative vehicles respectively; Step 2, constructing a cooperative perception model, which includes a voxel generator, a compression fusion module, and an occupancy prediction head connected in sequence; the voxel generator converts multi-view images into voxel features based on a probabilistic kernel truncated voxel generation mechanism; the compression fusion module fuses voxel features among cooperative vehicles based on a cardinality conditional attention mechanism, selectively aggregating key occupancy perception cues; Step 3, simultaneously inputting the acquired multi-view images of the driver vehicle and cooperative vehicles into the cooperative perception model, ultimately obtaining the perception results of the environment where the driver vehicle and cooperative vehicles are located. This method improves query accuracy while reducing computational overhead.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-vehicle collaborative perception method. Background Technology

[0002] Collaborative perception enables connected autonomous vehicles to exchange perception information via wireless communication, thereby alleviating the inherent limitations of single-vehicle autonomous driving perception systems, including limited perception range and obstructed field of vision.

[0003] In existing collaborative occupancy perception technologies, a vehicle constructs a voxel-level semantic occupancy grid centered on itself by fusing perception data from multiple collaborating vehicles. For each voxel, this grid indicates not only whether it is occupied but also which type of target occupies it. Existing technologies can improve the robustness of multi-vehicle fusion in severely occluded and complex dynamic traffic environments by jointly encoding 3D geometric structure and semantic information. Compared to bird's-eye view-based collaborative perception, voxel-level semantic occupancy provides a 3D voxel grid representation of the scene, which can support a more comprehensive and finer-grained understanding of the 3D scene and improve the reliability of downstream tasks such as motion planning, collision avoidance, and trajectory prediction.

[0004] While multi-vehicle collaboration can improve the completeness of scene information and alleviate occlusion problems, it also introduces additional computational load and wireless data communication overhead, significantly increasing overall perception latency. Excessive perception latency leads to lagging environmental representation, forcing the system to make decisions based on outdated information, thus increasing collision risk. In a cooperative perception system, each autonomous vehicle generates approximately 2MB of perception features per frame on average. When three autonomous vehicles participate in collaboration, with a 100Mbps communication link, the average communication latency per frame is approximately 480ms, and the computation latency is approximately 187ms, resulting in a total latency of approximately 667ms per frame. To maintain high perception accuracy under bandwidth-constrained conditions, the cooperative perception system should compress intermediate features and employ a selective transmission strategy to reduce communication and computational overhead.

[0005] Existing cooperative sensing methods for occupancy prediction have explored various filtering and compression strategies aimed at reducing communication overhead. In these methods, the cooperating vehicle independently selects, compresses, and transmits the most valuable features, while the autonomous vehicle passively receives and fuses these features for occupancy prediction. However, the above methods mainly suffer from the following two problems:

[0006] (1) It is difficult to accurately match the selected features with the vehicle's perception requirements;

[0007] The cooperating vehicle selects the most valuable features and transmits them to the autonomous vehicle. Typically, the value of these features is measured primarily based on the cooperating vehicle's local importance estimation, utilizing mechanisms such as spatial confidence maps, feature saliency, or information entropy. During this process, the autonomous vehicle passively receives features. However, since feature selection relies heavily on the cooperating vehicle's local spatial priors, it may not meet the autonomous vehicle's task requirements. Specifically, from the autonomous vehicle's perspective, some received features are redundant, while some key features are missing, leading to a decline in perception performance.

[0008] (2) Delay increases with the number of collaborators;

[0009] As the number of collaborators increases, each vehicle transmits only independently selected features without adjusting the sparsity of the features based on the current collaborator cardinality. This lack of adaptability leads to redundant transmission, significantly increasing communication bandwidth consumption and the computational overhead of the vehicle, thereby reducing real-time performance. Overall, most existing methods suffer from mismatch with the needs of the vehicle and lack of adaptation to the collaborator cardinality, ultimately resulting in decreased perception accuracy and increased system latency.

[0010] Therefore, further improvements to existing technologies are needed. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a multi-vehicle collaborative perception method that can reduce computational overhead and improve query accuracy, in contrast to the above-mentioned prior art.

[0012] The technical solution adopted by this invention to solve the above-mentioned technical problems is a multi-vehicle cooperative perception method, characterized by the following steps:

[0013] Step 1: Acquire multi-view images taken by the vehicle itself and the cooperating vehicles respectively;

[0014] Step 2: Construct a collaborative perception model. The constructed collaborative perception model includes a voxel generator, a compression fusion module, and an occupancy prediction head connected in sequence. The voxel generator converts multi-view images into voxel features based on a probabilistic kernel truncated voxel generation mechanism. The compression fusion module fuses voxel features between collaborative vehicles based on a cardinality conditional attention mechanism and selectively aggregates key occupancy perception cues.

[0015] Step 3: Simultaneously input the acquired multi-view images of the autonomous vehicle and the cooperative vehicle into the cooperative perception model to obtain the perception results of the environment in which the autonomous vehicle and the cooperative vehicle are located.

[0016] The specific processing steps of the collaborative sensing model are as follows:

[0017] The multi-view images of the vehicle are input into the voxel generator to generate voxel features of the vehicle; the multi-view images of the cooperating vehicles are input into the voxel generator to generate voxel features of the cooperating vehicles; the voxel features of the vehicle and the cooperating vehicles are then input into the compression fusion module to obtain the fused features; finally, the fused features are input into the occupancy prediction head to obtain the perception results of the environment in which the vehicle and the cooperating vehicles are located.

[0018] To improve query accuracy while reducing computational overhead, the specific processing procedure of the voxel generator in step 2 is as follows:

[0019] Step 2-1: Input the multi-view images into the feature extraction network to obtain the first feature map;

[0020] Step 2-2: Input the first feature map into a convolutional layer to obtain the second feature map. The convolutional layer is used to project the number of channels of the first feature map to the target dimension, that is: the number of channels of the second feature map is the target dimension;

[0021] Steps 2-3: Perform depth estimation on the first feature map to obtain the depth distribution map. , For pixel indices on the depth distribution map, For depth labeling, Let be the total number of depths, and calculate the posterior distribution of depth per pixel. The depth posterior distribution Including Posterior probability of depth;

[0022]

[0023] in, It is an exponential function with the natural constant e as its base;

[0024] Steps 2-4: For each pixel, sort all posterior probabilities in the depth posterior distribution in descending order to obtain a new depth posterior distribution. Then obtain the probability kernel according to the following formula. The probability kernel A collection consisting of multiple depths;

[0025]

[0026] in, This is the coverage threshold; This means that the total probability of obtaining the sorted posterior probability is greater than or equal to the depth set corresponding to the coverage threshold, and the depth set with the fewest depths is obtained from all depth sets;

[0027] Steps 2-5: Calculate the sparse cone features using the following formulas. ;

[0028]

[0029] in, For the second feature map Up pixel The channel vector at that location;

[0030] Steps 2-6: Use a differentiable sampler to sample each frustum element. ≡ Projected onto a voxel mesh, geometrically aligned voxel features are obtained, for index . voxels, their voxel characteristics The calculation formula is:

[0031]

[0032] in, For index The set of view cone elements contained within the trilinear interpolation neighborhood of a voxel, i.e.: The projection of each view frustum element falls on the index of Within the trilinear neighborhood of the voxel; It is a differentiable sampler for each view frustum element Assign non-negative mapping weights to its 8 neighboring voxels; for each frustum element In this case, the sum of all mapping weights over 8 neighboring voxels is 1.

[0033] Preferably, the feature extraction network in step 2-1 is a Unet network.

[0034] In order to eliminate redundancy caused by a fixed number of selections in simple regions and to avoid premature pruning in uncertain regions, steps 2-4... The calculation formula is:

[0035]

[0036] in, This is a truncation function. Indicates will Limited to [ Within the range, This is the minimum value of the coverage threshold. To cover the maximum value of the threshold, and All are weights. , ≥0; For normalized Shannon entropy, To normalize the depth variance, and All of these are constants that control sensitivity. ≥1;

[0037] The calculation formula is:

[0038] ;

[0039] The calculation formula is:

[0040] ;

[0041] Indicates the first The center depth of a depth set This represents the maximum standard deviation.

[0042] To achieve proactive on-demand querying at the voxel level and adaptive adjustment of feature sparsity, thereby resolving issues of misaligned feature requirements and communication redundancy, the specific processing procedure of the compression fusion module is as follows:

[0043] Step a: Calculate according to the following formula. Each query signal yields a query signal set. ;

[0044] ;

[0045] in, To capture the voxel features of the vehicle The result obtained by inputting into the global average pooling layer; The result is obtained by inputting the result of the global average pooling layer into the multilayer perceptron; The first query signal transpose; The first A query signal transpose;

[0046] Step b: Input the voxel features of each cooperating vehicle into the axial attention module to obtain refined voxel features, and decompose the refined voxel features into... Each axial slice is flattened into a tensor along the remaining dimensions, and then the key vector and value vector of each tensor are computed.

[0047] No. Key vectors of cooperative vehicles Sum value vector The calculation formula is:

[0048]

[0049] in, For the first The tensor corresponding to each cooperative vehicle. This indicates mapping the sequence to 3D key vector, This represents the corresponding value vector used to generate attention aggregation;

[0050] Step c: Generate the voxel feature Y of the queried cooperative vehicle according to the following formula;

[0051]

[0052] in, For the voxel features of the cooperative vehicle retrieved based on the first query signal transpose, According to the first The cooperative vehicle voxel features retrieved by the query signal transpose;

[0053] According to the The cooperative vehicle voxel features retrieved by the query signal The calculation formula is: ;

[0054] in, ∈{1, 2, ... }; For the number of vehicles involved in the collaboration. for The corresponding fusion weights; For the first The query signal was found to be the first Voxel characteristics of a collaborative vehicle , For the first The key vectors of all tensors of a cooperative vehicle. To calculate the dot product; Temperature is sensed based on a base value; For the first A vector of values ​​for all tensors of a cooperating vehicle;

[0055] Step d: Calculate the fused features according to the following formula. , The calculation formula is:

[0056]

[0057] in, An axis-aligned broadcast projection module is used to project features back to the voxel mesh; This is a semantic integration module used to rearrange and compress all features along the dimensions of the query signal; To capture the voxel features of the vehicle The result obtained by inputting into a 3D convolutional layer.

[0058] In order to adaptively suppress redundant cooperative vehicle information as the number of cooperative vehicles increases, the The calculation formula is:

[0059]

[0060] in, for The corresponding attention weights, , To query the signal and the first The correlation between collaborative vehicles ; , This is a lower bound for numerical stability. Based on the base temperature, ; For use in controlling the number of cooperating vehicles A constant representing the rate of decrease in effective temperature when the temperature increases; This indicates an indicator function that outputs 1 when a condition is met, and 0 otherwise; that is: The meaning is: when hour, =1; when At that time, =0; This is a relative threshold. , This is the preset minimum value.

[0061] To effectively mitigate the severe class imbalance problem while ensuring the accuracy of occupancy estimation, the training phase of the occupancy prediction head adopts a distribution-aligned reweighted unified loss, which is a combination of the class-balanced semantic occupancy cross-entropy with distribution-aligned reweighting and a binary occupancy term for geometric guidance.

[0062] Compared with the prior art, the advantages of the present invention are as follows:

[0063] 1. The voxel generator converts multi-view images into voxel features through a probabilistic kernel truncation voxel generation mechanism, which preserves the main geometric structure while suppressing long-tail noise; thereby reducing computational overhead and improving query accuracy.

[0064] 2. The compression and fusion module utilizes query signals combined with a cardinality-conditional attention mechanism to fuse voxel features among cooperating vehicles and selectively aggregate key perceptual cues. Therefore, it achieves voxel-level query and fusion of vehicle-conditional features through query signals, and performs cardinality-adaptive filtering and compression. This allows it to adaptively suppress redundant cooperating vehicle information as the number of cooperating vehicles increases, further reducing computational overhead. Attached Figure Description

[0065] Figure 1 This is a block diagram illustrating the principle of the multi-vehicle cooperative perception method in an embodiment of the present invention.

[0066] Figure 2 for Figure 1 Block diagram of the principle of the middle voxel generator;

[0067] Figure 3 for Figure 1 Block diagram of the compression fusion module. Detailed Implementation

[0068] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0069] like Figure 1 As shown, the multi-vehicle cooperative perception method in this embodiment includes the following steps:

[0070] Step 1: Acquire multi-view images taken by the vehicle itself and the cooperating vehicles respectively;

[0071] Step 2: Construct a collaborative perception model. The constructed collaborative perception model includes a voxel generator, a compression fusion module, and an occupancy prediction head connected in sequence. The voxel generator converts multi-view images into voxel features based on a probabilistic kernel truncated voxel generation mechanism. The compression fusion module fuses voxel features between collaborative vehicles based on a cardinality conditional attention mechanism and selectively aggregates key occupancy perception cues.

[0072] Step 3: Simultaneously input the multi-view images of the autonomous vehicle and the cooperative vehicle into the cooperative perception model to obtain the perception results of the environment where the autonomous vehicle and the cooperative vehicle are located.

[0073] The specific processing steps of the collaborative sensing model are as follows:

[0074] The multi-view images of the vehicle are input into the voxel generator to generate voxel features of the vehicle; the multi-view images of the cooperating vehicles are input into the voxel generator to generate voxel features of the cooperating vehicles; the voxel features of the vehicle and the cooperating vehicles are then input into the compression fusion module to obtain the fused features; finally, the fused features are input into the occupancy prediction head to obtain the perception results of the environment in which the vehicle and the cooperating vehicles are located.

[0075] like Figure 2 As shown, the specific processing procedure of the voxel generator in this embodiment is as follows:

[0076] Step 2-1: Input the multi-view images into the feature extraction network to obtain the first feature map;

[0077] In this embodiment, the feature extraction network is the Unet network, which is existing technology and will not be described in detail here.

[0078] Step 2-2: Input the first feature map into a convolutional layer to obtain the second feature map. The convolutional layer is used to project the number of channels of the first feature map to the target dimension, that is: the number of channels of the second feature map is the target dimension;

[0079] Steps 2-3: Perform depth estimation on the first feature map to obtain the depth distribution map. , For pixel indices on the depth distribution map, For depth labeling, Let be the total number of depths, and calculate the posterior distribution of depth per pixel. The depth posterior distribution Including Posterior probability of depth;

[0080]

[0081] in, It is an exponential function with the natural constant e as its base;

[0082] Steps 2-4: For each pixel, sort all posterior probabilities in the depth posterior distribution in descending order to obtain a new depth posterior distribution. Then obtain the probability kernel according to the following formula. The probability kernel A collection consisting of multiple depths;

[0083]

[0084] in, This is the coverage threshold; This means that the total probability of obtaining the sorted posterior probability is greater than or equal to the depth set corresponding to the coverage threshold, and the depth set with the fewest depths is obtained from all depth sets;

[0085] For ease of understanding The meaning of these terms is explained in the following examples in this embodiment:

[0086] Suppose that the depth of one pixel is {1m, 2m, 3m, 4m, 5m}; (m is an abbreviation for meter); the posterior probability corresponding to a depth of 1m is 0.5; the posterior probability corresponding to a depth of 2m is 0.05; the posterior probability corresponding to a depth of 3m is 0.15; the posterior probability corresponding to a depth of 4m is 0.2; and the posterior probability corresponding to a depth of 5m is 0.1.

[0087] First, sort all posterior probabilities in descending order, resulting in: 0.5, 0.2, 0.15, 0.1, 0.05; assuming... Then it satisfies The given depth sets are {1m, 4m}, {1m, 3m, 2m}, and {1m, 3m, 5m}. Since {1m, 4m} contains only 2 depths, while the other two depth sets contain 3 depths, the depth set with the fewest depths is {1m, 4m}. ={1m, 4m};

[0088] The calculation formula is:

[0089]

[0090] in, This is a truncation function. Indicates will Limited to [ Within the range, This is the minimum value of the coverage threshold. To cover the maximum value of the threshold, and All are weights. , ≥0; For normalized Shannon entropy, To normalize the depth variance, and All of these are constants that control sensitivity. ≥1;

[0091] The calculation formula is:

[0092] ;

[0093] The calculation formula is:

[0094] ;

[0095] Indicates the first The center depth of a depth set Indicates the maximum standard deviation;

[0096] Steps 2-5: Calculate the sparse cone features using the following formulas. ;

[0097]

[0098] in, For the second feature map Up pixel The channel vector at that location;

[0099] Steps 2-6: Use a differentiable sampler to sample each frustum element. ≡ Projected onto a voxel mesh, geometrically aligned voxel features are obtained, for index . voxels, their voxel characteristics The calculation formula is:

[0100]

[0101] in, For index The set of view cone elements contained within the trilinear interpolation neighborhood of a voxel, i.e.: The projection of each view frustum element falls on the index of Within the trilinear neighborhood of the voxel; It is a differentiable sampler for each view frustum element Assign non-negative mapping weights to its 8 neighboring voxels; for each frustum element In this case, the sum of all mapping weights over 8 neighboring voxels is 1;

[0102] In this embodiment, the differentiable sampler performs both geometric transfer and voxel accumulation simultaneously. Given the calibration extrinsic parameters, the induced mapping will map each frustum element... ≡ Projected into a voxel grid.

[0103] In steps 2-3, directly matching and calculating the complete posterior distribution for each pixel is computationally and storage-intensive, and is easily affected by noise from low-probability long tails. Therefore, in order to achieve a balance between fidelity and efficiency, this embodiment introduces an uncertainty-calibrated coverage threshold, i.e., the coverage threshold. Its value range is Furthermore, it adaptively increases with the increase of local posterior uncertainty. Therefore, this invention employs two complementary uncertainty measures: normalized Shannon entropy. With normalized depth variance Normalized Shannon entropy It concerns the posterior distribution. A distribution structure function independent of metric scale, used to quantify the dispersion and multimodality of a distribution, thereby reflecting the size of the effective support set and the number of competing modes; in contrast, normalized depth variance... It is relative to the depth set center Furthermore, the second-order central moment, which is related to the metric scale, can characterize the degree of geometric diffusion and modal separation, thus maintaining high accuracy for discretization scales and non-uniform binning. This is further enhanced by the combined use of normalized Shannon entropy. With normalized depth variance It can provide complementary descriptions of uncertainty from both probabilistic shape and geometric scale dimensions, thereby enabling more precise tuning of the coverage threshold in the voxel generator. and kernel size This design reduces computational and communication overhead while preserving the main geometric structure. The obtained kernel size about It is non-decreasing and can naturally adapt to uncertainty: for pixels with high confidence, the kernel size... It will shrink to a small depth set; while in the fuzzy region, It will expand to preserve the main geometry.

[0104] The above coverage threshold Calibration is performed based on local uncertainty. A larger posterior variance results in a higher coverage threshold, thus retaining more depth sets; conversely, for pixels with high confidence, the coverage threshold is lower, leading to earlier truncation. This adaptive truncation mechanism eliminates redundancy from fixed-number selections in simple regions and avoids premature pruning in uncertain regions. Capacity is concentrated in regions with more divergent posterior distributions, such as those with weak textures, grazing views, occlusion, and thin structures, thereby preserving finer-grained geometric details and stabilizing the supervisory signal. Without reducing accuracy, this distribution-driven sparsity further reduces downstream computation and communication overhead. Semantic features are coupled only to the posterior distribution on the retained depth sets, forming distribution-driven sparse frustum features that suppress low-confidence noise while preserving key geometric evidence. Subsequently, using calibrated camera-to-vehicle extrinsic parameters, a differentiable sampler projects the frustum features from each viewpoint onto a unified voxel grid, obtaining geometrically aligned voxel features.

[0105] To accurately match the feature requirements of autonomous vehicles while maintaining low latency when the number of cooperating vehicles changes, this embodiment proposes a cardinality-conditional compression fusion module. This module uses a query signal to perform voxel-level active query and utilizes a cardinality-adaptive attention mechanism to dynamically control the sparsity of transmitted features as the number of cooperating vehicles changes.

[0106] like Figure 3As shown, the specific processing procedure of the compression fusion module is as follows:

[0107] Step a: Calculate according to the following formula. Each query signal yields a query signal set. ;

[0108] ;

[0109] in, To capture the voxel features of the vehicle The result obtained by inputting into the global average pooling layer; The result is obtained by inputting the result of the global average pooling layer into the multilayer perceptron; The first query signal transpose; The first A query signal The transpose of the above query signals are all outputs of the multilayer perceptron. These query signals are used to guide the on-demand querying and aggregation of features of cooperative vehicles, thereby improving perception accuracy while suppressing redundancy. The semantic scope of the query signals is adaptive: each query signal will dynamically adjust its focus according to the scene context, so that different query signals can capture different geometric or semantic patterns on demand.

[0110] Step b: Input the voxel features of each cooperating vehicle into the axial attention module to obtain refined voxel features, and decompose the refined voxel features into... Each axial slice is flattened into a tensor along the remaining dimensions, and then the key vector and value vector of each tensor are computed.

[0111] In this embodiment, an axial attention module is used to enhance directional cues commonly found in road scenes, such as slender lane markings, vehicle body structures, and occlusion boundaries. The axial attention module can capture long-range dependencies and improve directional selectivity, thereby generating refined features that are more suitable for query signals to guide queries.

[0112] Figure 3 The Middle The voxel characteristics of each collaborative vehicle are , No. Key vectors of cooperative vehicles Sum value vector The calculation formula is:

[0113]

[0114] in, For the first The tensor corresponding to each cooperative vehicle. This indicates mapping the sequence to 3D key vector, This represents the corresponding value vector used to generate attention aggregation;

[0115] Step c: Generate the voxel feature Y of the queried cooperative vehicle according to the following formula;

[0116]

[0117] in, For the voxel features of the cooperative vehicle retrieved based on the first query signal transpose, According to the first The cooperative vehicle voxel features retrieved by the query signal transpose;

[0118] According to the The cooperative vehicle voxel features retrieved by the query signal The calculation formula is: ;

[0119] in, ∈{1, 2, ... }; For the number of vehicles involved in the collaboration. for The corresponding fusion weights; For the first The query signal was found to be the first Voxel characteristics of a collaborative vehicle

[0120] The calculation formula is:

[0121]

[0122] in, for The corresponding attention weights, , To query the signal and the first The correlation between collaborative vehicles ; , This is a lower bound for numerical stability. This serves as the base temperature, used to control the smoothness of attention distribution in scenarios with few vehicles cooperating. ; For use in controlling the number of cooperating vehicles A constant representing the rate of decrease in effective temperature when the temperature increases; This indicates an indicator function that outputs 1 when a condition is met, and 0 otherwise; that is: The meaning is: when hour, =1; when At that time, =0; This is a relative threshold. , This is the preset minimum value;

[0123] To achieve cardinality-conditional compression, this embodiment uses the sharpness of the attention distribution (i.e., cardinality-sensing temperature) as the basis for compression. Designed for the number of collaborative vehicles The function, and with Modulation is applied to changes; when there are fewer cooperating vehicles, a larger [modulation] is used. This results in smoother attention allocation, thus retaining richer and more diverse information; while in multi-vehicle collaborative scenarios, smaller... This will make the selection sharper, suppressing redundant information; relative threshold Further used to enhance sparsity; for each query signal Only attention weights are retained. Exceeding the relative threshold Collaborative vehicles; in practice, It is usually set as about A monotonically increasing function such that when When smaller, retain more collaborative vehicles, while as For larger plants, a more aggressive pruning strategy is adopted;

[0124] Therefore, cardinal sensing temperature With relative threshold Together, they constitute a selective compression mechanism at the vehicle level based on query signals, which can adaptively suppress redundant cooperative vehicle information as the number of cooperative vehicles increases.

[0125] Within each vehicle, the same query signal performs a fine-grained query on the axial component, in the form of:

[0126] ;

[0127] For the first The key vectors of all tensors of a cooperative vehicle. To calculate the dot product; Temperature is sensed based on a base value; For the first A vector of values ​​for all tensors of a cooperating vehicle;

[0128] Step d: Calculate the fused features according to the following formula. , The calculation formula is:

[0129]

[0130] in, An axis-aligned broadcast projection module is used to project features back to the voxel mesh; This is a semantic integration module used to rearrange and compress all features along the dimensions of the query signal; To capture the voxel features of the vehicle The result obtained by inputting into a 3D convolutional layer.

[0131] In this embodiment, only the compressed features that are queried are transmitted to the vehicle, thereby preserving key occupancy perception clues while reducing communication overhead as the cooperative cardinality changes. The vehicle recovers a unified voxel representation through a two-step mapping. The final result Geometric alignment is used for subsequent occupancy decoding.

[0132] In this embodiment, the training phase of the occupancy prediction head adopts the distribution-aligned reweighted unified loss. This distribution-aligned reweighted unified loss is a combination of the class-balanced semantic occupancy cross-entropy with distribution-aligned reweighting and the binary occupancy term used for geometric guidance, in order to alleviate class imbalance and calibrate semantic probabilities at the occupancy boundary, thereby improving the quality of query target and fused features.

[0133] The occupancy prediction head network outputs the predicted distribution of all semantic categories for each voxel (including free space and various occupancy semantics). Semantic terms use cross-entropy, but are not directly calculated with equal weights. Instead, the semantic supervision of each voxel is multiplied by two types of weights before summing. The first type of weight is a class balance weight, calculated based on the frequency of each semantic category in the training data, ensuring that rare categories have a higher weighting in the loss and common categories a lower weighting, while normalization ensures overall scale stability. The second type of weight is distribution alignment reweighting, which adaptively corrects the deviation between the model's current category prediction prior and the target category prior: if some categories are systematically underestimated in the prediction distribution, their weights are increased; if they are overestimated, their weights are decreased. To avoid training oscillations, distribution alignment typically uses moving average statistics to estimate the prior, and the reweighting coefficients are pruned and normalized, thereby improving the learning effect of long-tail categories while maintaining optimization stability.

[0134] Beyond the semantic term, the distribution-aligned reweighted unified loss also introduces a binary occupancy term as a geometric guide. In implementation, the semantic prediction distribution is converted into an occupancy probability (obtained by summing the probabilities of all non-free space categories), and this probability is used to calculate the binary cross-entropy with the true binary occupancy values ​​of the voxels. This term is particularly sensitive to occupancy boundaries, suppressing the problem of semantically correct but geometrically ambiguous or unclear boundaries, thus allowing the query target and fused features to focus more on truly occupiable structures and clear boundaries.

[0135] The final distribution alignment reweighted unified loss is a weighted sum of the semantic terms and the binary occupancy term. During training, it is typically calculated only on effective voxels and averaged over all voxels and batches. Long-tail semantic imbalance is addressed through class balance and distribution alignment, while geometric boundaries and probabilistic calibration are reinforced through the binary occupancy term, thereby improving the overall supervision quality and training stability of collaborative occupancy awareness.

[0136] This invention extracts key geometric structures and suppresses noise through a probabilistic kernel-truncated voxel generation mechanism. A cardinality-conditional attention mechanism is employed in the compression fusion module to achieve proactive on-demand querying at the voxel level and adaptive adjustment of feature sparsity, resolving issues of feature demand misalignment and communication redundancy. Simultaneously, distribution-aligned reweighted loss optimizes model training, thereby effectively improving occupancy prediction accuracy in dynamic collaborative scenarios, significantly reducing system overhead, and enhancing real-time performance. The method of this invention allows autonomous vehicles to proactively generate feature queries to guide collaborative vehicles in more accurate feature selection. Compared to passive feature reception methods, this invention improves collaborative perception occupancy prediction accuracy while effectively controlling communication and computational overhead as the number of collaborative vehicles increases.

[0137] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-vehicle cooperative perception method, characterized in that... Includes the following steps: Step 1: Acquire multi-view images taken by the vehicle itself and the cooperating vehicles respectively; Step 2: Construct a collaborative perception model. The constructed collaborative perception model includes a voxel generator, a compression fusion module, and an occupancy prediction head connected in sequence. The voxel generator converts multi-view images into voxel features based on a probabilistic kernel truncation voxel generation mechanism. The compression fusion module fuses voxel features among collaborative vehicles based on a cardinality-conditional attention mechanism, selectively aggregating key perceptual cues. Step 3: Simultaneously input the acquired multi-view images of the autonomous vehicle and the cooperative vehicle into the cooperative perception model to obtain the perception results of the environment in which the autonomous vehicle and the cooperative vehicle are located. The specific processing steps of the collaborative sensing model are as follows: The multi-view images of the vehicle are input into the voxel generator to generate voxel features of the vehicle; the multi-view images of the cooperating vehicles are input into the voxel generator to generate voxel features of the cooperating vehicles; the voxel features of the vehicle and the cooperating vehicles are then input into the compression fusion module to obtain the fused features; finally, the fused features are input into the occupancy prediction head to obtain the perception results of the environment in which the vehicle and the cooperating vehicles are located.

2. The multi-vehicle cooperative perception method according to claim 1, characterized in that: The specific processing procedure of the voxel generator in step 2 is as follows: Step 2-1: Input the multi-view images into the feature extraction network to obtain the first feature map; Step 2-2: Input the first feature map into a convolutional layer to obtain the second feature map. The convolutional layer is used to project the number of channels of the first feature map to the target dimension, that is: the number of channels of the second feature map is the target dimension; Steps 2-3: Perform depth estimation on the first feature map to obtain the depth distribution map. , For pixel indices on the depth distribution map, For depth labeling, Let be the total number of depths, and calculate the posterior distribution of depth per pixel. The depth posterior distribution Including Posterior probability of depth; in, It is an exponential function with the natural constant e as its base; Steps 2-4: For each pixel, sort all posterior probabilities in the depth posterior distribution in descending order to obtain a new depth posterior distribution. Then obtain the probability kernel according to the following formula. The probability kernel A collection consisting of multiple depths; in, This is the coverage threshold; This means that the total probability of obtaining the sorted posterior probability is greater than or equal to the depth set corresponding to the coverage threshold, and the depth set with the fewest depths is obtained from all depth sets. Steps 2-5: Calculate the sparse cone features using the following formulas. ; in, For the second feature map Up pixel The channel vector at that location; Steps 2-6: Use a differentiable sampler to sample each frustum element. ≡ Projected onto a voxel mesh, geometrically aligned voxel features are obtained, for index . voxels, their voxel characteristics The calculation formula is: in, For index The set of view cone elements contained within the trilinear interpolation neighborhood of a voxel, i.e.: The projection of each view frustum element falls on the index of Within the trilinear neighborhood of the voxel; It is a differentiable sampler for each view frustum element Assign non-negative mapping weights to its 8 neighboring voxels; for each frustum element In this case, the sum of all mapping weights over 8 neighboring voxels is 1.

3. The multi-vehicle cooperative perception method according to claim 2, characterized in that: The feature extraction network in step 2-1 is the Unet network.

4. The multi-vehicle cooperative perception method according to claim 2, characterized in that: In steps 2-4 The calculation formula is: in, This is a truncation function. Indicates will Limited to [ Within the range, This is the minimum value of the coverage threshold. To cover the maximum value of the threshold, and All are weights. , ≥0; For normalized Shannon entropy, To normalize the depth variance, and All of these are constants that control sensitivity. ≥1; The calculation formula is: ； The calculation formula is: ； Indicates the first The center depth of a depth set This represents the maximum standard deviation.

5. The multi-vehicle cooperative perception method according to claim 2, characterized in that: The specific processing procedure of the compression fusion module is as follows: Step a: Calculate according to the following formula. Each query signal yields a query signal set. ; ； in, To capture the voxel features of the vehicle The result obtained by inputting into the global average pooling layer; The result is obtained by inputting the result of the global average pooling layer into the multilayer perceptron; The first query signal Transpose of; The first A query signal Transpose of; Step b: Input the voxel features of each cooperating vehicle into the axial attention module to obtain refined voxel features, and decompose the refined voxel features into... Each axial slice is flattened into a tensor along the remaining dimensions, and then the key vector and value vector of each tensor are computed. No. Key vectors of cooperative vehicles Sum value vector The calculation formula is: in, For the first The tensor corresponding to each cooperative vehicle. This indicates mapping the sequence to 3D key vector, This represents the corresponding value vector used to generate attention aggregation; Step c: Generate the voxel feature Y of the queried cooperative vehicle according to the following formula; in, For the voxel features of the cooperative vehicle retrieved based on the first query signal transpose, According to the first The cooperative vehicle voxel features retrieved by the query signal Transpose of; According to the The cooperative vehicle voxel features retrieved by the query signal The calculation formula is: ; in, ∈{1, 2, ... }; For the number of vehicles involved in the collaboration. for The corresponding fusion weights; For the first The query signal was found to be the first Voxel characteristics of a collaborative vehicle , For the first The key vectors of all tensors of a cooperative vehicle. To calculate the dot product; Temperature is sensed based on a base value; For the first A vector of values ​​for all tensors of a cooperative vehicle; Step d: Calculate the fused features according to the following formula. , The calculation formula is: in, An axis-aligned broadcast projection module is used to project features back to the voxel mesh; This is a semantic integration module used to rearrange and compress all features along the dimensions of the query signal; To capture the voxel features of the vehicle The result obtained by inputting into a 3D convolutional layer.

6. The multi-vehicle cooperative perception method according to claim 5, characterized in that: The The calculation formula is: in, for The corresponding attention weights, , To query the signal and the first The correlation between collaborative vehicles ; , This is a lower bound for numerical stability. Based on the base temperature, ; For use in controlling the number of cooperating vehicles A constant representing the rate of decrease in effective temperature when the temperature increases; This indicates an indicator function that outputs 1 when a condition is met, and 0 otherwise; that is: The meaning is: when hour, =1; when At that time, =0; This is a relative threshold. , This is the preset minimum value.

7. The multi-vehicle cooperative perception method according to any one of claims 1 to 6, characterized in that: The training phase of the occupancy prediction head uses a distributed alignment reweighted unified loss, which is a combination of the category-balanced semantic occupancy cross-entropy with distributed alignment reweighting and a binary occupancy term for geometric guidance.

Images