A multi-agent collaborative perception feature enhancement method based on context aggregation

Abstract

The application discloses a multi-agent collaborative perception feature enhancement method based on context aggregation, relates to the technical field of automatic driving and intelligent networked vehicles, and comprises the following steps: acquiring current frame features and at least one frame of historical features; adaptively aligning the historical features based on a motion prediction network and a deformable convolution offset; generating semantic consistency weights and identifying time sequence discontinuous regions based on a correlation prediction network; fusing the current and historical features according to the weights and carrying out state space selective scanning, pooling, denoising and aggregation; generating multi-scale features, combining scene complexity, quality evaluation and time sequence consistency constraints to determine scale weights and then fusing; and inputting an LSTM to perform time sequence modeling and output enhanced context features. Invalid alignment is avoided through motion compensation and semantic consistency constraints, noise and scale jitter are suppressed through position-level denoising and multi-scale adaptive smoothing, and the collaborative perception robustness and detection accuracy are improved.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and intelligent connected vehicle technology, specifically to a multi-agent collaborative perception feature enhancement method based on context aggregation. Background Technology

[0002] Autonomous driving, as a crucial application of artificial intelligence and intelligent transportation systems, hinges on the accurate and real-time perception of complex traffic environments. Traditional single-vehicle perception systems rely on onboard sensors to acquire information about the surrounding environment; however, limitations such as sensor field of view, adverse weather conditions, and obstructions often result in blind spots and insufficient accuracy. To address these issues, perception technologies based on vehicle-to-everything (V2X) and multi-vehicle collaboration have emerged. Through information sharing and collaborative processing among multiple intelligent agents, these technologies significantly improve the coverage and detection accuracy of perception systems.

[0003] However, existing collaborative sensing methods still face many technical challenges in feature aggregation and temporal information processing:

[0004] First, traditional feature aggregation methods mainly employ simple concatenation, summation, or averaging operations, lacking the ability to intelligently identify the importance of features. These methods easily lead to the dilution of important features and indiscriminately introduce noise and redundant information into the fusion result, severely impacting the final perception accuracy. Especially in complex traffic scenarios, the quality of features acquired by different sensors varies significantly, making it difficult for simple aggregation strategies to leverage the advantages of collaborative perception.

[0005] Secondly, the effective utilization of temporal information is key to improving perception performance, but existing methods suffer from low alignment accuracy when processing historical frame features. Traditional geometric transformation methods assume rigid body motion, which cannot adapt to the complex motion patterns of dynamic targets such as vehicles and pedestrians, leading to the accumulation of feature alignment errors. Furthermore, when new targets appear in the scene or targets reappear after being occluded, temporal discontinuities arise. Traditional methods force motion compensation for all regions, which introduces ineffective noise interference.

[0006] Furthermore, existing pooling operations often employ fixed aggregation strategies, such as average pooling and max pooling. These methods either indiscriminately smooth all features, leading to the loss of important information, or they may amplify noise peaks and cannot adaptively adjust to specific data characteristics. In multi-source heterogeneous collaborative sensing data, the importance of features at different spatial locations varies significantly, making it difficult for fixed-strategy pooling methods to achieve accurate feature selection and noise suppression.

[0007] Finally, multi-scale feature processing is an important means of dealing with the detection of targets of different sizes, but traditional methods usually adopt a fixed scale selection strategy, which cannot be dynamically adjusted according to the complexity of the scene. This leads to a waste of computational resources in simple scenes and insufficient feature representation ability in complex scenes. More importantly, existing methods lack consideration for temporal consistency, and frame-by-frame independent scale selection is prone to abrupt changes, affecting the stability and continuity of perception results.

[0008] While existing technologies have proposed methods such as attention mechanisms, graph neural networks, and Transformers to improve feature aggregation, these methods still have shortcomings in terms of computational complexity, real-time requirements, and the unique challenges of collaborative perception. Attention mechanisms, although capable of learning feature weights, suffer from high computational overhead and are sensitive to noise; graph neural networks require explicit modeling of the relationship graph between agents, leading to complex graph structure updates in dynamic scenarios; and while Transformers excel in sequence modeling, their computational and storage requirements are enormous when processing high-dimensional feature maps.

[0009] Considering the aforementioned technical limitations, there is an urgent need for a collaborative perception feature enhancement method that can intelligently aggregate multi-source features, accurately process temporal information, adaptively suppress noise interference, and dynamically select the processing scale, in order to meet the urgent needs of autonomous driving systems for high-precision, high-real-time, and high-robust perception capabilities. Summary of the Invention

[0010] Based on the shortcomings of the prior art described above, the purpose of this invention is to provide a multi-agent collaborative perception feature enhancement method based on context aggregation to solve the above-mentioned technical problems.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a multi-agent collaborative perception feature enhancement method based on context aggregation, comprising:

[0012] Obtain the features of the current frame and the features of at least one historical frame;

[0013] Motion cues are extracted from the feature differences between adjacent frames and deformable convolution offsets are predicted by a motion prediction network, and historical features are adaptively aligned.

[0014] The semantic consistency between the current frame and the aligned historical features is evaluated through a correlation prediction network, which generates correlation weights and identifies temporally discontinuous regions.

[0015] Based on the relevance weights, the aligned historical features and the current frame features are spatially adaptively fused to obtain the fused features;

[0016] The fused features are input into the selective scanning pooling module, which learns and updates a strategy to suppress noise for each spatial location based on the state space mechanism, and outputs denoised fused features.

[0017] The denoised and fused features are used to generate multi-scale feature representations and perform scene complexity analysis. The scale selection weights are determined by combining the quality assessment network and temporal consistency constraints, and the transition is smoothed.

[0018] The multi-scale features are weighted and fused according to the weights and then input into the LSTM temporal processing module for dependency modeling. The output temporally enhanced contextual features are used for collaborative perception tasks.

[0019] The present invention is further configured such that the motion prediction network extracts motion cues by calculating the inter-frame feature differences between the current frame features and the historical frame features, predicts the sampling offset parameters of the deformable convolution based on the motion cues, and performs offset-based adaptive sampling alignment on the historical frame features to obtain aligned historical frame features.

[0020] The present invention is further configured such that the relevance prediction network combines the features of the current frame and the features of the historical frame in the channel dimension, and outputs a relevance weight map corresponding to the spatial location, which is used to characterize the semantic consistency between the current frame and the historical frame at the corresponding position, and determines the low relevance region as the temporally discontinuous region.

[0021] The present invention is further configured such that effective alignment guided by relevance weights includes: the aligned historical frame features and the current frame features are fused position by position according to the relevance weights, so that the aligned historical information is introduced first at high relevance positions and the current frame information is retained first at low relevance positions, in order to avoid invalid motion compensation in newly appearing targets or occlusion recurrence areas.

[0022] The present invention is further configured such that the selective scanning pooling module includes flattening the fused features from a spatial feature map into sequence features, generating a main information stream and a gated information stream through a learnable projection layer; performing a one-dimensional convolution on the main information stream to extract local spatial relationships, and learning an adaptive update strategy for each position in the sequence, which is used to enhance important positions and attenuate noisy positions in recursive updates.

[0023] The present invention is further configured such that the recursive update of the selective scanning pooling module is implemented using a state-space mechanism, and an adaptive time step parameter with positive values ​​is learned for each sequence position. A stable recursive update process is constructed based on the learnable state transition parameters. At the same time, a pass-through connection is set to suppress over-smoothing and improve numerical stability.

[0024] The present invention is further configured such that the multi-scale temporal adaptive selection mechanism includes generating feature representations at multiple scales, and evaluating the scene complexity by a scene complexity analyzer; the scene complexity evaluation includes at least texture complexity detection based on gradient response, edge complexity detection based on edge operators, and target density detection based on the proportion of activated pixels, and the multi-dimensional detection results are fused to obtain a scene complexity characterization, which is used to drive scale selection.

[0025] The present invention is further configured such that dynamic scale selection includes: the quality score of each scale feature is output by the quality assessment network, and prior preferences are assigned to different scales in combination with the scene complexity characterization to obtain scale selection weights.

[0026] The present invention is further configured such that the temporal consistency constraint includes a memory mechanism for maintaining historical scale selection, detecting the degree of scene change between the current frame and historical frames, and adaptively adjusting the temporal smoothing intensity according to the degree of change, so as to enhance scale selection consistency when the scene is stable to avoid jumps, and reduce the smoothing intensity to maintain fast response when the scene changes drastically.

[0027] The present invention is further configured such that the LSTM temporal processing module inputs the multi-scale feature sequence after scale selection frame by frame, and controls the information flow through the input gate, forget gate and output gate, utilizes cell state to store and transmit key temporal information across time, utilizes hidden state to represent the features of the current time, and outputs temporally enhanced contextual features for collaborative perception tasks.

[0028] This invention provides a multi-agent collaborative perception feature enhancement method based on context aggregation. The method involves: acquiring current frame features and at least one frame of historical features; extracting motion cues from the differences between adjacent frame features and predicting deformable convolutional offsets using a motion prediction network to adaptively align historical features; evaluating the semantic consistency between the current frame and aligned historical features using a correlation prediction network to generate correlation weights and identify temporally discontinuous regions; spatially adaptively fusing the aligned historical features and current frame features according to the correlation weights to obtain fused features; inputting the fused features into a selective scanning pooling module, learning and updating strategies for each spatial location based on a state-space mechanism to suppress noise, and outputting denoised fused features; generating multi-scale feature representations from the denoised fused features and performing scene complexity analysis, combining a quality assessment network and temporal consistency constraints to determine scale selection weights and smooth transitions; weighting and fusing the multi-scale features according to the weights and inputting them into an LSTM temporal processing module for dependency modeling, outputting temporally enhanced contextual features for collaborative perception tasks. The beneficial effects include:

[0029] The motion prediction network learns motion patterns between adjacent frames and performs adaptive spatial transformations, effectively compensating for feature position shifts caused by vehicle and target motion. The correlation prediction network evaluates semantic consistency between frames, identifies newly appearing or recurring occluded regions in the scene, and avoids invalid alignment in temporally discontinuous regions.

[0030] The selective scanning pooling module achieves intelligent feature denoising and information aggregation through a state-space mechanism. This module learns an independent feature selection strategy for each spatial location. Locations containing important target information are preserved and enhanced, while noise components are gradually attenuated at locations affected by noise. This approach effectively suppresses noise interference while retaining valuable information, significantly improving the robustness of the perception task.

[0031] The multi-scale temporal adaptive selection mechanism can dynamically adjust the feature granularity according to scene complexity. In simple scenes, a coarse-grained scale is selected to obtain a larger receptive field and higher computational efficiency, while in complex scenes, a fine-grained scale is selected to retain more spatial details for accurate detection of small and dense targets. The introduced temporal smoothing optimization mechanism maintains selection consistency when the scene is stable and quickly adapts to new features when the scene changes, effectively avoiding temporal jitter in scale selection by dynamically adjusting the smoothing factor.

[0032] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0034] Figure 1 This is a flowchart of a multi-agent collaborative perception feature enhancement method based on context aggregation, according to an embodiment of the present invention.

[0035] Figure 2 This is a flowchart of some motion prediction networks and correlation prediction networks in an embodiment of the present invention;

[0036] Figure 3 This is a flowchart of selective scan pooling in an embodiment of the present invention;

[0037] Figure 4 This is a flowchart of a multi-scale time-adaptive selection process according to an embodiment of the present invention. Detailed Implementation

[0038] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0039] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0040] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0041] A multi-agent collaborative perception feature enhancement method based on context aggregation, such as Figure 1 As shown, it includes:

[0042] S1: Obtain the current frame features and historical frame features as input;

[0043] In the vehicle-road cooperative perception system, the roadside unit (RSU) and on-board unit (OBU) respectively collect point cloud data through LiDAR sensors. First, the current time is obtained through a point cloud feature extraction network. Feature map and historical moments Feature map ,in Indicates batch size, and indicates the number of feature channels. and These represent the height and width of the feature map, corresponding to a perception range of approximately 80 meters × 70 meters in actual physical space. These raw features will serve as the common input for subsequent modules such as motion prediction and correlation analysis, forming the starting point for information transmission.

[0044] To reflect multi-agent collaborative perception, current frame features and historical frame features are extracted separately by the roadside unit and the vehicle-mounted unit to form a collaborative feature set. Collaborative input preparation includes: timestamp alignment and cache management of data collected from different agents; unifying the features of different agents into the same bird's-eye view grid coordinate system based on calibration parameters and pose information; handling the latency and packet loss introduced by the communication link with validity marking and replacement strategies, using the most recently available features or only the features of the current agent when some agent features are unavailable; after the collaborative feature set is formed, aggregated features of the current and historical moments are constructed as common input for motion prediction and correlation analysis, ensuring that subsequent alignment and fusion are performed under unified coordinates and unified temporal semantics across multiple agents. Historical features are not limited to a single previous frame; a sliding time window can be used to cache multiple frames of historical features and organize them chronologically. When using multiple frames of historical features, subsequent motion alignment and correlation analysis can be performed separately for each frame of historical features and then converged during the fusion stage, enabling the system to obtain stable context support even in scenarios with low frame rates, long occlusion durations, or communication interruptions.

[0045] S2: Calculate inter-frame motion cues and generate motion context features through a motion prediction network;

[0046] like Figure 2 The flowcharts for the motion prediction network and the correlation prediction network are shown. The core objective of the motion prediction network is to learn motion patterns between adjacent frames, thereby accurately aligning historical features. First, it calculates the feature differences between adjacent frames to capture motion cues:

[0047]

[0048] in For motion context features. ConvNet is a motion coding network consisting of multiple convolutional layers. This network first uses a 3×3 convolutional layer to convert the input channels from... Compress to The features are then normalized and nonlinearly transformed by the BatchNorm layer and the ReLU activation function, and finally refined by another 3×3 convolutional layer to extract richer motion semantic information.

[0049] The offset parameters of deformable convolution are learned through an offset prediction network:

[0050]

[0051] in Indicates the first Two-dimensional offset of a spatial location, The square of the convolution kernel size is usually For a corresponding 3×3 convolution kernel, the offset includes direction and The displacement in the direction, therefore the number of channels is OffsetNet is an offset prediction network that learns motion context features to predict the sampling offset that should be used for each spatial location.

[0052] Adaptive alignment of historical features is performed using deformable convolutions:

[0053]

[0054] in This represents the aligned historical features. Deformable convolution, by adding an offset to a regular sampling grid, allows the convolution kernel to adaptively adjust its sampling position, better adapting to the deformation and motion of the target.

[0055] To avoid anomalous jumps in offset predictions in sparse texture or noisy regions, smoothing and amplitude constraints can be applied to the offset results during training and inference. For example, consistency constraints can be set for offset changes in adjacent spatial positions, extreme large displacements can be pruned or reverted to conventional sampling strategies, and confidence gating can be introduced when the vehicle makes sharp turns, experiences bumps, or the point cloud is sparse to reduce the impact of unreliable offsets on the alignment results, thereby improving the stability and repeatability of motion alignment.

[0056] While motion-aligned features compensate for the motion of vehicles and targets, forced alignment may introduce noise in areas where new objects appear or occlusions reappear in the scene. Therefore, a correlation prediction network is needed to identify these temporally discontinuous regions and provide guidance for selective alignment.

[0057] S3: Identify temporally discontinuous regions and generate correlation weights through a correlation prediction network;

[0058] The correlation prediction network identifies which regions are suitable for using aligned historical features and which regions should retain current frame information by evaluating inter-frame semantic consistency. The correlation prediction network receives both current frame and historical frame features as input.

[0059]

[0060] in This is a correlation weighting plot, with values ​​ranging from [value range missing]. The CorrelationNet network structure includes a feature concatenation layer that concatenates features from the current frame and historical frames along the channel dimension. The channel characteristics are then analyzed, and the number of channels is reduced to 1×1 using a 1×1 convolutional layer. To reduce computational complexity, a 3×3 convolutional layer is used for spatial information aggregation after the ReLU activation function. Finally, a 1×1 convolutional layer outputs a single-channel correlation map, and the output value is normalized using the Sigmoid activation function. scope.

[0061] The historical features used for relevance evaluation can be selected as either pre-alignment or post-alignment historical features, depending on the implementation: when post-alignment historical features are used for relevance evaluation, the relevance weights place more emphasis on semantic consistency judgment and reduce geometric misalignment interference; when pre-alignment historical features are used for relevance evaluation, newly emerging regions and occluded recurrence regions can be more sensitively identified. In actual deployment, one method can be fixed, or both types of historical features can be input simultaneously during the training phase, and the network can adaptively learn a more robust relevance discrimination.

[0062] Relevance weight The physical meaning is when Time indicates position The current frame features are highly correlated with historical frame features, making them suitable for motion alignment. This indicates that a new target or semantic discontinuity has appeared at that location, and the original features of the current frame should be preserved.

[0063] The correlation weight will serve as a spatially adaptive modulation signal to guide the weighted fusion of historical and current features in S4, achieving selective alignment guided by correlation and avoiding the introduction of invalid historical information in regions of temporal discontinuity.

[0064] S4: Effective alignment of historical features based on relevance weights;

[0065] We use correlation weights to perform weighted fusion of motion-aligned features and current frame features:

[0066]

[0067] in This indicates element-wise multiplication. This is the final fusion feature. The advantage of this fusion strategy is that it mainly uses historical features after motion alignment when the correlation is high. Make full use of temporal information; when the correlation is low, mainly retain the features of the current frame. To avoid noise introduced by invalid alignment, a smooth transition is achieved through continuous weight values ​​to avoid discontinuities caused by hard switching.

[0068] To further reduce the risk of misfusion in temporally discontinuous regions, protection strategies can be introduced for low-correlation regions: in low-correlation regions, the current frame is forced to be the primary frame and the proportion of historical information injected is limited; local continuity processing is introduced at the correlation transition boundary to avoid fragmentation points in the weight map that cause local artifacts; when a target is detected to appear rapidly or large-area occlusion is restored, the proportion of the current frame can be temporarily increased and the utilization of historical information can be gradually restored after a few frames, thereby suppressing misalignment noise caused by sudden changes while ensuring temporal utilization.

[0069] While the quality of the aligned features is improved, sensor noise is not filtered out. The selective scan pooling module will further denoise and aggregate the features.

[0070] S5: Learn adaptive weights and update strategies for each spatial location through selective scanning pooling modules;

[0071] like Figure 3 The selective scan pooling flowchart shown demonstrates adaptive feature aggregation through the introduction of a state-space mechanism.

[0072] Fusion features Flatten the spatial form into a sequence form for subsequent sequence processing:

[0073]

[0074] in The Each location corresponds to a spatial location in the original feature map. satisfy .

[0075] The main information stream and the gated information stream are generated through a learnable linear projection layer:

[0076]

[0077] in For sequence length, Take internal feature dimension , This is the expansion factor. The main information flow is used for state-space computation. The gated information flow is used for final gated fusion.

[0078] Perform one-dimensional convolution on the main information stream to extract local spatial relationships:

[0079]

[0080] Where Conv1D is a one-dimensional convolution operation, and the kernel size is taken as... Grouped convolutions are used to reduce the number of parameters. SiLU is defined as the Swish activation function. .

[0081] The serialized spatial locations are at risk of scanning order bias. To reduce the uneven information propagation caused by a single scanning direction, a multi-directional scanning strategy can be adopted. For example, scanning can be performed in different orders of row priority and column priority, and the results of multiple scanning can be merged. Alternatively, bidirectional scanning in both forward and reverse directions can be adopted and merged, so that each spatial location can receive context from both the preceding and following locations, thereby improving the balance and stability of spatial information aggregation.

[0082] S6: Preserves important features and suppresses noise interference during selective scanning;

[0083] The core of selective scanning is the state-space equation, which achieves intelligent feature selection by learning adaptive time steps and state parameters. First, adaptive time step parameters are learned for each sequence position:

[0084]

[0085] The Softplus function Ensure the time step is positive. (Larger) A smaller value indicates that the feature at that location is changing rapidly and requires more frequent updates. The value indicates that the feature is relatively stable.

[0086] Calculate the state-space parameters of the input and output:

[0087]

[0088] in For the state space dimension. Control the degree to which the input signal affects the state. The degree to which the control state contributes to the output.

[0089] The state transition matrix is ​​defined as follows:

[0090]

[0091] in As a learnable parameter, negative exponentiation is used to ensure system stability.

[0092] A recursive update process involving selective scanning is performed, which simulates the discretization of a continuous-time state-space model:

[0093]

[0094]

[0095] in For a moment The hidden state, For the current input, This is the current output. To prevent gradient vanishing and oversmoothing, the weights of the pass-through connections are used.

[0096] The advantage of this recursive update mechanism is that important feature locations are updated more frequently. and The values ​​obtain stronger state updates and output weights, and the noise locations are gradually decayed through smaller weight values, with an adaptive time step. This results in different response speeds at different locations.

[0097] Finally, the processed features are obtained through gated fusion and output projection:

[0098]

[0099]

[0100] Among them, the gating mechanism This allows the model to selectively deliver information, further enhancing its feature selection capabilities.

[0101] The quality of the denoised features is significantly improved, but different scenarios have different requirements for feature granularity. The multi-scale temporal adaptive selection mechanism will dynamically select the optimal scale according to the scenario complexity, enabling the model to adapt to various road conditions.

[0102] Noise interference can originate from point cloud sparsity, reflection anomalies, dynamic occlusion, registration errors, and communication compression errors. In addition to relying on network self-learning, location-level update strategies can introduce weak constraints to enhance interpretability and generalization ability. For example, reducing update intensity in low-confidence or high-uncertainty regions, maintaining slow updates in long-term stable regions to reduce drift, and increasing update sensitivity in target edge regions to preserve structural details. These constraints make the denoising and aggregation process more aligned with the stability and boundary preservation requirements of perception tasks.

[0103] S7: Scenario complexity analysis is performed through a multi-scale temporal adaptive selection mechanism;

[0104] like Figure 4 The flowchart shown illustrates the multi-scale temporal adaptive selection process. Traditional methods typically employ fixed scale selection strategies, which cannot be dynamically adjusted according to scene characteristics. The multi-scale temporal adaptive selection mechanism proposed in this invention can intelligently select the optimal feature scale based on scene complexity.

[0105] First, the S6 denoised features are used to generate feature representations at four different scales. The scene complexity analyzer evaluates the complexity of the current scene across multiple dimensions, while texture complexity detection measures the drasticness of texture changes by calculating the gradient magnitude of the feature maps.

[0106]

[0107] Edge detection uses the Sobel operator to calculate gradients in the horizontal and vertical directions:

[0108]

[0109] in and These are the Sobel gradient responses in the horizontal and vertical directions, respectively.

[0110] Density detection assesses the density of a target distribution by statistically analyzing the proportion of pixels with activation values ​​exceeding a threshold in the feature map.

[0111]

[0112] in For indicator functions, The preset activation threshold is usually taken as the standard deviation of the mean of the feature map.

[0113] Density detection thresholds should be set in an interpretable and reproducible manner, such as adaptively updating based on quantiles or sliding window statistics of the current frame's feature distribution, and maintaining a consistent normalization caliber across different scales to avoid incomparable density scores due to scale variations. Texture, edge, and density scores should be uniformly normalized before fusion to ensure a stable range of values ​​for the overall complexity representation across different scenes, scales, and batches, facilitating the stable operation of subsequent scale selection strategies.

[0114] The overall complexity assessment uses a weighted fusion of scores across three dimensions:

[0115]

[0116] in The weighting parameters were determined through comparative experiments.

[0117] S8: Achieving dynamic scale selection and smooth transition based on quality assessment network and temporal consistency constraints;

[0118] The quality evaluation network quantifies the feature quality at each scale. This network uses a lightweight convolutional structure to avoid introducing excessive computational overhead.

[0119]

[0120] in Indicates the first The QualityNet network first unifies features at different scales into a fixed size using adaptive average pooling, then uses 1×1 convolutional layers to compress features along the channel dimension, and finally outputs quality evaluation scores using the Sigmoid activation function.

[0121] The initial selection weights combine quality assessment and complexity assessment:

[0122]

[0123] The ComplexityWeight function assigns prior weights to different scales based on the complexity of the scene. Complex scenes tend to choose fine-grained scales, while simple scenes tend to choose coarse-grained scales.

[0124] To avoid timing jitter in scale selection, a timing smoothing optimization mechanism is introduced. First, the degree of scene change is calculated:

[0125]

[0126] Then calculate the time-series smoothing factor:

[0127]

[0128] in To adjust the parameters and control the smoothing intensity, the final weights are obtained by a weighted average of the historical weights and the current weights.

[0129]

[0130] When the scene changes little Approaching 1 primarily maintains historical choices to achieve a smooth transition, especially when the scenario changes significantly. It can quickly adapt to new scene characteristics with near-zero speed.

[0131] The mapping from complexity to scale prior should clearly define a monotonic relationship: higher complexity increases the prior weights at finer-grained scales, and lower complexity increases the prior weights at coarser-grained scales. Several segmented intervals can be set to avoid frequent weight switching due to minor fluctuations in complexity. In addition to adaptively adjusting based on the degree of scene change, temporal smoothing should also provide anomaly degradation rules: when historical weights are unavailable, collaborative features are missing, communication delays are significant, or scene mutations exceed a threshold, the smoothing intensity should be temporarily reduced or the current weights should be used directly to avoid persistent bias caused by erroneous historical memory; once the scene stabilizes, the smoothing and memory mechanisms should be gradually restored to ensure that the smoothing mechanism can suppress jitter without masking real mutations.

[0132] By avoiding abrupt changes in scale selection through a temporal consistency constraint mechanism, a dynamic balance is achieved in which coarse-grained scales are selected to improve efficiency in simple scenarios and fine-grained scales are selected to ensure accuracy in complex scenarios. This provides high-quality and temporally stable multi-scale fusion features for subsequent LSTM temporal modeling.

[0133] S9: Input the selected multi-scale features into the LSTM time-series processing module for time-series modeling and dependency learning;

[0134] The scale selection weights obtained in step S8 are used to weight and fuse the multi-scale features before inputting them into a standard LSTM network for temporal processing. First, the features at the four scales are fused according to the weights:

[0135]

[0136] The fused features are then input into an LSTM network, where temporal information is modeled and long-term dependencies are learned through standard input, forget, and output gate mechanisms. LSTM effectively preserves important historical information and combines it with current observations, outputting feature representations rich in temporal context. .

[0137] To preserve the spatial structure and ensure that the temporal modeling output remains a two-dimensional feature map, the temporal processing module can employ a gated loop structure with convolutional operations to directly perform gated updates on the feature map to retain the spatial layout; alternatively, the fused features can be unfolded into a sequence according to their spatial positions for temporal modeling, and then rearranged back into the feature map form according to their original spatial positions. Regardless of the implementation method, it is essential to ensure that the input organization, hidden state update, and output write-back methods for temporal processing remain consistent throughout the system. This ensures consistent behavior during the training and inference phases and seamless integration with subsequent detection, segmentation, or trajectory prediction modules.

[0138] S10: Output temporally enhanced contextual features for autonomous driving perception tasks;

[0139] Features after LSTM time-series processing This feature, containing rich temporal contextual and spatial semantic information, will be output as an enhanced contextual feature to subsequent perception task modules. The dimensionality of the output feature remains consistent with the input feature. This ensures compatibility with existing perception frameworks. This feature can be directly input into detection networks such as PointPillars and SECOND, or into task modules such as semantic segmentation and trajectory prediction, providing stable and reliable technical support for collaborative perception systems.

[0140] During the training phase, an end-to-end joint training approach can be adopted, where the supervision signals from downstream perception tasks are passed back to the motion alignment, correlation discrimination, denoising aggregation, and scale selection modules, enabling each module to collaboratively optimize around the perception target. Simultaneously, auxiliary constraints such as alignment stability, correlation consistency, and scale smoothness consistency can be added to reduce instability and overfitting during training. The deployment phase executes according to a fixed data flow sequence: multi-agent feature alignment and aggregation, motion compensation alignment, correlation-guided fusion, selective scanning denoising, multi-scale adaptive selection and smoothing, temporal modeling enhancement, and downstream task inference. A fallback strategy is implemented for communication anomalies and sensor missing data to ensure the system's robustness in real-world road environments.

[0141] Compared to traditional single-frame features, this temporally enhanced contextual feature offers the following advantages: it incorporates motion information and temporal dependencies, leading to more accurate detection and tracking of dynamic targets; selective scanning pooling and correlation prediction effectively suppress noise interference, improving feature robustness; a multi-scale adaptive selection mechanism dynamically adjusts the granularity of feature representation based on scene complexity; and the long-term memory capability of LSTM enables the handling of complex scenes such as occlusion and re-enactment. The final output temporally enhanced features can be directly input into object detection networks, semantic segmentation networks, or other perception task modules, providing collaborative perception systems with more accurate and stable environmental understanding capabilities.

[0142] In practice, the parameter settings for each of the above steps can be adjusted according to the specific application scenario. For example, in a highway scenario, the weight of temporal smoothing can be increased to maintain stable perception results, while in a complex urban intersection scenario, the weight of temporal smoothing can be reduced to quickly adapt to environmental changes. In addition, the network structure of each submodule can also be scaled according to computing resources and accuracy requirements. A lighter network structure can be used on edge computing devices, while a more complex network can be used to achieve higher accuracy when processing in the cloud.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-agent collaborative perception feature enhancement method based on context aggregation, characterized in that, include: Obtain the features of the current frame and the features of at least one historical frame; Motion cues are extracted from the feature differences between adjacent frames and deformable convolution offsets are predicted by a motion prediction network, and historical features are adaptively aligned. The semantic consistency between the current frame and the aligned historical features is evaluated through a correlation prediction network, which generates correlation weights and identifies temporally discontinuous regions. Based on the relevance weights, the aligned historical features and the current frame features are spatially adaptively fused to obtain the fused features; The fused features are input into the selective scanning pooling module, which learns and updates a strategy to suppress noise for each spatial location based on the state space mechanism, and outputs denoised fused features. The denoised and fused features are used to generate multi-scale feature representations and perform scene complexity analysis. The scale selection weights are determined by combining the quality assessment network and temporal consistency constraints, and the transition is smoothed. The multi-scale features are weighted and fused according to the weights and then input into the LSTM temporal processing module for dependency modeling. The output temporally enhanced contextual features are used for collaborative perception tasks.

2. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 1, characterized in that, The motion prediction network extracts motion cues by calculating the inter-frame feature differences between the current frame features and the historical frame features. Based on the motion cues, it predicts the sampling offset parameters of the deformable convolution and performs offset-based adaptive sampling alignment on the historical frame features to obtain the aligned historical frame features.

3. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 1, characterized in that, The correlation prediction network combines the features of the current frame and the features of the historical frames in the channel dimension and outputs a correlation weight map corresponding to the spatial location. This map is used to characterize the semantic consistency between the current frame and the historical frames at the corresponding positions and to identify low-correlation regions as temporally discontinuous regions.

4. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 1, characterized in that, Effective alignment guided by relevance weights includes: weighted fusion of aligned historical frame features and current frame features according to relevance weights position by position, so that high relevance positions are given priority to incorporate aligned historical information and low relevance positions are given priority to retain current frame information, in order to avoid invalid motion compensation in newly appearing targets or occluded recurrence areas.

5. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 1, characterized in that, The selective scanning pooling module includes flattening the fused features from a spatial feature map into sequence features, generating a main information stream and a gated information stream through a learnable projection layer; performing one-dimensional convolution on the main information stream to extract local spatial relationships, and learning an adaptive update strategy for each position in the sequence, which is used to enhance important positions and attenuate noisy positions in recursive updates.

6. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 5, characterized in that, The recursive update of the selective scan pooling module is implemented using a state-space mechanism, and an adaptive time step parameter with positive values ​​is learned for each sequence position. A stable recursive update process is constructed based on the learnable state transition parameters. At the same time, a pass-through connection is set to suppress over-smoothing and improve numerical stability.

7. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 1, characterized in that, The multi-scale temporal adaptive selection mechanism includes generating feature representations at multiple scales and evaluating scene complexity using a scene complexity analyzer. The scene complexity evaluation includes at least texture complexity detection based on gradient response, edge complexity detection based on edge operators, and target density detection based on the proportion of activated pixels. The multi-dimensional detection results are fused to obtain a scene complexity representation, which is used to drive scale selection.

8. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 7, characterized in that, Dynamic scale selection includes: the quality score of each scale feature is output by the quality assessment network, and prior preferences are assigned to different scales in combination with the scene complexity characterization to obtain the scale selection weight.

9. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 8, characterized in that, Temporal consistency constraints include a memory mechanism for maintaining historical scale selection, detecting the degree of scene change between the current frame and historical frames, and adaptively adjusting the temporal smoothing intensity based on the degree of change. This is used to enhance scale selection consistency when the scene is stable to avoid jumps, and to reduce the smoothing intensity to maintain a fast response when the scene changes drastically.

10. The method for enhancing multi-agent collaborative perception features based on context aggregation according to claim 1, characterized in that, The LSTM temporal processing module inputs the scale-selected multi-scale feature sequence frame by frame, and controls the information flow through the input gate, forget gate and output gate. It uses cell state to store and transmit key temporal information across time steps, and uses hidden state to represent the features at the current time step, so as to output temporally enhanced contextual features for collaborative perception tasks.

Images