Heterogeneous video cooperative perception method based on stcppo and multi-vehicle cooperative vision perception

Abstract

A heterogeneous video collaborative perception method based on STCPPO is used for a heterogeneous multi-agent perception system. In the agent collaboration scene, the video stream data collected by the perception devices connected to each agent are shared and fused in real time. The method is to perform heterogeneous video collaborative perception under STCPPO. STCPPO includes an adaptive space-time compression STC module, a collaborative decision module based on proximal policy optimization PPO, and a feedback-driven collaborative fusion module. The method first constructs an adaptive space-time compression mechanism, jointly models information entropy and time sequence dynamics, filters key frames and key regions at the source data end, and reduces redundant feature transmission. Then, the collaborative node selection is modeled as PPO decision, and high-value collaborative nodes are selected through joint heuristic preliminary screening and strategy optimization. Finally, motion compensation is used to realize asynchronous BEV feature alignment, and a feedback mechanism based on feature consistency residual is used to suppress low-quality collaborative nodes.

Description

Technical Field

[0001] This invention relates to the field of vehicle networking technology, specifically a heterogeneous video collaborative perception method based on Spatio-Temporal Compression-driven Proximal Policy Optimization (STCPPO), and a multi-vehicle collaborative visual perception method for intelligent connected vehicles. Background Technology

[0002] Accurate environmental perception is a prerequisite for intelligent connected vehicles to achieve autonomous decision-making and collaborative control [1]. Benefiting from the breakthrough in deep learning technology, single-vehicle visual perception has made great progress in target detection and scene understanding in recent years [2]. However, the limited field of view of sensors and target occlusion make it difficult for single-vehicle perception systems to avoid blind spots and fragmented perception. In order to overcome this limitation, multi-vehicle collaborative perception reconstructs a global view with wide coverage and high confidence by sharing visual feature representations [3], thereby significantly improving the reliability and robustness of target detection.

[0003] Due to its rich texture details and fine-grained semantic information, the camera occupies a core position in various collaborative perception modalities. Compared with static images, video streams can effectively capture the spatiotemporal context of traffic participants' movements. This is crucial for building temporally consistent environmental models and predicting the behavioral intentions of dynamic targets. However, the high-dimensional and dense characteristics of video features pose a severe challenge to bandwidth-limited and fluctuating V2X communication. In order to efficiently transmit visual features under limited bandwidth, researchers have proposed various feature interaction schemes, mainly divided into early, intermediate and late fusion[4]. Among them, intermediate fusion can balance perception accuracy and communication overhead and has become the mainstream paradigm[5]. Nevertheless, intermediate fusion schemes for heterogeneous video collaborative perception still face the following challenges:

[0004] (1) High spatiotemporal redundancy of video features. Existing methods mostly focus on spatial feature screening of single frames [6], but there is a strong correlation between adjacent frames in a video stream. If a unified feature extraction and transmission strategy is adopted for all frames and regions, it is easy to cause a large number of low-value features to occupy limited resources, affecting the real-time transmission of key frames and regions.

[0005] (2) Difficulty in scheduling heterogeneous nodes. Vehicles and roadside units (RSUs) differ in perception perspective, communication status, computing power and spatial location. Not all nodes can provide stable perception gains for the ego vehicle [7]. If all neighboring nodes are included in the cooperation scope, it will introduce low-quality features and increase the communication burden.

[0006] (3) Asynchronous feature misalignment and insufficient collaborative feedback. Transmission delay, positioning error and sensor asynchrony can cause multi-source features to be misaligned in the bird's-eye view (BEV) space and affect the fusion quality [8]. At the same time, if node selection relies solely on front-end observations and lacks feedback on the fusion results, the system will have difficulty in timely identifying and suppressing low-quality collaborative nodes, resulting in a decline in collaborative sensing performance [9]

[10] . Summary of the Invention

[0007] To address the aforementioned challenges in existing technologies, this invention proposes a heterogeneous video collaborative perception method based on Spatio-Temporal Compression-driven Proximal Policy Optimization (STCPPO).

[0008] The present invention provides a heterogeneous video collaborative perception method based on STCPPO for use in heterogeneous multi-agent perception systems. The heterogeneous multi-agent perception system includes multiple agents, which serve as nodes in the system. In agent collaboration scenarios, video stream data collected by the sensing devices connected to each agent are shared and fused in real time.

[0009] This method performs heterogeneous video collaborative perception under the STCPPO framework, which is based on spatiotemporal compression-driven proximal policy optimization. The STCPPO consists of three core modules: an adaptive spatiotemporal compression STC module, a collaborative decision module based on proximal policy optimization PPO, and a feedback-driven collaborative fusion module. Under STCPPO, the alignment error is fed back to the collaborative decision module through a closed-loop feedback path to achieve dynamic policy adjustment.

[0010] The heterogeneous video collaborative perception method based on STCPPO is used for multi-vehicle collaborative visual perception, and the heterogeneous multi-agent perception system is V2X; in V2X, the agents include roadside units (RSUs) and onboard units (OBUs) of autonomous vehicles; the autonomous vehicles collect video data through onboard video acquisition equipment; the agents... The vehicle is the owner, and the collaborating nodes are other vehicles.

[0011] The main contributions of this invention are as follows:

[0012] (1) A video perception mechanism based on spatiotemporal compression is used to construct a dynamic compression module by combining YOLO-nano and inverse perspective mapping to reduce the cost of feature transmission; at the same time, information entropy and temporal dynamics are modeled to filter key frames and remove redundant frames. The generated low-dimensional metadata is used to support subsequent collaborative decision-making.

[0013] (2) The collaborative node selection mechanism based on proximal policy optimization models the scheduling of collaborative nodes under dynamic topology as a Markov decision process. Based on the collected metadata, the vehicle identifies high-quality collaborative nodes through heuristic initial selection and PPO (Proximal Policy Optimization) decision-making to enhance the perceived benefits of collaboration.

[0014] (3) A feedback optimization mechanism based on asynchronous alignment quality utilizes kinematic state compensation to achieve asynchronous alignment of multi-source BEV features, and quantifies the fusion quality through feature consistency residuals. The alignment error is fed back to the collaborative decision module to suppress interference from low-quality nodes and improve detection robustness. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the STCPPO framework for spatiotemporal compression-driven near-end strategy optimization;

[0016] Figure 2 This is a schematic diagram of the adaptive spatiotemporal compression module;

[0017] Figure 3 This is a schematic diagram of a collaborative decision-making (node ​​selection) module based on PPO;

[0018] Figure 4(a) and Figure 4(b) are the training and validation loss curves of STCPPO, respectively. Figure 4(a) corresponds to the DAIR-V2X dataset, and Figure 4(b) corresponds to the DAIR-V2X dataset.

[0019] Figure 5(a) and Figure 5(b) show the feature space distribution under different fusion strategies, where Figure 5(a) corresponds to V2VNet and Figure 5(b) corresponds to STCPPO.

[0020] Figures 6(a) to 6(d) are schematic diagrams of the evolution of multi-source feature distribution under training stages of 5, 20, 40 and 60 epochs, respectively.

[0021] Figure 7(a) and Figure 7(b) show the comparison of the detection accuracy of nine methods on the OPV2V and DAIR-V2X datasets, respectively. Figure 7(a) corresponds to the DAIR-V2X dataset, and Figure 7(b) corresponds to the DAIR-V2X dataset.

[0022] Figures 8(a) to 8(c) are schematic diagrams of BEV feature retention weights under different feature transmission budgets (100%, 75%, and 50%).

[0023] Figures 9(a) to 9(c) show the comparison of accuracy, computational cost, and communication bandwidth of the six methods under different input resolutions. Figure 9(a) compares the accuracy, Figure 9(b) compares the computational cost, and Figure 9(c) compares the communication bandwidth.

[0024] Figures 10(a) to 10(c) show the cooperative node selection process of STCPPO. Figure 10(a) shows the distribution of candidate vehicles, Figure 10(b) shows the complementarity matrix between the vehicles and candidate nodes, and Figure 10(c) shows the node priority ranking.

[0025] Figures 11(a) and 11(b) show the impact of the number of cooperative nodes on detection accuracy and processing speed, respectively. Figure 11(a) corresponds to detection accuracy, and Figure 11(b) corresponds to processing speed.

[0026] Figures 12(a) and 12(b) show the impact of communication latency on detection accuracy and accuracy retention rate, respectively. Figure 12(a) corresponds to detection accuracy, and Figure 12(b) corresponds to accuracy retention rate.

[0027] Figures 13(a) and 13(b) show the experimental results of progressive ablation of the core module, where Figure 13(a) corresponds to the detection accuracy and Figure 13(b) corresponds to the average bandwidth.

[0028] Figure 14 These are qualitative detection images of different methods in rain and fog interference scenarios. Detailed Implementation

[0029] 1. Overview

[0030] Multi-vehicle collaborative perception can extend the perception range of a single vehicle by sharing multi-source visual features. However, the high spatiotemporal redundancy of video streams, the difficulty of dynamic scheduling of heterogeneous nodes, and feature misalignment caused by communication latency still limit the real-time performance and robustness of collaborative perception systems.

[0031] To address the aforementioned problems, this invention proposes a heterogeneous video cooperative perception framework (STCPPO) based on spatiotemporal compression-driven near-end strategy optimization. This framework...

[0032] First, an adaptive spatiotemporal compression mechanism is constructed, which jointly models information entropy and temporal dynamics, and filters key frames and key regions at the source data end to reduce redundant feature transmission.

[0033] Secondly, the selection of collaborative nodes is modeled as a PPO decision, and high-value collaborative nodes are selected through joint heuristic screening and strategy optimization.

[0034] Finally, motion compensation is used to achieve asynchronous BEV feature alignment, and a feedback mechanism based on feature consistency residuals is used to suppress low-quality cooperative nodes.

[0035] Experimental results on the OPV2V and DAIR-V2X datasets demonstrate that STCPPO achieves competitive detection performance with low communication and computational overhead, and maintains good robustness under transmission latency and perturbation conditions.

[0036] 2. Related work

[0037] 2.1 Lightweighting of Video Detection Models

[0038] The high-dimensionality of video streams places high computational demands on in-vehicle devices. To reduce local overhead, existing lightweight research mainly focuses on three directions: network structure simplification, knowledge distillation, and dynamic computation. At the structural level, YOLOv7

[11] reduces inference costs through reparameterization, feature aggregation, and detector head optimization. In terms of knowledge distillation, CrossKD

[12] uses cross-detector head prediction distillation to improve the learning ability of lightweight detectors. SBAD

[13] for streaming video generates pseudo-labels with the help of the teacher model and selects high-quality samples online to distill the student model. However, consecutive video frames do not have the same perceptual value. The above methods perform a uniform computation process on all input frames and do not dynamically adjust the strategy according to the changes between frames, thus affecting resource utilization. Therefore, dynamic computation methods have received attention in recent years. TEE

[14] uses the temporal early dismissal module to judge the semantic changes of adjacent frames and reuses historical detection results for highly similar frames to avoid repeated inference. DFA

[15] dynamically selects supporting frames to participate in feature enhancement and reduces unnecessary feature aggregation. However, such methods are highly dependent on accurately judging the changes in features between frames

[16]

[17] . When a nearby vehicle suddenly accelerates or an obscured target suddenly reappears, the key change frame may not be captured in time, resulting in delayed detection results or missed detection. Therefore, lightweight video detection still needs to reduce the inference overhead of a single frame while using the spatiotemporal correlation between frames to achieve adaptive key frame screening and redundant frame filtering in order to compress costs to the maximum extent

[18] .

[0039] 2.2 Efficient V2X Feature Transmission Mechanism

[0040] Fluctuating V2X bandwidth is the main bottleneck restricting the real-time performance of collaborative video perception, especially the high-concurrency communication of dense vehicles exacerbates the link load. In order to improve transmission efficiency, existing research focuses on spatial compression and spatiotemporal joint compression. In the spatial dimension, Where2comm [6] transmits only the sparse features of key areas based on the spatial confidence map to save bandwidth. UMC

[19] adopts a multi-resolution region selection mechanism to jointly optimize feature representation in the communication, collaboration and reconstruction process to improve collaboration efficiency. For the temporal characteristics of video streams, Select2Col [7] evaluates the semantic importance of spatiotemporal domain to reduce redundant transmission. CodeFilling

[20] uses codebook-based message representation and information-filling message selection to map high-dimensional features into compact indexes to reduce information loss. In addition, research on compressed domain video surveillance shows that motion information can reduce redundant processing in the video analysis process

[21] .

[0041] While existing methods can reduce communication overhead, their transmission decisions largely rely on the spatial confidence of the current frame, compression ratio, or the importance of local messages, making it difficult to simultaneously adapt to bandwidth fluctuations and changes in video content. Especially in collaborative scenarios, the features of each node differ in spatial location, semantic importance, and temporal variation. A uniform compression or transmission strategy can easily lead to insufficient transmission of key areas or continuous resource consumption by low-contribution areas. Therefore, feature transmission in collaborative video perception needs to dynamically allocate resources based on changes in video content and differences in the features of collaborative nodes, under communication budget constraints, prioritizing the transmission of features in key areas.

[0042] 2.3 Feature Fusion Strategy for Collaborative Video Detection

[0043] Intermediate fusion projects multi-view features onto a unified bird's-eye view BEV space for alignment and aggregation

[22] . V2VNet[4] and DiscoNet

[23] are the basis for synchronous fusion of spatial features, which expands the perception range of a single vehicle. However, these methods mostly focus on single-frame static images and do not make sufficient use of the temporal information of dynamic scenes. To make up for this deficiency, recent studies have begun to explore video stream collaborative fusion. SCOPE

[24] aggregates the temporal context and spatial key features of multiple agents based on spatiotemporal perception differences. CoST

[25] unifies the modeling of multi-agent fusion and multi-temporal fusion to improve the collaborative perception performance of continuous scenes. Related BEV temporal modeling and collaborative sequence studies have shown that temporal information helps to enhance the stability of feature fusion in dynamic scenes

[26]

[27] .

[0044] While temporal modeling can improve perception stability under ideal synchronization conditions, transmission delays and sensor asynchrony in actual deployments still cause temporal misalignment of features from heterogeneous nodes. Existing spatiotemporal fusion methods, although attempting to model historical information, rely on accurate time synchronization and are insufficient in compensating for asynchronous feature misalignment. When the time difference is large, directly fusing features from different times can easily produce geometric offsets or target ghosting in the BEV space. Therefore, for real-world V2X collaborative fusion, under the influence of transmission delays and positioning perturbations, it is necessary to suppress geometric offsets and target ghosting through motion compensation and asynchronous alignment of historical features.

[0045] 2.4 Selection Mechanism for Heterogeneous Collaborative Vehicles

[0046] The heterogeneity of V2X means that not all candidate nodes can provide positive perception gains for the vehicle, making node scheduling particularly critical. Early methods relied on static rules such as communication distance, signal strength, or spatial location. While these methods are simple to implement, they are difficult to adapt to rapid changes in traffic topology. To improve scheduling flexibility, recent studies have begun to introduce learning-based selection mechanisms. Select2Col [7] uses graph neural networks to estimate the semantic importance of candidate nodes and selects high-contribution nodes to reduce redundant communication. SchedCP

[28] models node scheduling as reinforcement learning, with a dual deep Q network selecting cooperative nodes based on channel state and perception semantics

[29] . The above methods show that node selection needs to take into account geometric distance, communication quality, and perception contribution.

[0047] However, existing scheduling strategies mostly rely on current state observations and lack feedback on the actual contribution after fusion. While some candidate nodes may have high spatial complementarity or semantic importance to the vehicular vehicle during the initial selection phase, their feature quality can still be affected by communication latency, positioning errors, or observation noise. If selection is based solely on front-end observations, the vehicular vehicle may struggle to identify low-quality cooperative nodes in a timely manner. Therefore, node scheduling needs to introduce a feedback mechanism to suppress the continuous interference from low-quality nodes online, thereby improving cooperative robustness.

[0048] 3. Specific Implementation Methods of the Invention

[0049] like Figure 1 As shown, V2X cooperative perception is modeled as a heterogeneous multi-agent perception system, including... Roadside Units (RSUs) and A number of autonomous vehicles, whose index set is and STCPPO consists of three core modules: adaptive spatiotemporal compression, PPO-based collaborative decision-making, and feedback-driven collaborative fusion. Furthermore, a closed-loop feedback path is constructed to feed alignment errors back to the collaborative decision-making module, enabling dynamic policy adjustment.

[0050] 3.1 Adaptive Spatiotemporal Compression

[0051] Figure 2 This demonstrates an adaptive spatiotemporal compression module designed to alleviate the high computational and communication load of video streaming. Given a time window sequence... Set up sensing devices The captured video stream is ,in Indicates time The video frames. In order to balance the preservation of key information and the reduction of redundancy, this invention quantifies the perceived value of each frame from two dimensions: information complexity and temporal dynamics.

[0052] Lightweight detection networks (such as YOLO-nano

[30] ) detect the current frame Perform coarse detection to obtain a set of candidate bounding boxes. Let the total number of target categories be... Candidate boxes The classification probability distribution is .definition The information entropy index is the average Shannon entropy

[31] .

[0053] (1)

[0054] A higher value indicates a more uncertain distribution of candidate target categories, thus necessitating the preservation of fine-grained features. To further evaluate the current frame... With the previous frame Temporal correlation, defining pixel-level difference indicators

[0055] (2)

[0056] in and For the height and width of the frame, These are pixel values. A higher value indicates a more drastic environmental change. To avoid background pixel shift... Interference, the compression module calculates formula (2) using data from the velocimeter and gyroscope. Perform global motion compensation.

[0057] The combined equations (1) and (2) yield the following results: Perceived quality vector To achieve adaptive spatiotemporal compression, a dynamic threshold generation mechanism based on feature mapping maps high-dimensional perceptual features into a single decision threshold. Normalization and get and as well as Subsequently, a forward adaptive mapping function is defined to compute the optimal decision threshold in real time.

[0058] (3)

[0059] in As the baseline threshold, To adjust the parameters, the frame resolution switching is therefore modeled as a binary decision function. .when hour, If a frame is identified as a keyframe, the intermediate layer features retain their original resolution; otherwise, spatial downsampling (e.g., halving the resolution) is performed to reduce bandwidth overhead.

[0060] The generated metadata is based on the combination of driving status and perceived quality. ,in For timestamps, and These represent GPS position, heading angle, and velocity, respectively. This lightweight metadata is broadcast in real-time via a V2X channel. Let the set of candidate nodes be... Bicycle Cache the received metadata to the shared pool This provides a basis for node selection in the collaborative decision-making module.

[0061] 3.2 Node Selection Strategy Based on PPO

[0062] Figure 3 The collaborative decision-making module is designed to filter collaborative nodes from candidate nodes. To balance decision-making speed and quality, this module employs a cascading mechanism of initial scoring screening and PPO (Progressive Point of Purchase) decision-making.

[0063] The initial scoring screening narrows the state space by quickly eliminating low-value nodes. This requires quantifying the information gain that candidates bring to the autonomous vehicle, but this posterior metric cannot be accurately predicted before candidate nodes participate in collaborative fusion. It is noted that high-value collaborative nodes should possess two key characteristics: perceptual reliability (high signal-to-noise ratio and rich information) and spatial complementarity (low overlap with the autonomous vehicle's field of view). Furthermore, although the RSU has a wide field of view, V2I transmission latency is often greater than V2V. Therefore, a scoring function incorporating a latency penalty term is defined as follows:

[0064] (4)

[0065] in For bicycle With candidate nodes The overlap rate of the field of view. When the information gain provided by the candidate vehicle is insufficient (such as omnidirectional occlusion), the RSU is selected as a candidate node due to its high spatial complementarity. This is a cooperative strategy with V2V as the primary mode and V2I as the secondary mode. The autonomous vehicle sorts all candidate vehicles in descending order of their scores and extracts the top-ranked vehicles. High-value nodes serve as inputs for the PPO decision-making layer.

[0066] After initially screening out high-value collaborative nodes, the selection of the optimal collaborative node is modeled as a Markov decision (MDP) process

[32] . Collaborative strategies are learned using Actor-Critic Proximal Policy Optimization (PPO)

[33] , and tuples are defined. These correspond to the state space, action space, transition probability, and reward function, respectively.

[0067] state space Combining the metadata of candidate nodes in the shared pool with the vehicle's status, the input vector of the PPO decision layer is:

[0068] (5)

[0069] During the offline training phase, At any moment The detection accuracy is compared with that of real-world annotations. When the number of collaborating nodes is less than a preset K, zero-vector padding is used to flexibly handle variable-length inputs.

[0070] Action space Define actions as binary vectors ,in Represents a node Select as a collaborative node, otherwise To avoid the explosion of action space dimensions, restrictions are imposed. .

[0071] reward function To guide the policy network in balancing perception accuracy and communication overhead, a composite reward function is defined.

[0072] (6)

[0073] Where the constant Used for numerical smoothing to avoid instability in the early stages of training. Total bandwidth For nodes Required bandwidth The weights are denoted by . The normalization design of Equation (6) can mitigate the interference of reward value fluctuations on gradient updates.

[0074] Compared to traditional policy gradient methods, PPO avoids performance degradation caused by excessively large single update steps by constraining the policy update step size. It utilizes a Critic network to estimate the state-value function. And calculate the time-series difference residuals.

[0075] (7)

[0076] in For Critic network parameters, This is the discount factor. Generalized advantage estimation is introduced to calculate the advantage function

[34] .

[0077] (8)

[0078] in As a discount factor, To smooth the parameters, and to constrain the policy update magnitude of the Actor network and stabilize training, the probability ratio of the new and old policies is defined as...

[0079] (9)

[0080] And a loss function with clipping is used.

[0081] (10)

[0082] in For Actor network parameters, For the clipping radius. This design limits the policy update range to [a certain value]. To avoid training oscillations, the joint loss function is defined.

[0083] (11)

[0084] in For the actual returns of the Critic network, As weights. Policy entropy. Encourage strategy exploration to avoid premature convergence to a suboptimal solution. Perform gradient descent optimization to complete offline training of PPO.

[0085] During online reasoning, Input the Actor network to obtain the cooperation strategy, and select the optimal one from the candidate node set. There are collaborative nodes, and their set is: The vehicle acquires a compressed feature set. This is for use by the collaboration and integration module.

[0086] 3.3 Feedback-driven collaborative integration

[0087] Feedback-driven collaborative fusion modules mitigate the characteristics of autonomous vehicles. and collaborative feature set The spatiotemporal misalignment between them generates spatiotemporally consistent fusion features and forms a feedback calibration signal. Specifically, an inverse perspective mapping is established using the camera extrinsic matrix

[22] to transform the input features to a unified BEV coordinate system to eliminate viewpoint differences. Subsequently, an explicit alignment method based on motion estimation corrects the temporal misalignment caused by transmission delay. Query collaboration nodes At any moment Metadata, and position is corrected using a low-computational-cost linear motion model.

[0088] (12)

[0089] in The spatial offset is calculated according to equation (12). And construct the affine transformation matrix.

[0090] (13)

[0091] in It is the identity matrix. The hysteresis characteristics are obtained through bilinear interpolation. Spatial resampling is performed to obtain aligned calibration features. This allows historical features to be projected onto the current coordinate system, mitigating spatiotemporal inconsistencies caused by asynchronous transmission.

[0092] The feature fusion stage employs an adaptive weighting strategy based on node reliability to balance the quality differences among different feature sources. Unlike the information entropy in Section 3.1, the system introduces a node uncertainty index. This is used to characterize the degree to which the features of collaborative nodes are affected by motion blur, positioning drift, and asynchronous errors. Node The fusion weight is

[0093] (14)

[0094] in The adjustment coefficient is used. Therefore, the globally weighted fusion feature is...

[0095] (15)

[0096] Input the detection decoder to output the detection result .

[0097] STCPPO generates a feedback signal based on the fused feature consistency residuals, which is used for candidate node scoring calibration in the next time step. Specifically, the autonomous vehicle delineates the region overlapping with the field of vision of the cooperating nodes. The alignment error is characterized by the average feature distance per pixel.

[0098] (16)

[0099] in Indicates the vehicle's spatial position Features Represents a collaborative node After asynchronous alignment, in spatial position Calibration characteristics.

[0100] To incorporate error feedback into the candidate node selection process, the system will Normalization And define the collaboration confidence coefficient.

[0101] (17)

[0102] in This is the initial confidence level coefficient. Let be the attenuation coefficient. After receiving feedback information, the system only updates the confidence coefficient of the selected collaborative node, so the candidate node scoring function in equation (4) is rewritten as follows:

[0103] (18)

[0104] Equation (18) shows that the feedback information does not only correct spatial complementarity, but also recalibrates the collaborative value of candidate nodes by leveraging collaborative confidence. For nodes selected in the previous time step but with poor fusion quality, their confidence decreases, leading to a decrease in scores in subsequent time steps. This effectively reduces the probability of low-quality nodes being repeatedly selected by correcting candidate node scores online without updating the PPO network parameters, thereby enhancing the robustness of the vehicle detection model under localization noise and spatiotemporal asynchrony conditions.

[0105]

[0106] 4. Experimental Results and Analysis

[0107] To verify the effectiveness, robustness, and practicality of STCPPO, this paper designs a systematic comparative experiment.

[0108] 4.1 Experimental Data and Parameter Configuration

[0109] OPV2

[36] is built on the CARLA

[38] high-fidelity simulator and the OpenCDA

[39] cooperative driving framework. It can provide a controllable multi-vehicle cooperative perception environment for verifying the compression, transmission and asynchronous alignment capabilities of heterogeneous nodes.

[0110] DAIR-V2X

[37] is collected from real vehicle-road cooperative environments, including actual disturbance factors such as changes in illumination, GPS drift and sensor asynchrony, and is used to evaluate the generalization and robustness of each method.

[0111] All experiments were implemented using the PyTorch

[40] framework and were trained and tested on an Intel Core i9-14900K processor, 64GB DDR5-5200 memory, and NVIDIA RTX 4090 GPU platform. The PPO policy network used the Adam optimizer with a base learning rate of The total number of steps for strategy training is The maximum step size for each episode is 200.

[0112] This invention introduces a positioning perturbation to simulate real V2X communication and sets the default transmission latency to 100 ms to analyze asynchronous detection performance. Other main parameter settings are shown in Table 1.

[0113] Table 1 Main experimental parameter settings

[0114]

[0115] To comprehensively evaluate the detection performance of STCPPO, this experiment selected several representative collaborative sensing methods as a baseline for comparison. Table 2 summarizes the detection methods and technical characteristics of each method.

[0116] Table 2 Summary of Baseline Methods

[0117]

[0118] 4.2 Performance Evaluation and Comparison

[0119] Model training convergence and feature representation analysis

[0120] The training and validation loss curves of STCPPO on the OPV2V and DAIR-V2X datasets in Figures 4(a) and 4(b) show that the total loss decreases rapidly in the early stage and stabilizes after about 5 epochs. The confidence loss remains at a low level throughout, while the validation loss follows a similar trend to the total loss and increases later, indicating that the model training is stable and there is no obvious overfitting.

[0121] Next, Figures 5(a) and 5(b) compare the feature space distributions of the baseline fusion method and STCPPO. It is evident that there is significant mixing among vehicles, occluded vehicles, cooperative vehicle (CV) auxiliary targets, and background tokens. In contrast, STCPPO can form clearer clustering structures for different categories, significantly improving the separation between foreground targets and background tokens.

[0122] Finally, Figures 6(a), 6(b), 6(c), and 6(d) show the distribution of multi-source features at different training stages. As the number of training rounds increases, the clustering structure of the vehicle target, CV features, and RSU features gradually becomes clearer and more compact, indicating that the model can stably learn collaborative feature representations.

[0123] Overall performance comparison

[0124] Figures 7(a) and 7(b) show the target detection results of nine methods on two datasets. Compared with No Fusion and Late Fusion, the intermediate fusion methods all achieved higher accuracy, indicating that feature-level collaboration can effectively alleviate the occlusion problem and extend the perception range beyond the blind spot of a single vehicle. As a representative of spatiotemporal fusion methods, CoST

[25] maintains a significant accuracy advantage. On OPV2V, STCPPO's AP@0.7 reaches 44.1%, which is better than Where2comm, SCOPE, Select2Col and CodeFilling, and close to CoST, which verifies its collaborative detection advantage. On DAIR-V2X, STCPPO's AP@0.7 reaches 29.8%, which is better than all baseline methods except CoST. This dataset contains actual perturbation factors such as localization error, sensor asynchrony and illumination changes. Thanks to PPO node selection and asynchronous feature alignment, STCPPO can effectively suppress the interference of low-quality collaborative information and shows strong robustness.

[0125] 4.3 Performance Analysis of Core Modules

[0126] Effectiveness of adaptive spatiotemporal compression

[0127] Figures 8(a) to 8(c) visualize the BEV feature retention weights under different transmission budgets, where darker colors indicate higher retention weights. At 100% budget, the model effectively retains the road region, target neighborhood, and complementary regions provided by collaborating nodes. When the budget is reduced to 75% and 50%, most of the weights for the background region and low-contribution road region are reduced, while the target neighbor, temporal dynamics, and RSU support regions still maintain strong responses. This indicates that STCPPO retains key information while reducing transmission overhead, avoiding the loss of key features due to downsampling.

[0128] Figures 9(a) to 9(c) compare the performance of different methods at three input resolutions in terms of accuracy, GFLOPs, and bandwidth. As the resolution and budget decrease, the computational cost and communication overhead of all methods decrease. STCPPO maintains relatively stable accuracy under low budget conditions, outperforming V2VNet, Where2comm, SCOPE, and CodeFilling. Compared to CoST, STCPPO has slightly lower accuracy at 100% and 75% budgets, but lower computational cost and communication overhead. At 50% budget, its accuracy is comparable to, or even slightly higher than, CoST. This advantage stems from the model's preferential preservation of target-related and temporally variable regions.

[0129] Impact of the number of collaborating nodes on performance

[0130] Figures 10(a) to 10(c) illustrate the cooperative node selection process of STCPPO. Figure 10(a) shows the distribution of candidate vehicles, where blue dots and lines represent selected nodes, and green boxes represent unselected nodes. Figure 10(b) shows the complementarity matrix between the vehicle and candidate nodes; darker colors indicate stronger complementarity. Figure 10(c) shows the node priority ranking, where blue bars represent the three selected nodes, and green bars represent unselected nodes. STCPPO consistently tends to select nodes with stronger visual complementarity and higher cooperative gain, while suppressing low-contribution nodes.

[0131] Figures 11(a) and 11(b) compare the impact of selection strategies on detection accuracy and processing speed under different numbers of nodes. With... Increasing AP@0.7 while decreasing FPS indicates that more collaborating nodes can provide richer information, but incur additional overhead. Distance-Priority depends only on spatial distance, and the performance gain increases with... The increase followed by a decrease indicates that proximity does not necessarily imply a high perceptual contribution. After introducing semantic information or scheduling strategies, Select2Col and ScheduleCP outperform distance-first methods. STCPPO in all... The higher AP@0.7 values ​​indicate that PPO decision-making can more effectively select nodes with complementary perspectives and high perceived value. At that time, STCPPO achieved an AP of 44.1% at 28 FPS @ 0.7. Increasing the value to 5 only improved AP@0.7 to 44.4%, but reduced FPS to 21. Considering the limited improvement in accuracy while the significant loss in speed, It is the preferred option.

[0132] Delay robustness of asynchronous alignment

[0133] Transmission latency and network jitter are inherent problems in real-world V2X communication, meaning that features uploaded by collaborating nodes may originate from past moments. Directly performing BEV spatial fusion in this case can lead to spatial misalignment, target ghosting, or detection box offset. To evaluate the model's robustness, this paper sets communication latency of 0, 100, 200, 300, 400, and 500 ms.

[0134] For the ideal synchronization condition (0 ms), Figures 12(a) and 12(b) show that all collaborative methods achieve high accuracy, indicating that feature fusion effectively supplements the blind spots of single-vehicle perception. With increasing latency, the performance of all methods decreases, with V2VNet and Where2comm showing the most significant declines. At 300 ms, AP@0.7 drops to 24.2% and 28.7%, respectively; at 500 ms, it further decreases to 18.5% and 22.5%. This indicates that strategies relying solely on synchronous fusion or single-frame spatial selection are insufficient to address feature misalignment. In contrast, STCPPO achieves AP@0.7 of 43.1%, 41.8%, and 37.2% at 100 ms, 300 ms, and 500 ms, respectively, with a smaller performance degradation. This robustness stems from the asynchronous feature alignment module's use of velocity, heading angle, and timestamps to perform motion compensation on delayed features and remap them to the current BEV coordinate system. Although asynchronous alignment cannot completely eliminate the impact of transmission latency, it effectively mitigates the damage to detection performance caused by spatiotemporal misalignment.

[0135] 4.4 Ablation Experiment

[0136] This invention conducts ablation experiments on OPV2V to evaluate the contribution of the core modules. Table 3 lists different module combinations and their detection performance. Figures 13(a) and 13(b) show that Proposed-1, which only uses the vehicle's own observations, inevitably suffers from blind spots and fragmented observations due to its limited field of view and target occlusion. After introducing fixed neighbor cooperation and simple BEV feature fusion, Proposed-2 significantly improves detection accuracy, indicating that cooperative nodes can supplement blind spot information to some extent. However, transmitting uncompressed features leads to high communication overhead. Proposed-3 uses fixed-ratio compression, which significantly reduces communication overhead, but cannot distinguish between key frames and redundant frames. Proposed-4 introduces spatiotemporal compression, which further improves detection accuracy while maintaining low overhead, indicating that this module can retain key visual information according to changes in video content. Based on this, Proposed-5 adds a PPO cooperative decision module, which improves accuracy and verifies the important role of node selection quality in fusion performance. Proposed-6 adds an asynchronous feature alignment module, which uses motion state to compensate for delayed features, effectively alleviating BEV spatial misalignment caused by transmission delay. The Proposed implementation introduces a feedback mechanism and achieves the highest accuracy, demonstrating that feedback can suppress interference from low-quality collaborating nodes. Therefore, the performance improvement of STCPPO does not stem from a single module, but rather is the result of the combined effects of dynamic spatiotemporal compression, PPO collaborative decision-making, asynchronous alignment, and the feedback mechanism.

[0137] Table 3 Ablation results of STCPPO core module

[0138]

[0139] 4.5 Visualization of Qualitative Results

[0140] To compare the detection performance of different methods under challenging conditions, Figure 14 presents continuous temporal detection results in rain and fog scenarios. The first two rows show the camera views of the autonomous vehicle and collaborating vehicles, while the last five rows show the BEV detection results of V2VNet, Where2comm, CoST, STCPPO (with RSU), and STCPPO (without RSU). Under conditions of poor visibility and limited viewpoints, some baseline methods exhibit missed detections or localization errors. In contrast, STCPPO utilizes the complementary viewpoints of collaborating nodes and mitigates historical feature misalignment through asynchronous alignment, achieving more stable detection across consecutive frames. When RSU is further introduced, the model gains additional coverage in long-range and lateral regions, validating that infrastructure can provide effective perception assistance when the vehicle's viewpoint is limited.

[0141] 5. Conclusion

[0142] This invention proposes a collaborative sensing method for heterogeneous video perception, addressing issues of redundant transmission, node scheduling, and asynchronous feature misalignment. The framework integrates an adaptive spatiotemporal compression mechanism to reduce the transmission of low-value video features; introduces a PPO-based scheduling strategy to dynamically select high-value collaborative nodes; and combines motion compensation and feature consistency feedback to mitigate BEV feature misalignment caused by asynchronous communication. Experimental results on the OPV2V and DAIR-V2X datasets demonstrate that STCPPO achieves a good balance between communication overhead, computational cost, and latency robustness while maintaining stable detection performance.

[0143] References

[0144] [1]AH Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, O. Beijbom,PointPillars: Fast encoders for object detection from point clouds, in: Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 2019, pp. 12697–12705.

[0145] [2]T. Yin, X. Zhou, P. Krähenbühl, Center-based 3D object detection and tracking, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11784–11793.

[0146] [3]S. Liu, C. Gao, Y. Chen,

[0147] [4]T.-H. Wang, S. Manivasagam, M. Liang, B. Yang, W. Zeng, J. Tu, R.Urtasun, V2VNet: Vehicle-to-vehicle communication for joint perception andprediction, in: Proceedings of the European Conference on Computer Vision,2020, pp. 605–621.

[0148] [5]R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, J. Ma, V2X-ViT:Vehicle-to-everything cooperative perception with vision transformer, in:Proceedings of the European Conference on Computer Vision, 2022, pp. 107–124.

[0149] [6]Y. Hu, S. Fang, Z. Lei, Y. Zhong, S. Chen, Where2comm:Communication-efficient collaborative perception via spatial confidence maps,in: Advances in Neural Information Processing Systems, 2022.

[0150] [7]Y. Liu, Q. Huang, R. Li, X. Chen, Z. Zhao, S. Zhao, Y. Zhu, H.Zhang, Select2Col: Leveraging spatial-temporal importance of semanticinformation for efficient collaborative perception, IEEE Transactions onVehicular Technology 73 (9) (2024) 12556–12569.

[0151] [8]Z. Lei, S. Ren, Y. Hu, W. Zhang, S. Chen, Latency-awarecollaborative perception, in: Proceedings of the European Conference onComputer Vision, 2022, pp. 316–332.

[0152] [9]S. Wei, Y. Wei, Y. Hu, Y. Lu, Y. Zhong, S. Chen, Y. Zhang,Asynchrony-robust collaborative perception via bird’s eye view flow, in:Advances in Neural Information Processing Systems, 2023.

[0153]

[10] D. Yang, K. Yang, Y. Wang, J. Liu, Z. Xu, R. Yin, P. Zhai, L.Zhang, How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception, in: Advances in Neural Information Processing Systems,2023.

[0154]

[11] C.-Y. Wang, A. Bochkovskiy, H.-Y.M. Liao, YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors, in:Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 2023, pp. 7464–7475.

[0155]

[12] J. Wang, Y. Chen, Z. Zheng, X. Li, M.-M. Cheng, Q. Hou, CrossKD:Cross-Head Knowledge Distillation for Object Detection, in: Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024.

[0156]

[13] D. Manjah, D. Cacciarelli, M. Benkedadra, B. Standaert, G.R. deHertaing, B. Macq, Stream-based active distillation for scalable modeldeployment, in: Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition Workshops, 2023, pp. 4999–5007.

[0157]

[14] A. Sabet, J. Hare, B. Al-Hashimi, G.V. Merrett, Temporal earlyexits for efficient video object detection, arXiv preprint arXiv:2106.11208,2021.

[0158]

[15] Y. Cui, Dynamic feature aggregation for efficient video objectdetection, in: Proceedings of the Asian Conference on Computer Vision, 2022,pp. 944–960.

[0159]

[16] X. Zhu, Y. Xiong, J. Dai, L. Yuan, Y. Wei, Deep feature flow forvideo recognition, in: Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, 2017, pp. 4141–4150.

[0160]

[17] X. Zhu, Y. Wang, J. Dai, L. Yuan, Y. Wei, Flow-guided featureaggregation for video object detection, in: Proceedings of the IEEEInternational Conference on Computer Vision, 2017, pp. 408–417.

[0161]

[18] B. Wang, S. Tang, J.-B. Xiao, Q.-F. Yan, Y.-D. Zhang, Detectionand tracking based tubelet generation for video object detection, Journal ofVisual Communication and Image Representation 58 (2019) 102–111.

[0162]

[19] T. Wang, G. Chen, K. Chen, Z. Liu, B. Zhang, A. Knoll, C. Jiang,UMC: A unified bandwidth-efficient and multi-resolution based collaborativeperception framework, in: Proceedings of the IEEE / CVF InternationalConference on Computer Vision, 2023.

[0163]

[20] Y. Hu, J. Peng, S. Liu, J. Ge, S. Liu, S. Chen, Communication-efficient collaborative perception via information filling with codebook, in:Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 2024, pp. 15481–15490.

[0164]

[21] C. Poppe, S. De Bruyne, P. Lambert, R. Van de Walle, Movingobject detection in the H.264 / AVC compressed domain for video surveillanceapplications, Journal of Visual Communication and Image Representation 20 (6)(2009) 428–437.

[0165]

[22] J. Philion, S. Fidler, Lift, splat, shoot: Encoding images fromarbitrary camera rigs by implicitly unprojecting to 3D, in: Proceedings ofthe European Conference on Computer Vision, 2020.

[0166]

[23] Y. Li, S. Ren, P. Wu, S. Chen, C. Feng, W. Zhang, Learningdistilled collaboration graph for multi-agent perception, in: Advances inNeural Information Processing Systems, 2021, pp. 29541–29552.

[0167]

[24] K. Yang, D. Yang, J. Zhang, M. Li, Y. Liu, J. Liu, H. Wang, P.Sun, L. Song, Spatio-temporal domain awareness for multi-agent collaborativeperception, in: Proceedings of the IEEE / CVF International Conference onComputer Vision, 2023, pp. 23383–23392.

[0168]

[25] Z. Tang, Y. Liu, Y. Sun, Y. Gao, J. Chen, R. Xu, S. Liu, CoST:Efficient collaborative perception from unified spatiotemporal perspective,in: Proceedings of the IEEE / CVF International Conference on Computer Vision,2025.

[0169]

[26] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, J. Dai,BEVFormer: Learning bird’s-eye-view representation from multi-camera imagesvia spatiotemporal transformers, in: Proceedings of the European Conferenceon Computer Vision, 2022.

[0170]

[27] H. Yu, W. Yang, H. Ruan, Z. Yang, Y. Tang, X. Gao, X. Hao, Y.Shi, Y. Pan, N. Sun, J. Song, J. Yuan, P. Luo, Z. Nie, V2X-Seq: A large-scalesequential dataset for vehicle-infrastructure cooperative perception andforecasting, in: Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition, 2023.

[0171]

[28] Y. Liu, G. Liu, L. Liang, H. Ye, C. Guo, S. Jin, Deepreinforcement learning-based user scheduling for collaborative perception,IEEE Transactions on Mobile Computing 24 (12) (2025) 13351–13365.

[0172]

[29] H. van Hasselt, A. Guez, D. Silver, Deep reinforcement learningwith double Q-learning, in: Proceedings of the AAAI Conference on ArtificialIntelligence, 2016, pp. 2094–2100.

[0173]

[30] A. Wong, M. Famuori, M.J. Shafiee, F. Li, B. Chwyl, J. Chung,YOLO Nano: A highly compact you only look once convolutional neural networkfor object detection, arXiv preprint arXiv:1910.01271, 2019.

[0174]

[31] C.E. Shannon, A mathematical theory of communication, Bell SystemTechnical Journal 27 (3) (1948) 379–423.

[0175]

[32] R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction,2nd ed., MIT Press, Cambridge, MA, 2018.

[0176]

[33] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov,Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347,2017.

[0177]

[34] J. Schulman, P. Moritz, S. Levine, M.I. Jordan, P. Abbeel, High-dimensional continuous control using generalized advantage estimation, in:Proceedings of the International Conference on Learning Representations,2016.

[0178]

[35] H.A. Mallot, H.H. Bülthoff, J.J. Little, S. Bohrer, Inverseperspective mapping simplifies optical flow computation and obstacledetection, Biological Cybernetics 64 (3) (1991) 177–185.

[0179]

[36] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, J. Ma, OPV2V: An openbenchmark dataset and fusion pipeline for perception with vehicle-to-vehiclecommunication, in: Proceedings of the IEEE International Conference onRobotics and Automation, 2022, pp. 2583–2589.

[0180]

[37] H. Yu, Y. Luo, M. Shu, Y. Huo, Z. Yang, Y. Shi, Z. Guo, H. Li, X.Hu, J. Yuan, Z. Nie, DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3D object detection, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022.

[0181]

[38] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, CARLA:An open urban driving simulator, in: Proceedings of the Conference on RobotLearning, 2017, pp. 1–16.

[0182]

[39] R. Xu, Y. Guo, X. Han, X. Xia, H. Xiang, J. Ma, OpenCDA: An opencooperative driving automation framework integrated with co-simulation, in:Proceedings of the IEEE International Intelligent Transportation SystemsConference, 2021, pp. 1155–1162.

[0183]

[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan,T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., PyTorch: An imperativestyle, high-performance deep learning library, in: Advances in NeuralInformation Processing Systems, 2019, pp. 8024–8035.

Claims

1. A heterogeneous video collaborative perception method based on STCPPO, used in a heterogeneous multi-agent perception system; the heterogeneous multi-agent perception system includes multiple agents, which serve as nodes in the heterogeneous multi-agent perception system; In the scenario of intelligent agent collaboration, video stream data collected by sensing devices connected to various intelligent agents are shared and fused in real time; Its characteristic is heterogeneity Video collaboration awareness is performed within the STCPPO heterogeneous video collaboration awareness framework, which is based on spatiotemporal compression-driven near-end strategy optimization. The STCPPO consists of three core modules: an adaptive spatiotemporal compression STC module, a collaborative decision-making module based on near-end policy optimization PPO, and a feedback-driven collaborative fusion module. Under STCPPO, the alignment error is fed back to the collaborative decision-making module through a closed-loop feedback path to achieve dynamic policy adjustment. (i) The adaptive STC module is used to alleviate the high computational and communication load of the video stream, and its working process is as follows: 1.1) Given a time window sequence Set up sensing devices The captured video stream is ,in Indicates time The video frames; quantifying the perceived value of each frame from two dimensions: information complexity and temporal dynamics; Visual object detection network for the current frame Perform coarse detection to obtain a set of candidate bounding boxes. ; Let the total number of target categories be Candidate boxes The classification probability distribution is ;definition The information entropy index is the average Shannon entropy: (1)， in, A higher value indicates a more uncertain distribution of candidate target categories, and a stronger necessity to retain fine-grained features; To further measure the current frame With the previous frame The temporal correlation is defined by the pixel-level difference index as follows: (2)， in, and For the height and width of the frame, Pixel value; A higher value indicates a more drastic environmental change; The combined equations (1) and (2) yield the following results: Perceived quality vector ; 1.2) A dynamic threshold generation mechanism based on feature mapping maps high-dimensional perceptual features into a single decision threshold; First normalization and get and as well as ; Subsequently, a forward adaptive mapping function is defined to compute the optimal decision threshold in real time: (3) ， in, As the baseline threshold, To adjust the parameters; Therefore, switching frame resolution is modeled as a binary decision function. ;when hour, When a frame is identified as a keyframe, the intermediate layer features retain their original resolution; At that time, spatial downsampling is performed; 1.3) The generated metadata is ,in For timestamps, and These represent GPS position, heading angle, and velocity, respectively; metadata is broadcast in real time via the channel of a heterogeneous multi-agent perception system. Let the set of candidate nodes be intelligent agent Cache the received metadata to the shared pool This provides a basis for node selection in the collaborative decision-making module; (ii) The PPO-based collaborative decision-making module is used to screen collaborative nodes from candidate nodes; the collaborative decision-making module adopts a cascade mechanism of initial scoring screening and PPO decision-making, and the working process includes: 2.1) Initial scoring and screening: reducing the state space by quickly eliminating low-value nodes; Because high-value collaborative nodes possess perceptual reliability and spatial complementarity, where perceptual reliability represents high signal-to-noise ratio and rich information, and spatial complementarity represents low overlap with the vehicle's field of vision; the scoring function using the latency penalty term is defined as follows: (4) ， in For intelligent agents With candidate nodes The overlap rate of the field of view; intelligent agent Sort all other candidate nodes in descending order of their scores, and extract the top-ranked nodes. A high-value node; 2.2) After initially identifying high-value collaborative nodes, the optimal collaborative node is selected and modeled as a Markov Decision Programming (MDP) process; the Actor-Critic PPO learning collaboration strategy is adopted, and tuples are defined. These correspond to the state space, action space, transition probabilities, and reward function, respectively: state space Combining the metadata of candidate nodes in the shared pool with their own states, the input vector for the PPO decision layer is: (5)， During the offline training phase, At any moment The detection accuracy is compared with that of real-world annotations; when the number of collaborating nodes is less than the preset K, zero vectors are used to fill in the gaps. Action space Define actions as binary vectors. ,in Represents a node Select as a collaborative node, otherwise ;limit ; reward function To guide the policy network in balancing perception accuracy and communication overhead, a composite reward function is defined: (6)， Where, constant Used for numerical smoothing to avoid instability in the early stages of training. Total bandwidth For nodes Required bandwidth As weight; Estimating the state value function using a Critic network And calculate the time-series difference residuals: (7)， in For Critic network parameters, Discount factor; Introducing generalized advantage estimation to calculate the advantage function: (8)， in As a discount factor, For smoothing parameters; To constrain the policy update magnitude of the Actor network and ensure stable training, the probability ratio of the new and old policies is defined as: (9)， At the same time, a loss function with clipping is used: (10)， in, For Actor network parameters, The cutting radius; Finally, define the joint loss function: (11)， in, For the actual returns of the Critic network, Weights; policy entropy Encourage strategy exploration to avoid premature convergence to a suboptimal solution; Perform gradient descent optimization to complete offline training of PPO; During online reasoning, Input the Actor network to obtain the cooperation strategy, and select the optimal one from the candidate node set. There are 10 collaborative nodes, and their set is 10. intelligent agent Obtain the compressed feature set For use by the collaboration and integration module; (iii) The feedback-driven collaborative fusion module alleviates the problem of intelligent agents. feature and collaborative feature set The spatiotemporal misalignment between them generates spatiotemporally consistent fusion features and forms a feedback calibration signal. Its working process includes: 3.1) First, an inverse perspective mapping is established using the extrinsic parameter matrix of the sensing device to transform the input features into a unified bird's-eye view BEV coordinate system; then, an explicit alignment method based on motion estimation is used to correct the time misalignment caused by transmission delay. intelligent agent Query collaboration nodes At any moment The metadata, and the position corrected using a low-computational-cost linear motion model, is represented as: (12)， in, ; Calculate the spatial offset according to equation (12). And construct the affine transformation matrix: (13)， in, It is the identity matrix; Hysteresis characteristics were obtained through bilinear interpolation. Spatial resampling is performed to obtain aligned calibration features. This projects historical features onto the current coordinate system; 3.2) In the feature fusion stage, an adaptive weighting strategy based on node reliability is adopted to balance the quality differences between different feature sources; a node uncertainty index is introduced. This characterizes the degree to which the features of collaborative nodes are affected by motion blur, positioning drift, and asynchronous errors; nodes The fusion weights are: (14)， in, This is the adjustment coefficient; Therefore, the globally weighted fusion feature is: (15)， Input the detection decoder to output the detection result ; Finally, STCPPO generates a feedback signal based on the fused feature consistency residuals, which is used for candidate node score calibration in the next time step. Specifically, intelligent agents Delineate the area overlapping with the vision of the collaborating nodes. The alignment error is characterized by the average feature distance per pixel. (16)， in, Represents intelligent agents In spatial location Features Represents a collaborative node After asynchronous alignment, in spatial position Calibration characteristics; To incorporate error feedback into the candidate node selection process, Normalization And define the collaboration confidence coefficient: (17) ， in, This is the initial confidence level coefficient. The attenuation coefficient; After receiving the feedback information, only the confidence coefficient of the selected collaborative node is updated, so the candidate node scoring function in equation (4) is rewritten as: (18)， Equation (18) shows that the feedback information does not only correct spatial complementarity, but also recalibrates the collaborative value of candidate nodes by means of collaborative confidence. For nodes that were selected in the previous time step but have poor fusion quality, their confidence decreases, resulting in a lower score in subsequent time steps.

2. The heterogeneous video collaborative perception method based on STCPPO according to claim 1, characterized in that: If used for multi-vehicle collaborative visual perception, then the heterogeneous multi-agent perception system is a vehicle-to-everything (V2X) network. In V2X, intelligent agents include roadside units (RSUs) and onboard units (OBUs) of autonomous vehicles. Autonomous vehicles collect video data through onboard video capture equipment; intelligent agents The vehicle is the owner, and the collaborating nodes are other vehicles.

3. The heterogeneous video collaborative perception method based on STCPPO according to claim 1, characterized in that: In step 1.1), the data from the velocimeter and gyroscope are compared before the calculation formula (2). Perform global motion compensation.

4. The heterogeneous video collaborative perception method based on STCPPO according to claim 2, characterized in that: In step 2.1), when the information gain provided by the candidate vehicle is insufficient, the roadside unit (RSU) is used as a candidate node.

Images