An open heterogeneous vehicle-road cooperative target detection method

Abstract

This invention discloses an open heterogeneous vehicle-road cooperative target detection method. A vehicle-road cooperative system is constructed using selected sensors as the baseline modality. BEV features are extracted and combined with the detection head for training. After freezing the parameters of the fusion module and the detection head, a baseline cooperative feature space is established. When a new modality is introduced, features of an agent equipped with dual modalities are extracted. The total training loss is obtained after direction and amplitude alignment. Only the parameters of the new modality's backbone encoder are updated and projected onto the baseline cooperative feature space. A modality-aware low-rank adaptation module is injected into the frozen fusion module and the detection head. The generated low-rank residuals are combined with the output of the original fusion backbone network to obtain the fusion features adapted to the new modality, which are then input into the detection head. The low-rank parameters of this module are trained and uploaded to the vehicle-road cooperative system to complete incremental updates and activate the adaptation conditions for the new modality. Finally, an agent with the new modality is introduced into the system, and the new modality features are uploaded. The detection head then completes vehicle-road cooperative target detection based on the fusion features.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and autonomous driving technology, specifically relating to an open heterogeneous vehicle-road cooperative target detection method, which is particularly suitable for cooperative target detection among vehicles, roadside units and unmanned equipment with different sensors and model structures. Background Technology

[0002] With the development of vehicle-to-everything (V2X) and autonomous driving technologies, vehicle-road cooperative object detection has become a key technology for coping with complex traffic environments and improving perception safety and accuracy. Traditional single-vehicle perception, due to limited field of view and severe occlusion, struggles to meet the demands for global environmental perception in complex dynamic scenarios. Sharing perception information among vehicles, roadside units (RSUs), and other intelligent infrastructure can significantly improve the accuracy of environmental modeling. However, current cooperative perception systems primarily rely on homogeneous sensor configurations, such as uniformly using LiDAR or visual sensors, lacking the ability to handle the feature space differences between heterogeneous sensors (such as LiDAR, millimeter-wave radar, and cameras).

[0003] Existing heterogeneous cooperative target detection methods suffer from the following main problems: First, inconsistent feature spaces, with significant semantic and geometric differences in feature representations generated by different sensors, leading to poor cross-modal fusion performance; second, poor adaptability, requiring retraining of the entire cooperative model when new vehicles or sensor types are added to the system, resulting in high deployment costs and poor scalability; and third, limited model generalization ability, with existing models exhibiting severe performance fluctuations under different modal combinations and lacking the ability to adapt to dynamically changing sensor configurations. Therefore, there is an urgent need for a heterogeneous vehicle-road cooperative target detection method that can achieve unified cross-modal feature space construction, possess dynamic adaptability, and support open access. Summary of the Invention

[0004] In view of this, the present invention provides an open heterogeneous vehicle-road cooperative target detection method, which realizes open access of different types of vehicle nodes and different sensor nodes without the need to retrain the entire online cooperative model, and achieves target detection with high scalability, high robustness and high accuracy.

[0005] This invention provides an open heterogeneous vehicle-road cooperative target detection method, comprising the following steps:

[0006] Using a selected sensor as the baseline mode, a vehicle-road cooperative system is formed by multiple agents equipped with the baseline mode, a fusion module, and a detection head. The bird's-eye view features of the agents are extracted through the backbone encoder corresponding to the baseline mode, and the system training is completed by combining the detection head. The parameters of the fusion module and the detection head are frozen to establish a baseline cooperative feature space.

[0007] When a new modality is added, for the agent with the new modality, the bird's-eye view features under the baseline modality and the original features under the new modality are extracted and denoted as baseline features and new modality features, respectively. The baseline features and new modality features at each position in the bird's-eye view space are aligned by direction and aligned by amplitude, respectively, to obtain cosine similarity loss and feature norm squared difference. The two are fused to obtain the total training loss, which is used as the optimization target to update the backbone encoder of the new modality to minimize the loss value, thus obtaining the new modality encoder and new modality features.

[0008] A modality-aware low-rank adaptation module is injected into the frozen fusion module and the detection head. A binary mask is set to represent the active modality. The binary mask is converted into a rank-scaled vector, and the feature dimension of the baseline collaborative feature space is compressed to the low-rank space by dimensionality reduction calculation. The features in the low-rank space are modulated by the rank-scaled vector and then calculated in higher dimensions to generate low-rank residuals. The fusion features are obtained by combining the low-rank residuals with the output of the original fusion backbone network and input into the detection head to complete target detection. The modality-aware low-rank adaptation module is trained with the target detection loss of the detection head as the optimization objective to obtain low-rank parameters. The low-rank parameters are uploaded to the vehicle-road cooperative system. The modality-aware low-rank adaptation module in the fusion module and the detection head is updated incrementally with parameters. The adaptation conditions corresponding to the newly added modality are activated by modality mask encoding.

[0009] The system integrates intelligent agents with new modalities into the vehicle-road cooperative system, and fuses the new modal features with the original baseline modal node features into fused features. The detection head then uses these fused features to complete the detection of vehicle-road cooperative targets.

[0010] Furthermore, the method for performing directional alignment on the baseline features and new modal features at each location in the bird's-eye view space is as follows: for each location... The cosine similarity loss of the normalized feature vectors of the baseline mode and the new mode features is calculated as follows:

[0011] ,

[0012] in, and These represent the width and height of the location in the bird's-eye view space, respectively. As the baseline modal characteristics, For new modal features, The normalized baseline modal features, For the new modal features after normalization, To prevent the minimum value of division by zero, For vectors of Norm, For vectors of Norm.

[0013] Furthermore, the amplitude alignment method for the baseline features and new modal features at each location in the bird's-eye view space is as follows: for each location... The squared difference of the feature norms of the baseline mode and the new mode at a given location is calculated as follows:

[0014] ,

[0015] in, and These represent the width and height of the location in the bird's-eye view space, respectively. As the baseline modal characteristics, For new modal features, For vectors of Norm, For vectors of Norm.

[0016] Furthermore, orientation alignment and amplitude alignment are performed on the baseline features and new modal features at each location in the bird's-eye view space, respectively, to obtain the cosine similarity loss and the feature norm squared difference. The method of fusing the two to obtain the total training loss is as follows: based on the real occupancy map, a foreground weight mask is generated for each location in the bird's-eye view space, with the foreground region being 1 and the background region being 0. The cosine similarity loss and the feature norm squared difference are weighted and fused to obtain the total training loss, which is the foreground weighted normalized feature alignment loss.

[0017] Furthermore, based on the real occupancy map, a foreground weight mask is generated for each location in the bird's-eye view space, with the foreground region being 1 and the background region being 0. The cosine similarity loss and the feature norm squared difference are weighted and fused to obtain the total training loss, i.e., the foreground weighted normalized feature alignment loss, in the following way:

[0018] Location in space for each bird's-eye view based on the actual occupancy map Generate foreground weight mask The formula for total alignment loss is:

[0019] ,

[0020] in, and These represent directional consistency and its weight, respectively. and These represent amplitude consistency and its weight, respectively. Represents the foreground mask. To prevent the minimum value from being divided by zero.

[0021] Furthermore, the method of using rank scaling vectors to modulate the features of the low-rank space is as follows: combining the low-rank residuals with modal gating to obtain the output features of the fusion module, as shown in the following formula:

[0022] ,

[0023] in, It is a rank-scaled vector. This represents the number of channels after dimensionality reduction, corresponding to the low-rank dimension or the rank. Scaling factor For the input features of a certain layer or module, The output features of the original model are represented by D, where D is the convolution dimensionality reduction calculation and U is the convolution dimensionality increase calculation. It is a vector of all 1s, with dimensions equal to 1. Consistent.

[0024] Furthermore, D represents 1x1 convolution dimensionality reduction calculation, and U represents 1x1 convolution dimensionality increase calculation.

[0025] Furthermore, the incremental parameter update fusion module and the modality-aware low-rank adaptation module in the detection head are configured such that only the low-rank adaptation parameters corresponding to the new modality are written into the parameter region corresponding to the modality-aware low-rank adaptation module.

[0026] Beneficial effects:

[0027] This invention constructs a vehicle-road cooperative system based on selected sensors as the baseline modality. BEV features are extracted from the baseline modality backbone encoder and trained using a detection head. After freezing the parameters of the fusion module and the detection head, a baseline cooperative feature space is established. When a new modality is added, corresponding features are extracted from agents simultaneously equipped with dual modalities. Orientation and amplitude alignment are performed, and a foreground weighted normalized feature alignment loss is obtained by combining foreground weight masks. Only the parameters of the new modality backbone encoder are updated, and the new modality features are projected onto the baseline cooperative feature space while retaining the frozen parameters. The modality is then injected into the frozen fusion module and the detection head. The perception low-rank adaptation module generates low-rank residuals through modal masking encoding, feature dimensionality reduction modulation, and dimensionality increase. These residuals are then combined with the output of the original fusion backbone network to obtain fusion features adapted to the new modality, which are input into the detection head. The low-rank parameters of the module are obtained by training with the detection head loss as the target. These parameters are then uploaded to the vehicle-road cooperative system through a unified interface to complete the incremental update of the module and activate the adaptation conditions of the new modality. Finally, the agent with the new modality is connected to the online system, and the new modality features are uploaded and fused with the original baseline modality node features. The detection head then completes the vehicle-road cooperative target detection based on the fused features. Attached Figure Description

[0028] Figure 1 The flowchart of an open heterogeneous vehicle-road cooperative target detection method provided by the present invention.

[0029] Figure 2 A flowchart illustrating the full model update process for existing heterogeneous vehicle-road cooperative target detection.

[0030] Figure 3 The flowchart shows the training process of an open heterogeneous vehicle-road cooperative target detection method provided by the present invention. Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] This invention provides an open heterogeneous vehicle-road cooperative target detection method. The basic idea is as follows: A vehicle-road cooperative system is constructed using selected sensors as the baseline modality. BEV features are extracted from the baseline modality backbone encoder and trained with the detection head. After freezing the parameters of the fusion module and the detection head, a baseline cooperative feature space is established. When a new modality is added, corresponding features are extracted from the agent simultaneously equipped with dual modalities. The foreground weighted normalized feature alignment loss is obtained by aligning the orientation and amplitude and combining it with a foreground weight mask. Only the parameters of the new modality backbone encoder are updated, and the new modality features are projected onto the baseline cooperative feature space while retaining the frozen parameters. A modality-aware low-rank adaptation module is injected into the fusion module and the detection head. After modality masking, feature dimensionality reduction modulation, and dimensionality increase to generate low-rank residuals, the fusion features adapted to the new modality are obtained by combining them with the output of the original fusion backbone network and input into the detection head. The low-rank parameters of the module are obtained by training with the detection head loss as the target. The parameters are then uploaded to the vehicle-road cooperative system through a unified interface to complete the incremental update of the module and activate the adaptation conditions of the new modality. Finally, the agent with the new modality is connected to the online system, and the new modality features are uploaded and fused with the original baseline modality node features. The detection head completes the vehicle-road cooperative target detection based on the fused features.

[0033] This invention provides an open heterogeneous vehicle-road cooperative target detection method, the process of which is as follows: Figure 1 As shown, the specific steps include:

[0034] Step 1: Using the selected sensor as the baseline mode, construct a vehicle-road cooperative system consisting of multiple agents equipped with the baseline mode, a fusion module, and a detection head; extract the bird's-eye view (BEV) features of each agent through the backbone encoder corresponding to the baseline mode, and train it in combination with the detection head to minimize task loss; after training, freeze the parameters of the fusion module and the detection head to establish a baseline cooperative feature space.

[0035] In this invention, the reference mode is a standard reference or common language selected for the entire vehicle-to-infrastructure (V2I) system. Typically, a sensor with stable perception accuracy and the widest application in the V2I scenario is chosen, and its feature representation system becomes the unique reference rule for the entire V2I system. By using the selected sensor as the reference mode, a V2I system composed of multiple agents equipped with this reference mode is constructed. This solves the problem of feature space incompatibility between heterogeneous sensors, i.e., the significant differences in semantic and geometric feature representations between different sensors prevent direct fusion. Therefore, by selecting a unified reference, all subsequent newly added sensor modes only need to match their own features to the reference mode to integrate into the system, without needing to reconstruct the entire fusion system.

[0036] Multi-agent refers to the sensing nodes such as vehicles, roadside units (RSUs), and unmanned inspection equipment in vehicle-road cooperation. The initial system requires these nodes to be equipped with selected reference sensors to ensure that the sensing data and feature system of the initial system are completely unified.

[0037] The backbone encoder is a feature extraction network customized for baseline modal sensors. It is used to convert the raw sensing data (laser point clouds, images, radar traces, etc.) collected by the sensors into high-dimensional features that can be processed by the model and contain environmental semantics (such as vehicle, pedestrian, obstacle categories) and geometric information (such as position, size, and velocity).

[0038] Bird's-eye view (BEV) features are used in the field of autonomous driving to unify the perception data of intelligent agents from different positions and perspectives into features in a global coordinate system with a top-down view. It is the core carrier for multi-node feature fusion in vehicle-road cooperation, which completely solves the problem of feature misalignment caused by differences in the perspectives of different intelligent agents and occlusion, and ensures that multi-node features can be directly fused in the same geometric space.

[0039] The detection head is the output network of the target detection task, which is used to convert the fused BEV features into detection results (such as the target's category, 3D coordinates, size, motion speed, heading angle, etc.).

[0040] This invention uses a fully annotated real-world traffic scene dataset to train an end-to-end model consisting of a baseline modal encoder, a multi-node fusion module, and a detection head. Model parameters are optimized to minimize the deviation between the model's output detection results and the actual annotations, ensuring that the initial baseline system's target detection accuracy meets commercial standards for vehicle-road cooperative systems. After training, the weight parameters of the core network responsible for multi-agent feature fusion—the fusion module and the detection head—are completely fixed. Subsequent additions of any new sensor modal do not require modification of these core parameters. Existing solutions require a complete retraining of the entire system model when adding a new sensor modality, while the core modules of this invention require no modification, only optimization of a small number of parameters for the new modality, thus significantly reducing deployment costs.

[0041] The baseline collaborative feature space is a fixed and standardized feature space formed after freezing the parameters of the fusion module and the detection head. The feature format, semantic rules, and geometric coordinate system of the space are fixed. The fusion module and the detection head only recognize and process the features in this space. Subsequently added sensor modes only need to project their own features into this space through feature alignment, and they can be directly recognized and processed by the fusion module and the detection head without changing the core logic of fusion and detection.

[0042] Step 2: When a new modality is added to the system, for agents equipped with both a baseline modality and a new modality, the bird's-eye view features under the baseline modality and the original features under the new modality are extracted and denoted as baseline features and new modality features, respectively. For the baseline features and new modality features at each location in the bird's-eye view space, orientation alignment and amplitude alignment are performed to obtain cosine similarity loss and feature norm squared difference. Based on the real occupancy map, a foreground weight mask is generated for each location in the bird's-eye view space, with the foreground region set to 1 and the background region set to 0. The cosine similarity loss and feature norm squared difference are then added. Weighted fusion is performed to obtain the total training loss, which is the foreground weighted normalized feature alignment loss. The foreground weighted normalized feature alignment loss is used as the optimization objective. Only the parameters of the backbone encoder corresponding to the new modality are updated. When the loss value is minimized, it means that the features output by the backbone encoder corresponding to the new modality have conformed to the rules of the benchmark cooperative feature space. That is, the projection of the new modality features to the benchmark cooperative feature space has been completed. The new modality encoder with optimized parameters and the new modality features projected to the benchmark cooperative feature space are obtained. The fusion module and detector head parameters frozen in step 1 in the benchmark cooperative feature space are kept unchanged.

[0043] Furthermore, the orientation alignment method is as follows: for each bird's-eye view spatial position The cosine similarity loss of the normalized feature vectors of the baseline mode and the new mode features is calculated as follows:

[0044] ,

[0045] in, and These represent the width and height positions in the space from a bird's-eye view perspective. As the baseline modal characteristics, For new modal features, The normalized baseline modal features, These are the new modal features after normalization; To prevent extremely small values ​​from division by zero, set to .

[0046] Furthermore, the amplitude alignment method is as follows: for each bird's-eye view spatial location The squared difference of the feature norms of the baseline mode and the new mode at a given location is calculated as follows:

[0047] .

[0048] Furthermore, based on the real occupancy map, a foreground weight mask is generated for each location in the bird's-eye view space, with the foreground region set to 1 and the background region set to 0. The cosine similarity loss and the feature norm squared difference are weighted and fused to obtain the total training loss, i.e., the foreground weighted normalized feature alignment loss. This is achieved by: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] Generate foreground weight mask Where 0 represents the background region and 1 represents the foreground region, the total alignment loss formula is:

[0049] ,

[0050] in, and These represent directional consistency and its weight, respectively. and These represent amplitude consistency and its weight, respectively. Represents the foreground mask.

[0051] Step 3: Inject the modality-aware low-rank adaptation module into the frozen fusion module and detection head. Set a binary mask to represent the active modality, where 1 indicates the corresponding modality is in a working state and 0 indicates inactivity. Use a linear mapping to convert the binary mask into a rank-scaled vector. This vector serves as the modulation signal for the low-rank residual, enabling adaptive adjustment under different modality configurations. Set the low-rank dimension, i.e., the number of channels after dimensionality reduction, as the core dimension of the low-rank residual, with a value much smaller than the original feature channel number. Perform dimensionality reduction calculation on the features of the benchmark collaborative feature space through convolution, compressing the feature dimension to the low-rank space. Use the rank-scaled vector to modulate the features in the low-rank space, and then perform dimensionality increase calculation through convolution to restore the original feature dimension, generating the low-rank residual. Combine the low-rank residual with the output of the original fusion backbone network to obtain the fusion features adapted to the new modality, and input them into the detection head to complete target detection. Use the target detection loss of the detection head as the optimization objective to complete the training of the modality-aware low-rank adaptation module and obtain the low-rank parameters of the module.

[0052] The adaptive adjustment under different modal configurations is to combine low-rank residuals with modal gating to obtain the output features of the fusion module, and the specific formula is as follows:

[0053]

[0054] in, Represents a rank-scaled vector. This represents the number of channels in the lower-rank dimension or the rank corresponding to the reduced dimension. To control the scaling factor of the update intensity, This represents the input characteristics of a certain layer or module. The output features of the original model are represented by D, which is a 1*1 convolutional dimensionality reduction calculation, and U is a 1*1 convolutional dimensionality increase calculation. It is a vector of all 1s, with dimensions equal to 1. Consistent.

[0055] Step 4: Upload the parameters of the modal perception low-rank adaptation module to the vehicle-road cooperative system through the unified parameter transmission interface of the vehicle-road cooperative system. After receiving the parameters, the vehicle-road cooperative system performs incremental parameter updates on the fusion module and the modal perception low-rank adaptation module already injected in the detection head. Based on the modal mask encoding of the modal perception low-rank adaptation module, the vehicle-road cooperative system activates the adaptation conditions corresponding to the newly added modality.

[0056] Specifically, activating the adaptation conditions corresponding to the newly added modality involves the following steps: when the system recognizes the feature input of the newly added modality, the modality-aware low-rank adaptation module uses the features of the newly added modality as input and generates a low-rank residual using the updated low-rank parameters. This enables adaptive adjustment of the fusion and detection processes while maintaining the adaptation conditions of the original baseline modality unchanged.

[0057] The incremental parameter update method for the fusion module and the modality-aware low-rank adaptation module already injected in the detection head is as follows: only the low-rank adaptation parameters corresponding to the new modality are written into the parameter area corresponding to the modality-aware low-rank adaptation module.

[0058] Step 5: Connect the intelligent agent with the new modality to the online vehicle-road cooperative system using a preset unified collaborative interface. Upload the new modality features to the fusion module of the online vehicle-road cooperative system. After fusion with the features of the original baseline modality nodes in the system, the fused features are obtained. The detection head uses the fused features to complete the final vehicle-road cooperative target detection.

[0059] Example:

[0060] This embodiment employs an open heterogeneous vehicle-road cooperative target detection method provided by the present invention, realizing target detection when different types of vehicle nodes and different sensor nodes are accessed in an open manner. Specifically, it includes the following steps:

[0061] S1. Select a mainstream sensor (such as LiDAR) as the benchmark mode and construct an initial vehicle-road cooperative system composed of multiple agents equipped with the benchmark mode; extract the bird's-eye view (BEV) features of each agent through a modality-specific backbone encoder (such as PointPillar), and combine it with the detection head training to minimize task loss; after training, freeze the parameters of the fusion module and the detection head to establish a benchmark cooperative feature space.

[0062] S2. When a new sensor modality (such as a camera or millimeter-wave radar) is added to the system, for agents equipped with both the baseline modality and the new modality, a foreground weighted normalized feature alignment loss (FW-NFA) is added. This allows the features of the new modality to be projected onto the baseline collaborative feature space, achieving geometric and semantic consistency of cross-modal features. This step only optimizes the parameters of the backbone encoder (such as ResNet or EfficientNet) of the new modality and does not change the already frozen fusion and detection modules.

[0063] S3. Inject a modality-aware low-rank adaptation (MA-LoRA) module into the frozen fusion module and detection head; generate low-rank residuals of modality conditions by encoding the current active modality configuration through modality masking, and adaptively adjust the fusion and detection process; this step only trains the low-rank adaptation related parameters, completes the lightweight fine-tuning after the new modality is accessed, and realizes the optimization of target detection output.

[0064] S4. Upload the modality-aware low-rank adaptation (MA-LoRA) module parameters trained in S3 to the online collaborative system, and incrementally update the MA-LoRA structure in the fusion module and the detection head to achieve adaptive activation under new modal conditions. This process does not change the original baseline collaborative feature space and frozen parameters, but only updates the low-rank residual related parameters, thus completing the rapid expansion of the online model.

[0065] S5. After completing the MA-LoRA module update in S4, the newly added modality agent connects to the online system through a unified collaborative interface, performs collaborative fusion with existing node features, and outputs a unified target detection result.

[0066] The update process of the open heterogeneous vehicle-road cooperative target detection method provided by the present invention is as follows: Figure 3 As shown Figure 2The comparison with existing full-model update methods clearly demonstrates the lightweight and efficient advantages of this invention when introducing new modalities. The two figures respectively present the core processes and operational steps of the two technical routes. Specifically, the method proposed in this invention relies on Foreground Weighted Normalized Feature Alignment (FW-NFA) and Modality-Aware Low-Rank Adaptive (MA-LoRA) modules, optimizing only the encoder parameters, training low-rank parameters, and incrementally updating online for the new modality. It retains the original frozen fusion backbone network, detection head, and other core modules throughout the process, without requiring full parameter retraining or model replacement. Existing full-model update methods, on the other hand, require a complete retraining and parameter update of the entire vehicle-road cooperative system model when introducing new modalities, involving adjustments to all core modules such as fusion and detection. Through the comparison of the two methods' processes, operational nodes, and resource consumption, the technical advantages of this invention's method—few parameter changes, low training costs, high online update efficiency, and no interference with the original system operation—are clearly highlighted when expanding to new modalities. This also confirms the practicality and efficiency of this method in open heterogeneous vehicle-road cooperative scenarios.

[0067] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An open heterogeneous vehicle-road cooperative target detection method, characterized in that, Includes the following steps: Using a selected sensor as the baseline mode, a vehicle-road cooperative system is formed by multiple agents equipped with the baseline mode, a fusion module, and a detection head. The bird's-eye view features of the agents are extracted through the backbone encoder corresponding to the baseline mode, and the system training is completed by combining the detection head. The parameters of the fusion module and the detection head are frozen to establish a baseline cooperative feature space. When a new modality is added, for the agent with the new modality, the bird's-eye view features under the baseline modality and the original features under the new modality are extracted and denoted as baseline features and new modality features, respectively. The baseline features and new modality features at each position in the bird's-eye view space are aligned by direction and aligned by amplitude, respectively, to obtain cosine similarity loss and feature norm squared difference. The two are fused to obtain the total training loss, which is used as the optimization target to update the backbone encoder of the new modality to minimize the loss value, thus obtaining the new modality encoder and new modality features. A modality-aware low-rank adaptation module is injected into the frozen fusion module and the detection head. A binary mask is set to represent the active modality. The binary mask is converted into a rank-scaled vector, and the feature dimension of the baseline collaborative feature space is compressed to the low-rank space by dimensionality reduction calculation. The features in the low-rank space are modulated by the rank-scaled vector and then calculated in higher dimensions to generate low-rank residuals. The fusion features are obtained by combining the low-rank residuals with the output of the original fusion backbone network and input into the detection head to complete target detection. The modality-aware low-rank adaptation module is trained with the target detection loss of the detection head as the optimization objective to obtain low-rank parameters. The low-rank parameters are uploaded to the vehicle-road cooperative system. The modality-aware low-rank adaptation module in the fusion module and the detection head is updated incrementally with parameters. The adaptation conditions corresponding to the newly added modality are activated by modality mask encoding. The system integrates intelligent agents with new modalities into the vehicle-road cooperative system, and fuses the new modal features with the original baseline modal node features into fused features. The detection head then uses these fused features to complete the detection of vehicle-road cooperative targets.

2. The open heterogeneous vehicle-road cooperative target detection according to claim 1, characterized in that, The method for performing orientation alignment on the baseline features and new modal features at each location in the bird's-eye view space is as follows: for each location... The cosine similarity loss of the normalized feature vectors of the baseline mode and the new mode features is calculated as follows: ， in, and These represent the width and height of the location in the bird's-eye view space, respectively. As the baseline modal characteristics, For new modal features, The normalized baseline modal features, For the new modal features after normalization, To prevent the minimum value of division by zero, For vectors of Norm, For vectors of Norm.

3. The open heterogeneous vehicle-road cooperative target detection according to claim 1, characterized in that, The amplitude alignment method for the baseline features and new modal features at various locations in the bird's-eye view space is as follows: for each location... The squared difference of the feature norms of the baseline mode and the new mode at a given location is calculated as follows: ， in, and These represent the width and height of the location in the bird's-eye view space, respectively. As the baseline modal characteristics, For new modal features, For vectors of Norm, For vectors of Norm.

4. The open heterogeneous vehicle-road cooperative target detection according to claim 1, characterized in that, The baseline features and new modal features at each location in the bird's-eye view space are aligned by direction and by amplitude, respectively, to obtain the cosine similarity loss and the feature norm squared difference. The method of fusing the two to obtain the total training loss is as follows: Based on the real occupancy map, a foreground weight mask is generated for each location in the bird's-eye view space, with the foreground region being 1 and the background region being 0. The cosine similarity loss and the feature norm squared difference are weighted and fused to obtain the total training loss, which is the foreground weighted normalized feature alignment loss.

5. The open heterogeneous vehicle-road cooperative target detection according to claim 4, characterized in that, Based on the real occupancy map, a foreground weight mask is generated for each location in the bird's-eye view space, with the foreground region set to 1 and the background region set to 0. The cosine similarity loss and the feature norm squared difference are weighted and fused to obtain the total training loss, which is the foreground weighted normalized feature alignment loss. Location in space for each bird's-eye view based on the actual occupancy map Generate foreground weight mask The formula for total alignment loss is: ， in, and These represent directional consistency and its weight, respectively. and These represent amplitude consistency and its weight, respectively. Represents the foreground mask. To prevent the minimum value from being divided by zero.

6. The open heterogeneous vehicle-road cooperative target detection according to claim 1, characterized in that, The method of using rank scaling vector modulation of low-rank space features is as follows: the low-rank residual is combined with modal gating to obtain the output features of the fusion module, as shown in the following formula: ， in, It is a rank-scaled vector. This represents the number of channels after dimensionality reduction, corresponding to the low-rank dimension or the rank. Scaling factor For the input features of a certain layer or module, The output features of the original model are represented by D, where D is the convolution dimensionality reduction calculation and U is the convolution dimensionality increase calculation. It is a vector of all 1s, with dimensions equal to 1. Consistent.

7. The open heterogeneous vehicle-road cooperative target detection according to claim 6, characterized in that, D represents 1x1 convolution for dimensionality reduction, and U represents 1x1 convolution for dimensionality increase.

8. The open heterogeneous vehicle-road cooperative target detection according to claim 1, characterized in that, The incremental parameter update fusion module and the modality-aware low-rank adaptation module in the detection head work by only writing the low-rank adaptation parameters corresponding to the new modality into the parameter area corresponding to the modality-aware low-rank adaptation module.

Images