Vehicle-road cooperation 3D target detection method fusing time sequence information

Through the vehicle-road collaborative 3D object detection method that integrates timing information, the time context integration and motion-aware reconstruction module are used to solve the perceived performance problems of autonomous driving systems under extreme environments and communication instability, and improve detection accuracy and robustness. sex.

CN119091397APending Publication Date: 2024-12-06BEIJING INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411013384.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In extreme environments or complex urban layouts, remote perception difficulties of autonomous driving systems in building occlusion and harsh weather conditions, as well as perceived performance degradation caused by wireless communication instability.

Method used

A vehicle-road collaborative 3D object detection method that integrates timing information is adopted. Through the time context integration mechanism, the road-end and on-board image data are used to extract feature information, generate a high confidence timing query signal, and transmit it in V2X communication. If the communication is interrupted, Use the motion-aware reconstruction module to reconstruct the target reference point position and feature embedding. If the communication is normal, geometric and feature space alignment will be performed. Use Hungarian algorithms and feedforward neural networks to fuse, and finally input the detection head to output the detection result.

Benefits of technology

Improves the accuracy and stability of object detection in complex environments, enhances the robustness and reliability of the system, and ensures that collaborative detection performance can still be maintained during communication interruptions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091397A_ABST
    Figure CN119091397A_ABST
Patent Text Reader

Abstract

The invention discloses a time sequence information fused vehicle-road collaborative 3D target detection method, which comprises the following steps: firstly, extracting target features from a road end image and a vehicle, outputting a query by using a query generator, and meanwhile, retaining a historical query of a previous k1 time step; thirdly, the current query and the historical query are integrated through a time context aggregation module, a time sequence query is generated, and the time sequence query is filtered through the confidence coefficient; and transmitting first m queries of confidence through V2X communication, receiving and fusing the queries at a vehicle end, and storing k2 frame road end queries at the same time. And finally, fusing the fused features through a multi-layer sensor (MPL) and other methods to generate richer and more accurate target feature information. In the aspects of a multi-level time context integration mechanism and motion perception reconstruction, a historical query sequence is utilized to enhance the consistency and accuracy of a target state in a time dimension, enhance the time context information of a target, and capture and reconstruct the motion characteristics of the target, thereby improving the continuity and real-time performance of target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and specifically relates to a vehicle-road collaborative 3D target detection method integrating time series information. Background Art

[0002] In recent years, driven by the progress of artificial intelligence, autonomous driving has made great progress. Nevertheless, autonomous driving systems that rely entirely on the built-in sensors of a single vehicle still have to face complex and ever-changing real-world challenges, especially in extreme environments or complex urban layouts, such as building occlusion and remote perception problems in bad weather conditions. These problems are directly related to driving safety and passenger trust.

[0003] The coordination between vehicles and infrastructure depends on the close communication and data sharing between vehicles and elements in their driving environment, which greatly broadens the "field of vision" of vehicles and realizes cognitive capabilities beyond the scope of individual perception. Especially in this context, the widespread deployment of intelligent transportation systems and the popularity of sensors such as cameras provide strong support for the in-depth application of 3D object detection technology. 3D object detection can not only accurately identify and locate obstacles on the road, but also further analyze their dynamic behavior, providing crucial information support for autonomous driving systems to make immediate and accurate decisions. However, to fully realize the advantages of connected autonomous driving, a series of technical and implementation obstacles need to be overcome.

[0004] In addition, the instability of wireless communications is also an issue that cannot be ignored. Network interruptions or signal attenuation may affect the safe operation of autonomous vehicles at critical moments. Summary of the invention

[0005] In order to solve the technical problems existing in the background technology, the present invention aims to provide a vehicle-road collaborative 3D target detection method that integrates timing information, so as to solve the problem of insufficient utilization of timing information in the existing visual collaborative detection system and the problem of decreased perception performance caused by communication interruption.

[0006] In order to solve the technical problem, the technical solution of the present invention is:

[0007] A vehicle-road cooperative 3D target detection method integrating temporal information, based on a temporal context integration mechanism, comprising:

[0008] Extract feature information using road-side image data and vehicle-mounted image data, generate road-side and vehicle-mounted query signals through a query generator, and interact the road-side query signal with the stored historical query signal to obtain a time-series query signal;

[0009] After filtering the time series query signals by confidence, a high-confidence time series query signal is obtained and transmitted to the vehicle using V2X communication. If the communication is interrupted, the position and feature embedding of each target reference point at the current moment are reconstructed based on the motion perception reconstruction module.

[0010] If the communication is normal, the input vehicle query signal is aligned with the high-confidence time series query signal in the geometric space and feature space to obtain the aligned query signal; a cost matrix is ​​constructed based on the Euclidean distance between the reference points, and the Hungarian algorithm is used to obtain the best match. For each pair of queries, the feature embeddings are spliced ​​along the dimensional channel and then fused using the feedforward neural network MPL. The queries that are not successfully matched are directly inserted into the coarse fusion query set, and the stored road-end historical query sequence is used to guide the coarse fusion query to learn the time series features to obtain the collaborative query.

[0011] Furthermore, the obtained collaborative query is input into the detection head, and the final detection result is output.

[0012] Furthermore, if the communication is interrupted, the Kalman filter is used to track the state of the historical trajectory, and the position of each target reference point at the current moment is predicted based on the state of the trajectory; at the same time, the multi-head attention mechanism is used to predict the feature embedding of each target at the current moment based on the feature embedding of the historical query signal.

[0013] Furthermore, after extracting features from the input road-side image data through a preset perception model, the query signal is output through the query generator, while retaining the historical query signal generated from the previous k1 time steps; using the temporal context integration mechanism, the query signal is integrated with the historical query signal to generate a time-series query signal containing historical information; then, the contextual time-series query signal is filtered by confidence, and the top m queries with the highest confidence are transmitted through V2X communication. The vehicle receives the top m queries to obtain the fused road-side query, and stores the road-side query received in k2 frames.

[0014] Furthermore, the time context integration mechanism specifically includes:

[0015] Time Context Aggregation:

[0016] The query generator on the road side models the currently observed target, generates a query signal, and inputs the query signal into the temporal context aggregation module; this module uses a multi-head cross-attention mechanism to fuse the query signals of the past k1 frames with the current query signal to enhance the temporal expression ability of the query.

[0017] Timing Guided Fusion:

[0018] After the vehicle receives the query signal from the road, it performs spatiotemporal fusion. First, the geometric space is aligned and the reference point is converted into a unified coordinate system through the rotation and translation matrix.

[0019] Align the feature space, use MLP to convert the reference point and rotation matrix into position embedding, and then concatenate it with the query embedding to obtain the final unified query embedding;

[0020] The cost matrix is ​​constructed based on the Euclidean distance of the reference points through the Hungarian algorithm for optimal matching and fusion. Unmatched queries are directly inserted into the coarse fusion query set.

[0021] The historical queries at the road end are used to guide the fusion query to learn the temporal features, and this process is achieved through a multi-head cross-attention mechanism.

[0022] Furthermore, the motion perception reconstruction module extracts motion features from the road end historical query sequence frame by frame and reconstructs the motion features through the following steps:

[0023] State estimation: Use Kalman filter to estimate the state of historical trajectory;

[0024] Association: Construct a cost matrix based on the Euclidean distance between the query and the estimated state, and apply the Hungarian algorithm for matching;

[0025] State update: The Kalman filter updates the trajectory state based on the matched reference points;

[0026] Track management: create new instance tracks, delete disappeared tracks, and get the final track results;

[0027] By extracting and predicting query trajectories, vehicles are able to reconstruct and recover lost roadside queries when communication is interrupted and continue spatiotemporal collaborative detection.

[0028] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for vehicle-road collaborative 3D target detection integrating timing information described above is implemented.

[0029] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned vehicle-road collaborative 3D target detection methods integrating timing information.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] Introduction of multi-level temporal information: On the road side, a temporal context aggregation module is introduced to integrate historical information from continuous frames, which enhances the accuracy and stability of object detection in dynamic scenes. The integration of temporal context enables the detection system to understand the movement trajectory and changes of objects, thereby improving the robustness of detection in complex environments. On the vehicle side, a time-guided fusion module is introduced, which not only merges the vehicle-side query and the road-side query at the current moment, but also interacts with the road-side historical query sequence stored on the vehicle side to obtain a cross-perspective query set containing rich temporal clues. These two temporal modules can improve the model's ability to utilize time series and improve detection performance.

[0032] Dealing with communication interruptions: In response to communication interruptions that may be encountered in actual applications, the framework includes a motion perception reconstruction module. This module can use previously stored road-end historical sequences to reconstruct missing detection query information during communication interruptions, ensuring that the system can still maintain collaborative detection performance when there is no real-time road-end information input, thereby enhancing the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 , the main flow chart of a vehicle-road collaborative 3D target detection method integrating timing information according to the present invention. DETAILED DESCRIPTION

[0034] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0035] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0036] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0037] Embodiment 1:

[0038] like Figure 1As shown, the method proposed in the present invention utilizes the transmission signals from the infrastructure and the vehicle-mounted camera to output the collaborative detection results, solving the inherent defects in the single-vehicle 3D object detection. The communication transmission signal of the proposed method is the query, a multi-level temporal context integration mechanism is used, and a motion-aware reconstruction module is included.

[0039] 1. Transmission signal - query

[0040] In this system, each query can be considered as an abstract representation of a specific target, which carries rich information for subsequent fusion and detection. The following are its specific components and functions:

[0041] Each query contains:

[0042] 256-dimensional feature embedding:

[0043] Contains various information about the target, such as appearance features, behavior patterns, and categories.

[0044] Appearance features: color, shape, etc. describe the appearance of the target.

[0045] Behavioral pattern: dynamic characteristics of the target such as movement speed and direction.

[0046] Category: The category of the target, such as pedestrians, vehicles, etc.

[0047] Reference points:

[0048] Represents the exact location of the target in space, similar to the coordinates in the world coordinate system.

[0049] The reference points provide the physical location of the target in 3D space, which is very important for spatiotemporal alignment and matching.

[0050] Query-level fusion and matching:

[0051] Choosing to fuse and match at the query level in the detection system helps to effectively utilize rich target information while keeping the data lightweight. The following is the specific workflow:

[0052] Fusion and matching steps:

[0053] Query signal generation:

[0054] The infrastructure and vehicle systems generate initial query signals respectively.

[0055] These query signals carry the feature embeddings and reference points of the corresponding targets.

[0056] Confidence screening and transmission:

[0057] The infrastructure system retains high-confidence timing queries through confidence screening and transmits them to the vehicle through V2X communication.

[0058] Geometry and feature space alignment:

[0059] On the vehicle side, the onboard query signal is aligned with the received high-confidence timing query signal on the road side at the geometric (physical space) and feature (feature space) levels.

[0060] Geometric alignment ensures that queries from different sources are consistent in physical space.

[0061] Feature alignment ensures that the feature vector representation of the query is consistent.

[0062] Best match:

[0063] The Hungarian algorithm is used to perform the best query matching and achieve the best correspondence between the vehicle-mounted and road-side query signals.

[0064] Feature Fusion:

[0065] For successfully matched queries, their feature vectors are concatenated along the dimension channel.

[0066] The concatenated features are fused using methods such as Multi-Layer Perceptron (MLP) to generate richer and more accurate target feature information.

[0067] For unmatched queries, they are directly added to the rough fusion query set and continue the subsequent integration processing.

[0068] Multi-level temporal context integration mechanism and motion perception reconstruction:

[0069] The fused query signal provides a solid foundation for subsequent temporal context integration and motion perception reconstruction:

[0070] Temporal context integration mechanism:

[0071] Historical data integration:

[0072] Utilize historical query sequences to learn temporal features of the fused queries.

[0073] Increase the consistency and accuracy of target status in the time dimension.

[0074] Enhanced Timing Features:

[0075] Strengthen the temporal context information of the target and improve the understanding of the target trajectory and behavior pattern.

[0076] Motion-aware reconstruction:

[0077] Motion feature capture:

[0078] Based on the fused query, the target’s motion features, such as speed and direction, are captured and reconstructed.

[0079] Dynamic models such as Kalman filters are used to predict the future position of the target and enhance the continuity and real-time performance of detection.

[0080] Comprehensive indication:

[0081] Query generation:

[0082] The infrastructure system and the vehicle system generate initial query signals respectively.

[0083] The initial query signal goes through confidence screening and V2X communication transmission.

[0084] Geometry and feature alignment:

[0085] Alignment in geometry and feature space is performed on the vehicle side.

[0086] Best Matching and Feature Fusion:

[0087] Use the Hungarian algorithm for query matching.

[0088] Feature vector concatenation and fusion using MLP.

[0089] Temporal context integration and motion perception reconstruction:

[0090] Temporal context integration enhances temporal features.

[0091] Motion-aware reconstruction improves the continuity and real-time performance of target detection.

[0092] In this way, the system can effectively utilize the collaborative perception capabilities of the vehicle and infrastructure, improve the accuracy and robustness of target detection, and ultimately achieve high-precision perception of the environment.

[0093] The system includes two subsystems: smart vehicles and infrastructure;

[0094] In the infrastructure system, after extracting features from the input image information through the preset perception model, the query is output through the query generator, while retaining the historical queries generated from the previous k1 time steps.

[0095] The first step of the temporal context integration mechanism is to use the temporal context aggregation module to integrate the query with the historical query to generate a temporal query containing historical information. Then, the contextual temporal query is filtered by confidence, and the top m queries with the highest confidence are transmitted through V2X communication. The vehicle receives the top m queries to obtain the fused road-side query, and stores the road-side query received in k2 frames.

[0096] In the vehicle system, the target query is obtained from the input image, and then the final collaborative query is generated using the time-guided fusion module; then the spatial alignment process is performed;

[0097] Then, the vehicle query is spatially fused with the currently received road-end query, and then, based on the stored road-end historical query sequence, the temporal features of the query are further learned.

[0098] At this point, the temporal context integration mechanism has been completed. Finally, the fused collaborative query is input into the detection head to generate the detection output.

[0099] Embodiment 2:

[0100] The temporal context integration mechanism and motion-aware reconstruction module are introduced in detail below.

[0101] Temporal context integration mechanism:

[0102] Time Context Aggregation:

[0103] The query generator of the RDU models the currently observed targets, generates queries, and inputs these queries into the temporal context aggregation module.

[0104] This module uses a multi-head cross-attention mechanism to fuse the query of the past k1 frames with the current query to enhance the temporal expression ability of the query.

[0105] Timing Guided Fusion:

[0106] After the vehicle receives the query information from the road, it performs spatiotemporal fusion. First, the geometric space is aligned and the reference point is converted into a unified coordinate system through rotation and translation matrices.

[0107] The feature space is aligned, and the reference point and rotation matrix are converted into position embedding using MLP, which is then concatenated with the query embedding to obtain the final unified query embedding.

[0108] The cost matrix is ​​constructed based on the Euclidean distance of the reference points through the Hungarian algorithm for optimal matching and fusion. Unmatched queries are directly inserted into the coarse fusion query set.

[0109] The historical queries at the road end are used to guide the fusion query to learn the temporal features, and this process is achieved through a multi-head cross-attention mechanism.

[0110] In order to effectively utilize temporal information and improve performance, a multi-level interaction with the roadside history sequence is proposed. The roadside unit filters the query according to the confidence level before transmitting it to the vehicle. The reason for choosing the roadside sequence instead of the ego vehicle or fused sequence is that the roadside unit is installed overhead and fixedly, which inherently gives its sequence greater spatiotemporal correlation than the sequence from the ego vehicle or fused sequence. Specifically, the temporal context aggregation (TCA) module in the roadside unit and the temporal guided fusion (TGF) module in the intelligent vehicle jointly implement multi-level temporal context integration. The roadside unit stores the queries generated by its query generator, while the vehicle stores the queries received from the roadside. At each moment, they update the sequence according to the first-in-first-out principle. Next, the temporal context aggregation (TCA) module and the temporal guided fusion (TGF) module will be explained respectively:

[0111] Temporal context aggregation:

[0112] First, temporal information is introduced on the roadside to enhance the modeling capability of the query. Specifically, at each timestamp, the query generator of the roadside unit models the currently observed potential object to obtain the query. These queries are then input into the TCA module, where they perform a multi-head cross-attention mechanism using the continuous queries of the past k1 frames. This process is expressed as:

[0113]

[0114] where f linear is a linear projection with a fully connected layer, Concat is the concatenation along time, and f MHCA is a multi-head cross attention layer, f LN is the normalization of the layer. Please note that the amount of data in the final query is The amount of data for non-time series queries are equal, so this process does not increase the transmission load.

[0115] Time-guided fusion: After receiving the roadside query information, further time-space fusion is performed to extract the time information. At each timestamp, the vehicle first performs spatial fusion. Due to the different perspectives of the vehicle and the roadside unit, as well as the differences in model parameters, there are domain gaps between the two sets of queries in both geometric space and feature space. The domains need to be aligned before fusion.

[0116] For the geometric space, the vehicle’s lidar coordinate system is chosen as the unified coordinate system. The reference point is transformed into the unified coordinate system by calculating the rotation and translation matrices by considering the relative positions of the vehicle and the roadside unit as well as the calibration parameters. For the feature space, multiple MLPs are used to transform the reference point and rotation matrices into position embeddings. They are then concatenated with the query embedding and fed into another MLP to obtain the final unified query embedding.

[0117] Subsequently, the two different sets of queries are aggregated. Considering the shared observation area between vehicles and roadside units, it is necessary to identify instances detected simultaneously by both agents. A cost matrix is ​​constructed based on the Euclidean distance between reference points, and the Hungarian algorithm is used to obtain the best match. For each pair of queries, the embeddings are concatenated along the channel dimension and then fused using MLP. Unsuccessfully matched queries are directly inserted into the coarse fused query set.

[0118] To prevent the fused features from forgetting the context introduced by the roadside query, the historical sequence of the roadside is further utilized to guide the obtained fused query to learn temporal features. Considering that both vehicles and objects are moving, a motion encoding module is introduced to implicitly encode timestamps and ego poses into the query. Next, all k2 frames of the historical query are concatenated as keys and values, while the fused features obtained in the previous step are used as queries. Temporal interaction is performed via multi-head cross attention.

[0119] Motion-aware reconstruction:

[0120] This module extracts motion features from the roadside historical query sequence frame by frame and reconstructs it through the following steps:

[0121] State Estimation: Estimate the state of the historical trajectory using a Kalman filter.

[0122] Association: Construct a cost matrix based on the Euclidean distance between the query and the estimated state, and apply the Hungarian algorithm for matching.

[0123] State update: The Kalman filter updates the trajectory state based on the matched reference points.

[0124] Track management: create new instance tracks, delete disappeared tracks, and obtain the final track results.

[0125] By extracting and predicting query trajectories, vehicles are able to reconstruct and recover lost roadside queries when communication is interrupted and continue spatiotemporal cooperative detection.

[0126] In the case of communication interruption, the vehicle loses the additional information provided by the roadside unit and cannot benefit from cooperative sensing. To mitigate this negative impact, motion features are extracted frame by frame from the roadside historical query sequence to derive query trajectories, which are then used to predict the queries that should be received.

[0127] Specifically, we extract query trajectories based on reference points and predict reference points and query embeddings. For clarity, let’s consider query prediction at timestamp t. At timestamp t-1, to associate the current query with the historical trajectory, four basic steps are required:

[0128] (1) State estimation, where the Kalman filter is used to estimate the state of the historical trajectory, and a constant velocity model is used in this process. Each trajectory state uses a 6-dimensional vector [x, y, z, v x , v y , v z ], where x, y, z represent spatial coordinates, v x , v y and v z denotes the velocity along the respective axis;

[0129] (2) Association, a cost matrix is ​​constructed based on the Euclidean distance between the query and the estimated state. Subsequently, the Hungarian algorithm is applied to solve the bipartite matching problem.

[0130] (3) state update, where the Kalman filter updates the matching trajectory based on the paired reference points;

[0131] (4) Trajectory management: create new trajectories for newly appeared instances and delete disappeared trajectories to obtain the final trajectory results. It is worth noting that although the query is tracked based on the reference point, the embedding of each query is preserved. Therefore, the result is called query trajectory.

[0132] At timestamp t, the vehicle cannot receive the roadside query due to the interruption, so the reconstruction process is initiated. For the reference points, their current positions are directly determined by exploiting the state of the trajectory. Since instances in traffic scenes show continuous movement trends in a short period of time, a constant speed model is adopted for prediction. Due to the powerful sequence modeling ability, multi-head cross attention is adopted to predict the query embedding. First, the historical timestamps are encoded into time embeddings using cosine and sine functions. These embeddings are then added to the corresponding historical query embeddings, and the resulting features are concatenated along the time dimension. In addition, the current timestamp is encoded into a time embedding in the same way as the query. Note that in order to handle variable-length query trajectories, shorter trajectories are padded with zero values ​​and marked as mask O to prevent interference with them.

[0133] Finally, the lost roadside query is recovered by combining the predicted reference points and query embeddings.

[0134] The vehicles then continued to cooperate in space and time according to standard procedures, similar to normal circumstances.

[0135] It can be understood that the present invention proposes a multi-level temporal context integration mechanism, in which features containing temporal information are generated on the roadside and spatial and temporal fusion is performed on the vehicle side to enhance the representation ability of the query. The proposed temporal fusion module effectively improves the performance; a motion perception reconstruction module is designed to predict the lost features during communication interruption, making the collaborative perception system highly robust to communication interruption.

[0136] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0141] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A vehicle-road collaborative 3D target detection method integrating temporal information, It is characterized in that Based on the time context integration mechanism, the method includes: Extract feature information using road-side image data and vehicle-mounted image data, generate road-side and vehicle-mounted query signals through a query generator, and interact the road-side query signal with the stored historical query signal to obtain a time-series query signal; After filtering the time series query signals by confidence, a high-confidence time series query signal is obtained and transmitted to the vehicle using V2X communication. If the communication is interrupted, the position and feature embedding of each target reference point at the current moment are reconstructed based on the motion perception reconstruction module. If the communication is normal, the input vehicle query signal is aligned with the high-confidence time series query signal in the geometric space and feature space to obtain the aligned query signal; a cost matrix is ​​constructed based on the Euclidean distance between the reference points, and the Hungarian algorithm is used to obtain the best match. For each pair of queries, the feature embeddings are spliced ​​along the dimensional channel and then fused using the feedforward neural network MPL. The queries that are not successfully matched are directly inserted into the coarse fusion query set, and the stored road-end historical query sequence is used to guide the coarse fusion query to learn the time series features to obtain the collaborative query.

2. According to the vehicle-road cooperative 3D target detection method integrating time series information according to claim 1, It is characterized in that The obtained collaborative query is input into the detection head, and the final detection result is output.

3. According to the vehicle-road cooperative 3D target detection method integrating time series information according to claim 1, It is characterized in that If the communication is interrupted, the Kalman filter is used to track the status of the historical trajectory, and the position of each target reference point at the current moment is predicted based on the status of the trajectory; at the same time, the multi-head attention mechanism is used to predict the feature embedding of each target at the current moment based on the feature embedding of the historical query signal.

4. According to the vehicle-road cooperative 3D target detection method integrating time series information according to claim 1, It is characterized in that After extracting features from the input road-side image data through the preset perception model, the query signal is output through the query generator, while retaining the historical query signal generated from the previous k1 time steps; using the temporal context integration mechanism, the query signal is integrated with the historical query signal to generate a time-series query signal containing historical information; then, the contextual time-series query signal is filtered by confidence, and the top m queries with the highest confidence are transmitted through V2X communication. The vehicle receives the top m queries to obtain the fused road-side query, and stores the road-side query received in k2 frames.

5. According to the vehicle-road collaborative 3D target detection method integrating time series information as described in claim 1, It is characterized in that The time context integration mechanism specifically includes: Time Context Aggregation: The query generator on the road side models the currently observed target, generates a query signal, and inputs the query signal into the temporal context aggregation module; this module uses a multi-head cross-attention mechanism to fuse the query signals of the past k1 frames with the current query signal to enhance the temporal expression ability of the query. Timing Guided Fusion: After the vehicle receives the query signal from the road, it performs spatiotemporal fusion. First, the geometric space is aligned and the reference point is converted into a unified coordinate system through the rotation and translation matrix. Align the feature space, use MLP to convert the reference point and rotation matrix into position embedding, and then concatenate it with the query embedding to obtain the final unified query embedding; The cost matrix is ​​constructed based on the Euclidean distance of the reference points through the Hungarian algorithm for optimal matching and fusion. Unmatched queries are directly inserted into the coarse fusion query set. The historical queries at the road end are used to guide the fusion query to learn the temporal features, and this process is achieved through a multi-head cross-attention mechanism.

6. According to the vehicle-road cooperative 3D target detection method integrating time series information as described in claim 1, It is characterized in that The motion perception reconstruction module extracts motion features from the road-end historical query sequence frame by frame and reconstructs the motion features through the following steps: State estimation: Use Kalman filter to estimate the state of the historical trajectory; Association: Construct a cost matrix based on the Euclidean distance between the query and the estimated state, and apply the Hungarian algorithm for matching; State update: The Kalman filter updates the trajectory state based on the matched reference points; Track management: create new instance tracks, delete disappeared tracks, and get the final track results; By extracting and predicting query trajectories, vehicles are able to reconstruct and recover lost roadside queries when communication is interrupted and continue spatiotemporal collaborative detection.

7. A computer device, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a vehicle-road collaborative 3D target detection method integrating timing information as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a vehicle-road collaborative 3D target detection method integrating temporal information according to any one of claims 1 to 6.