Vehicle-road cooperation trajectory prediction method based on hierarchical space-time interaction modeling

Abstract

The invention relates to a vehicle-road cooperation trajectory prediction technology based on hierarchical space-time interaction modeling, and belongs to the technical field of automatic driving. The method comprises the following steps: acquiring sensing data historical track information of a vehicle-mounted sensor and roadside equipment, and constructing a track sequence containing a self-vehicle view angle and a roadside view angle and vectorized map representation according to high-precision map information; performing embedded coding on the historical track information and the vectorized map after the relative coordinate transformation through a multi-layer perceptron, and generating input feature representation; modeling a spatial relationship between the vehicle and the vectorized map through a map Transform; the method comprises the following steps of: extracting cross-block space-time interaction characteristics through alternate coding of a time Transform, a space Transform and a map Transform; and generating a multi-modal trajectory prediction result through an output layer, wherein the multi-modal trajectory prediction result comprises position coordinates of the target vehicle in a future time domain and uncertainty estimation. Through an individual-level and block-level two-layer processing structure, the problem that the calculation efficiency and feature extraction of a traditional method in long-time-sequence modeling are insufficient is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to a vehicle-road cooperative trajectory prediction technology based on hierarchical spatio-temporal interaction modeling, belonging to the technical field of autonomous driving. BACKGROUND

[0002] Accurate prediction of the motion trajectory of surrounding vehicles is crucial for autonomous vehicles. Currently, autonomous vehicles mainly rely on on-board sensors and computing power for environment perception and prediction. However, even with advanced perception devices and algorithms, the perception range of a single autonomous vehicle is still limited, which hinders a comprehensive understanding of the traffic situation.

[0003] In scenarios where vehicles are severely occluded or at a long distance, single-vehicle perception faces significant challenges, which can lead to sparse, unreliable, or even missing perception data, resulting in inaccurate trajectory prediction and unreasonable path planning. To solve this problem, vehicle-road cooperative autonomous driving solutions achieve information interconnection between autonomous vehicles and the surrounding environment through vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) cooperative technologies, thereby obtaining more comprehensive environmental perception capabilities.

[0004] However, existing cooperative trajectory prediction methods still have deficiencies in multi-view information fusion. Current technical solutions typically use simple data splicing methods to process multi-view trajectory sequences, or independently process different view information in the network architecture, which makes it difficult to fully exploit the complementarity of on-board and roadside view information. In addition, existing methods have limited ability to model spatio-temporal interaction features, especially in handling long-term temporal dependencies and dynamic interaction behaviors, which ultimately affects the accuracy of trajectory prediction. SUMMARY

[0005] To address the problems in the prior art, the present application provides a vehicle-road cooperative trajectory prediction technology based on hierarchical spatio-temporal interaction modeling, which fully integrates the complementary information of on-board and roadside views by constructing a hierarchical space-time Transformer architecture, effectively models the dynamic interaction between vehicles, and can provide more accurate trajectory prediction in a vehicle-road cooperative environment, providing reliable support for path planning and decision control of autonomous vehicles.

[0006] The present application adopts the following technical solutions:

[0007] The vehicle-road cooperative trajectory prediction method based on hierarchical spatio-temporal interaction modeling according to the present application,

[0008] S1: Obtain perception data from on-board sensors and roadside devices, extract historical trajectory information of target vehicles and surrounding vehicles, and obtain high-precision map information, construct trajectory sequences containing on-board and roadside views, and vectorized map representations;

[0009] S2: convert the vehicle position coordinates into relative coordinates centered on the ego vehicle, embed and encode the transformed historical trajectory information and the vectorized map through a multi-layer perception machine to generate input feature representations;

[0010] S3: divide the trajectory sequence in step S1 into multiple blocks according to time, in each block, extract time-dependent features through individual-level time Transformer, and extract spatial interaction features between vehicles through individual-level spatial Transformer;

[0011] S4: fuse the spatial interaction features from the ego vehicle perspective and the roadside perspective through a vehicle-road coordination pooling layer to generate vehicle-road coordination feature representations, and model the spatial relationship between the vehicle and the vectorized map through a map Transformer;

[0012] S5: decode the block-level spatio-temporal features extracted in step S4 using a block-level space-time Transformer, and extract cross-block spatio-temporal interaction features through the alternating encoding of time Transformer, space Transformer and map Transformer;

[0013] S6: generate multi-modal trajectory prediction results through the output layer, including the position coordinates and uncertainty estimates of the target vehicle in the future time domain.

[0014] The vehicle-road coordination trajectory prediction method based on hierarchical spatio-temporal interaction modeling according to the present application, step S2 includes:

[0015] S2.1: extract the trajectory coordinate sequence of N vehicles under the vehicle-mounted and roadside perspectives in the historical T h time steps, and convert it into relative coordinates centered on the ego vehicle, then input encode the trajectory at time t using a multi-layer perception machine; the expression is as follows:

[0016]

[0017] wherein, represents a rotation matrix, and represent the coordinates of vehicle i at time t under the vehicle-mounted and roadside perspectives, respectively, W MLP,* represents the learning parameters of the MLP network;

[0018] S2.2: Extract the vectorized map information under the corresponding scene, including lane line information Lane vectors, Lv, traffic control signal Traffic controls, Tc and turn direction information Turn directions, Td, denoted as Map = {Lv, Tc, Td}.

[0019] The vehicle-road cooperative trajectory prediction method based on hierarchical space-time interaction modeling described in the application, S3 step includes:

[0020] S3.1: The trajectory sequence is divided into B non-overlapping time blocks according to the fixed time step ΔT b Each block contains continuous time step trajectory information;

[0021]

[0022] Where, T h And T f represent the observation step and the prediction step respectively, B represents the number of divided blocks, represents the last timestamp of block b;

[0023] S3.2: Use individual-level time Transformer to time-encode the trajectories in each block through multi-head self-attention mechanism, capture the dependence relationship and vehicle motion pattern in the time dimension, and generate time feature vector

[0024]

[0025] Where, TempTF * () represents the time Transformer module, W TempTF,* represents the learnable parameters of TempTF * The Query vector Key vector and Value vector in the time Transformer module are defined as follows,

[0026]

[0027] Where, represents the coordinates at T b , and correspond to the linear change matrix of the three vectors respectively;

[0028] S3.3: Use individual-level space Transformer to model the spatial interaction relationship between different vehicles at the same time through cross-attention mechanism, and take the relative position information as a key-value pair, and the time feature As a query vector, and generate spatial features by fusing the ego vehicle features and interaction features through a gating function The expression is as follows:

[0029]

[0030] Wherein, SpatTF * () represents a spatial Transformer module, W SpatTF,* represent the learnable parameters of SpatTF * (). respectively represent the coordinates of vehicle i in the vehicle-mounted and roadside perspectives at T b time, the Query vector Key vector and Value vector in the spatial Transformer module are defined as follows,

[0031]

[0032] Wherein, respectively correspond to the linear change matrix of the three vectors.

[0033] The vehicle-road cooperative trajectory prediction method based on hierarchical space-time interaction modeling, the specific steps of step S4 are as follows:

[0034] S4.1: Through the vehicle-road cooperative pooling layer, the spatial interaction features in the vehicle-mounted perspective vic and the roadside perspective are weighted and fused by using the gating mechanism Pooling to generate vehicle-road cooperative feature representation

[0035]

[0036] W MLP,vic represent the learnable parameters of Pooling vic .

[0037] S4.2: Use the map Transformer module to model the spatial relationship between the vehicle trajectory features and the vectorized map elements Map through the cross-attention mechanism, integrate the road geometry constraints and traffic rules into the feature representation, and generate the final feature representation

[0038]

[0039] W MapTF,vic represent the learnable parameters of MapTF viclearnable parameters of the MLP

[0040] decoded block-level features are mapped to generate multi-modal trajectory prediction results with dimension [K, N, ΔT b , 4];

[0041] including position coordinates and uncertainty estimates of target vehicles in future time domain, where K represents the number of prediction trajectory modes, N represents the number of vehicles, ΔT b represents the time step, and the last dimension records the position and uncertainty estimate at each time step

[0042]

[0043] pi=MLP pi (Seq vic (pred);W MLP,pi )W MLP,output and W MLP,pi represent learnable parameters of the MLP output , MLP pi , respectively;

[0044] The loss function includes two parts of regression loss and classification loss, the regression loss is used to represent the coordinate error between the predicted trajectory and the real trajectory, and the classification loss is used to evaluate the uncertainty of the multi-modal trajectory prediction, and the learnable parameters of the model are updated through the back propagation of the loss function;

[0045] wherein the regression loss is defined by using the negative log-likelihood function of Laplace distribution, and the calculation formula is:

[0046]

[0047] wherein P(·|·) represents the Laplace distribution, represents the real trajectory of vehicle i at time t; and represent the predicted position and uncertainty estimate, respectively; represents a rotation matrix;

[0048] The classification loss adopts a cross-entropy loss function, and the total loss function is:

[0049]

[0050] Advantages

[0051] The hierarchical space-time interaction modeling architecture effectively solves the problems of calculation efficiency and insufficient feature extraction in long-time sequence modeling of traditional methods through two-layer processing structures of individual level and block level. The individual level Transformer focuses on extracting time-dependent and spatial interaction features of a single vehicle, and the block level Transformer is responsible for capturing global space-time relationships across time blocks. This hierarchical design not only ensures fine extraction of local features, but also realizes efficient modeling of long-time sequence dependence.

[0052] 2、The vehicle-road cooperation pooling layer of the application realizes deep fusion of the self-vehicle perspective and the road-side perspective information, breaking through the limitation of the existing method relying on a single perspective. Through feature-level fusion instead of simple data splicing, the local fine perception of the vehicle-mounted sensor and the global macroscopic perception advantage of the road-side device are fully utilized, and stable prediction performance can be maintained under abnormal conditions such as sensor failure or perspective obstruction.

[0053] 3、The application introduces a map Transformer to take the vectorized road structure information as an explicit constraint condition, ensuring that the predicted trajectory conforms to the actual road geometry and traffic rules. Unlike existing methods that treat map information as a simple background, this method models the spatial relationship between vehicles and road structures through deep feature interaction, significantly improving the rationality and executability of the predicted trajectory, and providing a more reliable input for path planning of autonomous vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a vehicle-road cooperation trajectory prediction process diagram based on hierarchical space-time interaction modeling.

[0055] Figure 2 is a vehicle-road cooperation trajectory prediction total model framework diagram based on hierarchical space-time interaction modeling.

[0056] Figure 3 is a frame diagram of a time-space Transformer network.

[0057] Figure 4 is a trajectory prediction visualization result under a typical intersection scenario.

[0058] Figure 5 is a trajectory prediction visualization result under an occlusion and over-the-horizon scenario. DETAILED DESCRIPTION

[0059] In order to make the purpose and technical solutions of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort belong to the scope of protection of the present application.

[0060] As shown in Figure 1 , Figure 2 , the present application discloses a vehicle-road cooperative trajectory prediction method based on hierarchical space-time interaction modeling, comprising the following steps:

[0061] S1: the model input contains trajectory sequences from the perspective of the ego vehicle and the roadside, and a vectorized map representation;

[0062] S2: the encoder converts the vehicle position coordinates into relative coordinates centered on the ego vehicle, and then embeds and encodes the transformed trajectory data and the vectorized map through a multi-layer perceptron (MLP) to generate input feature representation;

[0063] S2.1: extract the trajectory coordinate sequences of N vehicles under the vehicle-mounted and roadside perspectives in the past T h time steps, and convert them into relative coordinates centered on the ego vehicle, and then use a multi-layer perceptron for input encoding;

[0064]

[0065] wherein, represents a rotation matrix, and represent the coordinates of vehicle i at time t under the vehicle-mounted and roadside perspectives, respectively, and W MLP,* represents the learning parameters of the MLP network.

[0066] S2.2: extract the vectorized map information in the corresponding scene, including lane line vectors, traffic control signals, and turning direction information, denoted as Map={Lv,Tc,Td}.

[0067] S3: the encoder divides the trajectory sequence into multiple blocks according to time, and in each block, extracts time-dependent features through individual-level time Transformer and models the spatial interaction relationship between vehicles through individual-level space Transformer;

[0068] S3.1: divide the trajectory sequence into B non-overlapping time blocks according to a fixed time step ΔT b , each of which contains continuous time step trajectory information;

[0069]

[0070] where T h and T f represent the observation step and the prediction step, respectively, and B represents the number of divided blocks, denotes the last timestamp of block b;

[0071] S3.2: Time encoding of the trajectory sequence within each block by individual-level time Transformer through multi-head self-attention mechanism, capturing the dependence and vehicle motion pattern in the time dimension, and generating a time feature vector

[0072]

[0073] where TempTF * () represents the time Transformer module, and Query, Key and Value are defined as follows,

[0074]

[0075] S3.3: Modeling the spatial interaction relationship between different vehicles at the same time by individual-level space Transformer through cross-attention mechanism, taking the relative position information as the key-value pair, and the time feature as the query vector, and generating the spatial feature by fusing the ego feature and the interaction feature through the gating function

[0076]

[0077] where SpatTF * () represents the space Transformer module, and Query, Key and Value are defined as follows,

[0078]

[0079] S4: The encoder fuses the spatial interaction features of the ego vehicle perspective and the roadside perspective through the Vehicle-Infrastructure Cooperative (VIC) pooling layer to generate the Vehicle-Infrastructure Cooperative feature representation, and models the spatial relationship between the vehicle and the vectorized map through the map Transformer;

[0080] S4.1: Through the Vehicle-Infrastructure Cooperative pooling layer, the ego vehicle perspective and the roadside perspective ​The spatial interaction features of the vehicle and the road are fused by weighting to generate a vehicle-road coordination feature representation

[0081]

[0082] S4.2: The map Transformer module is used to model the vehicle trajectory features through cross-attention mechanism The spatial relationship between the vectorized map elements Map, the road geometry constraints and the traffic rules are integrated into the feature representation to generate the final feature representation

[0083]

[0084] S5: The decoder uses a block-level space-time Transformer to decode the encoded block-level time sequence features, and extracts cross-block space-time interaction features through the alternation of the time Transformer, the space Transformer and the map Transformer.

[0085] S6: The decoder generates a multi-modal trajectory prediction result through the output layer, including the position coordinates and uncertainty estimation of the target vehicle in the future time domain.

[0086] The output layer maps the decoded block-level features to generate a multi-modal trajectory prediction result with a dimension of [K, N, ΔT b , 4], including the position coordinates and uncertainty estimation of the target vehicle in the future time domain, where K represents the number of prediction trajectory modes, N represents the number of vehicles, ΔT b represents the number of time steps of each block, and 4 represents the position and uncertainty estimation

[0087]

[0088] pi=MLP pi (Seq vic (pred);W MLP,pi ) (10)

[0089] The loss function includes two parts of regression loss and classification loss, where the regression loss is defined by the negative log-likelihood function of the Laplace distribution, and the calculation formula is:

[0090]

[0091] where P(·|·) represents the Laplace distribution, represents the true trajectory of vehicle i at time t. and denote the predicted position and uncertainty estimate, respectively; denotes a rotation matrix.

[0092] The classification loss adopts a cross-entropy loss function, and the total loss function is:

[0093]

[0094] More specifically, the schematic diagram of the Transformer network framework described in steps S3, S4 and S5 is shown in Figure 3 The Transformer network adopts a hierarchical structure, including individual level and block level. The individual level Transformer processes the historical trajectory sequence within each block to extract time-dependent features and spatial interaction features; the block level Transformer processes the cross-block relationship to maintain the spatio-temporal consistency through alternating time, space and map Transformers.

[0095] Figure 4 The prediction results of the method in a typical intersection scenario of the V2X-Seq dataset are shown in Figure 4 In the actual application scenario of the V2X-Seq dataset, the method of the present application can accurately predict the multi-modal trajectory of the target vehicle. The prediction results of two typical scenarios are shown in the figure: the prediction performance in the turning scenario and the straight-ahead scenario. In the turning scenario, the method of the present application successfully handles the complexity of the environment, and can accurately identify the turning trajectory of the target vehicle even when the AV is waiting at the intersection and is far away from the target vehicle. For the straight-ahead scenario, the model shows higher prediction accuracy, and the multi-modal output shows significant consistency and alignment with the real trajectory. These accurate predictions are particularly valuable for autonomous vehicle systems, enabling more reliable decision-making and path planning, and ultimately improving operational safety. The experimental results show that the method of the present application has good effectiveness in different driving modes, verifying the superior performance of the hierarchical spatio-temporal interaction modeling framework.

[0096] Figure 5 The prediction results of the method in the occlusion and over-the-horizon complex scenarios involved in the V2X-Seq dataset are shown in Figure 5As shown, in complex scenes such as occlusion and over-the-horizon, the vehicle-road cooperative trajectory prediction method can effectively handle the case of limited single vehicle perception by fusing the complementary information of the self-vehicle perspective and the roadside perspective. The visualization results show the prediction performance of the model in several challenging scenarios. The first row shows the scenario of the target vehicle performing lane change in the rear area of the AV. From the perspective of the AV, the severe occlusion makes it difficult to obtain the complete historical trajectory of the target vehicle and surrounding vehicles, while the roadside perspective provides key complementary information, enabling the VIC strategy to obtain complete historical trajectory information. The second and third rows further show complex scenarios where the AV encounters multiple occluded vehicles at a long distance when predicting the trajectory of a left-turn target vehicle. In these scenarios, the AV's field of view is severely obstructed, making it difficult to accurately assess the upcoming intersection traffic conditions. The figure compares the prediction results of three strategies: vehicle-only perspective (Vehicle-Only), data-level fusion vehicle-road cooperation (Perception-prediction, PP-VIC), and feature-level vehicle-road cooperation (Feature-VIC). The superiority of the feature-level fusion strategy proposed in the present application over the data-level fusion method is verified, and the effectiveness of the hierarchical spatio-temporal Transformer in effectively utilizing multi-perspective trajectory complementary information for dynamic interaction modeling is demonstrated.

[0097] The above is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A vehicle-road cooperative trajectory prediction method based on hierarchical spatiotemporal interaction modeling, characterized in that: S1: Acquire perception data from vehicle-mounted sensors and roadside equipment, extract historical trajectory information of the target vehicle and surrounding vehicles, and acquire high-precision map information to construct a trajectory sequence and vectorized map representation that includes the vehicle's perspective and the roadside perspective. S2: Convert the vehicle's position coordinates into relative coordinates centered on the vehicle, and use a multilayer perceptron to embed and encode the historical trajectory information and vectorized map after the relative coordinate transformation to generate input feature representations; S3: Divide the trajectory sequence in step S1 into multiple blocks according to time. Within each block, extract time-dependent features using an individual-level temporal Transformer and extract spatial interaction features between vehicles using an individual-level spatial Transformer. S4: By fusing the spatial interaction features of the vehicle's perspective and the roadside perspective through the vehicle-road cooperative pooling layer, a vehicle-road cooperative feature representation is generated, and the spatial relationship between the vehicle and the vectorized map is modeled through the map Transformer. S5: The block-level spatiotemporal features extracted in step S4 are decoded using a block-level spatial-temporal Transformer. Cross-block spatiotemporal interaction features are extracted through alternating encoding of temporal Transformer, spatial Transformer and map Transformer. S6: Generates multimodal trajectory prediction results through the output layer, including the target vehicle's position coordinates and uncertainty estimates in the future time domain.

2. The vehicle-road cooperative trajectory prediction method based on hierarchical spatiotemporal interaction modeling according to claim 1, characterized in that: Step S2 includes: S2.1: Extract N vehicles from historical T h The trajectory coordinate sequence from the vehicle-mounted and roadside perspectives at each time step is converted into relative coordinates centered on the vehicle. Then, a multilayer perceptron is used to encode the trajectory at time t. The expression is as follows: in, Represents a rotation matrix. and W represents the coordinates of vehicle i at time t from the onboard and roadside perspectives, respectively. MLP,* These represent the learning parameters of the MLP network; S2.2: Extract vectorized map information for the corresponding scenario, including lane vectors (Lv), traffic controls (Tc), and turning directions (Td), denoted as Map = {Lv, Tc, Td}.

3. The vehicle-road cooperative trajectory prediction method based on hierarchical spatiotemporal interaction modeling according to claim 1, characterized in that: Step S3 includes: S3.1: Divide the trajectory sequence into fixed time steps ΔT b The time is divided into B non-overlapping time blocks, each containing continuous time step trajectory information; Among them, T h and T f These represent the observation step size and the prediction step size, respectively, and B represents the number of blocks. This represents the last timestamp of block b; S3.2: Using an individual-level temporal Transformer with a multi-head self-attention mechanism, the trajectory within each block is temporally encoded to capture temporal dependencies and vehicle motion patterns, generating a temporal feature vector. Among them, TempTF * () represents the time Transformer module, W TempTF,* Represents TempTF * Learnable parameters of (), Query vector in the time Transformer module. Key vector and Value vector They are defined as follows: in, Represents T b The coordinates of time, W Qtem p, W Ktem p, W Vtem p corresponds to the linear transformation matrix of each of the three vectors; S3.3: Using an individual-level spatial Transformer, a cross-attention mechanism is employed to model the spatial interaction relationships between different vehicles at the same time, integrating relative position information. As key-value pairs, time features As the query vector, spatial features are generated by fusing vehicle features and interaction features through a gating function. The expression is as follows: SpatTF * () represents the spatial Transformer module, W SpatTF,* Represents SpatTF * Learnable parameters of (), These represent the vehicle i in T from the in-vehicle and roadside perspectives, respectively. b The coordinates of time, the Query vector in the spatial Transformer module. Key vector and Value vector They are defined as follows: in, These are the linear transformation matrices corresponding to the three vectors, respectively.

4. The vehicle-road cooperative trajectory prediction method based on hierarchical spatiotemporal interaction modeling according to claim 1, characterized in that: The specific steps of step S4 are as follows: S4.1: Through the vehicle-road cooperative pooling layer, a gating mechanism is used to pool... vic From the perspective of a self-driving car and roadside view The spatial interaction features are weighted and fused to generate a vehicle-road cooperative feature representation. W MLP,vic Represents Pooling vic Learnable parameters; S4.2: Using the Map Transformer module, model vehicle trajectory features through a cross-attention mechanism. The spatial relationships between vectorized map elements (Map) are incorporated into the feature representation, integrating road geometric constraints and traffic rules to generate the final feature representation. W MapTF,vic Represents MapTF vic Learnable parameters; The decoded block-level features are processed through the output layer. Perform mapping to generate dimensions [K, N, ΔT] b Multimodal trajectory prediction results [4]; This includes the target vehicle's position coordinates and uncertainty estimation in the future time domain, where K represents the number of predicted trajectory modes, N represents the number of vehicles, and ΔT b This represents the time step, and the last dimension records the position of each time step. and uncertainty estimation pi=MLP pi (Seq vic (pred);In MLP,pi ) W MLP,output and W MLP,pi Representing MLP output MLP pi Learnable parameters; The loss function comprises two parts: regression loss and classification loss. The regression loss is defined using the negative log-likelihood function of the Laplace distribution, and its calculation formula is as follows: Where P(·|·) represents the Laplace distribution, This represents the actual trajectory of vehicle i at time t; and These represent the predicted location and the uncertainty estimate, respectively; Represents the rotation matrix; The classification loss uses the cross-entropy loss function, and the total loss function is:

Images