A trajectory prediction method based on a digital twin platform

Abstract

The application discloses a trajectory prediction method based on a digital twin platform, real-time collection of vehicle position, speed, driving state, surrounding environment state and other information; after initial processing of the information, the vehicle transmits the processed information to the trajectory prediction platform located in the cloud through the built-in communication module; then, the data features of the vehicle and the roadside unit are extracted through a multi-channel mechanism, and the extracted features are sent to the Transform module to predict the future trajectory; finally, the platform feeds back the prediction result to the vehicle, so that the driver can timely adjust the current trajectory and avoid traffic accidents. Compared with other schemes, the ADE and FDE indexes of the application are the lowest, the model efficiency is the best, and in addition, the average inference delay is 11.83ms within a 5-second trajectory prediction time step.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent vehicle technology, and specifically relates to a trajectory prediction method based on a digital twin platform. Background Art

[0002] Smart cars equipped with technologies such as V2X (Vehicle to Everything), cloud computing, and intelligent sensors provide users with high-quality services and a convenient travel experience. Trajectory prediction, which can perceive the vehicle's surroundings, demonstrates significant value in traffic safety. Currently, mainstream trajectory prediction solutions can be divided into two categories: single-vehicle trajectory prediction and vehicle-infrastructure collaborative trajectory prediction.

[0003] Single-vehicle trajectory prediction schemes rely on the vehicle's own collected historical trajectory and driving status information to predict the vehicle's trajectory at the next moment. However, this approach has some obvious limitations. First, because the data comes from a single vehicle, it leads to neglecting the interactions between traffic participants. Second, it may not accurately capture the driving styles of different drivers, which may vary due to factors such as personal habits and emotional state. Finally, due to the limited field of view and capabilities of the single-vehicle perception system, it may not be able to fully obtain the status information of surrounding vehicles, which may lead to inaccurate prediction results in complex traffic scenarios.

[0004] To overcome these shortcomings, researchers have proposed trajectory prediction schemes based on vehicle-infrastructure collaboration (V2I). By integrating resources such as connected vehicle technology, roadside infrastructure, and cloud computing, these schemes provide a more comprehensive understanding of the vehicle's surrounding environment. Specifically, they leverage information provided by the target vehicle, surrounding vehicles, and roadside units (ROUs) to more accurately predict the vehicle's future trajectory. Trajectory prediction schemes based on V2I collaboration are further divided into vehicle-to-vehicle (V2V) and vehicle-to-road collaborative (V2RS) approaches. V2V approaches analyze the historical trajectories and motion states of the target vehicle and surrounding vehicles to predict their next moves. This approach overcomes the lack of information about surrounding vehicles found in single-vehicle trajectory prediction schemes. However, its limitation lies in its limited information source, which may overlook surrounding road features such as road shape and traffic signs, which can affect the vehicle's trajectory. V2RS approaches effectively utilize road information, using data collected by RSUs to assist in trajectory prediction. This approach provides more comprehensive information about the traffic environment, including traffic flow and signal status, thereby improving prediction accuracy. However, it also has its limitations. First, the state information of surrounding vehicles used in this solution may be estimated rather than directly perceived, which may introduce additional uncertainty. Second, trajectory prediction solutions based on vehicle-infrastructure collaboration often rely on the vehicle's own computing power. However, onboard computing resources are generally limited, and communication bandwidth can become a bottleneck, limiting the application of complex algorithms and, in turn, affecting trajectory prediction accuracy.

[0005] While vehicle-infrastructure collaboration-based trajectory prediction solutions offer significant advantages in improving prediction accuracy, they still face challenges in terms of computing resources, communication bandwidth, and information accuracy. Future research needs to further optimize algorithms, improve computational efficiency, and explore more efficient information fusion and communication strategies to fully realize the potential of vehicle-infrastructure collaboration-based trajectory prediction solutions and provide users with a safer and more reliable travel experience. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the present invention provides a trajectory prediction method based on a digital twin platform. This method collects information such as the vehicle's location, speed, driving status, and surrounding environment in real time. After initial processing of this information, the vehicle uses its built-in communication module to transmit the processed information to a trajectory prediction platform located in the cloud. A multi-channel mechanism is then used to extract data features from the vehicle and roadside units, and these features are sent to a Transformer module to predict future trajectories. Finally, the platform feeds the prediction results back to the vehicle, allowing the driver to adjust the current trajectory in a timely manner to avoid traffic accidents. Compared with other solutions, the present invention achieves the lowest ADE and FDE indicators and the highest model efficiency. Furthermore, within a 5-second trajectory prediction time step, the average inference latency is 11.83ms.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] Step 1: Vectorize the collected vehicle information and map information;

[0009] For the vehicle, the dynamic state at time t is represented by x, y, v x ,v y ,a x ,a y ,φ, is represented by a high-dimensional vector of 7 features; (x, y) describes the spatial position of the target vehicle, (v x ,v y ,a x ,a y ) represents the tendency of the vehicle to move in the x and y directions, where v x 、a x Represent the vehicle's velocity and acceleration in the lateral direction, v y , a y denote the velocity and acceleration in the longitudinal direction, respectively, and φ denotes the direction of the vehicle;

[0010] Use D n 、D s 、D w 、D e 、D n-e 、D s-e 、D n-w 、D s-w Indicates the straight-line distance between vehicles in eight directions: north, south, west, east, northeast, southeast, northwest, and southwest;

[0011] Use V n 、V s 、V w 、V e 、V n-e 、V s-e 、V n-w 、V s-w Respectively represent the motion state differences between vehicles in eight directions: north, south, west, east, northeast, southeast, northwest, and southwest;

[0012] The auxiliary factor a is used to represent the acceleration difference between vehicles in the scene;

[0013] use represents the angle difference between vehicles in the selected scene;

[0014] For map information, the collected map information is converted into a direction P ψ and coordinates (P x ,P y), which is used to distinguish different road features such as straight traffic, circular lanes and intersections;

[0015] Step 2: Call the channel attention mechanism to extract the influence weights of different features from vehicles and roadside units (RSUs). The original vehicle and map feature vectors first pass through the fully connected layer, which further transforms and maps the feature vectors. The mapping relationship is shown in the following formula:

[0016] h=Wx+b

[0017] Where h represents the output vector of the vehicle and map, x represents the original input feature vector of the vehicle and map, W is the weight matrix, and b is the bias vector;

[0018] After the fully connected layer, an activation function is used to perform a nonlinear transformation on the previously transformed vector;

[0019] The Sigmoid function is used to normalize the result vector of the activation function; the Sigmoid function maps the original feature weight to a probability distribution that indicates the contribution of each channel to the prediction result;

[0020] Finally, the output weight of the Sigmoid function is multiplied by the original feature vectors of the vehicle and the roadside unit RSU to obtain the output vector;

[0021] Step 3: The output vector from the multi-channel attention mechanism is first passed through position encoding. To obtain the position encoding, the unique encoding vectors from different positions are merged with the output vector from step 2 into a new vector. Secondly, the new vector is used as the input of the transformer encoder, which consists of an attention mechanism, a feedforward neural network, and a residual layer. In the attention mechanism, the new vector is decomposed into query Q, key K, and value V. Then, the similarity between the query vector and the key vector is calculated to obtain the attention weight that reflects the importance of each query-key correspondence. The attention weight is assigned to the corresponding value vector, and the feature representation after the corresponding attention mechanism is obtained by summing. The calculation formula is as follows:

[0022]

[0023] Next, this process is repeated H times; finally, they are concatenated through the Concatenate layer to obtain the global self-attention feature map, which is calculated as follows:

[0024] Multihead(Q,K,V)=Concat(head1,···,head h )W o

[0025] head i=Attention(QW i Q ,KW i K ,VW i V ),

[0026] i=1,2,...,H

[0027] This weight matrix combines all outputs into the final output vector to capture richer feature representations;

[0028] After executing the attention mechanism, the transformer encoder introduces a set of feedforward neural networks with residual layers to compress the entire input sequence into a hidden state tensor HS; HS is used as the initial hidden state as the input of the transformer decoder; the transformer decoder gradually generates the target sequence through the attention mechanism; in the transformer decoder, the initial input X obtained at time t and HS is used t To predict the X of the next vehicle with step size t+1 t+1 ; On this basis, the output X at time t+1 is t+1 and hidden state HS+1 as the initial input and initial hidden state of Transformer Decoder, predicting X at time t+2 t+2 ; This prediction process will be repeated until the prediction of X is completed at prediction step t+n t+n The prediction result is then fed back to the vehicle to adjust the driving route.

[0029] Preferably, the activation function is a rectified linear unit (ReLU).

[0030] The beneficial effects of the present invention are as follows:

[0031] The present invention provides a trajectory prediction solution based on a digital twin platform. By combining the digital twin platform with trajectory prediction, it collects vehicle and environmental information in the scene in real time, solves the problems of limited vehicle field of view and insufficient vehicle computing resources, and further improves the accuracy of trajectory prediction. Specifically, through various sensors, the present invention can collect information such as the vehicle's position, speed, driving status, and surrounding environment status in real time. After initial processing of the information, the vehicle relies on its own built-in communication module to transmit the processed information to the trajectory prediction platform located in the cloud. Then, the present invention designs a multi-channel mechanism to extract data features of vehicles and roadside units, and sends the extracted features to the Transformer module to predict future trajectories. Finally, the platform feeds back the prediction results to the vehicle so that the driver can adjust the current trajectory in time to avoid traffic accidents. Compared with other solutions, the present invention has the lowest ADE and FDE indicators and the best model efficiency. In addition, within a trajectory prediction time step of 5 seconds, the average inference delay is 11.83ms. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart for implementing the trajectory prediction solution based on the digital twin platform in the present invention;

[0033] Figure 2 Comparison diagram of the average displacement error and the final displacement error in different scenarios of the present invention: (a) average displacement error of a three-lane straight road, (b) average displacement error of a four-lane straight road, (c) average displacement error of a nine-lane intersection, (d) final displacement error of a three-lane straight road, (e) final displacement error of a four-lane straight road, and (f) final displacement error of a nine-lane intersection.

[0034] Figure 3 Comparison diagram of the predicted trajectory and the actual trajectory in different scenarios of the present invention: (a) three-lane straight road, (b) nine-lane ring road, (c) four-lane ring road. DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings and examples.

[0036] The purpose of this invention is to provide a trajectory prediction solution based on a digital twin platform to solve the problems of limited vehicle field of view and insufficient vehicle computing resources, and to further improve the accuracy of trajectory prediction.

[0037] according to Figure 1 As shown, the present invention provides a trajectory prediction solution based on a digital twin platform, comprising the following steps:

[0038] Step 1: First, vectorize the collected vehicle information and map information. For a vehicle, the dynamic state at time t can be expressed as x, y, vx ,v y ,a x ,a y ,φ, is represented by a high-dimensional vector of 7 features. (x,y) describes the spatial position of the target vehicle, (v x ,v y ,a x ,a y ) represents the tendency of the vehicle to move in two directions, and φ represents the direction of the vehicle. In addition, the characteristics of the vehicle also include differences in position, motion state, and supplementary factors. The position of the vehicle is divided into eight directions: north, south, west, east, northeast, southeast, northwest, and southwest. In different directions, the present invention uses D n 、D s 、D w 、D e 、D n-e 、D s-e 、D n-w 、D s-w Indicates the straight-line distance between vehicles. In addition to the position of the vehicle, the difference in motion state between the target volume and nearby vehicles needs to be considered. The motion state difference is represented by the variable V n 、V s 、V w 、V e 、V n-e 、V s-e 、V n-w 、V s-w express.

[0039] In addition, auxiliary factors such as a represent the acceleration differences between vehicles in the scene.

[0040] For map information, the present invention converts the collected map information into a direction P ψ and coordinates (P x ,P y ). The point set converted by the present invention is used to distinguish different road features such as straight traffic, circular lanes, and intersections. The feature bodies in the traffic scene will help the model capture the influence weights of different factors in the next stage.

[0041] Step 2: The channel attention mechanism is called to extract the influence weights of different features from vehicles and roadside units (RSUs). The original vehicle and map feature vectors are first passed through the fully connected layer, which further transforms and maps the feature vectors. The mapping relationship is shown in the following formula:

[0042] h=Wx+b

[0043] Where h and x represent the output vector and original feature vector of the vehicle and map respectively, W is the weight matrix, and b is the bias vector. After the fully connected layer, an activation function such as the rectified linear unit (ReLU) is used to perform a nonlinear transformation on the previously transformed vector. It can learn more complex function equations. Then, the result vector of the activation function is normalized using the Sigmod function. The Sigmod function processes the obtained weight vector to ensure the rationality of the attention weight. The Sigmoid function maps the original feature weight to a probability distribution that indicates the contribution of each channel to the prediction result. Finally, the output weight of the Sigmoid function is multiplied by the original feature vector of the vehicle and roadside unit RSU to obtain the output vector.

[0044] In step 3, the output vector from the multi-channel attention mechanism will first pass through the position encoding part. In order to obtain the position encoding, the unique encoding vectors from different positions are merged with them into a new vector. Secondly, the new vector is used as the input of the transformer encoder, which consists of an attention mechanism, a feedforward neural network, and a residual layer. In the attention mechanism, these vectors are decomposed into query Q, key K, and value V. Then, the similarity between the Query vector and the Key vector is calculated to obtain the attention weight that reflects the importance of each Query-Key correspondence. These weights are assigned to the corresponding Value vector, and the feature representation after the corresponding attention mechanism is obtained by summing them. The calculation formula is as follows:

[0045]

[0046] Next, this process is repeated H times. Finally, they are concatenated through the Concatenate layer to obtain the global self-attention feature map. The calculation formula is as follows:

[0047] Multihead(Q,K,V)=Concat(head1,···,head h )W o

[0048] head i =Attention(QW i Q ,KW i K ,VW i V ),

[0049] i=1,2,...,H

[0050] This weight matrix combines all outputs into the final output vector to capture a richer feature representation.

[0051] After executing the attention mechanism, the transformer encoder introduces a set of feedforward neural networks with residual layers to compress the entire input sequence into a hidden state tensor HS. HS is used as the initial hidden state and as the input of the transformer decoder. The transformer decoder gradually generates the target sequence through the attention mechanism. In the transformer decoder, the initial input X obtained at time t and HS is used. t To predict the X of the next vehicle with step size t+1 t+1 On this basis, the output X at time t+1 is t+1 and hidden state HS+1 as the initial input and initial hidden state of Transformer Decoder, predicting X at time t+2 t+2 This prediction process will be repeated until the prediction of X is completed at prediction step t+n. t+n The trajectory prediction results are then fed back to the vehicle to adjust the driving route.

[0052] Example:

[0053] refer to Figure 2 As shown, Figure 2 The proposed trajectory prediction scheme is compared with LSTM, Social-LSTM, and CS-LSTM, and its accuracy is judged by visualizing the ADE / FDE results in three scenarios. The figure shows the prediction accuracy comparison for different scenarios within the next 5 seconds. The vertical axis represents the prediction accuracy of different schemes for the ADE / FDE value, and the horizontal axis represents the prediction time step. Based on the qualitative results, it can be seen that the scheme of the present invention has high accuracy and flexibility in various scenarios. As the compensation increases, the accuracy of trajectory prediction gradually decreases, but compared with other schemes, the scheme of the present invention still maintains high accuracy in three different scenarios. The results demonstrate the robustness of the scheme. In addition, the scheme fully utilizes information from different scales and sources, improving the performance and effectiveness of trajectory prediction.

[0054] As shown in Tables 1, 2, and 3, Tables 1, 2, and 3 respectively show the comparison between the proposed schemes and the current LSTM, Social-LSTM, and CS-LSTM in three scenarios: straight road, circular road, and intersection. The present invention analyzes the superiority of schemes with different time steps in different scenarios. CS-LSTM performs well in the first 15 seconds of the straight lane and the first 15 seconds of the intersection. However, as the prediction step size gradually increases, the index of the scheme of the present invention at 5 seconds is significantly lower than that of the benchmark, which shows that the scheme of the present invention has superior performance in long-term trajectory prediction in different scenarios.

[0055] Table 1 Prediction accuracy of different schemes (ADE / FDE) under straight track

[0056]

[0057]

[0058] Table 2 Prediction accuracy of different schemes (ADE / FDE) under loop

[0059]

[0060] Table 3 Prediction accuracy of different schemes (ADE / FDE) at intersections

[0061]

[0062] refer to Figure 3 As shown, Figure 3 The digital twin platform demonstrated trajectory prediction results for three typical scenarios: a three-lane straight, a four-lane circular, and a nine-lane intersection. Using the digital twin platform, it can predict the trajectories of all vehicles in an area. This means it can track vehicles and provide timely feedback, warning drivers to avoid accidents. Figure 3 In the figure, the red and green lines represent the actual and predicted trajectories within 5 seconds, respectively, and the yellow line represents the prediction deviation. As the compensation amount increases, the prediction deviation gradually increases, but the prediction accuracy remains high. Furthermore, the solution of the present invention performs well in all scenarios, demonstrating its robustness.

[0063] Refer to Table 4, which shows a comparison of the inference latency required for different time steps. The present invention defines inference time as the time cost of predicting the future trajectory of the target vehicle in a scenario. The results are shown in Table 2, which shows the distribution of inference time at different step sizes. In the experiments, the present invention set a prediction time step of 1-5 seconds. The difference in inference time between different time steps is relatively small. This is because the smaller model size of the present invention results in shorter inference time.

[0064] Table 4 Comparison of inference latency required for different time steps (ms)

[0065]

[0066] Without loss of generality, this example experiments with a trajectory prediction solution based on a digital twin platform on the INTERACTION dataset. Developed by the MSC Lab at the University of California, Berkeley, INTERACTION uses drones to collect data from cities in China, the United States, and Germany. It is divided into three parts: training, validation, and testing. The training, validation, and test sets contain 47,584, 11,794, and 22,644 cases, respectively. The diverse scenarios and driving data improve the model's robustness and accuracy.

[0067] The average displacement error (ADE) and final displacement error (FDE) are used to evaluate the positioning accuracy. All experiments are performed on a PC equipped with an AMD5000 CPU and a GeForce RTX3060 GPU.

Claims

1. A trajectory prediction method based on a digital twin platform, characterized in that: The steps include: Step 1: Vectorize the collected vehicle information and map information; For the vehicle, the dynamic state at time t is represented by x, y, v x ,v y ,a x ,a y ,φ, is represented by a high-dimensional vector of 7 features; (x, y) describes the spatial position of the target vehicle, (v x ,v y ,a x ,a y ) represents the tendency of the vehicle to move in the x and y directions, where v x 、a x Represent the vehicle's velocity and acceleration in the lateral direction, v y , a y denote the velocity and acceleration in the longitudinal direction, respectively, and φ denotes the direction of the vehicle; Use D n 、D s 、D w 、D e 、D n-e 、D s-e 、D n-w 、D s-w Indicates the straight-line distance between vehicles in eight directions: north, south, west, east, northeast, southeast, northwest, and southwest; Use V n 、V s 、V w 、V e 、V n-e 、V s-e 、V n-w 、V s-w Respectively represent the motion state differences between vehicles in eight directions: north, south, west, east, northeast, southeast, northwest, and southwest; The auxiliary factor a is used to represent the acceleration difference between vehicles in the scene; use represents the angle difference between vehicles in the selected scene; For map information, the collected map information is converted into a direction P ψ and coordinates (P x ,P y ), which is used to distinguish different road features such as straight traffic, circular lanes and intersections; Step 2: Call the channel attention mechanism to extract the influence weights of different features from vehicles and roadside units (RSUs). The original vehicle and map feature vectors first pass through the fully connected layer, which further transforms and maps the feature vectors. The mapping relationship is shown in the following formula: h=Wx+b Where h represents the output vector of the vehicle and map, x represents the original input feature vector of the vehicle and map, W is the weight matrix, and b is the bias vector; After the fully connected layer, an activation function is used to perform a nonlinear transformation on the previously transformed vector; The result vector of the activation function is normalized using the Sigmoid function; the Sigmoid function maps the original feature weights to a probability distribution that indicates the contribution of each channel to the prediction result; Finally, the output weight of the Sigmoid function is multiplied by the original feature vectors of the vehicle and the roadside unit RSU to obtain the output vector; Step 3: The output vector from the multi-channel attention mechanism is first passed through position encoding. To obtain the position encoding, the unique encoding vectors from different positions are merged with the output vector from step 2 into a new vector. Secondly, the new vector is used as the input of the transformer encoder, which consists of an attention mechanism, a feedforward neural network, and a residual layer. In the attention mechanism, the new vector is decomposed into query Q, key K, and value V. Then, the similarity between the query vector and the key vector is calculated to obtain the attention weight that reflects the importance of each query-key correspondence. The attention weight is assigned to the corresponding value vector, and the feature representation after the corresponding attention mechanism is obtained by summing. The calculation formula is as follows: Next, this process is repeated H times; finally, they are concatenated through the Concatenate layer to obtain the global self-attention feature map, which is calculated as follows: Multihead(Q,K,V)=Concat(head1,···,head h )W o head i =Attention(QW i Q ,KW i K ,VW i V ), i=1,2,...,H Weight matrix W o Merge all outputs into the final output vector to capture richer feature representations; After executing the attention mechanism, the transformer encoder introduces a set of feedforward neural networks with residual layers to compress the entire input sequence into a hidden state tensor HS; HS is used as the initial hidden state as the input of the transformer decoder; the transformer decoder gradually generates the target sequence through the attention mechanism; in the transformer decoder, the initial input X obtained at time t and HS is used t To predict the X of the next vehicle with step size t+1 t+1 ; On this basis, the output X at time t+1 is t+1 and hidden state HS+1 as the initial input and initial hidden state of Transformer Decoder, predicting X at time t+2 t+2 ; This prediction process will be repeated until the prediction of X is completed at prediction step t+n t+n The prediction result is then fed back to the vehicle to adjust the driving route.

2. A trajectory prediction method based on a digital twin platform according to claim 1, characterized in that: The activation function is the rectified linear unit ReLU.

3. An electronic device, characterized in that: include: processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

5. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method according to any one of claims 1 to 2.

6. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 2 is implemented.

Images