A method and device for generating vehicle traffic trajectory based on diffusion generation model

Through the diffusion generation model, the generation of real-distributed traffic trajectories is solved, and the problem of generating real and complete driving scenario data in the existing technology is solved, the decision-making algorithm performance and safety of the autonomous driving system is improved, and the cost and Manpower demand.

CN117237475BActive Publication Date: 2024-11-01UNIV OF SCI & TECH BEIJING
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202311024393.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-14
Publication Date
2024-11-01
Estimated Expiration
2043-08-14

AI Technical Summary

Technical Problem

The prior art is difficult to generate real, complete and long-term driving scenario data, resulting in limited performance and safety of decision-making algorithms of autonomous driving systems, and manual design methods require a lot of manpower and domain knowledge.

Method used

Using a vehicle traffic trajectory generation method based on a diffusion generation model, a traffic trajectory close to the real distribution can be generated from the real world data set through a noise prediction network, which can directly generate long-term driving scenario data, reduce time and cost, and improve sample diversity. .

Benefits of technology

The trajectory data generated by this method can be used as an effective data supplement, improving the performance of autonomous driving decision algorithms and the reliability and safety of the system, and does not require artificial design rules and conform to real-world data distribution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117237475B_ABST
    Figure CN117237475B_ABST
Patent Text Reader

Abstract

The present invention discloses a vehicle traffic trajectory generation method and device based on a diffusion generation model, which relates to the fields of traffic scene generation technology and autonomous driving technology. It includes: for any new trackless map scene where the trajectory is to be generated, pure noise is obtained according to the number of trajectories required to be generated for the map scene and the time span required for the trajectory, and the pure noise is used as the initial center position and initial speed; the initial center position and initial speed are input into a trained noise prediction network; according to the initial center position, initial speed and noise prediction network, the predicted center position noise and relative position noise are obtained, and then the vehicle traffic trajectory of the map scene is obtained. The present invention can generate longer and complete driving trajectories that are close to the real distribution in different scenes, enrich the existing data sets, provide data support for autonomous driving trajectory prediction model training and large-scale autonomous driving system evaluation, and improve the reliability and accuracy of autonomous driving decision algorithms.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of traffic scene generation technology and autonomous driving technology, and in particular to a vehicle traffic trajectory generation method and device based on a diffusion generation model. Background Art

[0002] Autonomous driving is a technology that uses advanced computer vision, sensors, and artificial intelligence technologies to enable vehicles to autonomously perceive and interpret the environment and safely perform driving tasks without human driver intervention. It includes perception tasks involving sensor technologies such as computer vision and radar, decision-making tasks such as trajectory prediction, path planning and obstacle avoidance, and execution tasks for control operations. The goal of autonomous driving technology is to improve traffic efficiency and alleviate traffic congestion, improve safety, reduce traffic accidents, and provide users with more convenient and efficient travel methods. One of the biggest obstacles to deploying autonomous driving in the real world is to ensure that vehicles controlled by decision-making algorithms operate safely and reliably in various traffic scenarios. Obtaining more diverse and comprehensive driving data in different scenarios with a longer time span has the potential to evaluate autonomous driving systems on a large scale, enabling rapid development and safe deployment.

[0003] The current mainstream trajectory dataset collection method generally requires specific data collection vehicles to convert the data collected by sensors into vehicle trajectories through target detection technology in the real world at different time periods and locations, and provide training and evaluation data support for decision-making algorithms to ensure the fidelity of driving scenarios. Large-scale trajectory data collection in this way is a time-consuming process. Secondly, since they are collected by data collection vehicles while driving, and these vehicles are often blocked by other traffic participants, the collected trajectories are mostly fragmented and incomplete, and most of the trajectories span a short time. For example, in the Waymo Open Dataset, only 30% of the trajectories last more than 10 seconds, and only 12% of the trajectories cover the entire scene. On the other hand, a solution to cover more traffic scenarios is to manually design test cases or use heuristic methods. However, manually designed scenarios generated in simulators do not reflect the complexity of real-world traffic and road structures well. In addition, it requires a lot of manpower and domain knowledge to design rules for placing vehicles, determine their initial states, and set trigger conditions to ensure the interaction between the self-vehicle and other traffic participants (DJ Fremont, T. Dreossi, S. Ghosh, XY ue, AL Sangiovanni-Vincentelli, and S. A. Seshia, "Scenic: A language for scenario specification and scene generation," in Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, 2019, pp. 63–78.). Currently, most simulators, such as CARLA and SMARTS, have hand-crafted traffic generation rules and maps, but the traffic scenarios available for testing are still far from enough to simulate the complexity of the real world. In order to generate more realistic driving scenarios, researchers have tried to apply data-driven methods to learn relevant data generation models from large-scale real-world datasets.Liang et al. learned the probability distribution of traffic flow, from which new scenes can be sampled, but this method requires encapsulating the scene grammar about traffic rules (M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, "Learning lane graph representations for motion forecasting," in European Conference on Computer Vision, Springer, 2020, pp. 541–556.). Tan et al. proposed a neural autoregressive model SceneGen, which inserts various categories of vehicles into the scene and integrates their size, direction, and speed to model the high-dimensional features of the scene, improving fidelity and scalability. However, this method needs to filter out vehicles that are inappropriately placed outside the road boundary and can only generate single-frame static scenes (S. Tan, K. Wong, S. Wang, S. Manivasagam, M. Ren, and R. Urtasun, "Scenegen: Learning to generate realistic traffic scenes," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 892–901.). Summary of the invention

[0004] The present invention proposes the problem of how to generate a new driving scenario simulating a real scenario, or enrich and supplement an existing scenario, based on context information such as an input map.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In one aspect, the present invention provides a method for generating vehicle traffic trajectories based on a diffusion generation model, the method being implemented by an electronic device, the method comprising:

[0007] S1. For any new trackless map scene where trajectories are to be generated, pure noise is obtained according to the number of trajectories required to be generated in the map scene and the time span required for the trajectories, and the pure noise is used as the initial center position and initial speed.

[0008] S2. Input the initial center position and initial velocity into the trained noise prediction network.

[0009] S3. According to the initial center position, initial speed and noise prediction network, the predicted center position noise and relative position noise are obtained, and then the vehicle traffic trajectory of the map scene is obtained.

[0010] Optionally, the training process of the noise prediction network in S2 includes:

[0011] S21. Obtain trajectory samples in the sample data set, regard the speed of the missing part between two fragmented trajectories in the trajectory sample as a uniform speed for data filling, and add a one-dimensional mask record as the filling data.

[0012] S22, obtain the center position range and trajectory speed range of the padded trajectory sample, perform S-step forward diffusion process to add noise to the center position and trajectory speed, and obtain the center position P with noise s and the noisy trajectory velocity val s .

[0013] S23. Obtaining the relative position noise of the trajectory according to the noisy trajectory speed.

[0014] S24. Learning noise prediction networks by optimizing the weighted sum of negative log-likelihood variational bounds.

[0015] Optionally, in S23, the relative position noise of the track is obtained according to the noisy track speed, as shown in the following formula (1):

[0016]

[0017] in, is the relative position noise at time t, t mid For the intermediate moment, is the velocity noise at the i-th moment.

[0018] Optionally, the weighted sum of negative log-likelihood variation bounds in S24 is as shown in the following formula (2):

[0019]

[0020] Among them, e pos Center position noise, P is the center position, is the predicted center position noise of the sth step, e traj is the relative position noise, traj is the relative position, is the relative position noise of the prediction at step s.

[0021] Optionally, in S1, pure noise is obtained according to the number of trajectories required to be generated according to the map scene and the time span required for the trajectories, and the pure noise is used as the initial center position and initial speed, including:

[0022] According to the number of trajectories N required to be generated in the map scene and the time span T required for the trajectories, N×(T-1) first pure noises that obey the normal distribution are sampled and the first pure noise is used as the initial velocity.

[0023] Sample N second pure noises that obey the normal distribution, and use the second pure noises as the initial center position.

[0024] Optionally, the step of obtaining predicted center position noise and relative position noise according to the initial center position, initial speed and noise prediction network in S3, and then obtaining the vehicle traffic trajectory of the map scene, includes:

[0025] S31, according to the initial center position and initial speed, obtain the center position P of the s-th step with noise s and trajectory velocity val s , P s and val s Input the noise prediction network to get the predicted center position noise of step s and relative position noise

[0026] S32, update the noisy center position P of the s-1th step reverse process s-1 and relative position traj s-1 .

[0027] S33, after S steps of reverse process denoising, the center position P of the trajectory is obtained 0 and relative position traj 0 , for P 0 and traj 0 Sum and get the absolute position pos of the track in the current map scene coordinate system 0 , and then obtain the vehicle traffic trajectory of the map scene.

[0028] Optionally, the noise prediction network in S2 includes an encoder and a decoder.

[0029] The encoder is used to obtain map coding, trajectory coding, neighbor coding and time coding.

[0030] The decoder is used to obtain the predicted central position noise and relative position noise according to the map coding, trajectory coding, neighbor coding and time coding.

[0031] Optionally, map codes, trajectory codes, neighbor codes, and time codes are obtained, including:

[0032] Perform Fourier transform on the noise coefficient to get the time code time s .

[0033] The trajectory velocity val with noise s Input the long short-term memory network LSTM to obtain the trajectory code emb.

[0034] The noisy relative position, trajectory speed and neighbor trajectory speed are input into LSTM to obtain the neighbor encoding nei.

[0035] The map scene is represented as a lane segment with a topological structure. The static and dynamic codes of the lane are obtained through a multi-layer perceptron (MLP). The static and dynamic codes are added together as lane features, and the map code is obtained based on the lane features.

[0036] Optionally, according to the map coding, the trajectory coding, the neighbor coding and the time coding, the predicted center position noise and relative position noise are obtained, including:

[0037] The trajectory encoding and time encoding are updated through cross attention to obtain the neighbor trajectory features, and the neighbor trajectory features are updated through self-attention to obtain new neighbor trajectory features.

[0038] For the center position P with noise s Attention is paid to the lanes in the vicinity to obtain the center position feature. The center position feature and time encoding are decoded through MLP to obtain the predicted center position noise.

[0039] Perform cross attention on all lane features to obtain trajectory features, decode the trajectory features and time encoding through MLP to obtain the predicted relative position noise

[0040] On the other hand, the present invention provides a vehicle traffic trajectory generation device based on a diffusion generation model, which is used to implement a vehicle traffic trajectory generation method based on a diffusion generation model, and the device includes:

[0041] The acquisition module is used to obtain pure noise for any new trackless map scene where the track is to be generated according to the number of tracks required to be generated in the map scene and the time span required for the track, and use the pure noise as the initial center position and initial speed.

[0042] The input module is used to input the initial center position and initial velocity into the trained noise prediction network.

[0043] The output module is used to obtain the predicted center position noise and relative position noise according to the initial center position, initial speed and noise prediction network, and then obtain the vehicle traffic trajectory of the map scene.

[0044] Optionally, the input module is further configured to:

[0045] S21. Obtain trajectory samples in the sample data set, regard the speed of the missing part between two fragmented trajectories in the trajectory sample as a uniform speed for data filling, and add a one-dimensional mask record as the filling data.

[0046] S22, obtain the center position range and trajectory speed range of the padded trajectory sample, perform S-step forward diffusion process to add noise to the center position and trajectory speed, and obtain the center position P with noise s and the noisy trajectory velocity val s .

[0047] S23. Obtaining the relative position noise of the trajectory according to the noisy trajectory speed.

[0048] S24. Learning noise prediction networks by optimizing the weighted sum of negative log-likelihood variational bounds.

[0049] Optionally, the relative position noise of the track is obtained according to the noisy track speed, as shown in the following formula (1):

[0050]

[0051] in, is the relative position noise at time t, t mid For the intermediate moment, is the velocity noise at the i-th moment.

[0052] Optionally, the weighted sum of negative log-likelihood variation bounds is as shown in equation (2):

[0053]

[0054] Among them, e pos Center position noise, P is the center position, is the predicted center position noise of the sth step, e traj is the relative position noise, traj is the relative position, is the relative position noise of the prediction at step s.

[0055] Optionally, the acquisition module is further used to:

[0056] According to the number of trajectories N required to be generated in the map scene and the time span T required for the trajectories, N×(T-1) first pure noises that obey the normal distribution are sampled and the first pure noise is used as the initial velocity.

[0057] Sample N second pure noises that obey the normal distribution, and use the second pure noises as the initial center position.

[0058] Optionally, the output module is further configured to:

[0059] S31, according to the initial center position and initial speed, obtain the center position Ps and trajectory speed val with noise in the sth step s , Ps and val s Input the noise prediction network to get the predicted center position noise of step s and relative position noise

[0060] S32, update the noisy center position Ps of the s-1th step reverse process -1 and relative position trajs -1 .

[0061] S33, after S steps of reverse process denoising, the center position P of the trajectory is obtained 0 and relative position traj 0 , for P 0 and traj 0 Sum and get the absolute position pos of the track in the current map scene coordinate system 0 , and then obtain the vehicle traffic trajectory of the map scene.

[0062] Optionally, the noise prediction network includes an encoder and a decoder.

[0063] The encoder is used to obtain map coding, trajectory coding, neighbor coding and time coding.

[0064] The decoder is used to obtain the predicted central position noise and relative position noise according to the map coding, trajectory coding, neighbor coding and time coding.

[0065] Optionally, the input module is further configured to:

[0066] Perform Fourier transform on the noise coefficient to get the time code time s .

[0067] The trajectory velocity val with noise s Input the long short-term memory network LSTM to obtain the trajectory code emb.

[0068] The noisy relative position, trajectory speed and neighbor trajectory speed are input into LSTM to obtain the neighbor encoding nei.

[0069] The map scene is represented as a lane segment with a topological structure. The static and dynamic codes of the lane are obtained through a multi-layer perceptron (MLP). The static and dynamic codes are added together as lane features, and the map code is obtained based on the lane features.

[0070] Optionally, the input module is further configured to:

[0071] The trajectory encoding and time encoding are updated through cross attention to obtain the neighbor trajectory features, and the neighbor trajectory features are updated through self-attention to obtain new neighbor trajectory features.

[0072] For the center position P with noise s Attention is paid to the lanes in the vicinity to obtain the center position feature. The center position feature and time encoding are decoded through MLP to obtain the predicted center position noise.

[0073] Perform cross attention on all lane features to obtain trajectory features, decode the trajectory features and time encoding through MLP to obtain the predicted relative position noise

[0074] On the one hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned vehicle traffic trajectory generation method based on the diffusion generation model.

[0075] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned vehicle traffic trajectory generation method based on the diffusion generation model.

[0076] Compared with the prior art, the above technical solution has at least the following beneficial effects:

[0077] The above scheme can generate a traffic trajectory generation model that is close to the real distribution by learning from real-world datasets through data-driven learning in the form of a diffusion model. Compared with the current method of using sensors to collect large-scale trajectory datasets, this method is low in time and cost, and can generate long-term and complete driving scene data to supplement the existing dataset. Compared with manually designed use cases, this method does not contain artificial constraints to make it more consistent with the data distribution of the real world, and does not require a lot of manpower and domain knowledge to design rules for placing vehicles. Compared with the method of learning probability distribution to generate traffic snapshots, this method can directly generate driving trajectories for a period of time, and with the help of a diffusion model, the training is stable and the diversity of samples is improved. The trajectories generated by this method can be used as an effective data supplement to train effective learning-based methods in the field of autonomous driving decision-making. This is of great significance for improving the performance of decision-making algorithms and improving the reliability and safety of AD systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 It is a flow chart of a method for generating vehicle traffic trajectories based on a diffusion generation model provided by an embodiment of the present invention;

[0080] Figure 2 is a schematic diagram of the training phase of the method for generating driving scene trajectories based on a diffusion model provided by an embodiment of the present invention;

[0081] Figure 3 Schematic diagram of the noise prediction network structure of the method for generating driving scene trajectories based on a diffusion model provided by an embodiment of the present invention;

[0082] Figure 4 is a schematic diagram of a trajectory generation phase of a method for generating driving scene trajectories based on a diffusion model provided by an embodiment of the present invention;

[0083] Figure 5 It is a block diagram of a vehicle traffic trajectory generation device based on a diffusion generation model provided by an embodiment of the present invention;

[0084] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] like Figure 1 As shown, an embodiment of the present invention provides a method for generating vehicle traffic trajectories based on a diffusion generation model, which can be implemented by an electronic device. Figure 1 The flow chart of the vehicle traffic trajectory generation method based on the diffusion generation model is shown, and the processing flow of the method may include the following steps:

[0087] S1. For any new trackless map scene where trajectories are to be generated, pure noise is obtained according to the number of trajectories required to be generated in the map scene and the time span required for the trajectories, and the pure noise is used as the initial center position and initial speed.

[0088] Optionally, in S1, pure noise is obtained according to the number of trajectories required to be generated according to the map scene and the time span required for the trajectories, and the pure noise is used as the initial center position and initial speed, including:

[0089] According to the number of trajectories N required to be generated in the map scene and the time span T required for the trajectories, N×(T-1) first pure noises that obey the normal distribution are sampled and the first pure noise is used as the initial velocity.

[0090] Sample N second pure noises that obey the normal distribution, and use the second pure noises as the initial center position.

[0091] In a feasible implementation, in the trajectory generation stage, the required amount of pure noise is sampled from a normal distribution with a standard deviation of the data range and a mean of zero, and is used as the initial center position and the initial relative position, which are input into the trained noise prediction network. After an S-step reverse diffusion process, the scene trajectory is restored.

[0092] The present invention divides each scene trajectory generation task into generating trajectory intermediate time t mid The center position of , and generate the relative trajectory with respect to the center position.

[0093] S2. Input the initial center position and initial velocity into the trained noise prediction network.

[0094] In a feasible implementation, Figure 2 As shown in the figure, during the training phase, according to the center position range and speed range of the trajectory, the center position and speed are noised by S-step forward diffusion process, and the relative position noise of the current step forward process is obtained by integrating the speed noise. By optimizing the weighted sum of negative log-likelihood variational bound loss, a noise prediction network that can predict the center position noise and relative position noise is learned.

[0095] Specifically, the training process of the noise prediction network in S2 includes S21-S23:

[0096] S21. Obtain trajectory samples in the sample data set, regard the speed of the missing part between two fragmented trajectories in the trajectory sample as a uniform speed for data filling, and add a one-dimensional mask record as the filling data.

[0097] In a feasible implementation, the statistical data range of the training sample data set is calculated, the sample scene coordinate system adopts the sampling vehicle coordinate system, and the middle time t of each trajectory of the sample scene is taken midThe position of is taken as the center position of the trajectory, and the relative trajectory position is expressed as the distance relative to the center position. The missing part between two fragmented trajectories of the same trajectory is regarded as uniform motion and filled. The center position and speed range of all trajectories in the training set are counted, and the e added to the center position and trajectory speed is pos and e va The standard deviation is the center position range σ p and speed range σ val , to ensure that the data results of the entire forward process have a close distribution range.

[0098] Furthermore, the scene coordinate system adopts the sampling vehicle coordinate system, and the intermediate time frame t mid The trajectory coordinates are taken as the center position P 0 For the missing parts of two fragmented trajectories of the same trajectory, the trajectory velocity val 0 According to the starting and ending positions of the missing frames, they are regarded as uniform motion for filling, and a one-dimensional mask is added to record whether it is obtained by filling. The maximum value of the two norms of all the center positions and velocities in the statistical sample data set is taken as the center position range σ p and speed range σ val .

[0099] S22, obtain the center position range and trajectory speed range of the padded trajectory sample, perform S-step forward diffusion process to add noise to the center position and trajectory speed, and obtain the center position P with noise s and the noisy trajectory velocity val s .

[0100] S23. Obtaining the relative position noise of the trajectory according to the noisy trajectory speed.

[0101] In a feasible implementation, the total number of noise addition steps in the parameter table is set to S, and the noise addition coefficient β s It increases linearly from 0.0005 to 0.05. According to the parameter table, the attenuation coefficient α in the sth step s =1-β s Therefore, the attenuation coefficient and noise coefficient after s steps of forward process are and

[0102] Furthermore, the noise coefficient is added in the current s step Bottom: The center position of the noise is The trajectory velocity with noise at time t is The mask is not affected by the forward noise process. pos ,e val The mean is 0, and the standard deviation is the center position range σ p and speed range σval Normal distribution. 0 ,val 0 is the real data without noise. mid The relative position noise is fixed to 0 and does not need to be learned. Integrate the velocity and velocity noise to obtain the relative position traj after adding noise s and trajectory relative position noise e traj , the relative position of the trajectory at time t and trajectory relative position noise The integral formula is as follows:

[0103]

[0104]

[0105] The simplified integral formula is as follows: mid time Is 0 and no prediction is needed:

[0106]

[0107]

[0108] in, is the relative position noise at time t, t mid For the intermediate moment, is the velocity noise at the i-th moment.

[0109] Furthermore, for each parameter update, s is randomly selected from 1 to S steps to determine the current step, and the noisy data P is obtained by the forward process. s ,val s . s ,val s Input the noise prediction network model to get the current s-step noise prediction value

[0110] S24. Learning noise prediction networks by optimizing the weighted sum of negative log-likelihood variational bounds.

[0111] In a feasible implementation, during the forward noise addition process, e is added to the center position and the relative position. pos and e traj As a label, by weighting so that the coefficients of the forward process of different steps of the variational bound of the negative log-likelihood are all 1, the loss function is obtained, as shown in the following formula (5), and the noise prediction network is optimized by optimizing the variational bound L(θ) of the weighted sum of negative log-likelihood.

[0112]

[0113] Among them, e pos Center position noise, P is the center position, is the predicted center position noise of the sth step, e traj is the relative position noise, traj is the relative position, is the relative position noise of the prediction at step s.

[0114] Optionally, the noise prediction network in S2 includes an encoder and a decoder.

[0115] In a feasible implementation, Figure 3 As shown in the figure, the noise prediction network model consists of two parts: an encoder and a decoder. The encoder extracts the encoding of the noisy trajectory and the neighbor trajectory through a temporal network, and extracts the dynamic and static features of the map through a graph convolutional network; the decoder obtains the neighbor lane and neighbor trajectory features through a time encoding that represents the current number of noisy steps and the trajectory encoding through cross-attention updates, and obtains higher-level neighbor trajectory features through a self-attention layer; finally, the center position noise is estimated by cross-attention to a large range of lanes near the center position, and the relative position noise is estimated by cross-attention to a small range of lanes near the entire trajectory.

[0116] The encoder is used to obtain map coding, trajectory coding, neighbor coding and time coding.

[0117] Specifically, the noise coefficient Using Fourier transform as time encoding s .

[0118] The trajectory encoder encodes the trajectory after the diffusion of the sth step, and the trajectory speed after noise is interacted in time sequence through LSTM to obtain the trajectory feature emb.

[0119] For the neighbor trajectories of a trajectory, the relative position, trajectory speed and neighbor trajectory speed at each time are interacted in time series through LSTM (Long Short-Term Memory) as the neighbor feature nei.

[0120] Encode the scene context and encode the static lane features in the map, including position, shape, and semantic information, through MLP (Multilayer Perceptron) to obtain static features. The trajectory features near the lane, the difference between the lane position and the center position of the trajectory are used as the dynamic features of the lane through MLP. The sum of the static features and the dynamic features is used as the lane feature. According to the connection relationship between lanes, the lanes are interacted through GCN (Graph Convolutional Networks) to obtain updated lane features.

[0121] In one feasible implementation, the map encoder of the noise prediction network represents the map as lane segments with topological structure. For each lane in the map, the center position, shape, and lane semantic features provided in the dataset are used to obtain the static encoding map of the lane through MLP. sta ∈R dim All m′ neighbor trajectories within 15m of lane i are taken as its neighbors, and the relative distance and trajectory features of the center position of the neighbor trajectory and the lane are connected in series, and MLP is used as the lane dynamic encoding map dyn ∈R m′×dim The static code and m′ dynamic codes are added as the lane feature of the current scene. According to the connection relationship between the map lanes, the features between the maps are interacted through the graph network to obtain the lane code map∈R after interaction dim .

[0122] In this embodiment, the lane graph network for updating the lane adopts the lane graph convolution used in LaneGCN (Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.), using three layers of lane graph convolution, and each layer of graph convolution connects the lanes with a distance of 1, 2, and 4 before and after and the left and right adjacent lanes.

[0123] The decoder is used to obtain the predicted central position noise and relative position noise according to the map coding, trajectory coding, neighbor coding and time coding.

[0124] Specifically, the trajectory encoding and time encoding are updated through cross-attention to obtain neighbor trajectory features, and the neighbor trajectory features are updated through self-attention to obtain new neighbor trajectory features.

[0125] For the center position P with noise s Attention is paid to the lanes in the vicinity to obtain the center position feature. The center position feature and time encoding are decoded through MLP to obtain the predicted center position noise.

[0126] Perform cross attention on all lane features to obtain trajectory features, decode the trajectory features and time encoding through MLP to obtain the predicted relative position noise

[0127] In one feasible implementation, the model decoder is based on the time code time representing the current noise coefficient s and trajectory features with noise, cross-attention to other elements of the current scene, and achieve decoding based on the current scene.

[0128] Specifically, the decoder of the noise prediction network encodes the time s Connect with the trajectory encoding emb and use it as the query of the decoder through the fully connected layer. The relative position of the center position of each trajectory and the surrounding lanes is used as the relative position encoding of the map through MLP. The relative position encoding and the map feature map are connected as the key to perform cross-attention to obtain the surrounding map information. Use the updated trajectory encoding as the query, connect the neighbor features and the relative distance between the center position of the neighbor trajectory, generate a key through MLP, and perform cross-attention with the neighbor features. Perform self-attention between the updated trajectory features to obtain higher-level neighbor information.

[0129] Furthermore, we perform cross-attention on lane segments and trajectories with a larger range of noisy center positions to obtain center position features. After connecting with the time code, we decode the center position noise of the current s noise adding step through MLP decoding. Cross attention is performed on the lane segments near the trajectory to obtain trajectory features. After connecting with the time code, the trajectory position noise of the current s noise adding step is obtained through MLP decoding.

[0130] In this embodiment, the trajectory encoding and time encoding are connected as the query, and the difference between the lane center point and the position center point of each trajectory is used as the relative position encoding dist of the trajectory and lane through MLP. am . Encode the relative position dist am The lane feature map is connected as a key to perform cross attention on the lanes. The difference in the center position between the trajectories is used as the relative position encoding dist of the trajectory and the neighboring trajectory through MLP. aa′ , dist aa ′ and the corresponding neighbor feature nei are connected as query, and cross-attention is performed on the neighbor features. The trajectory feature emb updated by cross-attention passes through the self-attention layer to obtain higher-level neighbor information.

[0131] In this embodiment, the noisy trajectory feature emb after cross-attention and self-attention update is cross-attentioned with the lane feature within 50m around the center of the trajectory to obtain the center position feature. After connecting with the time code, it is decoded by MLP to obtain the prediction value of the center position plus noise in the current t-step forward process. Perform cross attention on all lane features within 10m of the entire trajectory to obtain trajectory features, connect them with the time code, and then decode them through MLP to obtain the prediction value of the current t-step forward process trajectory plus noise.

[0132] In this embodiment, the data set can use the Argoverse 1 data set (Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al Argoverse: 3D tracking and forecasting with rich maps. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.), which collects 323,557 data sequences with a sampling rate of 10 Hz and a sequence length of 5 s from Miami and Pittsburgh. The map is divided into lane segments with topological connections, and each lane segment contains information such as the center line and curve, control, and overlap.

[0133] S3. According to the initial center position, initial speed and noise prediction network, the predicted center position noise and relative position noise are obtained, and then the vehicle traffic trajectory of the map scene is obtained.

[0134] Alternatively, if Figure 4 As shown, the above step S3 may include the following steps S31-S33:

[0135] S31, according to the initial center position and initial speed, obtain the center position P of the s-th step with noise s and trajectory velocity val s , P s and val s Input the noise prediction network to get the predicted center position noise of step s and relative position noise

[0136] Specifically, in the trajectory generation stage, the trained noise prediction network generates N trajectories with the required time span T in any new trackless map scene, specifically including: the required number of samples follows the normal distribution The pure noise of is taken as the velocity val of the initial S step of the reverse process s ∈R N×(T-1)×2, and the corresponding quantity follows a normal distribution The pure noise is used as the initial center position P S ∈R N×2 .

[0137] S32, update the noisy center position P of the s-1th step reverse process s-1 and relative position traj s-1 .

[0138] In a feasible implementation, the relative position of the reverse process in step s-1 is updated Central Location where α s , β s is the diffusion process parameter related to s, In each step of the reverse process, independent sampling is performed, with a mean of 0 and a standard deviation of the center position range σ p and speed range σ val The normal distribution of .

[0139] Furthermore, the trajectory velocity val at step s-1 is s-1 By traj s-1 It is obtained by taking the difference between adjacent moments.

[0140] S33, after S steps of reverse process denoising, the center position P of the trajectory is obtained 0 and relative position traj 0 , for P 0 and traj 0 Sum and get the absolute position pos of the track in the current map scene coordinate system 0 , and then obtain the vehicle traffic trajectory of the map scene.

[0141] In an embodiment of the present invention, a traffic trajectory generation model close to the real distribution can be generated by learning from a real-world dataset through data-driven learning in the form of a diffusion model. Compared with the current method of using sensors to collect large-scale trajectory datasets, this method is low in time and cost, and can generate long-term and complete driving scene data, thereby supplementing the existing dataset. Compared with manually designed use cases, this method does not contain artificial constraints to make it more consistent with the data distribution of the real world, and does not require a lot of manpower and domain knowledge to design rules for placing vehicles. Compared with the method of learning probability distribution to generate traffic snapshots, this method can directly generate driving trajectories for a period of time, and with the help of a diffusion model, the training is stable and the diversity of samples is improved. The trajectories generated by this method can be used as an effective data supplement to train effective learning-based methods in the field of autonomous driving decision-making. This is of great significance for improving the performance of decision-making algorithms and improving the reliability and safety of AD systems.

[0142] like Figure 5 As shown, an embodiment of the present invention provides a vehicle traffic trajectory generation device 500 based on a diffusion generation model. The device 500 is applied to implement a vehicle traffic trajectory generation method based on a diffusion generation model. The device 500 includes:

[0143] The acquisition module 510 is used to obtain pure noise according to the number of trajectories required to be generated in the map scene and the time span required for the trajectories for any new trackless map scene where the trajectories are to be generated, and use the pure noise as the initial center position and initial speed.

[0144] The input module 520 is used to input the initial center position and the initial velocity into the trained noise prediction network.

[0145] The output module 530 is used to obtain the predicted center position noise and relative position noise according to the initial center position, the initial speed and the noise prediction network, and then obtain the vehicle traffic trajectory of the map scene.

[0146] Optionally, the input module 520 is further configured to:

[0147] S21. Obtain trajectory samples in the sample data set, regard the speed of the missing part between two fragmented trajectories in the trajectory sample as a uniform speed for data filling, and add a one-dimensional mask record as the filling data.

[0148] S22, obtain the center position range and trajectory speed range of the padded trajectory sample, perform S-step forward diffusion process to add noise to the center position and trajectory speed, and obtain the center position P with noise s and the noisy trajectory velocity val s .

[0149] S23. Obtaining the relative position noise of the trajectory according to the noisy trajectory speed.

[0150] S24. Learning noise prediction networks by optimizing the weighted sum of negative log-likelihood variational bounds.

[0151] Optionally, the relative position noise of the track is obtained according to the noisy track speed, as shown in the following formula (1):

[0152]

[0153] in, is the relative position noise at time t, t mid For the intermediate moment, is the velocity noise at the i-th moment.

[0154] Optionally, the weighted sum of negative log-likelihood variation bounds is as shown in equation (2):

[0155]

[0156] Among them, e pos Center position noise, P is the center position, is the predicted center position noise of the sth step, e traj is the relative position noise, traj is the relative position, is the relative position noise of the prediction at step s.

[0157] Optionally, the acquisition module 510 is further configured to:

[0158] According to the number of trajectories N required to be generated in the map scene and the time span T required for the trajectories, N×(T-1) first pure noises that obey the normal distribution are sampled and the first pure noise is used as the initial velocity.

[0159] Sample N second pure noises that obey the normal distribution, and use the second pure noises as the initial center position.

[0160] Optionally, the output module 530 is further configured to:

[0161] S31, according to the initial center position and initial speed, obtain the center position P of the s-th step with noise s and trajectory velocity val s , P s and val s Input the noise prediction network to get the predicted center position noise of step s and relative position noise

[0162] S32, update the noisy center position P of the s-1th step reverse process s-1and relative position traj s-1 .

[0163] S33, after S steps of reverse process denoising, the center position P of the trajectory is obtained 0 and relative position traj 0 , for P 0 and traj 0 Sum and get the absolute position pos of the track in the current map scene coordinate system 0 , and then obtain the vehicle traffic trajectory of the map scene.

[0164] Optionally, the noise prediction network includes an encoder and a decoder.

[0165] The encoder is used to obtain map coding, trajectory coding, neighbor coding and time coding.

[0166] The decoder is used to obtain the predicted central position noise and relative position noise according to the map coding, trajectory coding, neighbor coding and time coding.

[0167] Optionally, the input module 520 is further configured to:

[0168] Perform Fourier transform on the noise coefficient to get the time code time s .

[0169] The trajectory velocity val with noise s Input the long short-term memory network LSTM to obtain the trajectory code emb.

[0170] The noisy relative position, trajectory speed and neighbor trajectory speed are input into LSTM to obtain the neighbor encoding nei.

[0171] The map scene is represented as a lane segment with a topological structure. The static and dynamic codes of the lane are obtained through a multi-layer perceptron (MLP). The static and dynamic codes are added together as lane features, and the map code is obtained based on the lane features.

[0172] Optionally, the input module 520 is further configured to:

[0173] The trajectory encoding and time encoding are updated through cross attention to obtain the neighbor trajectory features, and the neighbor trajectory features are updated through self-attention to obtain new neighbor trajectory features.

[0174] For the center position P with noise s Attention is paid to the lanes in the vicinity to obtain the center position feature. The center position feature and time encoding are decoded through MLP to obtain the predicted center position noise.

[0175] Perform cross attention on all lane features to obtain trajectory features, decode the trajectory features and time encoding through MLP to obtain the predicted relative position noise

[0176] In an embodiment of the present invention, a traffic trajectory generation model close to the real distribution can be generated by learning from a real-world dataset through data-driven learning in the form of a diffusion model. Compared with the current method of using sensors to collect large-scale trajectory datasets, this method is low in time and cost, and can generate long-term and complete driving scene data, thereby supplementing the existing dataset. Compared with manually designed use cases, this method does not contain artificial constraints to make it more consistent with the data distribution of the real world, and does not require a lot of manpower and domain knowledge to design rules for placing vehicles. Compared with the method of learning probability distribution to generate traffic snapshots, this method can directly generate driving trajectories for a period of time, and with the help of a diffusion model, the training is stable and the diversity of samples is improved. The trajectories generated by this method can be used as an effective data supplement to train effective learning-based methods in the field of autonomous driving decision-making. This is of great significance for improving the performance of decision-making algorithms and improving the reliability and safety of AD systems.

[0177] Figure 6 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 and one or more memories 602, wherein the memory 602 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 601 to implement the following vehicle traffic trajectory generation method based on the diffusion generation model:

[0178] S1. For any new trackless map scene where trajectories are to be generated, pure noise is obtained according to the number of trajectories required to be generated in the map scene and the time span required for the trajectories, and the pure noise is used as the initial center position and initial speed.

[0179] S2. Input the initial center position and initial velocity into the trained noise prediction network.

[0180] S3. According to the initial center position, initial speed and noise prediction network, the predicted center position noise and relative position noise are obtained, and then the vehicle traffic trajectory of the map scene is obtained.

[0181] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned vehicle traffic trajectory generation method based on the diffusion generation model. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0182] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle traffic trajectory generation method based on a diffusion generation model, characterized in that: The method comprises: S1. For any new trackless map scene where tracks are to be generated, pure noise is obtained according to the number of tracks required to be generated in the map scene and the time span required for the tracks, and the pure noise is used as the initial center position and initial speed; S2, inputting the initial center position and initial velocity into a trained noise prediction network; S3, obtaining predicted center position noise and relative position noise according to the initial center position, initial speed and noise prediction network, and then obtaining the vehicle traffic trajectory of the map scene; The noise prediction network in S2 includes an encoder and a decoder; The encoder is used to obtain map code, trajectory code, neighbor code and time code: The decoder is used to obtain predicted central position noise and relative position noise according to the map code, trajectory code, neighbor code and time code; The obtaining of the map code, the trajectory code, the neighbor code and the time code comprises: Perform Fourier transform on the noise coefficient to get the time code time s ; The trajectory velocity val with noise s Input the long short-term memory network LSTM to obtain the trajectory code emb; The relative position, trajectory speed and neighbor trajectory speed with noise are input into LSTM to obtain the neighbor encoding nei: The map scene is represented as a lane segment with a topological structure, a static code and a dynamic code of the lane are obtained by a multi-layer perceptron MLP, the static code and the dynamic code are added as lane features, and a map code is obtained according to the lane features; The method of obtaining predicted center position noise and relative position noise according to the map coding, trajectory coding, neighbor coding and time coding includes: The trajectory code and the time code are updated through cross attention to obtain neighbor trajectory features, and the neighbor trajectory features are updated through self-attention to obtain new neighbor trajectory features; For the center position P with noise s Attention is paid to the intersection of lanes in the nearby range to obtain the center position feature, and the center position feature and time encoding are decoded through MLP to obtain the predicted center position noise Perform cross attention on all lane features to obtain trajectory features, decode the trajectory features and time encoding through MLP to obtain the predicted relative position noise 2. The method according to claim 1, characterized in that The training process of the noise prediction network in S2 includes: S21, obtaining a trajectory sample in the sample data set, treating the speed of the missing part between two fragmented trajectories in the trajectory sample as a uniform speed for data filling, and adding a one-dimensional mask record as the filling data; S22, obtain the center position range and trajectory speed range of the padded trajectory sample, perform s-step forward diffusion process to add noise to the center position and trajectory speed, and obtain the center position P with noise s and the noisy trajectory velocity val s ; S23, obtaining the relative position noise of the track according to the noisy track speed; S24. Learning noise prediction networks by optimizing the weighted sum of negative log-likelihood variational bounds.

3. The method according to claim 2, characterized in that In S23, the relative position noise of the track is obtained according to the noisy track speed, as shown in the following formula (1): in, is the relative position noise at time t, t mid For the intermediate moment, is the velocity noise at the i-th moment.

4. The method according to claim 2, characterized in that: The weighted sum of negative log-likelihood variation bounds in S24 is as shown in the following formula (2): Among them, e pos Center position noise, P is the center position, is the predicted center position noise of the sth step, e traj is the relative position noise, traj is the relative position, is the relative position noise of the prediction at step s.

5. The method according to claim 1, characterized in that The step S1 obtains pure noise according to the number of trajectories required to be generated in the map scene and the time span required for the trajectories, and uses the pure noise as the initial center position and initial speed, including: According to the number N of trajectories required to be generated in the map scene and the time span T required for the trajectories, N×(T-1) first pure noises obeying a normal distribution are sampled, and the first pure noises are used as the initial velocity; N second pure noises obeying normal distribution are sampled, and the second pure noises are used as initial center positions.

6. The method according to claim 1, characterized in that The step S3 obtains predicted center position noise and relative position noise according to the initial center position, initial speed and noise prediction network, and further obtains the vehicle traffic trajectory of the map scene, including: S31, according to the initial center position and initial speed, obtain the center position P of the s-th step with noise s and trajectory velocity val s , the P s and val s Input the noise prediction network to get the predicted center position noise of step s and relative position noise S32, update the noisy center position P of the s-1th step reverse process s-1 and relative position traj s-1 ; S33, after S steps of reverse process denoising, the center position P of the trajectory is obtained 0 and relative position traj 0 , for the P 0 and traj 0 Sum and get the absolute position pos of the track in the current map scene coordinate system 0 , and then obtain the vehicle traffic trajectory of the map scene.

7. A vehicle traffic trajectory generation device based on a diffusion generation model, characterized in that: The device comprises: An acquisition module, for obtaining pure noise according to the number of trajectories required to be generated in any new map scene without trajectories for which trajectories are to be generated and the time span required for the trajectories, and using the pure noise as the initial center position and initial speed; An input module, used for inputting the initial center position and initial velocity into a trained noise prediction network; An output module, used to obtain predicted center position noise and relative position noise according to the initial center position, initial speed and noise prediction network, and further obtain the vehicle traffic trajectory of the map scene; The noise prediction network includes an encoder and a decoder; Wherein, the encoder is used to obtain map coding, trajectory coding, neighbor coding and time coding; The decoder is used to obtain predicted central position noise and relative position noise according to the map code, trajectory code, neighbor code and time code; The obtaining of the map code, the trajectory code, the neighbor code and the time code comprises: Perform Fourier transform on the noise coefficient to get the time code time s ; The trajectory velocity val with noise s Input the long short-term memory network LSTM to obtain the trajectory code emb; The relative position, trajectory speed and neighbor trajectory speed with noise are input into LSTM to obtain the neighbor code nei; The map scene is represented as a lane segment with a topological structure, a static code and a dynamic code of the lane are obtained by a multi-layer perceptron MLP, the static code and the dynamic code are added as lane features, and a map code is obtained according to the lane features; The method of obtaining predicted center position noise and relative position noise according to the map coding, trajectory coding, neighbor coding and time coding includes: The trajectory code and the time code are updated through cross attention to obtain neighbor trajectory features, and the neighbor trajectory features are updated through self-attention to obtain new neighbor trajectory features; For the center position P with noise s Attention is paid to the intersection of lanes in the nearby range to obtain the center position feature, and the center position feature and time encoding are decoded through MLP to obtain the predicted center position noise Perform cross attention on all lane features to obtain trajectory features, decode the trajectory features and time encoding through MLP to obtain the predicted relative position noise

Citation Information

Cited By

  • Large-scale trajectory generation modeling algorithm and platform system supporting multi-scene migration

    CN122452910A