Trajectory prediction method, device, electronic device and storage medium

Abstract

The present disclosure provides a trajectory prediction method, device, electronic device and storage medium. In the trajectory prediction method provided by the embodiment of the present disclosure, after obtaining environmental features through the sensor data of the vehicle and the existing map data, the environmental features are first converted into an instance tag set, which includes not only the vehicle tag but also the tags of the agents representing other dynamic entities in the current scene, and then the future trajectory data of the vehicle is generated through the instance tag set and the vehicle trajectory latent space learned by the generative model. The present disclosure realizes end-to-end processing from sensor data to trajectory generation, reduces the data conversion overhead between modules, improves computing efficiency, and adopts generative modeling to improve accuracy, real-time, security and reliability.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a trajectory prediction method, device, electronic device and storage medium. Background Art

[0002] In the field of autonomous driving, end-to-end autonomous driving systems are gradually becoming a hot topic of research. Such systems are designed to generate driving decisions directly from raw sensor data (such as cameras, radars, lidars, etc.) without the need for complex manual feature engineering. At present, end-to-end autonomous driving systems usually adopt a staged processing approach, with perception, prediction, and planning processed as independent models. However, due to the limited interaction and synergy between these independent modules such as perception, prediction, and planning, and the inability to fully predict dynamic changes in complex scenarios such as urban traffic, the execution efficiency and accuracy of trajectory prediction and its fit with actual driving habits are all insufficient. Summary of the invention

[0003] In view of this, the present disclosure provides a trajectory prediction method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a trajectory prediction method is provided, the method comprising:

[0005] Use the vehicle's sensor data and existing map data to obtain the environmental characteristics of the current scene in which the vehicle is located;

[0006] Converting the environmental features of the current scene into an instance tag set, wherein the instance tag set includes multiple instance tags, each of which represents information of an entity in the current scene, the entities in the current scene include the ego vehicle and at least one agent in the current scene, and each of which represents another dynamic entity in the current scene except the ego vehicle;

[0007] The future trajectory data of the ego vehicle is generated using the instance label set and a previously obtained ego vehicle trajectory latent space, where the ego vehicle trajectory latent space is learned by a pre-trained variational autoencoder based on the historical trajectory data of the ego vehicle.

[0008] In some embodiments of the first aspect of the present disclosure, the environmental features of the current scene include BEV features of the current scene and map information of the current scene; converting the environmental features into a set of instance markers includes: converting the BEV features of the current scene into map markers and proxy markers; generating an ego vehicle marker; forming an instance marker set using the map marker, the proxy marker and the ego vehicle marker; and injecting the map information of the current scene into the instance marker set.

[0009] In some embodiments of the first aspect of the present disclosure, it also includes: updating the instance tag set using the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent, so that each instance tag in the instance tag set contains high-order interaction information between instances, and the high-order interaction information is used to indicate the dynamic relationship and interaction between the ego vehicle and one or more of the other dynamic entities in the current scene.

[0010] In some embodiments of the first aspect of the present disclosure, the use of the historical trajectory data of the vehicle and the historical trajectory data of the at least one agent to update the instance label set includes: using a self-attention mechanism to process the historical trajectory data of the vehicle and the historical trajectory data of the at least one agent to obtain a first attention score for each instance, the first attention score of each instance indicating the degree of trajectory correlation between the instance and other instances; and using the first attention score of each instance to update the instance label set.

[0011] In some embodiments of the first aspect of the present disclosure, the method of updating the instance tag set using the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent includes: using the historical trajectory data of the ego vehicle as a query sequence of the cross-attention mechanism and the historical trajectory data of at least one agent as a key sequence and a value sequence of the cross-attention mechanism, performing calculations of the cross-attention mechanism to obtain a second attention score for each instance, the second attention score of each instance indicating the degree of correlation between the trajectory of the ego vehicle and the trajectory of the agent; and updating the instance tag set using the second attention score of each instance.

[0012] In some embodiments of the first aspect of the present disclosure, the method of generating future trajectory data of the ego-vehicle using the instance label set and the previously obtained ego-vehicle trajectory latent space includes: using a gated recurrent unit to determine a potential state of the ego-vehicle within a predetermined future duration based on the instance label set and the ego-vehicle trajectory latent space; and using a decoder in the variational autoencoder to decode the potential state of the ego-vehicle within the predetermined future duration to obtain trajectory point information of the ego-vehicle within the predetermined future duration.

[0013] In some implementations of the first aspect of the present disclosure, the method further includes: collecting historical trajectory data of the vehicle in real time and using the historical trajectory data of the vehicle to update parameters of the variational autoencoder in real time.

[0014] According to a second aspect of the present disclosure, there is provided a trajectory prediction device, comprising:

[0015] A perception unit, used to obtain the environmental characteristics of the current scene where the vehicle is located by using the sensor data of the vehicle and the existing map data;

[0016] A scene representation unit, configured to convert the environmental features of the current scene into an instance tag set, wherein the instance tag set includes a plurality of instance tags, each of which represents information of an entity in the current scene, the entities in the current scene include the ego vehicle and at least one agent in the current scene, and each of which represents another dynamic entity in the current scene except the ego vehicle;

[0017] A trajectory generation unit is used to generate future trajectory data of the ego vehicle using the instance tag set and a previously obtained ego vehicle trajectory latent space, wherein the ego vehicle trajectory latent space is learned by a pre-trained variational autoencoder based on the historical trajectory data of the ego vehicle.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising: one or more processors and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0019] According to a fourth aspect of the present disclosure, a computer-readable storage medium storing a program is provided, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the above method.

[0020] It can be seen from the above technical solution that after acquiring the environmental features of the current scene, the disclosed embodiment converts the environmental features into an instance tag set, and then obtains the future trajectory data of the vehicle through the instance tag set combined with the variational autoencoder and its pre-learned vehicle trajectory latent space. As a result, the disclosed embodiment realizes end-to-end processing from sensor data to trajectory generation, avoids the complex data conversion and interface docking problems between independent modules in related technologies, reduces the data conversion overhead between modules, and improves computing efficiency; at the same time, the disclosed embodiment adopts generative modeling such as variational autoencoders to realize trajectory prediction, which improves the accuracy, real-time performance, security and reliability of trajectory prediction compared to related technologies that rely on deterministic models and manual feature engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of a flow chart of a trajectory prediction method provided in an embodiment of the present disclosure;

[0023] Figure 2A schematic diagram of a process of converting environmental features into an instance tag set in a trajectory prediction method provided in an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of the structure of a trajectory prediction device provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0027] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0028] As used herein, the words "if," "if," and the like may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0029] First, the relevant technology is described in detail.

[0030] As mentioned earlier, current end-to-end autonomous driving technology usually adopts a phased processing approach, where perception, prediction, and planning are processed as independent modules.

[0031] In related technologies, the trajectory prediction process in end-to-end autonomous driving technology usually includes: first using models such as convolutional neural networks for environmental perception, then using models such as recurrent neural networks for motion prediction, and finally generating trajectories through a planning network.

[0032] The relevant technologies mainly have the following problems:

[0033] 1) The related technology cannot add relevant information of actual driving behavior to trajectory prediction, resulting in the trajectory prediction result not being consistent with actual driving habits.

[0034] 2) Treating perception, prediction, and planning as independent modules, due to the limited interaction and synergy between these independent modules, lack of effective information integration and interaction, and inability to fully predict the dynamic changes of complex scenarios such as urban traffic, the execution efficiency and accuracy of trajectory prediction are insufficient and cannot meet the needs of practical applications.

[0035] In view of this, the embodiments of the present disclosure provide the following trajectory prediction method, device, electronic device and storage medium. After obtaining the environmental features of the current scene, the environmental features are converted into an instance tag set, and then the future trajectory data of the vehicle is obtained by combining the instance tag set with the variational autoencoder and its pre-learned vehicle trajectory latent space. As a result, the embodiments of the present disclosure realize end-to-end processing from sensor data to trajectory generation, avoiding the complex data conversion and interface docking problems between independent modules in related technologies, reducing the data conversion overhead between modules, and improving computing efficiency; at the same time, the embodiments of the present disclosure use generative modeling such as variational autoencoders to realize trajectory prediction, which further improves safety and reliability compared to related technologies that rely on deterministic models and manual feature engineering.

[0036] The disclosed embodiments can be applied to the control of various types of equipment such as multiple wheeled mobile robots, wheeled mobile robots, mobile robots, vehicles, aircraft, ships, and intelligent rail rapid transit systems (ART, Autonomous rail Rapid Transit). The vehicles can be, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans, etc.), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid electric vehicles), and special vehicles (e.g., garbage trucks, sprinkler trucks, etc.). In other words, the "vehicle" in the disclosed embodiments is equivalent to the aforementioned various equipment.

[0037] The embodiments of the present disclosure can be applied to scenes such as urban transportation, highways, ports, mines, farms, and closed parks, and can be applied to many aspects such as travel by car, public transportation, logistics distribution, unmanned transportation, terminal distribution, automated agricultural operations, and automated sanitation. Of course, the embodiments of the present disclosure can also be applied to any other intelligent control scenes involving equipment such as vehicles, and the present disclosure does not limit the application scenes and applicable fields of the embodiments of the present disclosure.

[0038] Exemplarily, the embodiments of the present disclosure may be applied to the following four aspects:

[0039] 1) Urban roads: In complex urban traffic environments, the embodiments of the present disclosure can handle variable traffic flows and diverse traffic participants and provide accurate trajectory prediction.

[0040] 2) Highways: On highways, the disclosed embodiments can efficiently, quickly and accurately predict the trajectory of vehicles traveling at high speeds.

[0041] 3) Closed areas such as ports and parks: In closed or semi-closed environments, applying the embodiments of the present disclosure to autonomous driving vehicles can achieve efficient, fast and accurate navigation of autonomous driving vehicles, improve efficiency and reduce labor costs.

[0042] 4) Shared travel services: Integrating the embodiments of the present disclosure into self-driving taxis or shared vehicles will help provide safe and reliable travel services.

[0043] The specific implementation of the embodiment of the present disclosure is described in detail below.

[0044] Figure 1 The flow chart of the trajectory prediction method provided by the embodiment of the present disclosure is shown. The trajectory prediction method can be executed by the electronic device below. The electronic device can be implemented as but not limited to a domain controller or other similar devices. Figure 1 The trajectory prediction method of the embodiment of the present disclosure includes the following steps:

[0045] Step 101, using the sensor data of the vehicle and the existing map data to obtain the environmental characteristics of the current scene where the vehicle is located;

[0046] Step 102, converting the environmental features of the current scene into an instance tag set, the instance tag set includes multiple instance tags, each instance tag represents information of an entity in the current scene, the entities in the current scene include the ego vehicle and at least one agent in the current scene, and each agent represents another dynamic entity in the current scene except the ego vehicle;

[0047] Step 103 , generating future trajectory data of the ego vehicle using the instance label set and the previously obtained ego vehicle trajectory latent space, where the ego vehicle trajectory latent space is learned by a pre-trained variational autoencoder based on the ego vehicle's historical trajectory data.

[0048] The following is an exemplary description of the specific implementation of each step in the trajectory prediction method provided in the embodiment of the present disclosure.

[0049] In step 101, the sensor data of the vehicle may include: multi-view images of the current scene of the vehicle captured by multiple surround cameras carried by the vehicle. Environmental features may include: bird's-eye view (BEV) features of the current scene. Step 101 may include: using image recognition models based on convolutional neural networks such as Feature Pyramid Networks (FPNs) and Residual Networks 50 (ResNets) to process the multi-view images to obtain multi-scale image features, initialize the multi-scale image features as queries in the BEV space, and transfer the image features from the multi-scale image space to the BEV space through a deformable cross-attention mechanism to generate a BEV feature representation, which is the BEV feature of the current scene.

[0050] In step 101, the environmental features may also include map information of the current scene, and the map information may include information such as lane lines and road signs in the current scene. Existing map data may include but is not limited to locally stored map data, external map data, and map data provided by the cloud. In step 101, the following may be included: extracting map information of the current scene from the existing map data according to the current position of the vehicle. The current position of the vehicle may be obtained by sensors such as IMU and GPS installed on the vehicle.

[0051] It should be noted that the environmental features involved in the embodiments of the present disclosure may also include other contents, not limited to the aforementioned BEV features and map information. The embodiments of the present disclosure do not limit the specific contents of the environmental features and the methods of obtaining them.

[0052] In the embodiments of the present disclosure, dynamic entities may be, but are not limited to, dynamically changing entities such as vehicles and pedestrians, and static entities may include, but are not limited to, static objects, lane lines, road signs, and other static entities. The embodiments of the present disclosure do not limit the types and numbers of dynamic entities and static entities.

[0053] Figure 2 FIG. 2 shows an exemplary flow chart of converting environmental features into instance tag sets in an embodiment of the present disclosure. Figure 2 In some implementations, step 102 may include:

[0054] Step 201, converting the BEV features of the current scene into map markers and proxy markers;

[0055] Specifically, the cross-attention mechanism can be used to convert BEV features into map tokens and agent tokens.

[0056] Map markers represent markers of semantic map information, which may include but are not limited to lane lines, road boundaries, pedestrian crossings, etc. Map markers are decoded by a map decoder into points in the BEV (bird's eye view) space, indicating the category of map elements and their corresponding positions.

[0057] The agent tag represents the information of each agent (such as other vehicles, pedestrians, other moving objects and other dynamic entities) in the environment around the vehicle (i.e., the current scene). The agent information includes but is not limited to the agent's position, direction, category, etc.

[0058] Step 202, generating a vehicle tag;

[0059] Specifically, on the basis of obtaining the proxy token, the ego token (Ego Token) is introduced. The ego token represents the status information of the ego vehicle in the current scene. The status information may include but is not limited to position, speed, direction, etc. The status information of the ego vehicle can be obtained through the sensor data of the ego vehicle (for example, data collected by sensors such as IMU and GPS mounted on the ego vehicle).

[0060] Step 203, using the map mark, the proxy mark and the vehicle mark to form an instance mark set;

[0061] Specifically, the vehicle token, map token and agent token are combined to form an instance token set (InstanceTokens). In the instance token set, each instance token represents a specific entity in the current scene, which can be but not limited to the vehicle, other vehicles, pedestrians, etc. The instance token not only contains agent information, but also integrates map information, thereby constructing an instance-centric scene representation.

[0062] The instance tag set formed by combining the ego vehicle tag, map tag and agent tag can fully and efficiently represent the current scene. This set not only contains the instance information of the agent and the ego vehicle, but also integrates the semantic map information to better understand and predict the dynamic changes and interactions in the current scene. In this way, the instance tag set can support motion prediction and planning tasks in the autonomous driving system.

[0063] Step 204: inject the map information of the current scene into the instance tag set.

[0064] Specifically, the cross-attention mechanism can be used to inject map information into each instance tag in the instance tag set to generate instance tags with map awareness. Therefore, the instance tags not only contain the information of the entity or object, but also integrate rich map context information, which can provide comprehensive contextual support for subsequent trajectory prediction.

[0065] Through steps 101 to 102, the conversion of sensor raw data into map-aware instance labels is achieved, providing a structured and rich scene representation for motion prediction and trajectory planning in autonomous driving scenarios. This scene representation method provides more comprehensive contextual support by integrating map information and dynamic agent information, which helps to achieve more accurate and robust driving decisions in complex traffic environments.

[0066] In complex urban traffic environments, it is necessary to accurately predict the dynamic interactions between the ego vehicle and other traffic participants (e.g., other vehicles, pedestrians, etc.) in order to accurately predict the trajectory of the ego vehicle. However, related technologies cannot effectively simulate such dynamic interactions, which may lead to inaccurate trajectory prediction results.

[0067] In view of this, the trajectory prediction method of the embodiment of the present disclosure further includes: using the historical trajectory data of the self-vehicle and the historical trajectory data of at least one agent to update the instance tag set, so that each instance tag in the instance tag set contains high-order interaction information between instances, and the high-order interaction information can be used to indicate the dynamic relationship and interaction between the self-vehicle and one or more other dynamic entities in the current scene. Specifically, "high-order interaction" refers to how the behavior of the self-vehicle affects the behavior of other vehicles, and how the behavior of other dynamic entities (such as other vehicles, pedestrians, etc.) in turn affects the decision-making of the self-vehicle. The interaction involves dynamic dependencies between multiple agents, and the high-order interaction information can reflect the high-order interaction between the self-vehicle and other dynamic entities.

[0068] In specific applications, the instance tag set can be updated through the attention mechanism. In some examples, considering that the self-attention mechanism can identify and value the direct interaction between the vehicle and other agents, and considering that the cross-attention mechanism allows the entire traffic environment and road structure to be considered in future trajectory prediction, the instance tag set can be updated through the self-attention mechanism and the cross-attention mechanism.

[0069] In some embodiments, updating the instance label set using the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent includes: using a self-attention mechanism to process the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent to obtain a first attention score for each instance, the first attention score of each instance indicating the degree of trajectory correlation between the instance and other instances, that is, the first attention score indicates how much attention one instance should give to another instance when predicting future trajectories; and, updating the corresponding instance label using the first attention score of each instance.

[0070] For example, the instance tag set may be updated based on the first attention score of each instance in a weighted manner.

[0071] Specifically, the historical trajectory data of each instance (i.e., the vehicle or any agent) can be represented as a series of vectors (called tags). The self-attention mechanism involves calculating the relationship between the query (Q), key (K) and value (V). The self-attention mechanism is used to calculate the first attention score of each instance. The first attention score of each instance is normalized by the softmax function and multiplied with the value vector of the corresponding instance tag in the instance tag set to obtain an updated instance tag set. The importance of each instance in the updated instance tag set is determined by the first attention score.

[0072] Updating the instance tag set through the self-attention mechanism can capture the dependencies within the sequence in the instance tag set, especially when processing text, speech or time series data, it can capture long-distance dependencies. The updated instance tag set integrates the information of all instances, and the information of all instances contains relevant information emphasized by the self-attention mechanism. The contribution of each instance is determined by the relationship between them. Using the updated instance tag set to perform subsequent processing can avoid collisions or consider traffic rules, thereby achieving safer and more reasonable trajectory prediction, and further improving the accuracy of trajectory prediction.

[0073] It should be understood that the method of updating the instance tag set using the first attention score is not limited to this, and any other applicable method may be used.

[0074] In some embodiments, updating the instance tag set using the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent includes: using the historical trajectory data of the ego vehicle as a query sequence (Q) of a cross-attention mechanism, the historical trajectory data of at least one agent as a key sequence (K) and a value sequence (V) of the cross-attention mechanism, performing calculations of the cross-attention mechanism to obtain a second attention score for each agent, and updating the instance tag set using the second attention score of each instance. The second attention score of each agent indicates the degree of correlation between the trajectory of the ego vehicle and the trajectory of the agent, and the second attention score of a certain agent indicates how much attention should be given to the trajectory of the agent when predicting the future trajectory of the ego vehicle.

[0075] Specifically, the calculation of the cross-attention mechanism includes calculating the dot product between the query sequence and the key sequence. The second attention score of each agent calculated by the cross-attention mechanism is normalized by the softmax function and multiplied with the value sequence representing the instance tag set to obtain the updated instance tag set. Therefore, the instance tag set can be weighted using the second attention score to obtain the updated instance tag set.

[0076] The criss-cross attention mechanism can process two related but different sequences, which is particularly useful in multimodal tasks. The relationship between the ego-car trajectory and the agent trajectory can be integrated into the instance labeling through the criss-cross attention mechanism.

[0077] The updated instance tag set through the cross-attention mechanism integrates the information of the surrounding agents that are most relevant to the state of the ego vehicle, reflects the contribution of each agent (including the ego vehicle and other agents) to the state of the ego vehicle, and can provide the ego vehicle with a new representation that includes the environmental context. Using the updated instance tag set to perform subsequent processing can further improve the accuracy of trajectory prediction.

[0078] It should be understood that the method of updating the instance tag set using the second attention score is not limited to this, and any other applicable method may be used.

[0079] As can be seen from the above, the disclosed embodiment simulates the high-order interaction between the vehicle and other traffic participants through the self-attention mechanism, integrates the relationship between the vehicle trajectory and the trajectory of other traffic participants through the cross-attention mechanism, and further enriches the instance tags in the instance tag set of the current scene so that it can capture complex traffic dynamics, which helps to improve the dynamics and accuracy of subsequent trajectory prediction. However, related technologies may only consider simple interactions or completely ignore such interactions.

[0080] Before step 103, the trajectory prediction method of the embodiment of the present disclosure may further include: learning the vehicle trajectory latent space based on the historical trajectory data of the vehicle by using a trained variational autoencoder.

[0081] Here, the historical trajectory data of the self-vehicle may include the state information of the self-vehicle at different time points in the past (e.g., position, speed, acceleration, etc.), which may be collected by sensors such as IMU and GPS mounted on the self-vehicle. In addition, the historical trajectory data of the self-vehicle may also include the environmental information of the self-vehicle at different time points in the past (e.g., the position and speed of other vehicles, etc.).

[0082] The variational autoencoder (VAE) can include: an encoder, a reparameterization trick, and a decoder. Among them, the encoder of VAE is a deep neural network that receives the historical trajectory data of the ego vehicle and outputs the distribution parameters of the ego vehicle trajectory latent space, namely the mean (μ) and variance (σ^2). In order to make VAE trainable, VAE uses the reparameterization trick to generate a latent vector Z by sampling noise ε from a standard normal distribution and combining it with the distribution parameters of the ego vehicle trajectory latent space (ie, mean and variance). This vector Z represents the representation of the input data (ie, the historical trajectory data of the ego vehicle) in the ego vehicle trajectory latent space. This process is differentiable, allowing gradients to be passed in back propagation. The decoder of VAE is also a deep neural network, which is responsible for sampling points from the ego vehicle trajectory latent space and mapping them back to the original data space to generate the future trajectory point information of the ego vehicle.

[0083] Specifically, the process of learning the latent space of the vehicle trajectory by the variational autoencoder may include: inputting the historical trajectory data of the vehicle into the encoder in the variational autoencoder, and the encoder in the variational autoencoder maps the historical trajectory data of the vehicle to the distribution of a latent space to obtain the distribution parameters (i.e., mean and variance) of the latent space, and the latent space is the latent space of the vehicle trajectory. Since the dimension and structure of the latent space of the vehicle trajectory are learned through the trajectory data of the real world, the uncertainty and prior knowledge of the vehicle trajectory (i.e., the possible range and trend of the change of the vehicle trajectory) are captured, and the latent space of the vehicle trajectory reflects the driving behavior pattern of the vehicle, therefore, the latent space of the vehicle trajectory can be used to generate a trajectory that is more in line with the actual driving habits of the vehicle.

[0084] Furthermore, before learning the latent space of the vehicle trajectory, the method of the embodiment of the present disclosure may also include: training the variational autoencoder with the goal of maximizing the evidence lower bound (ELBO) to determine the parameters of the variational autoencoder. Maximizing the evidence lower bound involves reconstruction error and KL divergence. The reconstruction error is used to measure the similarity between the reconstructed data of the variational autoencoder and the real data, while the KL divergence can be used to measure the difference between the distribution of the potential representation (i.e., the latent space) and the prior distribution.

[0085] Furthermore, the method of the disclosed embodiment may also include: collecting the historical trajectory data of the vehicle in real time and using the historical trajectory data of the vehicle in real time to update the parameters of the variational autoencoder to achieve the goal of optimizing the variational autoencoder parameters, so that the vehicle trajectory latent space learned by the variational autoencoder is as close as possible to the prior distribution (for example, the standard normal distribution), thereby obtaining the vehicle trajectory latent space that is more in line with the actual driving behavior pattern. In this way, the variational autoencoder can better adapt to the ever-changing traffic environment, new traffic rules, new road conditions and driving behavior, so that the variational autoencoder has the ability to quickly adapt to the new environment and its performance can be continuously improved, so that the trajectory prediction is more in line with the actual driving behavior pattern and actual scenario, thereby further improving the accuracy of trajectory prediction,

[0086] As can be seen from the above, the disclosed embodiment adopts a generative modeling method, and learns the potential distribution of the real trajectory of the vehicle through VAE, which can take into account the uncertainty and prior knowledge of the trajectory while generating a trajectory that is more in line with actual driving habits. Traditional methods often rely on deterministic models and manual feature engineering, and cannot effectively simulate the trajectory generation process. It can be seen that the disclosed embodiment generates trajectories by learning the latent space of the vehicle's trajectory through VAE, which can improve the accuracy of trajectory prediction and make the trajectory prediction better in line with actual driving habits.

[0087] In step 103, the future trajectory data of the vehicle may include future trajectory point information of the vehicle within a predetermined time period in the future, and the future trajectory data may be directly applied to the planning and control of the autonomous driving system.

[0088] In some implementations, step 103 may include the following steps a1 and a2:

[0089] Step a1: Use a gated recurrent unit to determine the potential state of the ego vehicle within a predetermined time period in the future based on the instance tag set and the ego vehicle trajectory potential space.

[0090] Specifically, the instance label set obtained in step 102 is mapped to the vehicle trajectory latent space to sample the latent variables, and a gated recurrent unit is used to simulate the time evolution of the instance (including the vehicle and at least one agent) based on the latent variables to gradually obtain the potential state of the vehicle at each time point within a predetermined time period in the future.

[0091] The gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN) and is used to solve the gradient vanishing or gradient exploding problems of traditional RNNs when processing long sequence data. The core idea of ​​GRU is to control the flow of information by introducing update gates and reset gates, so as to better capture long-distance dependencies. The disclosed embodiment uses a gated recurrent unit to obtain the potential state of the vehicle at each time point within a predetermined time in the future. It can not only apply the high-order interaction information contained in the instance tag set, the information of each agent in the current scene, etc. to the generation of real-time trajectories, but also can perform effective real-time trajectory generation in complex traffic environments, while meeting real-time needs, improving the accuracy of trajectory prediction and making trajectory prediction better in line with actual driving habits.

[0092] Step a2, using the decoder in the variational autoencoder to decode the potential state of the vehicle within a predetermined time in the future to obtain the trajectory point information of the vehicle within a predetermined time in the future.

[0093] The decoder in the variational autoencoder can be, but is not limited to, a multi-layer perceptron (MLP) decoder. The decoding process is performed step by step, and the trajectory point information of the vehicle at each time in the future predetermined time period is gradually decoded based on the potential state of the vehicle at each time point in the future predetermined time period. The trajectory point information may include information such as the position and direction of the trajectory point.

[0094] The disclosed embodiment uses GRU as the core time series model to gradually generate future trajectory points, while taking into account long-term dependencies and real-time performance. At the same time, combined with a simple MLP decoder, specific trajectory points can be quickly decoded from the vehicle's trajectory potential space. Therefore, combined with the gated recurrent unit (GRU) and a simple MLP decoder, the generation and planning of real-time trajectories can be efficiently and quickly achieved under limited computing resources, which can better meet the real-time requirements of the autonomous driving system. At the same time, combined with the future trajectory prediction of the generative model, while conforming to the driving behavior pattern, it takes into account the uncertainty and prior knowledge of the trajectory, which can better meet the needs of safety, efficiency, comfort and other aspects.

[0095] The trajectory prediction method provided by the embodiment of the present disclosure not only realizes end-to-end processing from sensor data to trajectory generation, reduces the data conversion overhead between modules, and improves the calculation efficiency, but also has the following beneficial effects:

[0096] 1) Improved trajectory prediction accuracy, that is, by simulating high-order interactions and utilizing structured ego vehicle trajectory potential space, the future movement of the ego vehicle and other traffic participants can be predicted more accurately. It has been verified that in complex urban traffic scenarios, compared with related technologies, the disclosed embodiments improve trajectory prediction accuracy by about 20%, with lower trajectory prediction error and higher prediction accuracy.

[0097] 2) Better real-time performance, that is, generative modeling and real-time trajectory generation significantly improve the efficiency of trajectory prediction and reduce computing delay. It has been verified that in complex urban traffic scenarios, the trajectory planning delay of the embodiment of the present disclosure is reduced from 400 milliseconds in related technologies to less than 80 milliseconds, which can better meet the real-time requirements of autonomous driving.

[0098] 3) Driving safety is improved, that is, the embodiments of the present disclosure can select the safest and most efficient driving trajectory by comprehensively considering multiple possible future situations, thereby improving overall driving safety. It has been verified that in a simulated environment, the performance of the present disclosure in avoiding collisions and complying with traffic rules is better than that of related technologies, reducing potential safety accidents by about 37%.

[0099] It can be seen from the above that the embodiments of the present disclosure have great application potential in the field of autonomous driving, especially in terms of improving prediction accuracy, planning efficiency and driving safety.

[0100] Figure 3 FIG. 1 shows a schematic diagram of the structure of a trajectory prediction device provided by an embodiment of the present disclosure. Figure 3 , the trajectory prediction device 300 of the embodiment of the present disclosure may include:

[0101] A perception unit 301 is used to obtain environmental features of the current scene where the vehicle is located using sensor data of the vehicle and existing map data;

[0102] A scene representation unit 302, configured to convert environmental features of the current scene into an instance tag set, the instance tag set including multiple instance tags, each instance tag representing information of an entity in the current scene, the entities in the current scene including the ego vehicle and at least one agent in the current scene, each agent representing another dynamic entity in the current scene except the ego vehicle;

[0103] The trajectory generating unit 303 is used to generate future trajectory data of the ego vehicle using the instance tag set and the previously obtained ego vehicle trajectory latent space, where the ego vehicle trajectory latent space is learned by a pre-trained variational autoencoder based on the ego vehicle's historical trajectory data.

[0104] Furthermore, the environmental features of the current scene include BEV features of the current scene and map information of the current scene; the scene representation unit 302 can be specifically used to: convert the BEV features of the current scene into map tags and proxy tags; generate ego vehicle tags; use the map tags, the proxy tags and the ego vehicle tags to form an instance tag set; and inject the map information of the current scene into the instance tag set.

[0105] Furthermore, the scene representation unit 302 can also be used to: update the instance tag set using the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent, so that each instance tag in the instance tag set contains high-order interaction information between instances, and the high-order interaction information is used to indicate the dynamic relationship and interaction between the ego vehicle and one or more of the other dynamic entities in the current scene.

[0106] Furthermore, the scene representation unit 302 can also be used to: use the self-attention mechanism to process the historical trajectory data of the vehicle and the historical trajectory data of at least one agent to obtain a first attention score for each instance, the first attention score of each instance indicating the trajectory correlation degree between the instance and other instances; and use the first attention score of each instance to update the instance tag set.

[0107] Furthermore, the scene representation unit 302 can also be used to: use the historical trajectory data of the ego-vehicle as the query sequence of the cross-attention mechanism and the historical trajectory data of at least one agent as the key sequence and value sequence of the cross-attention mechanism, perform the calculation of the cross-attention mechanism to obtain the second attention score of each instance, and the second attention score of each instance represents the degree of correlation between the trajectory of the ego-vehicle and the trajectory of the agent; and use the second attention score of each instance to update the instance tag set.

[0108] Furthermore, the trajectory generation unit 303 can be specifically used to: use a gated recurrent unit to determine the potential state of the ego-vehicle within a predetermined future time period based on the instance tag set and the ego-vehicle trajectory potential space; use a decoder in the variational autoencoder to decode the potential state of the ego-vehicle within a predetermined future time period to obtain the trajectory point information of the ego-vehicle within the predetermined future time period.

[0109] In a specific application, the trajectory prediction device 300 can be implemented by software, hardware, or a combination of the two. For example, the trajectory prediction device 300 can be implemented as software running in the electronic device 400 described below.

[0110] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of the aforementioned trajectory prediction method are executed.

[0111] Figure 4 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present disclosure. Figure 4 The electronic device 400 may include: one or more processors 401, and a memory 402 storing one or more programs, which are executed by the one or more processors 401 to implement the method flow shown in the above embodiments of the present disclosure and / or program units corresponding to each unit in the device.

[0112] The various components are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 401 can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memories if desired.

[0113] Processor 401 may include one or more single-core processors or multi-core processors. Processor 401 may include any combination of general-purpose processors or dedicated processors (such as image processors, application processors, baseband processors, etc.).

[0114] The memory 402 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the following in the embodiments of the present disclosure: Figure 1 The processor 401 executes the non-transient software programs, instructions and units stored in the memory 402, thereby executing the above method embodiments. Figure 1 The programs, instructions and units corresponding to the trajectory prediction method shown.

[0115] The electronic device 400 may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0116] The input device 403 can receive input digital or character information, and generate signal input related to user settings and function control, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 404 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0117] The above-mentioned programs (also referred to as software, software applications, or codes) include machine instructions for programmable processors, and these computer programs can be implemented using object-oriented programming languages, assembly or machine languages.

[0118] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, and can also be downloaded directly from the network, etc. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be used but not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by or in combination with instruction execution systems, devices or devices.

[0119] The technical solution provided by the present disclosure is described in detail above. The principles and implementation methods of the present disclosure are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present disclosure. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present disclosure.

[0120] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A trajectory prediction method, characterized in that: The method comprises: Use the vehicle's sensor data and existing map data to obtain the environmental characteristics of the current scene in which the vehicle is located; Converting the environmental features of the current scene into an instance tag set, wherein the instance tag set includes multiple instance tags, each of which represents information of an entity in the current scene, the entities in the current scene include the ego vehicle and at least one agent in the current scene, and each of which represents another dynamic entity in the current scene except the ego vehicle; The future trajectory data of the ego vehicle is generated using the instance label set and a previously obtained ego vehicle trajectory latent space, where the ego vehicle trajectory latent space is learned by a pre-trained variational autoencoder based on the historical trajectory data of the ego vehicle.

2. The method according to claim 1, characterized in that The environmental features of the current scene include BEV features of the current scene and map information of the current scene; The converting of the environmental features into an instance tag set includes: converting the BEV features of the current scene into a map tag and a proxy tag; generating an ego vehicle tag; using the map tag, the proxy tag and the ego vehicle tag to form an instance tag set; and injecting the map information of the current scene into the instance tag set.

3. The method according to claim 1 or 2, characterized in that: Also includes: The instance tag set is updated using the historical trajectory data of the ego vehicle and the historical trajectory data of the at least one agent, so that each instance tag in the instance tag set contains high-order interaction information between instances, and the high-order interaction information is used to indicate the dynamic relationship and interaction between the ego vehicle and one or more of the other dynamic entities in the current scene.

4. The method according to claim 3, characterized in that The updating of the instance tag set by using the historical trajectory data of the vehicle and the historical trajectory data of the at least one agent comprises: Processing the historical trajectory data of the ego vehicle and the historical trajectory data of at least one agent using a self-attention mechanism to obtain a first attention score for each instance, wherein the first attention score of each instance indicates the trajectory correlation degree between the instance and other instances; The instance label set is updated using the first attention score of each instance.

5. The method according to claim 3, characterized in that: The updating of the instance tag set by using the historical trajectory data of the vehicle and the historical trajectory data of the at least one agent comprises: Using the historical trajectory data of the ego vehicle as the query sequence of the cross-attention mechanism and the historical trajectory data of at least one agent as the key sequence and value sequence of the cross-attention mechanism, the calculation of the cross-attention mechanism is performed to obtain a second attention score of each instance, wherein the second attention score of each instance represents the degree of correlation between the trajectory of the ego vehicle and the trajectory of the agent; The instance label set is updated using the second attention score of each instance.

6. The method according to claim 1, characterized in that The generating future trajectory data of the ego vehicle by using the instance tag set and the previously obtained ego vehicle trajectory latent space includes: Determine, using a gated recurrent unit, a potential state of the ego-vehicle within a predetermined time period in the future based on the instance tag set and the ego-vehicle trajectory potential space; The decoder in the variational autoencoder is used to decode the trajectory point information of the vehicle within the future predetermined time period based on the potential state of the vehicle within the future predetermined time period.

7. The method according to claim 1 or 6, characterized in that: The method also includes: collecting historical trajectory data of the vehicle in real time and using the historical trajectory data of the vehicle to update parameters of the variational autoencoder in real time.

8. A trajectory prediction device, characterized in that: include: A perception unit, used to obtain the environmental characteristics of the current scene where the vehicle is located by using the sensor data of the vehicle and the existing map data; A scene representation unit, configured to convert the environmental features of the current scene into an instance tag set, wherein the instance tag set includes a plurality of instance tags, each of which represents information of an entity in the current scene, the entities in the current scene include the ego vehicle and at least one agent in the current scene, and each of which represents another dynamic entity in the current scene except the ego vehicle; A trajectory generation unit is used to generate future trajectory data of the ego vehicle using the instance tag set and a previously obtained ego vehicle trajectory latent space, wherein the ego vehicle trajectory latent space is learned by a pre-trained variational autoencoder based on the historical trajectory data of the ego vehicle.

9. An electronic device, characterized in that: include: One or more processors and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, wherein the program comprises instructions, and when the instructions are executed by one or more processors of a computing device, the instructions cause the computing device to execute the method according to any one of claims 1 to 7.

Images