Training method of trajectory prediction model and trajectory prediction method

Abstract

The invention provides a training method of a trajectory prediction model and a trajectory prediction method, and relates to the technical field of artificial intelligence such as automatic driving and intelligent transportation. Wherein the initial trajectory prediction model comprises an embedded layer, an encoder and a decoder, and the method comprises the following steps: performing feature extraction on historical trajectory information of a sample vehicle and local map information associated with the historical trajectory information by using the embedded layer to obtain vehicle trajectory features and map features; encoding the vehicle track features and the map features by using an encoder to obtain vehicle track codes and map element codes; decoding the vehicle track code and the map element code by using a decoder to obtain a target prediction track of the target vehicle; and training the initial trajectory prediction model according to the loss between the target prediction trajectory and the real driving trajectory of the target vehicle to obtain a trajectory prediction model. According to the method provided by the invention, the prediction accuracy of the trajectory prediction model and the storage performance of the trajectory prediction model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of autonomous driving and intelligent transportation technology, and in particular to a trajectory prediction model training method and a trajectory prediction method. Background Art

[0002] In an autonomous driving system, the vehicle trajectory prediction results determine the vehicle's path planning and self-control. The current series of trajectory prediction solutions focus on scene modeling and interaction, that is, encoder architecture, such as VectorNet (Trajectory Prediction Vectorized Scene Representation Network), PointNet encoding method, LaneGCN, etc. Summary of the invention

[0003] The present disclosure provides a trajectory prediction model training method and a trajectory prediction method.

[0004] According to a first aspect of the present disclosure, a method for training a trajectory prediction model is provided, wherein the initial trajectory prediction model includes: an embedding layer, an encoder and a decoder, and the method includes: using the embedding layer to extract features of historical trajectory information of a sample vehicle and local map information associated with the historical trajectory information to obtain vehicle trajectory features and map features, wherein the sample vehicle includes a target vehicle and an obstacle vehicle; using the encoder to encode the vehicle trajectory features and the map features to obtain a vehicle trajectory code and a map element code; using the decoder to decode the vehicle trajectory code and the map element code to obtain a target predicted trajectory of the target vehicle; training the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

[0005] According to a second aspect of the present disclosure, a trajectory prediction method is provided, comprising: obtaining a first current driving trajectory of a target vehicle and a second current driving trajectory of an obstacle vehicle; inputting the first current driving trajectory and the second current driving trajectory into a trajectory prediction model, and outputting a predicted trajectory of the target vehicle, wherein the trajectory prediction model is trained using the method described in any implementation method of the first aspect.

[0006] According to a third aspect of the present disclosure, a training device for a trajectory prediction model is provided, wherein the initial trajectory prediction model includes: an embedding layer, an encoder and a decoder, and the device includes: an embedding module, configured to use the embedding layer to extract features of historical trajectory information of a sample vehicle and local map information associated with the historical trajectory information to obtain vehicle trajectory features and map features, wherein the sample vehicles include target vehicles and obstacle vehicles; an encoding module, configured to use the encoder to encode the vehicle trajectory features and the map features to obtain vehicle trajectory encoding and map element encoding; a decoding module, configured to use the decoder to decode the vehicle trajectory encoding and the map element encoding to obtain a target predicted trajectory of the target vehicle; and a training module, configured to train the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

[0007] According to a fourth aspect of the present disclosure, a trajectory prediction method is provided, comprising: an acquisition module, configured to acquire a first current driving trajectory of a target vehicle and a second current driving trajectory of an obstacle vehicle; a prediction module, configured to input the first current driving trajectory and the second current driving trajectory into a trajectory prediction model, and output a predicted trajectory of the target vehicle, wherein the trajectory prediction model is trained using the method described in any implementation manner in the first aspect.

[0008] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect or the second aspect.

[0009] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect or the second aspect.

[0010] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to perform the method described in any implementation manner of the first aspect or the second aspect.

[0011] According to an eighth aspect of the present disclosure, an autonomous driving vehicle is provided, which includes the electronic device as described in the fifth aspect.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1 is a flowchart of a first embodiment of a method for training a trajectory prediction model according to the present disclosure; Figure 2 is a flow chart of a second embodiment of a method for training a trajectory prediction model according to the present disclosure; Figure 3 is a flowchart of a third embodiment of a method for training a trajectory prediction model according to the present disclosure; Figure 4 is a flowchart of a fourth embodiment of a method for training a trajectory prediction model according to the present disclosure; Figure 5-1 is an application block diagram of a training method for a trajectory prediction model according to the present disclosure; Figure 5-2 It is a schematic diagram of the structure of another decoder; Figure 5-3 It is a structural diagram of another decoder; Figure 6 is a flow chart of an embodiment of a trajectory prediction method according to the present disclosure; Figure 7 is a structural schematic diagram of an embodiment of a training device for a trajectory prediction model according to the present disclosure; Figure 8 is a structural schematic diagram of an embodiment of a trajectory prediction device according to the present disclosure; Fig. 9 It is a block diagram of an electronic device used to implement the trajectory prediction model training method or trajectory prediction method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0015] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0016] An exemplary system architecture for implementing the training method of the trajectory prediction model or the trajectory prediction method provided by the present disclosure may include a terminal device, a network, and a server. The network is used to provide a communication link between the terminal device and the server, and may include various connection types, such as a wired communication link, a wireless communication link, or an optical fiber cable, etc.

[0017] The user can use the terminal device to interact with the server through the network to receive or send information, etc. Various client applications can be installed on the terminal device, for example, map, navigation, entertainment and other client applications.

[0018] The terminal device may be, for example, a vehicle computer system of a self-driving car, a delivery robot, or the like. The system may be implemented by hardware, by software, or by a combination of hardware and software.

[0019] The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0020] It should be pointed out that the training method of the trajectory prediction model or the execution subject of the trajectory prediction method provided by the present disclosure (hereinafter referred to as the "execution subject") can be executed by the server in the above-mentioned system architecture, or it can be implemented by the above-mentioned terminal device. For example, when the server executes, the server sends a control command to the client, such as the vehicle's on-board system, and then the vehicle's trajectory is predicted according to the control command through the on-board system, such as predicting the vehicle's driving trajectory, and controlling the vehicle's driving according to the predicted trajectory. For another example, when the terminal device (such as the vehicle's on-board system) executes, the terminal device predicts the vehicle's trajectory according to the control command, such as predicting the vehicle's driving trajectory, and controlling the vehicle's driving according to the predicted trajectory.

[0021] Figure 1 A process 100 of a first embodiment of a method for training a trajectory prediction model according to the present disclosure is shown. In this embodiment, the initial trajectory prediction model includes an embedding layer, an encoder, and a decoder, and the method for training the trajectory prediction model includes the following steps: Step 101 , using an embedding layer to extract features from historical trajectory information of a sample vehicle and local map information associated with the historical trajectory information, to obtain vehicle trajectory features and map features.

[0022] In this embodiment, the execution entity uses the embedding layer to extract features from the historical trajectory information of the sample vehicle and the local map information associated with the historical trajectory information to obtain vehicle trajectory features and map features, wherein the sample vehicle includes a target vehicle and an obstacle vehicle.

[0023] Here, the target vehicle is the ego vehicle, and the obstacle vehicle is the obstacle around the ego vehicle. The number of obstacle vehicles can be multiple. Since the sample vehicles include the ego vehicle and the obstacle vehicles, the historical trajectory information of the sample vehicles includes the historical trajectory information of the ego vehicle and the historical trajectory information of the obstacle vehicles. The historical trajectory information can be the trajectory information within a preset time range, such as the trajectory information within one hour from the current time; the historical trajectory information can be the trajectory information within a preset distance, such as the trajectory information within five kilometers from the current position.

[0024] In addition, the above-mentioned execution entity will obtain local map information associated with the historical trajectory information from the pre-built high-precision map. For example, the above-mentioned execution entity can first determine a location range based on the historical trajectory information, and then obtain the map corresponding to the location range from the high-precision map to obtain a local map.

[0025] It should be noted that the embedding layer is a special layer in the neural network, which is used to map discrete input data into a continuous vector space.

[0026] In this embodiment, the historical trajectory of the sample vehicle and the local map corresponding to the historical trajectory are input into the embedding layer. The embedding layer extracts features from the historical trajectory to obtain vehicle trajectory features, and also extracts features from the local map to obtain map features.

[0027] Step 102: Encode the vehicle trajectory features and the map features using an encoder to obtain a vehicle trajectory code and a map element code.

[0028] In this embodiment, the execution subject may encode the vehicle trajectory features and the map features by using an encoder, thereby obtaining the vehicle trajectory codes and the map element codes after encoding.

[0029] In a neural network, the task of an encoder is to encode an input sequence into a context vector. An encoder is usually composed of one or more layers of neurons, each of which receives input data and converts it into output through an activation function. The role of the encoder is to compress the input data into a lower-dimensional representation for subsequent processing by the neural network.

[0030] The encoder in this embodiment is composed of multiple layers of MHA (Multi-Head Attention), for example, the encoder can be obtained by stacking 6 layers of MHA. Here, the multi-head attention mechanism (MHA) captures different features in the input sequence by calculating multiple attention heads in parallel. Each attention head has its own query (Query, Q), key (Key, K) and value (Value, V) matrices, and their main functions are as follows: Query matrix Q: The query matrix is ​​the 'question' that you want to find a certain information. The query matrix is ​​a projection of the input, indicating the current token (the smallest discrete unit in text or sequence data) The demand for other tokens, it can help determine its position in the sequence and what content needs to be paid attention to; Key matrix K: The key matrix is ​​the 'information' or 'identifier' provided by each token. Each token has a key associated with it, which is used to compare with the query to determine its relevance to the query; Value matrix V: The value is the actual information, providing the content of the word vector. Depending on the degree of match between Q and K, V is ultimately used to generate the output vector.

[0031] In this embodiment, the vehicle trajectory features and the map features are input into the encoder, and the multi-layer MHA in the encoder captures the different features in the two sequences and encodes the features, thereby outputting the vehicle trajectory code and the map element code.

[0032] Step 103: Decode the vehicle trajectory code and the map element code using a decoder to obtain a target predicted trajectory of the target vehicle.

[0033] In this embodiment, the execution subject uses a decoder to decode the vehicle trajectory code and the map element code, thereby obtaining the target predicted trajectory of the target vehicle after the decoding process.

[0034] In a neural network, the task of a decoder is to generate an output sequence from a context vector. The decoder usually reads the context vector of the encoder first, and then starts to generate an output sequence. In the trajectory prediction scenario of this embodiment, the decoder decodes the feature vector output by the encoder into specific trajectory data, thereby outputting the target predicted trajectory of the target vehicle.

[0035] The decoder in this embodiment is also composed of multiple layers of MHA (Multi-Head Attention), and the output of the previous attention layer is the input of the next attention layer.

[0036] Step 104 , training the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

[0037] In this embodiment, the execution entity trains the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain the trajectory prediction model.

[0038] The model training samples in this embodiment include the historical trajectory information of the sample vehicle (i.e., the historical trajectory information of the target vehicle and the historical trajectory information of the obstacle vehicle) and the actual driving trajectory of the target vehicle. The historical trajectory information and the actual driving trajectory here are for the current moment (also called the prediction moment), that is, if the trajectory of the target vehicle is to be predicted at the current moment, then the trajectory before the current moment is called the historical trajectory, and the trajectory after the current moment is called the actual driving trajectory. The above-mentioned execution entity will generate a target prediction trajectory according to the above steps, and the target prediction trajectory is the predicted trajectory that the target vehicle will travel after the current moment.

[0039] The above-mentioned execution entity will calculate the loss between the target predicted trajectory and the actual driving trajectory. For example, the regression loss between the target predicted trajectory and the actual driving trajectory is calculated according to the regression loss function, and then the parameters of the embedding layer, encoder, and decoder of the initial trajectory prediction model are adjusted using the loss until the preset conditions are met to obtain a trained trajectory prediction model.

[0040] The training method of the trajectory prediction model provided by the embodiment of the present disclosure first uses an embedding layer to extract features from the historical trajectory information of the sample vehicle and the local map information associated with the historical trajectory information to obtain vehicle trajectory features and map features; then, an encoder is used to encode the vehicle trajectory features and map features to obtain vehicle trajectory encoding and map element encoding; then, a decoder is used to decode the vehicle trajectory encoding and map element encoding to obtain a target predicted trajectory of the target vehicle; finally, the initial trajectory prediction model is trained according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model. When training the trajectory prediction model, the method encodes based on a local connection graph (rather than a global connection graph), so that the local structure can be better maintained and the prediction accuracy of the trajectory prediction model can be improved; in addition, a larger map encoding is achieved through more efficient storage, thereby improving the storage performance of the trajectory prediction model.

[0041] In addition, in the technical solutions involved in the present disclosure, the acquisition, storage, use, processing, transportation, provision and disclosure of the vehicle trajectory information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0042] Continue to refer Figure 2 , Figure 2A process 200 of a second embodiment of a method for training a trajectory prediction model according to the present disclosure is shown. The method for training a trajectory prediction model comprises the following steps: Step 201, input the historical trajectory information and the local map information into the normalization layer, and output the vehicle trajectory matrix and the map element matrix.

[0043] In this embodiment, the embedding layer of this embodiment includes a normalization layer and a conversion layer. The execution body of the trajectory prediction model training method first inputs the historical trajectory information of the sample vehicle (historical trajectory information of the target vehicle and historical trajectory information of the obstacle vehicle) and the local map information into the normalization layer to normalize the historical trajectory information and the local map information.

[0044] Specifically, the local map information and obstacle trajectory information (i.e., vehicle trajectory information) can be converted into a local normalized representation centered on the target obstacle by the following formula. This type of information is collectively referred to as a polyline. After normalization, the vehicle trajectory matrix can be obtained: and the map element matrix :

[0045] in, is the vehicle historical trajectory information, is the local map information, and MLP is a multi-layer perceptron.

[0046] After normalization The dimension is Agent[N,T,C], where N represents the number of obstacles, which is 128; T represents the number of historical frames, which is 30; and C represents the number of vehicle features, which is 15. Vehicle features include x, y, heading, vel, etc. x and y represent coordinates, heading represents the direction of the vehicle body, and vel represents the velocity component.

[0047] After normalization The dimension is Map[M,N,C], where M is the number of map elements, which is 768, N is the number of points in each map element, which is 20, and C is the number of map features, which is 6. Map features include x, y, type, etc. x and y represent coordinates, and type represents the type of map element.

[0048] Step 202: input the vehicle trajectory matrix and the map element matrix into a conversion layer, and output the vehicle trajectory features and the map features.

[0049] In this embodiment, the execution entity inputs the vehicle trajectory matrix and the map element matrix into the conversion layer, that is, the normalized polyline is encoded using MLP (Multilayer Perceptron) and MaxPooling. The vehicle trajectory matrix and the map element matrix are encoded as the original features, and the encoded vehicle trajectory features are finally and map features The tensor dimensions are [N,D] and [M,D], where D=256. The agent (obstacle) MLP consists of three layers of 256 and two layers of 256. The map (map) MLP consists of five layers of 64-dimensional MLP and two layers of 256-dimensional MLP.

[0050] The vehicle trajectory information and local map information are processed through the embedding layer, so that the relationship between the data can be captured through the embedding layer, and the data can be converted into a low-dimensional vector for subsequent processing.

[0051] Step 203: input the vehicle trajectory features and the map features into the encoding network, and output the vehicle trajectory code and the map element code.

[0052] In this embodiment, the encoder in this embodiment includes an encoding network and a regression output layer. The above execution body first inputs the vehicle trajectory features and the map features into the encoding network, thereby outputting the vehicle trajectory code and the map element code.

[0053] The encoder in this embodiment is composed of multiple layers of MHA. For example, the encoder can be obtained by stacking 6 layers of MHA, and the output of the previous layer of MHA is the input of the next layer of MHA. The vehicle trajectory features and map features are input to the encoding network of MHA, that is, the input is [Agent, Map], and the dimension is [N+M, D], that is, [128+768, 256]. It can be expressed as a formula: MultiHeadAttn(query= +P ,key=k( )+P ,value=k( ) Among them, the input of the first layer =[ ], k() means using KNN (K-NearestNeighbor, k nearest neighbor) algorithm to find the 16 nearest Map polylines (map element lines) of each Agent polyline (obstacle trajectory line), PE represents the position code, and finally outputs the vehicle trajectory code A_past and the map element code M.

[0054] Step 204 , the vehicle trajectory encoding is input into the regression output layer, and the initial predicted trajectory and predicted speed of the sample vehicle are output.

[0055] In this embodiment, the above-mentioned execution body will input the vehicle trajectory encoding into the regression output layer, thereby outputting the initial predicted trajectory and predicted speed of the sample vehicle, which can be expressed as: =MLP( ) in, represents the initial predicted trajectory and predicted speed of the obstacle from time 1 to T, and i represents the number of future time steps.

[0056] Step 205 , performing local normalization processing and encoding conversion on the initial predicted trajectory and predicted speed to obtain the vehicle predicted trajectory encoding.

[0057] In this embodiment, the execution subject will use the embedded layer coding method to Processing is performed, that is, local normalization and encoding conversion are performed on the initial predicted trajectory and predicted speed to obtain the vehicle predicted trajectory encoding, that is, Coded as A_future.

[0058] Step 206 , concatenate the vehicle prediction trajectory code and the vehicle trajectory code to obtain the vehicle trajectory code.

[0059] In this embodiment, the above execution entity concatenates the vehicle prediction trajectory code A_future and the vehicle trajectory code A_past, and then wraps 3 layers of MLP (the full connection layer dimension is 512) as an enhanced representation of A, which can be expressed as: A=MLP([ ]) Among them, A is the vehicle trajectory code.

[0060] Here, future trajectories (predicted) are used to model interaction. The obtained scene encoding information is used to output the future trajectories and speeds of all obstacles, and the vehicle trajectory encoding is generated based on the future trajectory and speed. Therefore, the vehicle trajectory encoding generated by the encoder provides additional future context information to the decoder network, so that the trajectory prediction model can predict more future trajectories that match the scene when predicting the target obstacle.

[0061] Step 207: Decode the vehicle trajectory code and the map element code using a decoder to obtain a target predicted trajectory of the target vehicle.

[0062] Step 208 , training the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

[0063] Steps 207-208 are basically consistent with steps 103-104 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 103-104, which will not be repeated here.

[0064] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiments, the training method of the trajectory prediction model in this embodiment highlights the step of encoding using an encoder, and uses future trajectories (predicted) for modeling interaction, that is, using the obtained scene encoding information, a simple regression output head is connected to output the future trajectories and speeds of all obstacles, and the vehicle trajectory encoding is generated based on the future trajectory and speed. Therefore, the vehicle trajectory encoding generated by the encoder provides additional future context information for the decoder network, so that the trajectory prediction model can predict more future trajectories that conform to the scene when predicting the target obstacle.

[0065] Continue to refer Figure 3 , Figure 3 A process 300 of a third embodiment of a method for training a trajectory prediction model according to the present disclosure is shown. The method for training a trajectory prediction model comprises the following steps: Step 301 : extracting features from historical trajectory information of a sample vehicle and local map information associated with the historical trajectory information using an embedding layer to obtain vehicle trajectory features and map features.

[0066] Step 302: Encode the vehicle trajectory features and the map features using an encoder to obtain a vehicle trajectory code and a map element code.

[0067] Steps 301-302 are basically consistent with steps 101-102 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 101-102, which will not be repeated here.

[0068] Step 303: For each decoding layer, the vehicle trajectory code, the map element code, and the pre-generated global intent vectorized representation and auxiliary vectorized representation are input into the decoding network of the decoding layer, and the decoded feature information is output.

[0069] In this embodiment, the decoder includes N decoding layers, each decoding layer includes a decoding network and a decoding output layer, where N is a positive integer. For example, the decoder may include 6 decoding layers.

[0070] For each of the N decoding layers, a global intent vectorization representation and an auxiliary vectorization representation are pre-generated, and the global intent vectorization representation (also called a static intent query vector) and the auxiliary vectorization representation (also called a dynamic search query vector) are used as an intent query vector pair, so as to use the intent query vector pair to model the prediction task as a joint optimization task of global intent positioning and local motion optimization. Specifically, the above-mentioned execution subject will input the vehicle trajectory encoding, map element encoding, and the pre-generated intent query vector pair into the decoding network of the decoding layer, thereby outputting the decoded feature information.

[0071] In some optional implementations of the present embodiment, a vectorized representation of global intent is generated by the following steps: clustering the end points of the predicted trajectory to obtain a plurality of cluster centers; and transforming the position codes corresponding to the plurality of cluster centers using a fully connected layer to obtain a vectorized representation of global intent.

[0072] In this implementation, the execution subject clusters the end points of the future trajectory of the obstacle through the k-means algorithm, thereby obtaining multiple clusters, and the obtained cluster center is the intention representation. The clustering dimension considers both direction and speed, represented by I. I is position-encoded and fully linked to obtain the global intention vector representation. , which can be expressed as: =MLP(PE(I)) Among them, PE(I) represents the position code corresponding to I.

[0073] Each intent query is responsible for predicting the trajectory of a specific motion mode, which stabilizes the training process and facilitates predicting multimodal trajectories, since each motion mode has its own learnable embedding. Therefore, only 64 query vectors are needed to achieve good results, and dense target points are not required to cover the intent targets of obstacles.

[0074] In some optional implementations of this embodiment, the auxiliary vectorized representation is generated by the following steps: determining from the local map information a map element line corresponding to the initial predicted trajectory and satisfying preset conditions, and generating the auxiliary vectorized representation based on the map element line.

[0075] In this implementation, we aim to refine the trajectory by iteratively collecting fine-grained trajectory features to complement the global intent localization.

[0076] Each dynamic search query vector is also the position embedding code of the spatial point, initialized with its corresponding intent point, and dynamically updated according to the predicted trajectory of each decoder layer. For each motion query pair, a dynamic map collection module is designed to extract fine-grained trajectory features (query map features from the local area aligned with the trajectory, and the 128 map polylines closest to the predicted trajectory), that is, the fine-grained trajectory is used to query the nearest map polyline, which can be specifically expressed as: =MLP(PE( ) Where j is the number of layers of the decoder, is the trajectory feature corresponding to time T, and PE represents the position encoding.

[0077] Since the behavior of obstacles depends heavily on the map, this strategy is able to continuously focus on the latest local contextual information for iterative motion optimization.

[0078] In some optional implementations of this embodiment, the decoding network includes: a first decoding subnetwork, a second decoding subnetwork, a third decoding subnetwork and a decoding fully connected layer, each decoding subnetwork is an MHA; and step 303 further includes steps 3031-3034, specifically: Step 3031, input the global intent vectorized representation into the first decoding subnetwork, and output the first decoding information.

[0079] Here, the global intent will be vectorized first. Input to the first decoding subnetwork, so as to output the first decoding information , the first decoding subnetwork can be expressed as: MultiHeadAttn(query= + ,key= + ,value= ) Among them, j is the number of layers of the decoder, is the output of the previous decoder. It should be noted that Initialized to 0.

[0080] That is, for this MHA, the query is the output of the previous decoder and the vectorized representation of the global intent. , key is the output of the previous decoder and , value is the output of the previous decoder, and through this MHA we can get .

[0081] Step 3032: input the first decoding information, the auxiliary vectorized representation, the vehicle trajectory code, and the position code corresponding to the vehicle trajectory code into a second decoding subnetwork, and output the second decoding information.

[0082] Here, the above execution entity will , , vehicle trajectory code A and position code P corresponding to A Input to the second decoding subnetwork, so as to output the second decoding information , the second decoding subnetwork can be expressed as: MultiHeadAttn(query= ,key= ,value= ) That is, for this MHA, the query is and , keys are A and P , value is A, through this MHA we can get .

[0083] Step 3033: input the first decoding information, the auxiliary vectorized representation, the map element line, and the position code corresponding to the map element line into the third decoding subnetwork, and output the third decoding information.

[0084] Here, the above execution entity will , , map element line M and the position code P corresponding to M Input to the third decoding sub-network, so as to output the third decoding information , the third decoding subnetwork can be expressed as: MultiHeadAttn(query= ,key= ,value=α(M)) Among them, α(M) is the output of the dynamic map collection module, that is, M.

[0085] That is, for this MHA, the query is and , the keys are M and P , value is M, through this MHA we can get .

[0086] Step 3034: input the second decoded information and the third decoded information into a decoding fully connected layer, and output the decoded feature information.

[0087] Finally, and Input to the decoding fully connected layer, and the decoded feature information can be output , which can be expressed as: =MLP([ ]) Therefore, decoding information corresponding to the encoding information is generated through each decoding sub-network of the decoding layer and the decoding fully connected layer, and the accuracy of the obtained decoding information is improved through the multi-layer MHA structure.

[0088] Step 304: input the decoded feature information to the decoding output layer of the decoding layer, and output the probability distribution of each position point in the initial predicted trajectory.

[0089] In this embodiment, the above-mentioned execution subject inputs the decoded feature information into the decoding output layer of the decoding layer, thereby outputting the probability distribution of each position point in the initial predicted trajectory. The decoding output layer here includes a mixed Gaussian model, which can be expressed as: =MLP( ) =

[0090] in, Including N1:k ( ; ), i is between 1-T, P1:k has 6 elements, (o) is the probability that an obstacle appears at position o, and the predicted trajectory is generated by simply extracting the prediction center of the Gaussian distribution.

[0091] It should be noted that and yes and The mean of and yes and The standard deviation of is the correlation coefficient, which means and of relevance.

[0092] Step 305 , generating a target prediction trajectory according to the probability distribution of each position point in the initial prediction trajectory output by the N decoding layers.

[0093] In this embodiment, since the decoding output layer of each decoding layer will output the probability distribution of each position point in the initial prediction trajectory, the above-mentioned execution body will comprehensively generate the target prediction trajectory based on the probability distribution of each position point in the initial prediction trajectory output by the N decoding layers, for example, using NMS (non-maximum suppression principle) according to the trajectory end position, 6 final output trajectories are selected from 64 candidate trajectories, and then the target prediction trajectory is determined from these 6 final output trajectories. Here, a probability distribution of each coordinate value x, y is output instead of a determined trajectory coordinate value, so as to further determine the target prediction trajectory based on the probability distribution of multiple layers, thereby improving the accuracy of the target prediction trajectory.

[0094] Step 306 , training the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

[0095] Step 306 is basically the same as step 104 in the aforementioned embodiment. For the specific implementation method, reference may be made to the aforementioned description of step 104 and will not be repeated here.

[0096] from Figure 3 It can be seen that Figure 1 Compared with the corresponding embodiments, the training method of the trajectory prediction model in this embodiment highlights the step of decoding using a decoder. Specifically, the prediction task is modeled as a joint optimization task of global intent positioning and local motion optimization using intention query vector pairs. The learnable intention query pairs not only eliminate a large number of target candidates, but also predict different motion patterns through specific pattern motion query pairs, thereby reducing the optimization burden of the model.

[0097] Continue to refer Figure 4 , Figure 4 The process 400 of the fourth embodiment of the method for training a trajectory prediction model according to the present disclosure is shown. The method for training a trajectory prediction model comprises the following steps: Step 401 : extracting features from historical trajectory information of a sample vehicle and local map information associated with the historical trajectory information using an embedding layer to obtain vehicle trajectory features and map features.

[0098] Step 402: Encode the vehicle trajectory features and the map features using an encoder to obtain a vehicle trajectory code and a map element code.

[0099] Step 403, for each decoding layer, the vehicle trajectory code, the map element code, and the pre-generated global intent vectorized representation and auxiliary vectorized representation are input into the decoding network of the decoding layer, and the decoded feature information is output.

[0100] Step 404: input the decoded feature information to the decoding output layer of the decoding layer, and output the probability distribution of each position point in the initial predicted trajectory.

[0101] Step 405 , generating a target prediction trajectory according to the probability distribution of each position point in the initial prediction trajectory output by the N decoding layers.

[0102] Steps 401-405 are basically consistent with steps 301-305 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 301-305, which will not be repeated here.

[0103] Step 406 , calculating the regression loss between the target predicted trajectory and the actual driving trajectory.

[0104] In this embodiment, the execution body of the trajectory prediction model training method may use the L1 regression loss function shown below to calculate the regression loss L1 between the target predicted trajectory and the actual driving trajectory: L1=

[0105] Where n is the number of samples, Predict the trajectory for the target, Real driving trajectory.

[0106] Step 407, generating a comprehensive loss according to the regression loss and the target Gaussian loss.

[0107] In this embodiment, since each decoding layer in the decoder includes a mixed Gaussian model, the execution body also calculates the Gaussian loss in the entire decoding process, that is, the target Gaussian loss. Finally, the comprehensive loss Ltotal is calculated based on the regression loss L1 and the target Gaussian loss LG.

[0108] In some optional implementations of this embodiment, the target Gaussian loss is obtained by the following steps: for each decoding layer, calculating the Gaussian regression loss of the decoding layer; adding the Gaussian regression losses of N decoding layers to obtain the target Gaussian loss.

[0109] In this implementation, for each decoding layer, the execution entity calculates the Gaussian loss of the decoding layer according to the following formula:

[0110] =log +log +0.5log(1- ( -2 )-

[0111] Then, the Gaussian regression losses of the N decoding layers are added together to obtain the target Gaussian loss LG. By first calculating the process Gaussian loss of each decoding layer and then adding the Gaussian losses of the N decoding layers to obtain the target Gaussian loss, the Gaussian loss of the entire decoding process is accurately calculated.

[0112] In some optional implementations of this embodiment, step 407 includes: determining a first weight corresponding to the regression loss and a second weight corresponding to the target Gaussian loss; and calculating a comprehensive loss based on the regression loss, the first weight, the target Gaussian loss, and the second weight.

[0113] In this implementation, the execution entity sets corresponding weights for L1 and LG, namely, the first weight and the second weight, respectively, and then calculates Ltotal according to L1 and the first weight, LG and the second weight.

[0114] For example, if the first weight and the second weight are both set to 0.5, Ltotal can be calculated based on the following formula: Ltotal=0.5 L1+0.5 LG Of course, the values ​​of the first weight and the second weight may be set according to actual conditions, and this embodiment does not specifically limit this.

[0115] By setting corresponding weights for the regression loss and Gaussian loss, and obtaining the final comprehensive loss through weighted calculation, and using the comprehensive loss for model training, the training efficiency and accuracy of the trajectory prediction model are improved.

[0116] Step 408 , train the initial trajectory prediction model using the comprehensive loss to obtain a trajectory prediction model.

[0117] In this embodiment, the above-mentioned execution entity trains the embedding layer, encoder and decoder in the initial trajectory prediction model according to the comprehensive loss between the target predicted trajectory and the actual driving trajectory of the target vehicle, so as to obtain the trajectory prediction model.

[0118] from Figure 4 It can be seen that Figure 3Compared with the corresponding embodiment, the training method of the trajectory prediction model in this embodiment, the method has the step of training the initial trajectory prediction model, the method first generates the regression loss between the predicted trajectory and the true trajectory, and then obtains the overall target Gaussian loss according to the Gaussian loss of each layer of decoder, and finally uses the regression loss and the target Gaussian loss to generate the total loss, and adjusts the parameters of the embedding layer, encoder and decoder according to the total loss to obtain the trajectory prediction model, thereby improving the prediction accuracy of the trajectory prediction model.

[0119] Figure 5-1 An application block diagram of the training method of the trajectory prediction model disclosed in the present invention is shown. As can be seen from the block diagram, it includes the following parts: 1. Perform Agent Embedding (vehicle feature encoding) and Map Embedding (map feature encoding). Specifically, take Agent_in as the original data, whose dimension is [N, T, C], use MLP and MaxPooling to extract features and convert them, so as to obtain the vehicle trajectory feature Agent_p. Take Map_in as the original data, whose dimension is [M, N, C], use MLP and MaxPooling to extract features and convert them, so as to obtain the map trajectory feature Map_p.

[0120] 2. Take Agent_p and Map_p as input data, pass through the encoder, and output Gi, which includes the vehicle trajectory encoding Agent Feature and the map element encoding Map Feature. The Encoder includes multiple layers of MHA. Specifically, Agent_p and Map_p as well as the position encodings corresponding to Agent_p and Map_p are used as the query of MHA (abbreviated as q), the line drawing of the map element found by the KNN algorithm is used as the key of MHA (abbreviated as k), and the position encoding corresponding to the line drawing of the map element is used as the value of MHA (abbreviated as v), so as to obtain the output Gi of MHA.

[0121] Then, based on the dense future prediction task and MLP, an enhanced representation A of the Agent Feature is generated.

[0122] 3. Construct the intent query pair, i.e., the static query vector Q_i and the dynamic query vector Q_s. Specifically, kmeans clustering can be performed based on the terminal speed and terminal position of the predicted trajectory, so as to obtain Q_i through the corresponding position encoding and MLP. Q_s is obtained based on the corresponding position encoding and MLP of the intended target point.

[0123] 4. Output the predicted trajectory through the decoder. The Decoder includes multiple layers of MHA. Each MHA in the Decoder includes multiple sub-networks and output layers. The sub-networks are C_sa, C_A, C_M and C_j respectively, and the output layer is GMM (Gaussian Mixture Model).

[0124] Specifically, for C_sa, the query is the output C_j-1 and Q_i of the previous decoder, the key is C_j-1 and Q_i, and the value is C_j-1. C_sa can be obtained through the MHA.

[0125] For C_A, the query is C_sa and Q_s, the key is A and the position code corresponding to A, and the value is A. C_A can be obtained through the MHA.

[0126] For C_M, the query is C_sa and Q_s, and the key is the output of the dynamic map module (updated from MapFeature) and The corresponding position code, value is , C_M can be obtained through this MHA.

[0127] Afterwards, C_A and C_M are connected through multiple layers of MLP to obtain C_j.

[0128] After that, C_j can be passed through GMM to obtain the probability distribution of each position point in the predicted trajectory, and then the output of N decoding layers can be combined to obtain the final predicted trajectory.

[0129] 5. Calculate the L1 loss between the predicted trajectory and the actual driving trajectory; calculate the Gaussian loss of each decoding layer to obtain the overall Gaussian loss; obtain the target loss based on the L1 loss and the overall Gaussian loss; finally, use the target loss to adjust the parameters of the initial trajectory prediction model (including Embedding, Encoder, and Decoder) to obtain the trajectory prediction model.

[0130] Further reference 5-2, Figure 5-2 The structure of another decoder is shown. The Decoder also includes a multi-layer MHA. Each MHA in the Decoder includes multiple sub-networks and an output layer. The sub-networks are C_sa, C_A_M and C_j, and the output layer is GMM (Gaussian Mixture Model).

[0131] Specifically, we will first construct an intent query pair, namely the static query vector Q_i and the dynamic query vector Q_s. Specifically, we can perform kmeans clustering based on the terminal speed and terminal position of the predicted trajectory, and then obtain Q_i through the corresponding position encoding and MLP. We can obtain Q_s based on the corresponding position encoding and MLP of the intended target point.

[0132] Then, for C_sa, the query is the output C_j-1 and Q_i of the previous layer decoder, the key is C_j-1 and Q_i, and the value is C_j-1. C_sa can be obtained through the MHA.

[0133] For C_A_M, the query is C_sa and Q_s, the key is A and the position code corresponding to A and and The corresponding position code, value is A and , C_A_M can be obtained through this MHA.

[0134] Afterwards, C_A_M is connected to MLP through multiple layers of full connection to obtain C_j.

[0135] Specifically, it can be expressed as: MultiHeadAttn(query= + ,key=[ + ,value= ) MultiHeadAttn(query= , ,key=[ ,value= ) =MLP( ) Among them, concatenating C_sa and Q_s can get the input query vector of C_A_M, and concatenating A and The key vector can be obtained by splicing A and Concatenate to get the value vector.

[0136] Further reference 5-3, Figure 5-3 The structure of another decoder is shown. The decoder also includes multiple layers of MHA. Each MHA in the decoder includes a C_A_M_SA subnetwork, a C_j subnetwork and an output layer. The output layer is a GMM (Gaussian Mixture Model).

[0137] Specifically, we will first construct an intent query pair, namely the static query vector Q_i and the dynamic query vector Q_s. Specifically, we can perform kmeans clustering based on the terminal speed and terminal position of the predicted trajectory, and then obtain Q_i through the corresponding position encoding and MLP. We can obtain Q_s based on the corresponding position encoding and MLP of the intended target point.

[0138] Then, for C_A_M_SA, the query is the output of the previous decoder C_j-1, Q_i and Q_s, and the key is the position code corresponding to C_j-1, Q_i, A and A. and The corresponding position code, value is C_j-1, A and , C_AM_SA can be obtained through this MHA.

[0139] After that, C_AM_SA is connected to MLP through multiple layers of full connection to obtain C_j.

[0140] Specifically, it can be expressed as: MultiHeadAttn(query= ,key=[ ,value= ) =MLP(

[0141] Among them, the input query vector of C_AM_SA can be obtained by concatenating C_j-1, Q_i and Q_s, the key vector can be obtained by concatenating the position codes corresponding to C_j-1, Q_i, A and A, and the key vector can be obtained by concatenating C_j-1, A and Concatenate to get the value vector.

[0142] from Figure 5-2 and Figure 5-3 It can be seen from the figure that the sub-networks contained in the MHA of these two decoders are Figure 5-1 The sub-networks contained in the MHA in the decoder shown are different. Therefore, the process of decoding the encoded information is also different, and the accuracy of the decoded information obtained is also different. Specifically, through Figure 5-1 The decoding accuracy obtained by the Decoder in is greater than that obtained by Figure 5-2 The decoding accuracy obtained by the Decoder in is greater than that obtained by Figure 5-3 The decoding accuracy obtained by the Decoder in .

[0143] Figure 6A process 600 of an embodiment of a trajectory prediction method according to the present disclosure is shown. The trajectory prediction method comprises the following steps: Step 601, obtaining a first current driving trajectory of the target vehicle and a second current driving trajectory of the obstacle vehicle.

[0144] In this embodiment, the execution entity will obtain the current driving trajectory of the target vehicle (i.e., the own vehicle), i.e., the first current driving trajectory. The current driving trajectory can be a trajectory within a preset time from the current moment or within a preset distance from the current position. For example, the first current driving trajectory can be the driving trajectory of the own vehicle within one hour before the current moment. The first current driving trajectory can also be the driving trajectory of the own vehicle within three kilometers from the current distance.

[0145] Correspondingly, the execution entity will also obtain the current driving trajectory of the obstacle vehicle, that is, the second current driving trajectory. The second current driving trajectory can be the driving trajectory of the obstacle within one hour before the current moment, and the second current driving trajectory can also be the driving trajectory of the obstacle within three kilometers from the current distance.

[0146] Step 602: Input the first current driving trajectory and the second current driving trajectory into a trajectory prediction model, and output a predicted trajectory of the target vehicle.

[0147] In this embodiment, the execution subject inputs the first current driving trajectory and the second current driving trajectory into the trajectory prediction model, thereby outputting the predicted trajectory of the target vehicle, wherein the trajectory prediction model can be trained using the method described in the above embodiment. That is, the trajectory prediction model can predict the future driving trajectory of the vehicle based on the driving trajectory of the vehicle (i.e., the first current driving trajectory) and the driving trajectory of the obstacle vehicle (i.e., the second current driving trajectory), thereby outputting the predicted trajectory of the vehicle.

[0148] The trajectory prediction method provided by the embodiment of the present disclosure inputs a first current driving trajectory of the target vehicle and a second current driving trajectory of the obstacle vehicle into a trajectory prediction model, so that the trajectory prediction model predicts the trajectory of the vehicle to obtain a predicted trajectory, thereby integrating the trajectory information of the obstacle vehicle when predicting the trajectory, thereby improving the accuracy of the trajectory prediction.

[0149] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training device for a trajectory prediction model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0150] like Figure 7As shown, the training device 700 of the trajectory prediction model of this embodiment includes: an embedding module 701, an encoding module 702, a decoding module 703 and a training module 704. The embedding module 701 is configured to extract features of the historical trajectory information of the sample vehicle and the local map information associated with the historical trajectory information using the embedding layer to obtain vehicle trajectory features and map features, and the sample vehicles include target vehicles and obstacle vehicles; the encoding module 702 is configured to encode the vehicle trajectory features and map features using an encoder to obtain vehicle trajectory encoding and map element encoding; the decoding module 703 is configured to decode the vehicle trajectory encoding and map element encoding using a decoder to obtain a target predicted trajectory of the target vehicle; the training module 704 is configured to train the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

[0151] In this embodiment, the specific processing of the embedding module 701, the encoding module 702, the decoding module 703 and the training module 704 in the training device 700 of the trajectory prediction model and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of steps 101 - 104 in the corresponding embodiment are not repeated here.

[0152] In some optional implementations of this embodiment, the embedding layer includes a normalization layer and a conversion layer; and the embedding module 701 is further configured to: input historical trajectory information and local map information into the normalization layer, and output a vehicle trajectory matrix and a map element matrix; input the vehicle trajectory matrix and the map element matrix into the conversion layer, and output vehicle trajectory features and map features.

[0153] In some optional implementations of this embodiment, the encoder includes an encoding network and a regression output layer; and the encoding module 702 is further configured to: input vehicle trajectory features and map features into the encoding network, and output vehicle trajectory codes and map element codes; input the vehicle trajectory codes into the regression output layer, and output the initial predicted trajectory and predicted speed of the sample vehicle; perform local normalization and encoding conversion on the initial predicted trajectory and predicted speed to obtain vehicle predicted trajectory codes; and concatenate the vehicle predicted trajectory codes and the vehicle trajectory codes to obtain vehicle trajectory codes.

[0154] In some optional implementations of the present embodiment, the decoder includes N decoding layers, the decoding layer includes a decoding network and a decoding output layer, wherein N is a positive integer; and the decoding module 703 includes: a first decoding submodule, configured to, for each decoding layer, input the vehicle trajectory encoding, the map element encoding, and the pre-generated global intent vectorized representation and the auxiliary vectorized representation into the decoding network of the decoding layer, and output the decoded feature information; input the decoded feature information into the decoding output layer of the decoding layer, and output the probability distribution of each position point in the initial predicted trajectory; and a second decoding submodule, configured to generate a target predicted trajectory according to the probability distribution of each position point in the initial predicted trajectory output by the N decoding layers.

[0155] In some optional implementations of the present embodiment, the training device 700 of the above-mentioned trajectory prediction model also includes: a clustering module, configured to cluster the end points of the predicted trajectory to obtain multiple cluster centers; a first representation module, configured to use a fully connected layer to transform the position codes corresponding to the multiple cluster centers to obtain a global intention vectorized representation; a second representation module, configured to determine from the local map information a map element line that corresponds to the initial predicted trajectory and meets preset conditions, and generate an auxiliary vectorized representation based on the map element line.

[0156] In some optional implementations of this embodiment, the decoding network includes: a first decoding subnetwork, a second decoding subnetwork, a third decoding subnetwork and a decoding fully connected layer; and the first decoding submodule is further configured to: input the global intent vectorized representation into the first decoding subnetwork, and output the first decoding information; input the first decoding information, the auxiliary vectorized representation, the vehicle trajectory encoding and the position encoding corresponding to the vehicle trajectory encoding into the second decoding subnetwork, and output the second decoding information; input the first decoding information, the auxiliary vectorized representation, the map element line and the position encoding corresponding to the map element line into the third decoding subnetwork, and output the third decoding information; input the second decoding information and the third decoding information into the decoding fully connected layer, and output the decoded feature information.

[0157] In some optional implementations of this embodiment, the training module 704 includes: a regression loss calculation submodule, configured to calculate the regression loss between the target predicted trajectory and the actual driving trajectory; a comprehensive loss calculation submodule, configured to generate a comprehensive loss based on the regression loss and the target Gaussian loss; and a training submodule, configured to use the comprehensive loss to train the initial trajectory prediction model to obtain a trajectory prediction model.

[0158] In some optional implementations of this embodiment, the above-mentioned trajectory prediction model training device 700 also includes: a Gaussian loss calculation module, configured to calculate the Gaussian regression loss of each decoding layer; add the Gaussian regression losses of N decoding layers to obtain the target Gaussian loss.

[0159] In some optional implementations of this embodiment, the comprehensive loss calculation submodule is further configured to: determine a first weight corresponding to the regression loss and a second weight corresponding to the target Gaussian loss; and calculate the comprehensive loss based on the regression loss, the first weight, the target Gaussian loss, and the second weight.

[0160] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a trajectory prediction device. Figure 6 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0161] like Figure 8 As shown, the trajectory prediction device 800 of this embodiment includes: an acquisition module 801 and a prediction module 802. The acquisition module 801 is configured to acquire a first current driving trajectory of the target vehicle and a second current driving trajectory of the obstacle vehicle; the prediction module 802 is configured to input the first current driving trajectory and the second current driving trajectory into a trajectory prediction model, and output a predicted trajectory of the target vehicle, and the trajectory prediction model is trained using the method described in the above embodiment.

[0162] In this embodiment, the specific processing of the acquisition module 801 and the prediction module 802 in the trajectory prediction device 800 and the technical effects thereof can be referred to in Figure 6 The relevant descriptions of steps 601-602 in the corresponding embodiment are not repeated here.

[0163] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle.

[0164] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers and other suitable devices with computing functions. The electronic device can also represent various forms of mobile devices suitable for performing calculations. In particular, the electronic device can also be an electronic device integrated in a vehicle, such as a vehicle computer.

[0165] The components shown herein, their connections and relationships, and their functions, are intended only as examples, and are not meant to limit implementations of the disclosure described and / or claimed herein.

[0166] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0167] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc. It should be understood that the input and output units listed here are only examples that exist in some special scenarios. In some other application scenarios, the input and output units can be in other forms. For example, when the device is a car device, the input unit can be a touch screen, a rotary button, etc.

[0168] The communication unit 909 , such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a training method or a trajectory prediction method for a trajectory prediction model. For example, in some embodiments, the training method or the trajectory prediction method for a trajectory prediction model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method or the trajectory prediction method for the trajectory prediction model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the trajectory prediction model training method or the trajectory prediction method in any other appropriate manner (for example, by means of firmware).

[0170] The autonomous driving vehicle provided by the present disclosure may include: Fig. 9 The electronic device shown above can implement the trajectory prediction model training method or trajectory prediction method described in any of the above embodiments when executed by its processor.

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0176] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a trajectory prediction model, wherein: The initial trajectory prediction model includes an embedding layer, an encoder and a decoder, and the method includes: Using the embedding layer to extract features from historical trajectory information of sample vehicles and local map information associated with the historical trajectory information to obtain vehicle trajectory features and map features, wherein the sample vehicles include target vehicles and obstacle vehicles; Encoding the vehicle trajectory feature and the map feature using the encoder to obtain a vehicle trajectory code and a map element code; Decoding the vehicle trajectory code and the map element code using the decoder to obtain a target predicted trajectory of the target vehicle; The initial trajectory prediction model is trained according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

2. The method according to claim 1, wherein: The embedding layer includes a normalization layer and a conversion layer; and The method of using the embedding layer to extract features from the historical trajectory information of the sample vehicle and the local map information associated with the historical trajectory information to obtain vehicle trajectory features and map features includes: Inputting the historical trajectory information and the local map information into the normalization layer, and outputting a vehicle trajectory matrix and a map element matrix; The vehicle trajectory matrix and the map element matrix are input into the conversion layer, and the vehicle trajectory features and the map features are output.

3. The method according to claim 1, wherein: The encoder includes an encoding network and a regression output layer; as well as The step of encoding the vehicle trajectory feature and the map feature by using the encoder to obtain a vehicle trajectory code and a map element code includes: Inputting the vehicle trajectory feature and the map feature into the encoding network, and outputting the vehicle trajectory code and the map element code; Inputting the vehicle trajectory code into the regression output layer, and outputting the initial predicted trajectory and predicted speed of the sample vehicle; Performing local normalization processing and encoding conversion on the initial predicted trajectory and predicted speed to obtain a vehicle predicted trajectory encoding; The vehicle prediction trajectory code and the vehicle trajectory code are concatenated to obtain the vehicle trajectory code.

4. The method according to claim 3, wherein: The decoder comprises N decoding layers, each of which comprises a decoding network and a decoding output layer, wherein N is a positive integer; and The step of decoding the vehicle trajectory code and the map element code by using the decoder to obtain the target predicted trajectory of the target vehicle includes: For each decoding layer, the vehicle trajectory code, the map element code, and the pre-generated global intent vectorized representation and auxiliary vectorized representation are input into the decoding network of the decoding layer, and decoded feature information is output; the decoded feature information is input into the decoding output layer of the decoding layer, and the probability distribution of each position point in the initial predicted trajectory is output; The target prediction trajectory is generated according to the probability distribution of each position point in the initial prediction trajectory output by the N decoding layers.

5. The method according to claim 4, wherein: The method further comprises: Clustering the endpoints of the predicted trajectory to obtain multiple cluster centers; Transforming the position codes corresponding to the multiple cluster centers using a fully connected layer to obtain a vectorized representation of the global intent; A map element line corresponding to the initial predicted trajectory and satisfying a preset condition is determined from the local map information, and the auxiliary vectorized representation is generated based on the map element line.

6. The method according to claim 5, wherein: The decoding network includes: a first decoding subnetwork, a second decoding subnetwork, a third decoding subnetwork and a decoding fully connected layer; and The step of inputting the vehicle trajectory code, the map element code, and the pre-generated global intent vectorized representation and auxiliary vectorized representation into the decoding network of the decoding layer, and outputting decoded feature information, comprises: Inputting the global intent vectorized representation into the first decoding subnetwork, and outputting first decoding information; Inputting the first decoding information, the auxiliary vectorized representation, the vehicle trajectory code, and the position code corresponding to the vehicle trajectory code into the second decoding subnetwork, and outputting second decoding information; Inputting the first decoding information, the auxiliary vectorized representation, the map element line, and the position code corresponding to the map element line into the third decoding subnetwork, and outputting the third decoding information; The second decoded information and the third decoded information are input into the decoding fully connected layer, and the decoded feature information is output to obtain the decoded feature information.

7. The method according to claim 4, wherein: The initial trajectory prediction model is trained according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain the trajectory prediction model, including: Calculating the regression loss between the target predicted trajectory and the actual driving trajectory; Generate a comprehensive loss based on the regression loss and the target Gaussian loss; The initial trajectory prediction model is trained using the comprehensive loss to obtain the trajectory prediction model.

8. The method according to claim 7, wherein: The method further comprises: For each decoding layer, calculating the Gaussian regression loss of the decoding layer; The Gaussian regression losses of the N decoding layers are added together to obtain the target Gaussian loss.

9. The method according to claim 7, wherein: The comprehensive loss is generated according to the regression loss and the target Gaussian loss, including: Determine a first weight corresponding to the regression loss and a second weight corresponding to the target Gaussian loss; The comprehensive loss is calculated according to the regression loss, the first weight, the target Gaussian loss, and the second weight.

10. A trajectory prediction method, comprising: Acquire a first current driving trajectory of the target vehicle and a second current driving trajectory of the obstacle vehicle; The first current driving trajectory and the second current driving trajectory are input into a trajectory prediction model, and the predicted trajectory of the target vehicle is output, wherein the trajectory prediction model is trained using the method described in any one of claims 1 to 9.

11. A training device for a trajectory prediction model, wherein: The initial trajectory prediction model includes an embedding layer, an encoder and a decoder, and the device includes: An embedding module is configured to extract features from historical trajectory information of sample vehicles and local map information associated with the historical trajectory information using the embedding layer to obtain vehicle trajectory features and map features, wherein the sample vehicles include target vehicles and obstacle vehicles; an encoding module, configured to encode the vehicle trajectory feature and the map feature using the encoder to obtain a vehicle trajectory code and a map element code; A decoding module is configured to decode the vehicle trajectory code and the map element code using the decoder to obtain a target predicted trajectory of the target vehicle; The training module is configured to train the initial trajectory prediction model according to the loss between the target predicted trajectory and the actual driving trajectory of the target vehicle to obtain a trajectory prediction model.

12. A trajectory prediction device, comprising: An acquisition module is configured to acquire a first current driving trajectory of the target vehicle and a second current driving trajectory of the obstacle vehicle; The prediction module is configured to input the first current driving trajectory and the second current driving trajectory into a trajectory prediction model, and output a predicted trajectory of the target vehicle, wherein the trajectory prediction model is trained using the method described in any one of claims 1 to 9.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

16. An autonomous driving vehicle comprising the electronic device as claimed in claim 13.

Images