Trajectory prediction method, trajectory prediction model training method and device

Abstract

The embodiment of the invention provides a trajectory prediction method, a trajectory prediction model training method and device, electronic equipment and a storage medium, and belongs to the technical field of trajectory prediction. The method comprises the following steps: generating a current hidden state through a long-short term memory encoder according to input data at the current moment and historical track data; generating a feature vector from the current hidden state and the historical data coding information through a model layer; inputting the feature vector into a soft maximization function to obtain a moving trajectory probability; and generating a prediction trajectory and parameters of two-dimensional Gaussian distribution according to the feature vector and the moving trajectory probability through a long-short-term memory decoder. According to the embodiment of the invention, the accuracy of trajectory prediction can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of trajectory prediction, and in particular to a trajectory prediction method, a trajectory prediction model training method and a device. Background Art

[0002] With the development of economy and technology, it is of great significance to accurately predict the movement trajectory of intelligent devices to achieve motion obstacle avoidance and make reasonable decisions and safety plans during the movement.

[0003] In related technologies, trajectory prediction methods are basically based on physical models, but the accuracy of physical models will decrease in complex road conditions or motion environments.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a trajectory prediction method and device, an electronic device and a storage medium, aiming to improve the accuracy of trajectory prediction.

[0006] To achieve the above object, an embodiment of the present application provides a trajectory prediction method, which includes the following steps:

[0007] Generate a current hidden state according to input data at a current moment and historical trajectory data through a long short-term memory encoder, wherein the input data at the current moment includes current moment features of the smart device and historical features of the smart device, and the historical trajectory data indicates the state of the predicted object at a historical time point;

[0008] Generate a feature vector by using the current hidden state and historical data encoding information through a model layer, wherein the model layer includes a Gaussian density model-transformer layer;

[0009] Inputting the feature vector into a soft maximization function to obtain a running trajectory probability;

[0010] A long short-term memory decoder is used to generate a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability.

[0011] In some embodiments, the method further comprises:

[0012] Acquire current moment features of the smart device, wherein the current moment features include a relative lateral position of the smart device and the predicted object, a relative longitudinal position of the smart device and the predicted object, a speed of the smart device, and an acceleration of the smart device;

[0013] Obtaining historical characteristics of the smart device;

[0014] The input data at the current moment is generated according to the current moment feature and the historical feature of the smart device.

[0015] In some embodiments, the expression of the current moment feature is as follows:

[0016] C t ={(x 0,t ,y 0,t ,v 0,t ,a 0,t ),(x 1,t ,y 1,t ,v 1,t ,a 1,t ),...(x n,t ,y n,t ,v n,t ,a n,t )},

[0017] Among them, C t is the current moment feature, x n,t is the relative horizontal position between the smart device with sequence number n and the predicted object at time t, is the relative vertical position between the smart device with sequence number n and the predicted object at time t, v n,t is the speed of the smart device with serial number n at time t, a n,t is the acceleration of the smart device with sequence number n at time t, where n is a natural number.

[0018] In some embodiments, the expression of the predicted trajectory is as follows:

[0019]

[0020] Where P represents the predicted trajectory, is the future time point t+1, t+2, …, t+t f The corresponding trajectory position, t+t f Represents the tth relative to time t f moment, t f The value is an integer.

[0021] In some embodiments, the running trajectory probability includes a running trajectory type and a trajectory probability, and the generating of a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability by a long short-term memory decoder includes:

[0022] Determine the running trajectory type with the largest trajectory probability as the prediction type;

[0023] A prediction trajectory and parameters of a two-dimensional Gaussian distribution are generated according to the feature vector and the prediction type through a long short-term memory decoder.

[0024] An aspect of an embodiment of the present application provides a trajectory prediction method, the method comprising the following steps:

[0025] Collecting vehicle trajectory data of different scenarios, and generating a sample data set according to the vehicle trajectory data, wherein the sample data set includes a training set, a validation set, and a test set;

[0026] The trajectory prediction model is trained according to the sample data set, the mean square error loss function and the negative log-likelihood loss function to obtain the trained trajectory prediction model, and the trained trajectory prediction model is used to perform trajectory prediction through the above-mentioned trajectory prediction method.

[0027] In some embodiments, the step of training the trajectory prediction model according to the sample data set, the mean square error loss function, and the negative log-likelihood loss function to obtain the trained trajectory prediction model includes:

[0028] Training the trajectory prediction model according to the sample data set;

[0029] Calculating the mean square error loss function of the trajectory prediction model;

[0030] Training the trajectory prediction model according to the mean square error loss function;

[0031] Training the trajectory prediction model according to the sample data set;

[0032] Calculating the negative log-likelihood loss function of the trajectory prediction model;

[0033] Training a trajectory prediction model according to the negative log-likelihood loss function;

[0034] The trained trajectory prediction model is obtained.

[0035] To achieve the above object, another aspect of an embodiment of the present application provides a trajectory prediction device, the device comprising:

[0036] A long short-term memory encoding module is used to generate a current hidden state according to the input data at the current moment and the historical trajectory data through a long short-term memory encoder;

[0037] A feature vector generation module, used to generate a feature vector by using the current hidden state and the historical data encoding information through a model layer, wherein the model layer includes a Gaussian density model-transformer layer;

[0038] A trajectory probability generation module, used for inputting the feature vector into a soft maximization function to obtain a running trajectory probability;

[0039] The long short-term memory decoding module is used to generate a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability through a long short-term memory decoder.

[0040] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0041] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0042] The trajectory prediction method and device, electronic device and storage medium of the embodiments of the present application include at least the following beneficial effects: the scheme generates a current hidden state according to the input data at the current moment and the historical trajectory data through a long short-term memory encoder. The long short-term memory encoding can effectively process time data, which helps to improve the accuracy of trajectory prediction; the current hidden state and the historical data encoding information are used to generate a feature vector through the model layer, which helps to model the continuity and distribution of the data and improve the prediction accuracy of the model; the feature vector is input into the soft maximization function to obtain the running trajectory probability; the predicted trajectory and the parameters of the two-dimensional Gaussian distribution are generated according to the feature vector and the running trajectory probability through a long short-term memory decoder, and the trajectory probability is combined with the historical information to generate a more realistic and accurate trajectory prediction.

[0043] The trajectory prediction model training method of the embodiment of the present application includes at least the following beneficial effects: the scheme collects vehicle trajectory data of different scenarios and generates a sample data set according to the vehicle trajectory data; the trajectory prediction model is trained according to the sample data set, the mean square error loss function and the negative log-likelihood loss function, which can handle complex trajectory changes in different scenarios and optimize the continuity and accuracy of the trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a trajectory prediction method provided in an embodiment of the present application;

[0045] Figure 2 is a flowchart of the step of generating input data of the trajectory prediction method provided in an embodiment of the present application;

[0046] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0047] Figure 4 is a flow chart of a trajectory prediction model training method provided in an embodiment of the present application;

[0048] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0049] Figure 6 is an optional flow chart of applying the trajectory prediction method provided in an embodiment of the present application to a trajectory prediction model;

[0050] Figure 7 It is a schematic diagram of the RMSE comparison table of different prediction models provided in the embodiment of the present application on the test set;

[0051] Figure 8 It is a schematic diagram of a table of data set division and comparison results provided in an embodiment of the present application;

[0052] Fig. 9 is a schematic diagram of the structure of a trajectory prediction device provided in an embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0055] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0056] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0059] 1) Long Short-Term Memory (LSTM) is a special RNN (recurrent neural network) structure that aims to solve the gradient vanishing or gradient exploding problems that occur when traditional RNNs deal with long-distance dependency problems.

[0060] With the development of economy and technology, it is of great significance to accurately predict the movement trajectory of intelligent devices to achieve motion obstacle avoidance and make reasonable decisions and safety plans during the movement.

[0061] In related technologies, trajectory prediction methods are basically based on physical models, but the accuracy of physical models will decrease in complex road conditions or motion environments.

[0062] In summary, the technical problems existing in the relevant technologies need to be improved.

[0063] In view of this, a trajectory prediction method, apparatus, device and medium are provided in an embodiment of the present application. The scheme generates a current hidden state according to the input data at the current moment and the historical trajectory data through a long short-term memory encoder. The long short-term memory encoding can effectively process time data, which helps to improve the accuracy of trajectory prediction; the current hidden state and the historical data encoding information are used to generate a feature vector through a model layer, which helps to model the continuity and distribution of the data and improve the prediction accuracy of the model; the feature vector is input into a soft maximization function to obtain the running trajectory probability; the predicted trajectory and the parameters of the two-dimensional Gaussian distribution are generated according to the feature vector and the running trajectory probability through a long short-term memory decoder, and the trajectory probability is combined with the historical information to generate a more realistic and accurate trajectory prediction.

[0064] The present application provides a trajectory prediction model training method, which collects vehicle trajectory data in different scenarios and generates a sample data set based on the vehicle trajectory data; trains the trajectory prediction model based on the sample data set, mean square error loss function and negative log-likelihood loss function, which can handle complex trajectory changes in different scenarios and optimize the continuity and accuracy of the trajectory.

[0065] The trajectory prediction method and trajectory prediction model training method provided in the embodiments of the present application relate to the field of trajectory prediction technology. The trajectory prediction method and trajectory prediction model training method provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements the trajectory prediction method and the trajectory prediction model training method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Figure 1is an optional flowchart of the trajectory prediction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0069] Step S101, generating a current hidden state according to the input data at the current moment and the historical trajectory data through a long short-term memory encoder.

[0070] Specifically, the input data at the current moment includes the current moment features of the smart device and the historical features of the smart device, and the historical trajectory data represents the predicted state of the object at the historical time point.

[0071] In some embodiments, a long short-term memory encoder (i.e., LSTM encoder) generates a current hidden state based on the input data at the current moment and the historical trajectory data. The long short-term memory encoder maintains long-term dependencies through its memory cells, thereby being able to capture long-term dependent patterns in the data.

[0072] Optionally, obtain the current moment feature of the smart device, obtain the historical feature of the smart device; generate the current moment feature according to the smart device information and the historical feature of the smart device; generate the input data of the current moment according to the current moment feature and the historical feature of the smart device. The smart device information includes the relative horizontal position of the smart device and the predicted object, the relative vertical position of the smart device and the predicted object, the speed of the smart device, and the acceleration of the smart device.

[0073] In this embodiment, the current hidden state is generated according to the input data at the current moment and the historical trajectory data through the long short-term memory encoder, which is beneficial to process data, maintain long-term dependencies, and help to accurately represent and predict trajectories.

[0074] Step S102, the current hidden state and the historical data encoding information are used to generate a feature vector through the model layer.

[0075] Specifically, the model layer includes a Gaussian density model-transformer layer.

[0076] In some embodiments, the Gaussian density model models the probability distribution of data points in the feature space, while the transformer layer models the input features through a self-attention mechanism to generate a richer feature representation.

[0077] It should be noted that the Gaussian density model assumes that each point of the trajectory can be represented by a mixture of multiple Gaussian distributions. The transformer layer can better capture long-range dependencies while improving processing power.

[0078] In this embodiment, by generating a feature vector through a model layer by encoding information of the current hidden state and historical data, the discriminative power of the feature vector can be improved, thereby improving the prediction accuracy of the model when processing complex input data.

[0079] Step S103, input the feature vector into the soft maximization function to obtain the running trajectory probability.

[0080] Specifically, the running trajectory probability includes the running trajectory type and the trajectory probability.

[0081] In some embodiments, the soft maximization function is improved to convert the feature vector into the running trajectory type and its corresponding trajectory probability.

[0082] The soft maximization function maps the feature vector to a probability space so that each element represents the probability of the corresponding trajectory. The output of the soft maximization function is the probability distribution of different trajectory types.

[0083] Furthermore, each feature is exponentially normalized by a softmax function so that the output is a probability distribution (the sum of all probabilities is 1).

[0084] It can be understood that if there are multiple categories in trajectory prediction, a multi-classification soft maximization function can be used to handle the probabilities of these categories.

[0085] In this embodiment, the feature vector is input into the soft maximization function to obtain the running trajectory probability. The probability distribution is generated by the maximization function to make the prediction result more flexible, thereby improving the diversity and accuracy of the predicted trajectory.

[0086] Step S104, generating a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability through a long short-term memory decoder.

[0087] Specifically, the parameters of the two-dimensional Gaussian distribution include a mean vector, a covariance matrix, a standard deviation, and a weight.

[0088] In some embodiments, an LSTM decoder (i.e., long short-term memory decoder) generates a predicted trajectory based on the feature vector generated by the encoder and the running trajectory probability.

[0089] The decoder gradually generates future trajectory points by inputting the feature vector and the predicted probability into an LSTM network.

[0090] It can be understood that the LSTM decoder not only relies on the information of the current moment, but also guides the trajectory generation through the trajectory probability to improve the consistency of the generated trajectory in the entire sequence.

[0091] Furthermore, in the process of trajectory prediction, parameters of a two-dimensional Gaussian distribution are generated, wherein the parameters of the two-dimensional Gaussian distribution are used to further model the predicted trajectory and describe the spatial position distribution of the trajectory points.

[0092] It is understandable that the parameters of the two-dimensional Gaussian distribution can be regressed through a deep network.

[0093] Optionally, the parameters of a two-dimensional Gaussian distribution are predicted by the output of the LSTM decoder.

[0094] The mean represents the expected position of the trajectory points, and the covariance matrix represents the uncertainty of the predicted position.

[0095] In this embodiment, a long short-term memory decoder is used to generate a predicted trajectory and parameters of a two-dimensional Gaussian distribution based on the feature vector and the running trajectory probability. The trajectory probability is combined with historical information, which is conducive to generating a more accurate trajectory prediction. The two-dimensional Gaussian distribution can take into account position fluctuations and deviations when generating the trajectory.

[0096] In steps S101 to S104 shown in the embodiment of the present application, a current hidden state is generated according to the input data at the current moment and the historical trajectory data through a long short-term memory encoder. The long short-term memory encoding can effectively process time data, which helps to improve the accuracy of trajectory prediction; the current hidden state and the historical data encoding information are used to generate a feature vector through a model layer, which helps to model the continuity and distribution of the data and improve the prediction accuracy of the model; the feature vector is input into a soft maximization function to obtain the running trajectory probability; a predicted trajectory and parameters of a two-dimensional Gaussian distribution are generated according to the feature vector and the running trajectory probability through a long short-term memory decoder, and the trajectory probability is combined with historical information to generate a more realistic and accurate trajectory prediction.

[0097] See also Figure 2 In some embodiments, the trajectory prediction method provided in the embodiment of the present application further includes a step of generating input data, and the step of generating input data may include but is not limited to steps S201 to S203:

[0098] Step S201, obtaining the current moment characteristics of the smart device.

[0099] Specifically, the current moment feature includes the smart device information at the current moment. The smart device is used to obtain sensor data related to the object (such as position, speed, acceleration, etc.) and perform predictive calculations. The smart device information includes the relative lateral position of the smart device and the predicted object, the relative longitudinal position of the smart device and the predicted object, the speed of the smart device and the acceleration of the smart device. The predicted object refers to the entity for which trajectory prediction is required.

[0100] By way of example, smart devices include smart phones, wearable devices, vehicles, drones, surveillance cameras, and Internet of Things (IoT) devices, etc. Predicted objects include vehicles, drones, robots, and organisms, etc.

[0101] In some embodiments, the relative lateral position, relative longitudinal position, speed and acceleration of the smart device and the predicted object are acquired through sensors or smart devices.

[0102] Specifically, the expression of the current moment feature is as follows:

[0103] C t ={(x 0,t ,y 0,t ,v 0,t ,a 0,t ),(x 1,t ,y 1,t ,v 1,t ,a 1,t ),...(x n,t ,y n,t ,v n,t ,a n,t )},

[0104] Among them, C t is the current moment feature, x n,t is the relative horizontal position of the smart device with sequence number n and the predicted object at time t, is the relative vertical position of the smart device with sequence number n and the predicted object at time t, v n,t is the speed of the smart device with serial number n at time t, a n,t is the acceleration of the smart device with serial number n at time t.

[0105] Step S202, obtaining historical features of the smart device.

[0106] Specifically, the historical features of the smart device include the smart device information at historical moments.

[0107] In some embodiments, historical features of the smart device within a time length are obtained, wherein the time length may be preset or dynamically set by the user.

[0108] Step S203: generating input data at the current moment according to the current moment characteristics and the historical characteristics of the smart device.

[0109] In step S203 of some embodiments, the expression of the input data at the current moment is as follows:

[0110]

[0111] Among them, I t Represents the input data at the current moment, th Indicates the length of time, C t Represents the current moment characteristics, Represents the historical characteristics of a smart device.

[0112] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S302:

[0113] Step S301: determine the running trajectory type with the highest trajectory probability as the prediction type.

[0114] Specifically, the running trajectory probability includes the running trajectory type and the trajectory probability.

[0115] In step S301 of some embodiments, the running trajectory type with the largest trajectory probability among the running trajectory probabilities is determined as the predicted type.

[0116] In this embodiment, the running trajectory type with the highest trajectory probability is determined as the prediction type, so that the most appropriate trajectory can be selected and the error can be reduced.

[0117] Step S302 , generating a prediction trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the prediction type through a long short-term memory decoder.

[0118] In step S302 of some embodiments, the current feature vector and the prediction type are input into a long short-term memory decoder to generate a prediction trajectory.

[0119] Furthermore, in trajectory prediction, in order to better describe the uncertainty of the trajectory, a two-dimensional Gaussian distribution is used to model the position distribution of the trajectory.

[0120] Optionally, the LSTM decoder also outputs the mean and covariance of the trajectory position distribution, which are used to estimate the uncertainty of the trajectory.

[0121] In this embodiment, the prediction trajectory and the parameters of the two-dimensional Gaussian distribution are generated according to the feature vector and the prediction type through the long short-term memory decoder, which can help the system consider uncertainty when making decisions and avoid over-reliance on a single prediction, thereby improving the robustness and flexibility of the system.

[0122] Figure 4 is an optional flowchart of the trajectory prediction model training method provided in the embodiment of the present application. Figure 4 The method may include but is not limited to steps S401 to S402.

[0123] Step S401 , collecting vehicle trajectory data of different scenarios, and generating a sample data set according to the vehicle trajectory data.

[0124] Specifically, the sample data set includes a training set, a validation set, and a test set.

[0125] In step S401 of some embodiments, the sample data set sources include road sections with high and low traffic or other sports scenes, and data on light, moderate and congested conditions in different road sections. Each data set is 1 hour long, with a sampling time of 1 second and a second sampling interval of 2 seconds. The sliding window method is used, with the current frame as the origin, the historical features of the first 30 seconds are used as the model input, and the future trajectory of the last 50 seconds is used for prediction.

[0126] Furthermore, considering the diversity of samples, such as the high proportion of linear motion, 100,000 sample points are randomly selected from each type of sample set, totaling 300,000, to achieve sample balance. 80% of the processed data set is used as a training set, 10% as a validation set, and 10% as a test set. The other data will be used only for comparison.

[0127] Step S402 , training the trajectory prediction model according to the sample data set, the mean square error loss function and the negative log-likelihood loss function to obtain the trained trajectory prediction model.

[0128] Specifically, the trained trajectory prediction model is used to perform trajectory prediction through the above trajectory prediction method.

[0129] In step S402 of some embodiments, the mean square error (MSE) is used as the loss function for pre-training in the first training round, and the negative log-likelihood function NLL is used as the loss function for training in the second training round.

[0130] It is understandable that the number of the first training rounds and the number of the second training rounds may be predetermined or dynamically set by the user.

[0131] Exemplarily, the training strategy for data set sampling pre-training uses mean square error (MSE) as the loss function for pre-training in the first 10 rounds, and uses negative log-likelihood function NLL as the loss function for training in the next 5 rounds.

[0132] See also Figure 5 In some embodiments, step S402 may also include but is not limited to steps S501 to S509:

[0133] Step S501, determine whether the number of training rounds exceeds the first number of training rounds, if yes, execute step S505, if no, execute step S502.

[0134] It is understandable that when the model is first trained, the initial number of training rounds is 1.

[0135] In step S501 of some embodiments, the first number of training rounds is obtained, and each time the training process is performed, it is first checked whether the current number of training rounds is greater than the first number of training rounds. If so, jump directly to step S505. If not, continue to execute step S502.

[0136] The number of first training rounds may be preset or dynamically adjusted according to the training state of the model. For example, an early stopping strategy may be used to terminate the training in advance when the loss function no longer changes significantly.

[0137] Step S502: training a trajectory prediction model according to a sample data set.

[0138] In step S502 of some embodiments, a trajectory prediction model is trained according to the sample data set. For example, a supervised learning algorithm may be used to train the trajectory prediction model.

[0139] Step S503, calculating the mean square error loss function of the trajectory prediction model.

[0140] In step S503 of some embodiments, the gap between the model prediction value and the true value is measured by the mean square error (MSE) loss function.

[0141] Furthermore, the weighted mean square error (W-MSE) can be used to deal with the importance of different samples, or combined with other loss functions for multi-task learning.

[0142] Step S504, training the trajectory prediction model according to the mean square error loss function, and then executing step S509.

[0143] In step S504 of some embodiments, the parameters of the trajectory prediction model are adjusted by using a mean square error loss function through an optimization algorithm so that the prediction result of the model is closer to the true value. Exemplarily, the optimization algorithm can use batch gradient descent (BGD), stochastic gradient descent (SGD), Adam optimizer, etc.

[0144] Step S505, determining whether the number of training rounds exceeds the second number of training rounds, if yes, the process ends, if no, executing step S506.

[0145] In step S505 of some embodiments, the second number of training rounds is obtained. Each time the training process is performed, it is first checked whether the current number of training rounds is greater than the second number of training rounds. If so, the process ends and a trained trajectory prediction model is obtained. If not, step S506 is continued.

[0146] The number of the second training rounds can be preset or dynamically adjusted according to the training state of the model. For example, an early stopping strategy can be used to terminate the training in advance when the loss function no longer changes significantly.

[0147] Step S506: training a trajectory prediction model according to the sample data set.

[0148] In step S506 of some embodiments, the model is retrained using a negative log-likelihood loss function.

[0149] Step S507, calculating the negative log-likelihood loss function of the trajectory prediction model.

[0150] In step S507 of some embodiments, a negative log-likelihood loss function of the trajectory prediction model is calculated. Further, a weighted negative log-likelihood loss may be selected.

[0151] Step S508, training the trajectory prediction model according to the negative log-likelihood loss function, and then executing step S509.

[0152] In step S508 of some embodiments, an optimization algorithm such as gradient descent is used to minimize the negative log-likelihood loss function and adjust the model parameters.

[0153] Step S509, the number of training rounds is increased by one, and then step S501 is executed.

[0154] It can be understood that each time a complete training is completed, the current number of training rounds is increased by 1, and the process returns to step S501 to perform a new round judgment.

[0155] Figure 6 is an optional flow chart of applying the trajectory prediction method provided in the embodiment of the present application to a trajectory prediction model. Figure 6 The method may include but is not limited to the following steps.

[0156] Step 1: Data preprocessing.

[0157] In some embodiments, the data set comes from sections with high and low traffic or other sports scenes, and data on mild, moderate and congested conditions in different sections. Each data set is 1 hour long, with a sampling time of 1 second and a second sampling interval of 2 seconds. The sliding window method is used, with the current frame as the origin, and the historical features of the first 30 seconds are used as model input, and the future trajectory of the last 50 seconds is used for prediction. Taking into account the diversity of samples, such as a large proportion of linear motion, 100,000 sample points are randomly selected from each type of sample set, totaling 300,000, to achieve sample balance, and 80% of the processed data set is used as a training set, 10% as a validation set, and 10% as a test set. The other data will only be used for comparison.

[0158] It is understandable that considering the total sample diversity, for moving smart devices, only the two-dimensional plane is considered here. The device motion trajectory interacts with the surrounding environment. Device trajectory prediction needs to learn the potential impact of this interaction. Therefore, the model input information includes the impact of historical information and various parameter information of the surrounding device environment.

[0159] Among them, the input features of the model are:

[0160] Input features of the model I t Including surrounding intelligence setting time length t h The historical characteristics and forecasts of the device, as well as the current moment C t The historical characteristics are as follows:

[0161] C t ={(x 0,t ,y 0,t ,v 0,t ,a 0,t ),(x 1,t ,y 1,t ,v 1,t ,a 1,t ),...(x n,t ,y n,t ,v n,t ,a n,t )}, (1.1)

[0162] Among them, x n,t ,y n,t are the relative lateral and longitudinal positions of the vehicle (i.e. the predicted object) and the predicted smart device at time t, n=0 is the original node predicted by the smart device, and 1 to n are the range of 10*10 before and after the predicted smart device and the adjacent device; v n,t ,a n,t They are the speed and acceleration of the smart device respectively.

[0163]

[0164] Furthermore, the output features of the model are:

[0165] Based on the above input features, the model will output the probabilities of the three motion trajectories (KS, LKS, RKS), and output the corresponding length t accordingly. p The future trajectory is as follows:

[0166]

[0167] W = {x 0,r+i ,y 0,t+i},(1.4)

[0168] Among them, P represents the predicted trajectory, W represents the position of the vehicle at different time points, and x 0,t+i Indicates the coordinate y of the object on the x-axis at time t+i 0,t+i represents the coordinate of the object on the y-axis at time t+i, t+t f Represents the tth relative to time t f moment, t f The value is an integer.

[0169] Step 2: GDN-Transformer-LSTM model construction.

[0170] Specifically, the trajectory prediction model adopts the GDN-Transformer-LSTM model.

[0171] It should be noted that the GDN-Transformer-LSTM model consists of an LSTM codec and a GDN-Transformer layer. The future trajectory of the vehicle predicted by this application is used for subsequent lane change detection and quantification. It is impossible to completely and accurately predict the complex distribution of the future vehicle driving trajectory using the output form of a single point or a single trajectory line in the conventional vehicle trajectory prediction model, and it cannot meet the requirements of this application for lane change decision-making through quantification. The Gaussian density model GDN combines traditional neural networks and can model data relationships with non-determinism and multimodality. GDN combines multiple Gaussian probability density distributions to change the original single position information output into a probability density function of a tensor, which is of great value in precise applications.

[0172] Among them, the construction is based on the GDN-Transformer layer, which fully utilizes the advantages of Transformer in trajectory prediction. At the same time, the Gaussian density model is introduced, so that the trajectory prediction model no longer outputs a single future position of the motion trajectory, but outputs the probability distribution density of the future position fitted by multiple Gaussian functions, reflecting the uncertainty of the object trajectory prediction results, laying the foundation for the subsequent detection and quantification of potential motion lane changes.

[0173] It can be understood that this model obtains the result by combining multiple Gaussian probability functions to obtain the probability density and provide robustness for the prediction of uncertain results. The position prediction result output by the model at a certain moment obeys the bivariate Gaussian distribution. If each single-dimensional variable is set to be independently distributed, the output result of the model at a certain moment is:

[0174] v u =(v ux ,v iy ),w i (2.1)

[0175]

[0176] Among them, v i is the component vector of the ith Gaussian, v ix ,v iy is the predicted value of x and y coordinates, ∑i is the covariance matrix of the i-th Gaussian, and w i Each Gaussian mixture weight component, represents the pre-difference variance function at the y coordinate, W p represents the matrix, It is the hidden state at the previous moment.

[0177] Furthermore, the model fits the mean, standard deviation and mixing weight of the above bivariate Gaussian distribution through the hidden state of the previous moment (t-1 moment), and the matrix W p The linear layer is trained, and the probability density function expression is as follows:

[0178]

[0179] Among them, p is the probability density function, which represents the trajectory model performance, ν x is the displacement coordinate predicted for the next time step, is the estimated standard deviation of coordinate x, λ is the correlation coefficient of the coordinate step, k is the cost function, which reflects the error between the current predicted position and the actual position, Describe the x coordinate and the predicted value v x The error between .

[0180] It should be noted that LSTM in this model involves input and output encoding and decoding. During the input encoding period, at the current moment, the input value I t As well as the state at the previous time point, update the current hidden state, encode the historical data encoding information of the smart device, and then pass it through the model layer node to obtain the interactive feature vector, and then input it into the softmanx function to output the probability event, that is, the probability of the three operating modes. The decoder is responsible for extracting the encoded information and predicting the location of the target device that will appear in the future. After coupling the interactive information encoding vector and the probability to obtain the vector, the decoder will pass the output vector through the fully connected layer at the previous moment to generate the trajectory point coordinates of different behaviors in the future time and their two-dimensional Gaussian distribution parameters.

[0181] Step 3: Model training and predictive analysis.

[0182] In some embodiments, the training strategy of the dataset sampling pre-training uses the mean square error (MSE) as the loss function L in the first 10 rounds. m Pre-training is performed, and the negative log-likelihood function NLL is used as the loss function L in the last 5 rounds nPerform training. The loss function expression is as follows:

[0183]

[0184] Among them, L m is the mean square error loss function, L n is the negative log-likelihood loss function, For a given motion behavior, the model predicts a trajectory that is compared to the past The probability of Represents a binary Gaussian function, trajectory model is the output trajectory, z p is the future intelligent driving behavior, X represents the historical characteristic value, F(z p |X) is the probability that the smart device movement may have different behaviors under given historical characteristics, Represents the predicted trajectory and actual trajectory samples at the i-th sample j.

[0185] It is understandable that since this model provides the results of multimodal trajectory prediction, the mean square error (RMSE) between the predicted position with the highest probability of the intelligent device's motion behavior and the actual position is used as a standard for evaluating the accuracy of the model, and is also an indicator for comparing model performance. However, the root mean square error has reference value when evaluating the single-mode prediction accuracy of the model, but it is insufficient when measuring the probability distribution of multimodal. Therefore, NLL is used to compare single- and multimodal performance. For example, the RMSE comparison table of different prediction models on the test set is as follows: Figure 7 shown.

[0186] Among them, by comparing the changes in prediction errors of the above models within 4 seconds, it can be found that the prediction accuracy is improved under the name of LSTM after adding the convolutional pooling layer. In addition, the multimodal prediction ADI-DCS-GRU using attention and deep interaction based on CS-LSTM also improves the prediction accuracy. The CS-LSTM model with the introduction of the GCN layer can retain background information while weighting features, thereby further improving the prediction accuracy. The above RMSE is appropriate for evaluating the average prediction results, but multimodal prediction needs to rely on NLL to predict more accurately. However, the above RMSE can also show that this method is more accurate than other algorithms as a whole, so it will not be explained.

[0187] Furthermore, various scenarios were compared and analyzed. Due to the different congestion conditions during the movement of smart devices, the data set was divided into three scenarios: mild, moderate, and severe. The comparison results are shown in the table below: Figure 8 shown.

[0188] Among them, the table includes a comparison of the prediction effects of the two algorithm models in three different scenarios. It can be understood that the prediction accuracy of this model is significantly improved. In the 4s long time, the RMSE of the three scenarios is reduced by 5%-10%.

[0189] The embodiments of the present application collect vehicle trajectory data in different scenarios and generate a sample data set based on the vehicle trajectory data; train the trajectory prediction model based on the sample data set, the mean square error loss function and the negative log-likelihood loss function, so as to handle complex trajectory changes in different scenarios, provide a more accurate prediction for the motion trajectory of the smart mobile device, and optimize the continuity and accuracy of the trajectory.

[0190] See also Fig. 9 The present application also provides a trajectory prediction device, which can implement the above trajectory prediction method, and the device includes:

[0191] The long short-term memory encoding module 901 is used to generate the current hidden state according to the input data at the current moment and the historical trajectory data through the long short-term memory encoder;

[0192] A feature vector generating module 902, configured to generate a feature vector by using the current hidden state and the historical data encoding information through a model layer, wherein the model layer includes a Gaussian density model-transformer layer;

[0193] A trajectory probability generation module 903 is used to input the feature vector into a soft maximization function to obtain a running trajectory probability;

[0194] The long short-term memory decoding module 904 is used to generate a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability through a long short-term memory decoder.

[0195] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0196] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above trajectory prediction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0197] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0198] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0199] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0200] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the trajectory prediction method of the embodiment of this application;

[0201] Input / output interface 1003, used to implement information input and output;

[0202] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0203] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0204] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0205] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned trajectory prediction method is implemented.

[0206] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0207] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0208] The trajectory prediction method, trajectory prediction device, electronic device and storage medium provided in the embodiments of the present application collect vehicle trajectory data of different scenarios and generate a sample data set according to the vehicle trajectory data; train the trajectory prediction model according to the sample data set, the mean square error loss function and the negative log-likelihood loss function, so as to handle complex trajectory changes in different scenarios and optimize the continuity and accuracy of the trajectory.

[0209] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0210] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0211] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0213] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0214] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0215] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0216] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0218] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0219] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A trajectory prediction method, characterized in that: The method comprises the following steps: Generate a current hidden state according to input data at a current moment and historical trajectory data through a long short-term memory encoder, wherein the input data at the current moment includes current moment features of the smart device and historical features of the smart device, and the historical trajectory data indicates the state of the predicted object at a historical time point; Generate a feature vector by using the current hidden state and historical data encoding information through a model layer, wherein the model layer includes a Gaussian density model-transformer layer; Inputting the feature vector into a soft maximization function to obtain a running trajectory probability; A long short-term memory decoder is used to generate a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability.

2. The method according to claim 1, characterized in that The method further comprises: Acquire current moment features of the smart device, wherein the current moment features include a relative lateral position of the smart device and the predicted object, a relative longitudinal position of the smart device and the predicted object, a speed of the smart device, and an acceleration of the smart device; Obtaining historical characteristics of the smart device; The input data at the current moment is generated according to the current moment feature and the historical feature of the smart device.

3. The method according to claim 2, characterized in that The expression of the current moment feature is as follows: C t ={(x 0，t ,and 0,t ,v 0,t ,to 0，t ),(x 1,t ,and 1,t ,v 1，t ,to 1，t ),...(x n，t ,and n，t ,v n，t ,to n，t )}, Among them, C t is the current moment feature, x n,t is the relative horizontal position between the smart device with sequence number n and the predicted object at time t, is the relative vertical position between the smart device with sequence number n and the predicted object at time t, v n,t is the speed of the smart device with serial number n at time t, a n,t is the acceleration of the smart device with sequence number n at time t, where n is a natural number.

4. The method according to claim 1, characterized in that The expression of the predicted trajectory is as follows: Where P represents the predicted trajectory, is the future time point t+1, t+2, …, t+t f The corresponding trajectory position, t+t f Represents the tth relative to time t f moment, t f The value is an integer.

5. The method according to claim 1, characterized in that The running trajectory probability includes a running trajectory type and a trajectory probability, and the generating of a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability by a long short-term memory decoder includes: Determine the running trajectory type with the largest trajectory probability as the prediction type; A prediction trajectory and parameters of a two-dimensional Gaussian distribution are generated according to the feature vector and the prediction type through a long short-term memory decoder.

6. A trajectory prediction model training method, characterized in that: The method comprises the following steps: Collecting vehicle trajectory data of different scenarios, and generating a sample data set according to the vehicle trajectory data, wherein the sample data set includes a training set, a validation set, and a test set; The trajectory prediction model is trained according to the sample data set, the mean square error loss function and the negative log-likelihood loss function to obtain the trained trajectory prediction model, and the trained trajectory prediction model is used to perform trajectory prediction by the method according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The step of training the trajectory prediction model according to the sample data set, the mean square error loss function, and the negative log-likelihood loss function to obtain the trained trajectory prediction model includes: Training the trajectory prediction model according to the sample data set; Calculating the mean square error loss function of the trajectory prediction model; Training the trajectory prediction model according to the mean square error loss function; Training the trajectory prediction model according to the sample data set; Calculating the negative log-likelihood loss function of the trajectory prediction model; Training a trajectory prediction model according to the negative log-likelihood loss function; The trained trajectory prediction model is obtained.

8. A trajectory prediction device, characterized in that: The device comprises: A long short-term memory encoding module is used to generate a current hidden state according to the input data at the current moment and the historical trajectory data through a long short-term memory encoder; A feature vector generation module, used to generate a feature vector by using the current hidden state and the historical data encoding information through a model layer, wherein the model layer includes a Gaussian density model-transformer layer; A trajectory probability generation module, used for inputting the feature vector into a soft maximization function to obtain a running trajectory probability; The long short-term memory decoding module is used to generate a predicted trajectory and parameters of a two-dimensional Gaussian distribution according to the feature vector and the running trajectory probability through a long short-term memory decoder.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images