A trajectory prediction method based on denoising and related equipment

Abstract

The application relates to the technical field of trajectory prediction, and provides a trajectory prediction method based on denoising and related equipment, which comprises the following steps: encoding a historical trajectory of a target vehicle to obtain historical trajectory hidden encoding of the target vehicle, and encoding a historical trajectory of each neighbor vehicle to obtain historical trajectory hidden encoding of each neighbor vehicle; calculating final social encoding of the target vehicle according to the historical trajectory hidden encoding of all the neighbor vehicles; performing trajectory prediction on the target vehicle based on the historical trajectory hidden encoding and the final social encoding of the target vehicle to obtain a future trajectory distribution; sampling the future trajectory distribution to obtain a noisy future trajectory, and denoising the noisy future trajectory by using all the historical trajectories to obtain a final future trajectory of the target vehicle. The method can improve the accuracy of trajectory prediction.

Description

Technical Field

[0001] The present application relates to the technical field of trajectory prediction, and in particular to a trajectory prediction method based on denoising and related equipment. Background Art

[0002] The goal of vehicle trajectory prediction is to predict the future trajectory of a target vehicle based on its own and its surrounding neighbors' past trajectories. Accurately predicting a vehicle's future trajectory is crucial for many autonomous driving applications, including optimizing driving path planning, making accurate driving decisions in dynamic environments, and improving driving safety. Traditionally, statistical models predict future trajectories based on the historical trajectories of a single agent. However, these models fail to consider the interactions between the target agent and its neighbors, which reduces prediction performance. To address this issue, various deep learning-based models have been proposed to simulate the spatial interactions between vehicles. However, existing deep learning-based methods are unable to model trajectory uncertainty, which is common in real-world driving scenarios. Generally speaking, uncertainty in autonomous driving can be roughly divided into driving behavior uncertainty and driving scenario uncertainty. Designing uncertainty-aware trajectory prediction models remains underexplored and an open research topic. This uncertainty leads to low trajectory prediction accuracy. Summary of the Invention

[0003] The present application provides a trajectory prediction method and related equipment based on denoising, which can solve the problem of low accuracy of trajectory prediction.

[0004] In a first aspect, an embodiment of the present application provides a trajectory prediction method based on denoising, the trajectory prediction method comprising:

[0005] Get the historical trajectory of the target vehicle and the historical trajectory of each neighboring vehicle of the target vehicle;

[0006] The target vehicle's historical trajectory is encoded to obtain a hidden historical trajectory code of the target vehicle, and the historical trajectory of each neighboring vehicle is encoded to obtain a hidden historical trajectory code of each neighboring vehicle; the hidden historical trajectory code is used to describe the dynamic information of the historical trajectory;

[0007] The final social code of the target vehicle is calculated based on the hidden codes of the historical trajectories of all neighboring vehicles. The final social code is used to describe the interaction information between the target vehicle and all neighboring vehicles.

[0008] Based on the target vehicle's historical trajectory hidden code and final social code, the target vehicle's trajectory is predicted to obtain the future trajectory distribution. The future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuver mode at the future moment, as well as the probability of the target vehicle executing each maneuver mode at the future moment.

[0009] The future trajectory distribution is sampled to obtain the noisy future trajectory, and all historical trajectories are used to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0010] Optionally, the historical trajectory of the target vehicle is encoded to obtain a hidden encoding of the historical trajectory of the target vehicle, including:

[0011] Encoding the historical trajectory of the target vehicle to obtain the historical trajectory code of the target vehicle;

[0012] The historical trajectory code is re-encoded to obtain the historical trajectory hidden code.

[0013] Optionally, the final social code of the target vehicle is calculated based on the hidden codes of the historical trajectories of all neighboring vehicles, including:

[0014] For each neighbor vehicle, a social code corresponding to the neighbor vehicle is generated based on the hidden code of the neighbor vehicle's historical trajectory, and the amplitude embedding and phase embedding of the neighbor vehicle are calculated using the social code.

[0015] Compute the spatial representation of each neighbor vehicle based on all amplitude embeddings and all phase embeddings;

[0016] Construct an initial social encoding of the target vehicle based on all spatial representations;

[0017] The initial social code is calculated using the attention mechanism to obtain the final social code of the target vehicle.

[0018] Optionally, use social coding to calculate the amplitude embedding and phase embedding of neighbor vehicles, including:

[0019] By formula:

[0020] z j =Plain-FC(h j , W z )

[0021] θ j =Plain-FC(h j , W θ )

[0022] Calculate the amplitude embedding z of the jth neighbor vehicle j and phase embedding θ j ;

[0023] where h j denotes the social code of the jth neighbor vehicle, Plain-FC() denotes plain fully connected, W z denotes the learnable parameter matrix for computing the amplitude embedding, W θ denotes the learnable parameter matrix for computing the phase embedding, j = 1, 2,..., n, n denotes the number of neighbor vehicles.

[0024] Optionally, the spatial representation of each neighbor vehicle is calculated based on all amplitude embeddings and all phase embeddings, including:

[0025] Through the formula:

[0026]

[0027]

[0028] The spatial representation of the jth neighbor vehicle o j is calculated.

[0029] where denotes the complex value, denotes the amplitude embedding and the phase embedding of all neighbor vehicles, denotes the combination of the amplitude embedding and the phase embedding of the 1st neighbor vehicle, denotes the combination of the amplitude embedding and the phase embedding of the 2nd neighbor vehicle, denotes the combination of the amplitude embedding and the phase embedding of the nth neighbor vehicle, M pos denotes the neighbor vehicle position mask matrix, W t , and all denote learnable weight matrices, z k denotes the amplitude embedding of the kth neighbor vehicle, θ k denotes the phase embedding of the kth neighbor vehicle, z k ⊙cosθ k denotes the real value of the complex value , z k ⊙sinθ k denotes the imaginary value of the complex value , k ∈ (1, 2,..., n), k ≠ j.

[0030] Optionally, the trajectory of the target vehicle is predicted based on the historical trajectory hidden code and the final social code of the target vehicle, and a future trajectory distribution is obtained, including:

[0031] The historical trajectory hidden code and the final social code of the target vehicle are added and normalized to obtain an interaction information vector.

[0032] Adaptively fusing the interaction information vector and the mapping matrix of all maneuver modes to obtain a fusion vector;

[0033] Multiplying the fusion vector and the interaction information vector to obtain a final vector, and calculating the final vector to obtain a future trajectory distribution wherein, represents a mean value of the position of the target vehicle at a future time, represents a variance of the position of the target vehicle at a future time, represents a correlation coefficient.

[0034] Optionally, adaptively fusing the interaction information and the mapping matrix of all maneuver modes to obtain a fusion vector, comprising:

[0035] Through the formula:

[0036]

[0037] calculating the fusion vector

[0038] wherein, c t-t′ represents an element corresponding to the t-th historical time in the interaction information, represents a combination vector of the mapping matrix of all maneuver modes, represents an element corresponding to the first T h historical time in the mapping matrix containing all maneuver modes, represents an element corresponding to the first 2 historical time in the mapping matrix containing all maneuver modes, represents an element corresponding to the first 1 historical time in the mapping matrix containing all maneuver modes, represents the interaction information vector, represents the interaction information corresponding to the t-T h historical time, c t-2 represents the interaction information corresponding to the t-2 historical time, c t-1 represents the interaction information corresponding to the t-1 historical time, and t represents the current time.

[0039] Optionally, sampling the future trajectory distribution to obtain a noise future trajectory, comprising:

[0040] Taking the maneuver mode corresponding to the probability with the largest value in the future trajectory distribution as the final maneuver mode, and taking the future trajectory generated by the target vehicle performing the final maneuver mode as the noise future trajectory.

[0041] Optionally, the noise future trajectory is denoised by using all historical trajectories to obtain a final future trajectory of the target vehicle, including:

[0042] By the formula:

[0043]

[0044]

[0045]

[0046] The noise future trajectory after the rth step of denoising is calculated

[0047] Wherein, α r , Both represent parameters of the diffusion process, represents the noise future trajectory after the r+1th step of denoising, represents the estimated noise, z represents the noise, z~N(z;0,I), I represents the unit matrix, f ∈ () represents the noise estimation model, represents the spatio-temporal embedding, X tar represents the historical trajectory of the target vehicle, represents the historical trajectory of all neighbor vehicles, f context () represents the information encoder, r=1,2,...,R, R represents the number of steps of the denoising process, when r=1, the noise future trajectory after the 1st step of denoising is taken as the final future trajectory of the target vehicle.

[0048] In a second aspect, the embodiments of the present application provide a trajectory prediction device based on denoising, including:

[0049] The acquisition module acquires the historical trajectory of the target vehicle and acquires the historical trajectory of each neighbor vehicle of the target vehicle;

[0050] The encoding module encodes the historical trajectory of the target vehicle to obtain the historical trajectory hidden code of the target vehicle, and encodes the historical trajectory of each neighbor vehicle to obtain the historical trajectory hidden code of each neighbor vehicle; the historical trajectory hidden code is used to describe the dynamic information of the historical trajectory;

[0051] The calculation module calculates the final social code of the target vehicle according to the historical trajectory hidden code of all neighbor vehicles; the final social code is used to describe the interaction information between the target vehicle and all neighbor vehicles;

[0052] a trajectory prediction module, configured to perform trajectory prediction on the target vehicle based on the hidden encoding of the historical trajectory of the target vehicle and the final social encoding, to obtain a future trajectory distribution; the future trajectory distribution is used to describe a future trajectory generated by the target vehicle performing each maneuver mode at a future time, and a probability of the target vehicle performing each maneuver mode at the future time;

[0053] a sampling module, configured to sample the future trajectory distribution to obtain a noisy future trajectory, and denoise the noisy future trajectory using all the historical trajectories to obtain a final future trajectory of the target vehicle.

[0054] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the denoising-based trajectory prediction method when executing the computer program.

[0055] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to implement the denoising-based trajectory prediction method.

[0056] The above scheme of the present application has the following advantages:

[0057] In the embodiments of the present application, the historical trajectory of the target vehicle is obtained, and the historical trajectory of each neighbor vehicle of the target vehicle is obtained, then the historical trajectory of the target vehicle is encoded to obtain the hidden encoding of the historical trajectory of the target vehicle, and the historical trajectory of each neighbor vehicle is encoded to obtain the hidden encoding of the historical trajectory of each neighbor vehicle, then the final social encoding of the target vehicle is calculated according to the hidden encoding of the historical trajectory of all the neighbor vehicles, then the trajectory prediction is performed on the target vehicle based on the hidden encoding of the historical trajectory of the target vehicle and the final social encoding, to obtain the future trajectory distribution, finally the sampling is performed on the future trajectory distribution to obtain the noisy future trajectory, and the noisy future trajectory is denoised using all the historical trajectories to obtain the final future trajectory of the target vehicle. Wherein, the final social encoding of the target vehicle is calculated according to the hidden encoding of the historical trajectory of the neighbor vehicle, the interaction information between the target vehicle and the neighbor vehicle is fully analyzed, so that the final social encoding can accurately describe the interaction information between the target vehicle and all the neighbor vehicles, the accuracy of the future trajectory distribution calculated according to the accurate final social encoding is high, then the sampling and denoising are performed on the future trajectory distribution, which can reduce the uncertainty of the trajectory, and further improve the accuracy of the trajectory prediction.

[0058] In addition, the process of obtaining the future trajectory distribution first and then obtaining the final future trajectory by sampling and denoising realizes the two-stage trajectory prediction from coarse to fine, can well capture the interaction between vehicles, can also model the multi-modal trajectory of the vehicle, and most importantly, can gradually reduce the uncertainty of the future trajectory by sampling and gradually denoising.

[0059] Other benefits of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0061] Figure 1 The flow chart of the trajectory prediction method based on denoising provided by an embodiment of the present application;

[0062] Figure 2 The structure schematic diagram of the trajectory prediction device based on denoising provided by an embodiment of the present application;

[0063] Figure 3 The structure schematic diagram of the terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, specific details are set forth in order to provide a thorough understanding of embodiments of the present application. However, persons of ordinary skill in the art will readily recognize that embodiments of the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application.

[0065] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0066] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations thereof, and includes these combinations.

[0067] ​As used in the description of the application and the appended claims, the term “if’ can be interpreted as meaning “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [the described condition or event] is detected” can be interpreted as meaning “upon determining” or “in response to determining” or “upon detecting [the described condition or event]” or “in response to detecting [the described condition or event],” depending on the context.

[0068] In addition, in the description of the application and the appended claims, the terms “first,” “second,” “third,” and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0069] Reference in the specification to “one embodiment” or “some embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in additional embodiments,” and the like in various places in the specification are not necessarily all referring to the same embodiment, although they can. The terms “comprising,” “including,” “having,” and the like are meant to be interpreted as “including but not limited to,” unless otherwise indicated by the language of the description.

[0070] To solve the problem of low accuracy of existing trajectory prediction, the embodiment of the application provides a trajectory prediction method based on denoising. The trajectory prediction method comprises the following steps: obtaining a historical trajectory of a target vehicle, obtaining a historical trajectory of each neighbor vehicle of the target vehicle, encoding the historical trajectory of the target vehicle to obtain a hidden encoding of the historical trajectory of the target vehicle, encoding the historical trajectory of each neighbor vehicle to obtain a hidden encoding of the historical trajectory of each neighbor vehicle, calculating a final social encoding of the target vehicle according to the hidden encodings of the historical trajectories of all the neighbor vehicles, performing trajectory prediction on the target vehicle based on the hidden encoding of the historical trajectory of the target vehicle and the final social encoding to obtain a future trajectory distribution, and finally sampling the future trajectory distribution to obtain a noisy future trajectory, and denoising the noisy future trajectory by using all the historical trajectories to obtain a final future trajectory of the target vehicle. Wherein, the final social encoding of the target vehicle is calculated according to the hidden encodings of the historical trajectories of the neighbor vehicles, which realizes sufficient analysis of the interaction information between the target vehicle and the neighbor vehicles, so that the final social encoding can accurately describe the interaction information between the target vehicle and all the neighbor vehicles. The accuracy of the future trajectory distribution calculated according to the accurate final social encoding is high, and then the future trajectory distribution is sampled and denoised, which can reduce the uncertainty of the trajectory and further improve the accuracy of the trajectory prediction.

[0071] In addition, the process of first obtaining the future trajectory distribution and then obtaining the final future trajectory through sampling and denoising realizes a two-stage trajectory prediction from coarse to fine. It can well capture the interaction between vehicles and model the multimodal trajectory of vehicles. Most importantly, it can gradually reduce the uncertainty of the future trajectory through sampling and gradual denoising.

[0072] Next, the trajectory prediction method based on denoising provided by this application is exemplified.

[0073] like Figure 1 As shown, the trajectory prediction method based on denoising provided by this application includes the following steps:

[0074] Step 11: Obtain the historical trajectory of the target vehicle and the historical trajectory of each neighboring vehicle of the target vehicle.

[0075] The target vehicle is the vehicle for which trajectory prediction is required, the neighbor vehicle is the vehicle adjacent to the target vehicle, and the historical trajectory is the movement trajectory of the vehicle in the historical time period before the current time. For example, if the current time is 9 o'clock, the historical trajectory can be the movement trajectory of the vehicle from 7 o'clock to 8:30. The neighbor vehicle is the adjacent vehicle to the target vehicle in the historical time period.

[0076] In some embodiments of the present application, historical trajectories may be acquired using the positioning systems of the target vehicle and neighboring vehicles.

[0077] Step 12: Encode the historical trajectory of the target vehicle to obtain a hidden historical trajectory code of the target vehicle, and encode the historical trajectory of each neighboring vehicle to obtain a hidden historical trajectory code of each neighboring vehicle.

[0078] The above historical trajectory hidden code is used to describe the dynamic information of the historical trajectory, such as the moving direction and speed of the vehicle when traveling along the historical trajectory.

[0079] In some embodiments of the present application, the step of encoding the historical trajectory of the target vehicle to obtain the hidden historical trajectory code of the target vehicle is specifically as follows:

[0080] The first step is to encode the historical trajectory of the target vehicle to obtain the historical trajectory code of the target vehicle.

[0081] Exemplarily, a multi-layer perceptron may be used to encode the historical trajectory of the target vehicle to obtain the historical trajectory encoding of the target vehicle.

[0082] The second step is to perform secondary encoding on the historical trajectory code to obtain the hidden code of the historical trajectory.

[0083] Exemplarily, the historical trajectory encoding can be encoded again by using a long short-term memory network to obtain historical trajectory hidden encoding.

[0084] It should be noted that the process of encoding the historical trajectory of each neighbor vehicle to obtain the historical trajectory hidden encoding of each neighbor vehicle is the same as the process of obtaining the historical trajectory hidden encoding of the target vehicle described above, that is, the historical trajectory of the neighbor vehicle is encoded to obtain the historical trajectory encoding of the neighbor vehicle, and then the historical trajectory encoding is encoded again to obtain the historical trajectory hidden encoding.

[0085] Step 13, calculating the final social encoding of the target vehicle according to the historical trajectory hidden encodings of all neighbor vehicles.

[0086] The final social encoding described above is used to describe the interaction information between the target vehicle and all neighbor vehicles, such as the distance between the neighbor vehicle and the target vehicle, the relative speed between the neighbor vehicle and the target vehicle, etc.

[0087] In some embodiments of the present application, the step of calculating the final social encoding of the target vehicle according to the historical trajectory hidden encodings of all neighbor vehicles is specifically:

[0088] First, for each neighbor vehicle, generate the corresponding social encoding of the neighbor vehicle based on the historical trajectory hidden encoding of the neighbor vehicle, and calculate the amplitude embedding and phase embedding of the neighbor vehicle by using the social encoding.

[0089] In some embodiments of the present application, the step of generating the corresponding social encoding of the neighbor vehicle based on the historical trajectory hidden encoding of the neighbor vehicle is specifically: expanding the mask matrix mask representing the positional relationship between the target vehicle and each neighbor vehicle to a shape consistent with the historical time period corresponding to the historical trajectory. Then create a new tensor representing the social encoding, which is initialized as a zero tensor with the same shape as the mask matrix. After that, use the masked_scatter_ function of pytorch to fill the historical trajectory hidden encoding of the neighbor vehicle into the social encoding according to the mask matrix. Specifically, masked_scatter_ allows to insert specific values of one tensor into another tensor according to a Boolean mask, that is, for the positions where the mask is True, the corresponding value in the historical trajectory hidden encoding of the neighbor vehicle is filled into the social encoding, and for the positions where the mask is False, the social encoding remains unchanged.

[0090] The step of calculating the amplitude embedding and phase embedding of the neighbor vehicle by using the social encoding is specifically:

[0091] Through the formula:

[0092] z j =Plain-FC(hj , W z )

[0093] θ j = Plain-FC(h j , W θ )

[0094] Calculate the amplitude embedding z j and phase embedding θ j of the jth neighbor vehicle.

[0095] where h j denotes the social code of the jth neighbor vehicle, Plain-FC() denotes plain fully connected, W z denotes the learnable weight matrix of the amplitude embedding, W θ denotes the learnable weight matrix of the phase embedding, j = 1, 2, …, n, and n denotes the number of neighbor vehicles.

[0096] Second, calculate the spatial representation of each neighbor vehicle based on all amplitude embeddings and all phase embeddings.

[0097] Specifically, the spatial representation o j of the jth neighbor vehicle is calculated by the formula:

[0098]

[0099]

[0100]

[0101] where denotes the complex value, denotes all amplitude embeddings and phase embeddings of neighbor vehicles, denotes the combination of the amplitude embedding and the phase embedding of the 1st neighbor vehicle, denotes the combination of the amplitude embedding and the phase embedding of the 2nd neighbor vehicle, denotes the combination of the amplitude embedding and the phase embedding of the nth neighbor vehicle, M pos denotes the neighbor vehicle position mask matrix, W t , and both denote the learnable weight matrix, z k denotes the amplitude embedding of the kth neighbor vehicle, θ k denotes the phase embedding of the kth neighbor vehicle, z k ⊙cosθ k denotes the true value of the complex value , and z k ⊙sinθ k denotes the complex value​ imaginary value, k∈(1, 2, ..., n), k≠j.

[0102] It should be noted that the Surrounding-FC calculation described above involves extracting phase and amplitude embeddings for neighboring vehicles through convolution operations. These embeddings are then expanded using the Euler formula to capture the relative interactions and spatiotemporal relationships between vehicles. These features are then reweighted using adaptive pooling, automatically adjusting the relative importance of each feature. Finally, they are further processed using a multilayer perceptron (MLP) to generate a spatial representation with enhanced phase information and interaction features.

[0103] In the third step, an initial social encoding of the target vehicle is constructed based on all spatial representations.

[0104] It should be noted that for the spatial representation o j , which is a vector composed of multiple elements, each of which corresponds to multiple historical moments, and is used to describe the spatial information of neighbor vehicles at each historical moment. For a historical moment, the elements corresponding to the historical moment in all spatial representations are integrated to obtain the social interaction representation, and the social interaction representations of all historical moments are integrated to obtain the initial social coding in, Indicates the tT h social interaction representation of a historical moment, h t-2 represents the social interaction representation at the t-2th historical moment, h t - 1 Represents the social interaction representation at the t-1th historical moment.

[0105] In the fourth step, the initial social code is calculated using the attention mechanism to obtain the final social code of the target vehicle.

[0106] Specifically, the initial social code is used as the input of the attention mechanism, and the query matrix (Q), key matrix (K) and value matrix (V) are calculated based on the self-attention mechanism. Then, the dot product of the query matrix and the key matrix is ​​calculated to obtain the attention score, which is used to capture the dependency between historical moments, that is, the relationship between the features of different historical moments. Then, the attention score is normalized using softmax to ensure that the attention distribution at each historical moment is a probability distribution and the sum is 1. The value matrix is ​​weighted and summed according to the calculated attention score to obtain the new historical moment feature. Finally, the results of different attention heads are merged to obtain the final social code. The expression is:

[0107]

[0108] where Concat denotes a concatenation operation, head1 denotes the output of the first attention head, head2 denotes the output of the second attention head, head k denotes the output of the kth attention head.

[0109] It is worth mentioning that the final social encoding of the target vehicle is calculated according to the historical trajectory hidden encoding of the neighbor vehicles, which fully analyzes the interaction information between the target vehicle and the neighbor vehicles, so that the final social encoding can accurately describe the interaction information between the target vehicle and all neighbor vehicles.

[0110] Step 14, based on the historical trajectory hidden encoding and the final social encoding of the target vehicle, trajectory prediction is performed on the target vehicle to obtain a future trajectory distribution.

[0111] The future trajectory distribution is used to describe the future trajectory generated by the target vehicle performing each maneuver mode at a future time, and the probability of the target vehicle performing each maneuver mode at the future time. Maneuver mode is used to describe the motion state of the target vehicle, such as left lane change, right lane change, keep lane, accelerate, keep speed, etc.

[0112] In some embodiments of the present application, the step of predicting the trajectory of the target vehicle based on the historical trajectory hidden encoding and the final social encoding of the target vehicle to obtain the future trajectory distribution is specifically:

[0113] First, add the historical trajectory hidden encoding and the final social encoding of the target vehicle and perform normalization processing to obtain an interaction information vector.

[0114] For example, the existing normalization formula can be used for normalization processing.

[0115] Second, the interaction information vector and the mapping matrix of all maneuver modes are adaptively fused to obtain a fusion vector.

[0116] Specifically, the fusion vector is calculated by the formula:

[0117]

[0118]

[0119] where c t-t′ denotes the element corresponding to the t-t' historical moment in the interaction information, denotes the combination vector of the mapping matrix of all maneuver modes, denotes the element corresponding to the first T h historical moment in the mapping matrix containing all maneuver modes,​ represents an element in the mapping matrix corresponding to the first 2 historical time points, represents an element in the mapping matrix corresponding to the first 1 historical time point, represents an interaction information vector, represents the interaction information corresponding to the t-T h historical time point, c t-2 represents the interaction information corresponding to the t-2 historical time point, c t-1 represents the interaction information corresponding to the t-1 historical time point, t represents the current time.

[0120] It should be noted that the mapping matrix of the maneuver mode is obtained according to the maneuver condition of the maneuver mode, which has a plurality of elements, and the plurality of elements correspond to a plurality of historical time points one by one.

[0121] Thirdly, the fusion vector and the interaction information vector are multiplied to obtain a final vector, and the final vector is calculated to obtain a future trajectory distribution wherein, represents the mean value of the position of the target vehicle at the future time point, represents the variance of the position of the target vehicle at the future time point, represents the correlation coefficient.

[0122] For example, the long short-term memory network can be used to calculate the final vector to obtain the future trajectory distribution. The future trajectory distribution is a Gaussian distribution.

[0123] It is worth mentioning that the accuracy of the future trajectory distribution calculated according to the accurate final social encoding is high, which can accurately describe the probability of the target vehicle executing each maneuver mode at the future time point.

[0124] Step 15, sampling the future trajectory distribution to obtain a noise future trajectory, and using all historical trajectories to denoise the noise future trajectory to obtain a final future trajectory of the target vehicle.

[0125] In some embodiments of the present application, the above-mentioned step of sampling the future trajectory distribution to obtain a noise future trajectory, and using all historical trajectories to denoise the noise future trajectory to obtain a final future trajectory of the target vehicle is specifically:

[0126] Firstly, sampling the future trajectory distribution to obtain a noise future trajectory.

[0127] Specifically, the maneuver mode corresponding to the probability with the largest value in the future trajectory distribution is taken as the final maneuver mode, and the future trajectory generated by the target vehicle executing the final maneuver mode is taken as the noise future trajectory.

[0128] Second, the noise future trajectory is denoised using all historical trajectories to obtain the final future trajectory of the target vehicle.

[0129] Specifically, by the formula:

[0130]

[0131]

[0132]

[0133] The rth step denoised noise future trajectory

[0134] wherein α r , represent the parameters of the diffusion process, represents the r+1th step denoised noise future trajectory, represents the estimated noise, z represents the noise, z ~ N(z; 0, I), I represents the unit matrix, f ∈ () represents the noise estimation model, represents the spatio-temporal embedding, X tar represents the historical trajectory of the target vehicle, represents the historical trajectory of all neighbor vehicles, f context () represents the information encoder, r = 1, 2,..., R, R represents the number of steps of the denoising process, when r = 1, the first step denoised noise future trajectory is taken as the final future trajectory of the target vehicle.

[0135] It should be noted that in the above formula, when r = R, is the input noise future trajectory. The above denoising process is a reverse process from the last step to the first step (similar to the reverse diffusion process of the diffusion model). The above information encoder can be a transformer-based encoder.

[0136] Exemplarily, the method of the present application can be implemented on an NVIDIA 3090 GPU server using the Pytorch framework. The parameter settings are as follows: a 13x5 grid is used, which is defined with the target vehicle as the center, wherein each column corresponds to a lane and each row is spaced 15 feet apart. When calculating the historical trajectory hidden code, a multi-layer perceptron and a long short-term memory network are used, the hidden feature of the multi-layer perceptron is set to 32, and the activation function is ReL uTo train the framework corresponding to the method of the present application, a two-stage training strategy is considered, where the first stage trains the denoising module (i.e. the formula involved in step 15 of the present application), and the second stage focuses on training the spatio-temporal interaction module (i.e. the formula, model, etc. involved in steps 11-14 of the present application). Each historical trajectory is segmented into segments over a sensing horizon (i.e. 8s), which contains past (3s) and future (5s) positions at 5Hz. The dataset consisting of all the segmented historical trajectories is divided into training, validation and test sets in the ratio 7:2:1 for performing the method of the present application and training, validation and testing.

[0137] It is worth mentioning that the final social encoding of the target vehicle is calculated according to the historical trajectory hidden encoding of the neighbor vehicles, which fully analyzes the interaction information between the target vehicle and the neighbor vehicles, so that the final social encoding can accurately describe the interaction information between the target vehicle and all the neighbor vehicles, and the accuracy of the future trajectory distribution calculated according to the accurate final social encoding is high. Then, sampling and denoising the future trajectory distribution can reduce the uncertainty of the trajectory and further improve the accuracy of the trajectory prediction.

[0138] In addition, the process of first obtaining the future trajectory distribution and then obtaining the final future trajectory through sampling and denoising realizes the two-stage trajectory prediction from coarse to fine, which can well capture the interaction between vehicles, can model the multi-modal trajectory of the vehicle, and most importantly, can gradually reduce the uncertainty of the future trajectory through sampling and gradual denoising.

[0139] The denoising-based trajectory prediction device provided by the present application is exemplarily described below.

[0140] As shown in Figure 2 The denoising-based trajectory prediction device 200 provided by the embodiment of the present application comprises:

[0141] The acquisition module 201 acquires the historical trajectory of the target vehicle and acquires the historical trajectory of each neighbor vehicle of the target vehicle;

[0142] The encoding module 203 encodes the historical trajectory of the target vehicle to obtain the historical trajectory hidden encoding of the target vehicle, and encodes the historical trajectory of each neighbor vehicle to obtain the historical trajectory hidden encoding of each neighbor vehicle; the historical trajectory hidden encoding is used to describe the dynamic information of the historical trajectory;

[0143] The calculation module 204 calculates the final social encoding of the target vehicle according to the historical trajectory hidden encoding of all the neighbor vehicles; the final social encoding is used to describe the interaction information between the target vehicle and all the neighbor vehicles;

[0144] The trajectory prediction module 205 predicts the target vehicle's trajectory based on the target vehicle's historical trajectory hidden code and final social code to obtain a future trajectory distribution. The future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuver mode at a future time, as well as the probability of the target vehicle executing each maneuver mode at a future time.

[0145] The sampling module 206 samples the future trajectory distribution to obtain a noisy future trajectory, and uses all historical trajectories to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0146] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0148] like Figure 3 As shown, an embodiment of the present application provides a terminal device, and the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above method embodiments when executing the computer program D102.

[0149] Specifically, the processor D100 executes the computer program D102, obtains the historical trajectory of the target vehicle and the historical trajectory of each neighbor vehicle of the target vehicle, encodes the historical trajectory of the target vehicle to obtain the historical trajectory hidden code of the target vehicle, encodes the historical trajectory of each neighbor vehicle to obtain the historical trajectory hidden code of each neighbor vehicle, calculates the final social code of the target vehicle according to the historical trajectory hidden codes of all neighbor vehicles, performs trajectory prediction on the target vehicle based on the historical trajectory hidden code and the final social code of the target vehicle to obtain a future trajectory distribution, samples the future trajectory distribution to obtain a noisy future trajectory, and denoises the noisy future trajectory using all historical trajectories to obtain the final future trajectory of the target vehicle. The final social code of the target vehicle is calculated according to the historical trajectory hidden code of the neighbor vehicle, which fully analyzes the interaction information between the target vehicle and the neighbor vehicle, so that the final social code can accurately describe the interaction information between the target vehicle and all neighbor vehicles. The accuracy of the future trajectory distribution calculated according to the accurate final social code is high, and then sampling and denoising the future trajectory distribution can reduce the uncertainty of the trajectory and further improve the accuracy of the trajectory prediction.

[0150] The processor D100 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0151] The storage D101 can be an internal storage unit of the terminal device D10 in some embodiments, such as a hard disk or a memory of the terminal device D10. The storage D101 can also be an external storage device of the terminal device D10 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the storage D101 can include both an internal storage unit and an external storage device of the terminal device D10. The storage D101 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of the computer program, and the like. The storage D101 can also be used to temporarily store data that has been output or will be output.

[0152] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments.

[0153] The embodiments of the present application provide a computer program product. When the computer program product is run on a terminal device, the terminal device is caused to implement the steps in the above-mentioned various method embodiments.

[0154] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the embodiments of the present application can implement all or part of the processes in the above-mentioned method embodiments by means of a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments. The computer program includes computer program codes, which can be in the form of source codes, object codes, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program codes of the based on denoising trajectory prediction method, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0155] In the above embodiments, the description of each embodiment is focused on, and the part not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0156] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0157] The above is the preferred embodiment of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the principles described in the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A trajectory prediction method based on denoising, characterized in that: include: Obtain a historical trajectory of a target vehicle and a historical trajectory of each neighboring vehicle of the target vehicle; Encoding the historical trajectory of the target vehicle to obtain a hidden historical trajectory code of the target vehicle, and encoding the historical trajectory of each of the neighboring vehicles to obtain a hidden historical trajectory code of each of the neighboring vehicles; The historical trajectory hidden code is used to describe the dynamic information of the historical trajectory; Calculate the final social code of the target vehicle based on the hidden codes of the historical trajectories of all neighboring vehicles; The final social code is used to describe the interaction information between the target vehicle and all neighboring vehicles; Based on the target vehicle's historical trajectory hidden code and final social code, the target vehicle's trajectory is predicted to obtain a future trajectory distribution; the future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuver mode at a future moment, as well as the probability of the target vehicle executing each maneuver mode at a future moment; Sampling the future trajectory distribution to obtain a noisy future trajectory, and denoising the noisy future trajectory using all historical trajectories to obtain a final future trajectory of the target vehicle; The sampling of the future trajectory distribution to obtain the noisy future trajectory includes: The maneuver mode corresponding to the maximum probability of the median value of the future trajectory distribution is used as the final maneuver mode, and the future trajectory generated by the target vehicle executing the final maneuver mode is used as the noise future trajectory; Denoising the noisy future trajectory using all historical trajectories to obtain the final future trajectory of the target vehicle includes: By formula: Calculate the future trajectory of the noise after denoising in step r Among them, α r 、 are the parameters of the diffusion process, represents the future trajectory of the noise after the r+1th step denoising, Represents estimated noise, z represents noise, z~N(z;0,I), I represents the unit matrix, f ∈ () represents the noise estimation model, represents spatiotemporal embedding, X tar represents the historical trajectory of the target vehicle, represents the historical trajectories of all neighbor vehicles, f context () represents the information encoder, r=1,2,...,R, R represents the number of steps in the denoising process, when r=1, the future trajectory of the noise after the first step of denoising is converted to as the final future trajectory of the target vehicle.

2. The trajectory prediction method according to claim 1, characterized in that The encoding of the historical trajectory of the target vehicle to obtain the hidden historical trajectory code of the target vehicle includes: Encoding the historical trajectory of the target vehicle to obtain the historical trajectory code of the target vehicle; The historical trajectory code is re-encoded to obtain a historical trajectory hidden code.

3. The trajectory prediction method according to claim 1, characterized in that The calculation of the final social code of the target vehicle based on the hidden codes of the historical trajectories of all neighboring vehicles includes: For each of the neighbor vehicles, generating a social code corresponding to the neighbor vehicle based on the hidden code of the neighbor vehicle's historical trajectory, and using the social code to calculate the amplitude embedding and phase embedding of the neighbor vehicle; Compute the spatial representation of each neighbor vehicle based on all amplitude embeddings and all phase embeddings; constructing an initial social encoding of the target vehicle based on all spatial representations; The initial social code is calculated using an attention mechanism to obtain a final social code of the target vehicle.

4. The trajectory prediction method according to claim 3, characterized in that The calculating the amplitude embedding and phase embedding of the neighbor vehicle by using the social coding includes: By formula: z j =Plain-FC(h j ,W z ) θ j =Plain-FC(h j ,W θ ) Calculate the amplitude embedding z of the jth neighbor vehicle j and phase embedding θ j ; Among them, h j represents the social coding of the j-th neighbor vehicle, Plain-FC() represents the naive full connection, W z represents the learnable weight matrix for computing amplitude embedding, W θ represents the learnable weight matrix for computing phase embedding, j = 1, 2, ..., n, where n represents the number of neighbor vehicles.

5. The trajectory prediction method according to claim 4, characterized in that The spatial representation of each neighbor vehicle is calculated based on all amplitude embeddings and all phase embeddings, including: By formula: Calculate the spatial representation o of the jth neighbor vehicle j ; in, represents a complex value, represents the amplitude embedding and phase embedding of all neighbor vehicles, represents the combination of amplitude embedding and phase embedding of the first neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the second neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the nth neighbor vehicle, M pos represents the neighbor vehicle position mask matrix, W t 、 and Both represent learnable weight matrices, z k represents the amplitude embedding of the kth neighbor vehicle, θ k represents the phase embedding of the kth neighbor vehicle, z k ⊙cosθ k Represents a complex value The true value of z k ⊙sinθ k Represents a complex value imaginary value, k∈(1,2,…,n), k≠j.

6. The trajectory prediction method according to claim 1, characterized in that The target vehicle trajectory is predicted based on the historical trajectory hidden code and the final social code to obtain the future trajectory distribution, including: Adding the hidden code of the target vehicle's historical trajectory and the final social code and performing normalization processing to obtain an interactive information vector; Adaptively fusing the interaction information vector and the mapping matrices of all maneuver modes to obtain a fusion vector; Multiply the fusion vector and the mutual information vector to obtain the final vector, and calculate the final vector to obtain the future trajectory distribution in, represents the mean value of the target vehicle’s position at the future time, represents the variance of the target vehicle's position at the future moment, Represents the correlation coefficient.

7. The trajectory prediction method according to claim 6, characterized in that: Adaptively fusing the interaction information vector and the mapping matrices of all maneuver modes to obtain a fusion vector includes: By formula: Calculate the fusion vector Among them, c t-t' Represents the element corresponding to the t-t'th historical moment in the interaction information, Represents the combined vector of the mapping matrices of all maneuver modes, Represents the -Tth in the mapping matrix containing all maneuver modes h The elements corresponding to each historical moment, represents the element corresponding to the -2th historical moment in the mapping matrix containing all maneuver modes, represents the element corresponding to the -1th historical moment in the mapping matrix containing all maneuver modes, represents the mutual information vector, Indicates the tT h The interaction information corresponding to the historical moment, c t-2 represents the interaction information corresponding to the t-2th historical moment, c t-1 represents the interaction information corresponding to the t-1th historical moment, and t represents the current moment.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the trajectory prediction method based on denoising is implemented according to any one of claims 1 to 7.

Images