Denoising-based trajectory prediction method and related equipment

Abstract

The invention relates to the technical field of trajectory prediction, and provides a denoising-based trajectory prediction method and related equipment, and the method comprises the steps: encoding a historical trajectory of a target vehicle to obtain a historical trajectory hidden code of the target vehicle, and encoding a historical trajectory of each neighbor vehicle to obtain a historical trajectory hidden code of the target vehicle; obtaining a historical track hidden code of each neighbor vehicle; calculating a final social code of the target vehicle according to the historical track hidden codes of all neighbor vehicles; performing trajectory prediction on the target vehicle based on the historical trajectory hidden code and the final social code of the target vehicle to obtain future trajectory distribution; and sampling the future trajectory distribution to obtain a noise future trajectory, and de-noising the noise future trajectory by using all historical trajectories to obtain a final future trajectory of the target vehicle. According to the method, the accuracy of trajectory prediction can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of trajectory prediction, and in particular to a trajectory prediction method based on denoising and related equipment. Background Art

[0002] The goal of vehicle trajectory prediction is to predict the future trajectory of a target vehicle based on its historical trajectories and those of its neighbors. Accurately predicting the future trajectory of a vehicle is critical to many autonomous driving applications, including optimizing driving path planning, making accurate driving in dynamic environments, and improving driving safety. Traditionally, statistical models predict future trajectories based on the historical trajectories of a single agent. However, these models do not consider the interactions between the target agent and its neighbors, which reduces the prediction performance. To address this issue, various deep learning-based models have been proposed to model the spatial interactions between vehicles. However, existing deep learning-based methods are unable to model the uncertainty of trajectories, which is common in real-world driving scenarios. In general, the uncertainty in autonomous driving can be roughly divided into the uncertainty of driving behavior and the uncertainty of driving scenarios. How to design uncertainty-aware trajectory prediction models has not been fully explored and is still a research topic to be solved. Due to the existence of uncertainty, there is a problem of low accuracy in trajectory prediction. Summary of the invention

[0003] The present application provides a trajectory prediction method and related devices based on denoising, which can solve the problem of low accuracy of trajectory prediction.

[0004] In a first aspect, an embodiment of the present application provides a trajectory prediction method based on denoising, the trajectory prediction method comprising:

[0005] Obtain the historical trajectory of the target vehicle and the historical trajectory of each neighboring vehicle of the target vehicle;

[0006] The historical trajectory of the target vehicle is encoded to obtain the historical trajectory hidden code of the target vehicle, and the historical trajectory of each neighbor vehicle is encoded to obtain the historical trajectory hidden code of each neighbor vehicle; the historical trajectory hidden code is used to describe the dynamic information of the historical trajectory;

[0007] According to the hidden codes of the historical trajectories of all neighboring vehicles, the final social code of the target vehicle is calculated; the final social code is used to describe the interaction information between the target vehicle and all neighboring vehicles;

[0008] Based on the target vehicle's historical trajectory hidden code and final social code, the target vehicle's trajectory is predicted to obtain the future trajectory distribution; the future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuvering mode at the future moment, as well as the probability of the target vehicle executing each maneuvering mode at the future moment;

[0009] The future trajectory distribution is sampled to obtain the noisy future trajectory, and all historical trajectories are used to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0010] Optionally, the historical trajectory of the target vehicle is encoded to obtain a hidden historical trajectory encoding of the target vehicle, including:

[0011] Encoding the historical trajectory of the target vehicle to obtain the historical trajectory encoding of the target vehicle;

[0012] The historical trajectory code is re-encoded to obtain the historical trajectory hidden code.

[0013] Optionally, the final social code of the target vehicle is calculated based on the hidden codes of the historical trajectories of all neighboring vehicles, including:

[0014] For each neighbor vehicle, generate the corresponding social code based on the hidden code of the neighbor vehicle's historical trajectory, and use the social code to calculate the amplitude embedding and phase embedding of the neighbor vehicle;

[0015] Compute the spatial representation of each neighbor vehicle based on all amplitude embeddings and all phase embeddings;

[0016] Construct an initial social encoding of the target vehicle based on all spatial representations;

[0017] The initial social code is calculated using the attention mechanism to obtain the final social code of the target vehicle.

[0018] Optionally, use social coding to calculate the amplitude embedding and phase embedding of neighbor vehicles, including:

[0019] By formula:

[0020] z j =Plain-FC(h j , W z )

[0021] θ j =Plain-FC(h j , W θ )

[0022] Calculate the amplitude embedding z of the jth neighbor vehicle j and phase embedding θ j ;

[0023] Among them, h j represents the social encoding of the jth neighbor vehicle, Plain-FC() represents the naive full connection, W z represents the learnable parameter matrix for computing amplitude embedding, W θ represents the learnable parameter matrix for computing phase embedding, j = 1, 2, ..., n, where n represents the number of neighbor vehicles.

[0024] Optionally, a spatial representation of each neighbor vehicle is calculated based on all amplitude embeddings and all phase embeddings, including:

[0025] By formula:

[0026]

[0027]

[0028] Calculate the spatial representation o of the jth neighbor vehicle j ;

[0029] in, represents a complex value, represents the amplitude embedding and phase embedding of all neighbor vehicles, represents the combination of amplitude embedding and phase embedding of the first neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the second neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the nth neighbor vehicle, M pos represents the neighbor vehicle position mask matrix, W t , and Both represent learnable weight matrices, z k represents the amplitude embedding of the kth neighbor vehicle, θ k represents the phase embedding of the kth neighbor vehicle, z k ⊙cosθ k Represents complex value The true value of z k ⊙sinθ k Represents complex value imaginary value, k∈(1, 2, ..., n), k≠j.

[0030] Optionally, based on the historical trajectory hidden code and the final social code of the target vehicle, the trajectory of the target vehicle is predicted to obtain the future trajectory distribution, including:

[0031] The target vehicle's historical trajectory hidden code and final social code are added and normalized to obtain the interactive information vector;

[0032] Adaptively fuse the interactive information vector and the mapping matrices of all maneuver modes to obtain a fusion vector;

[0033] Multiply the fusion vector and the mutual information vector to get the final vector, and calculate the final vector to get the future trajectory distribution in, represents the mean position of the target vehicle at the future time. represents the variance of the target vehicle’s position at the future moment, Represents the correlation coefficient.

[0034] Optionally, the interactive information and the mapping matrices of all maneuvering modes are adaptively fused to obtain a fusion vector, including:

[0035] By formula:

[0036]

[0037] Calculate the fusion vector

[0038] Among them, c t-t′ represents the element corresponding to the tt′th historical moment in the interaction information, Represents the combination vector of the mapping matrices of all maneuvering modes, Indicates the first T in the mapping matrix containing all maneuver modes h The elements corresponding to each historical moment, represents the elements corresponding to the first two historical moments in the mapping matrix containing all maneuver modes, represents the element corresponding to the first historical moment in the mapping matrix containing all maneuver modes, represents the mutual information vector, Indicates the first h The interaction information corresponding to the historical moment, c t-2 represents the interaction information corresponding to the t-2th historical moment, c t-1 represents the interaction information corresponding to the t-1th historical moment, and t represents the current moment.

[0039] Optionally, the future trajectory distribution is sampled to obtain a noisy future trajectory, including:

[0040] The maneuvering mode corresponding to the maximum probability of the median of the future trajectory distribution is taken as the final maneuvering mode, and the future trajectory generated by the target vehicle executing the final maneuvering mode is taken as the noise future trajectory.

[0041] Optionally, all historical trajectories are used to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle, including:

[0042] By formula:

[0043]

[0044]

[0045]

[0046] Calculate the future trajectory of the noise after denoising in step r

[0047] Among them, α r , are the parameters of the diffusion process. represents the future trajectory of the noise after denoising in the r+1th step, represents the estimated noise, z represents the noise, z~N(z;0,I), I represents the unit matrix, f ∈ () represents the noise estimation model, represents spatiotemporal embedding, X tar represents the historical trajectory of the target vehicle, represents the historical trajectories of all neighboring vehicles, f context () represents the information encoder, r = 1, 2, ..., R, R represents the number of steps in the denoising process, when r = 1, the future trajectory of the noise after the first step of denoising is as the final future trajectory of the target vehicle.

[0048] In a second aspect, an embodiment of the present application provides a trajectory prediction device based on denoising, comprising:

[0049] An acquisition module is used to acquire the historical trajectory of the target vehicle and the historical trajectory of each neighboring vehicle of the target vehicle;

[0050] The encoding module encodes the historical trajectory of the target vehicle to obtain the hidden historical trajectory code of the target vehicle, and encodes the historical trajectory of each neighbor vehicle to obtain the hidden historical trajectory code of each neighbor vehicle; the hidden historical trajectory code is used to describe the dynamic information of the historical trajectory;

[0051] The calculation module calculates the final social code of the target vehicle based on the hidden codes of the historical trajectories of all neighboring vehicles. The final social code is used to describe the interaction information between the target vehicle and all neighboring vehicles.

[0052] The trajectory prediction module predicts the target vehicle’s trajectory based on the target vehicle’s historical trajectory hidden code and final social code to obtain the future trajectory distribution; the future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuvering mode at the future moment, as well as the probability of the target vehicle executing each maneuvering mode at the future moment;

[0053] The sampling module samples the future trajectory distribution to obtain the noisy future trajectory, and uses all historical trajectories to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0054] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned denoising-based trajectory prediction method when executing the above-mentioned computer program.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned denoising-based trajectory prediction method is implemented.

[0056] The above solution of the present application has the following beneficial effects:

[0057] In an embodiment of the present application, the historical trajectory of the target vehicle is obtained, and the historical trajectory of each neighbor vehicle of the target vehicle is obtained, and then the historical trajectory of the target vehicle is encoded to obtain the hidden code of the historical trajectory of the target vehicle, and the historical trajectory of each neighbor vehicle is encoded to obtain the hidden code of the historical trajectory of each neighbor vehicle, and then the final social code of the target vehicle is calculated based on the hidden code of the historical trajectory of all neighbor vehicles, and then the trajectory of the target vehicle is predicted based on the hidden code of the historical trajectory of the target vehicle and the final social code to obtain the future trajectory distribution, and finally the future trajectory distribution is sampled to obtain the noisy future trajectory, and the noisy future trajectory is denoised using all historical trajectories to obtain the final future trajectory of the target vehicle. Among them, the final social code of the target vehicle is calculated based on the hidden code of the historical trajectory of the neighbor vehicles, so as to fully analyze the interaction information between the target vehicle and the neighbor vehicles, so that the final social code can accurately describe the interaction information between the target vehicle and all neighbor vehicles, and the accuracy of the future trajectory distribution calculated based on the accurate final social code is high, and then the future trajectory distribution is sampled and denoised, which can reduce the uncertainty of the trajectory and further improve the accuracy of trajectory prediction.

[0058] In addition, the process of first obtaining the future trajectory distribution and then obtaining the final future trajectory through sampling and denoising realizes the trajectory prediction from coarse to fine stages, which can well capture the interaction between vehicles and model the multimodal trajectory of vehicles. Most importantly, the uncertainty of the future trajectory can be gradually reduced by sampling and gradually denoising.

[0059] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 A flowchart of a trajectory prediction method based on denoising provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of the structure of a trajectory prediction device based on denoising provided in one embodiment of the present application;

[0063] Figure 3 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0065] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0066] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0067] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0068] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0069] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0070] In view of the low accuracy of existing trajectory prediction, the embodiment of the present application provides a trajectory prediction method based on denoising, which obtains the historical trajectory of the target vehicle and the historical trajectory of each neighbor vehicle of the target vehicle, then encodes the historical trajectory of the target vehicle to obtain the hidden code of the historical trajectory of the target vehicle, and encodes the historical trajectory of each neighbor vehicle to obtain the hidden code of the historical trajectory of each neighbor vehicle, and then calculates the final social code of the target vehicle based on the hidden code of the historical trajectory of all neighbor vehicles, and then predicts the trajectory of the target vehicle based on the hidden code of the historical trajectory of the target vehicle and the final social code to obtain the future trajectory distribution, and finally samples the future trajectory distribution to obtain the noisy future trajectory, and uses all historical trajectories to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle. Among them, the final social code of the target vehicle is calculated based on the hidden code of the historical trajectory of the neighbor vehicles, so as to fully analyze the interactive information between the target vehicle and the neighbor vehicles, so that the final social code can accurately describe the interactive information between the target vehicle and all neighbor vehicles, and the future trajectory distribution calculated based on the accurate final social code is highly accurate, and then the future trajectory distribution is sampled and denoised, which can reduce the uncertainty of the trajectory and further improve the accuracy of trajectory prediction.

[0071] In addition, the process of first obtaining the future trajectory distribution and then obtaining the final future trajectory through sampling and denoising realizes the trajectory prediction from coarse to fine stages, which can well capture the interaction between vehicles and model the multimodal trajectory of vehicles. Most importantly, the uncertainty of the future trajectory can be gradually reduced by sampling and gradually denoising.

[0072] Next, the trajectory prediction method based on denoising provided in this application is exemplified.

[0073] like Figure 1 As shown, the trajectory prediction method based on denoising provided by the present application includes the following steps:

[0074] Step 11, obtain the historical trajectory of the target vehicle, and obtain the historical trajectory of each neighbor vehicle of the target vehicle.

[0075] The target vehicle is the vehicle that needs to be predicted, the neighbor vehicle is the vehicle adjacent to the target vehicle, and the historical trajectory is the movement trajectory of the vehicle in the historical time period before the current time. For example, if the current time is 9 o'clock, the historical trajectory can be the movement trajectory of the vehicle from 7 o'clock to 8:30, and the neighbor vehicle is the adjacent vehicle of the target vehicle in the historical time period.

[0076] In some embodiments of the present application, the historical trajectory may be acquired using the positioning systems of the target vehicle and neighboring vehicles.

[0077] Step 12: Encode the historical trajectory of the target vehicle to obtain a hidden code of the historical trajectory of the target vehicle, and encode the historical trajectory of each neighboring vehicle to obtain a hidden code of the historical trajectory of each neighboring vehicle.

[0078] The above historical trajectory hidden code is used to describe the dynamic information of the historical trajectory, such as the moving direction and speed of the vehicle when traveling along the historical trajectory.

[0079] In some embodiments of the present application, the above-mentioned step of encoding the historical trajectory of the target vehicle to obtain the hidden code of the historical trajectory of the target vehicle is specifically:

[0080] The first step is to encode the historical trajectory of the target vehicle to obtain the historical trajectory encoding of the target vehicle.

[0081] Exemplarily, a multi-layer perceptron may be used to encode the historical trajectory of the target vehicle to obtain the historical trajectory encoding of the target vehicle.

[0082] The second step is to re-encode the historical trajectory code to obtain the hidden code of the historical trajectory.

[0083] Exemplarily, the historical trajectory encoding can be re-encoded using a long short-term memory network to obtain a hidden encoding of the historical trajectory.

[0084] It should be noted that the process of encoding the historical trajectory of each neighbor vehicle to obtain the hidden code of the historical trajectory of each neighbor vehicle is the same as the process of obtaining the hidden code of the historical trajectory of the target vehicle, that is, the historical trajectory of the neighbor vehicle is encoded to obtain the historical trajectory code of the neighbor vehicle, and then the historical trajectory code is re-encoded to obtain the hidden code of the historical trajectory.

[0085] Step 13, calculate the final social code of the target vehicle based on the hidden codes of the historical trajectories of all neighboring vehicles.

[0086] The above final social coding is used to describe the interaction information between the target vehicle and all neighbor vehicles, such as the distance between the neighbor vehicles and the target vehicle, the relative speed between the neighbor vehicles and the target vehicle, etc.

[0087] In some embodiments of the present application, the above step of calculating the final social code of the target vehicle based on the historical trajectory hidden codes of all neighboring vehicles is specifically as follows:

[0088] In the first step, for each neighbor vehicle, the social code corresponding to the neighbor vehicle is generated based on the hidden code of the neighbor vehicle’s historical trajectory, and the amplitude embedding and phase embedding of the neighbor vehicle are calculated using the social code.

[0089] In some embodiments of the present application, the above-mentioned step of generating the social code corresponding to the neighbor vehicle based on the hidden code of the historical trajectory of the neighbor vehicle is specifically as follows: the mask matrix mask representing the positional relationship between the target vehicle and each neighbor vehicle is expanded to a shape consistent with the historical time period corresponding to the historical trajectory. Then a new tensor is created to represent the social code, and it is initialized to an all-zero tensor of the same shape as the mask matrix. Afterwards, the masked_scatter_ function of pytorch is used to fill the hidden code of the historical trajectory of the neighbor vehicle into the social code according to the mask matrix. Specifically, masked_scatter_ allows specific values ​​of a tensor to be inserted bit by bit into another tensor according to a Boolean mask, that is, for positions where the mask is True, the corresponding values ​​in the hidden code of the historical trajectory of the neighbor vehicle are filled into the social code, and for positions where the mask is False, the social code remains the same.

[0090] The above steps of using social coding to calculate the amplitude embedding and phase embedding of neighbor vehicles are as follows:

[0091] By formula:

[0092] z j =Plain-FC(hj , W z )

[0093] θ j =Plain-FC(h j , W θ )

[0094] Calculate the amplitude embedding z of the jth neighbor vehicle j and phase embedding θ j .

[0095] Among them, h j represents the social encoding of the jth neighbor vehicle, Plain-FC() represents the naive full connection, W z Represents the learnable weight matrix of amplitude embedding, W θ represents the learnable weight matrix of phase embedding, j = 1, 2, ..., n, where n represents the number of neighbor vehicles.

[0096] In the second step, the spatial representation of each neighbor vehicle is calculated based on all amplitude embeddings and all phase embeddings.

[0097] Specifically, through the formula:

[0098]

[0099]

[0100] Calculate the spatial representation o of the jth neighbor vehicle j .

[0101] in, represents a complex value, represents the amplitude embedding and phase embedding of all neighbor vehicles, represents the combination of amplitude embedding and phase embedding of the first neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the second neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the nth neighbor vehicle, M pos represents the neighbor vehicle position mask matrix, W t , and Both represent learnable weight matrices, z k represents the amplitude embedding of the kth neighbor vehicle, θ k represents the phase embedding of the kth neighbor vehicle, z k ⊙cosθ k Represents complex value The true value of z k ⊙sinθ k Represents complex value imaginary value, k∈(1, 2, ..., n), k≠j.

[0102] It should be noted that the calculation of the above Surrounding-FC is as follows: through convolution operations, phase embedding and amplitude embedding are extracted for neighboring vehicles. Then the phase embedding and amplitude embedding are expanded through the Euler formula to capture the relative interactions and spatiotemporal relationships between vehicles. After that, these features are re-weighted using adaptive pooling, so that each feature can be automatically adjusted according to the relative importance. Finally, after further processing by a multi-layer perceptron (MLP), a spatial representation with enhanced phase information and interaction features is generated.

[0103] In the third step, an initial social encoding of the target vehicle is constructed based on all spatial representations.

[0104] It should be noted that for the spatial representation o j , which is a vector composed of multiple elements, each of which corresponds to multiple historical moments, and is used to describe the spatial information of neighbor vehicles at each historical moment. For a historical moment, the elements corresponding to the historical moment in all spatial representations are integrated to obtain the social interaction representation, and the social interaction representations of all historical moments are integrated to obtain the initial social coding in, Indicates the first h A social interaction representation of a historical moment, h t-2 represents the social interaction representation at the t-2th historical moment, h t - 1 Represents the social interaction representation at the t-1th historical moment.

[0105] In the fourth step, the initial social code is calculated using the attention mechanism to obtain the final social code of the target vehicle.

[0106] Specifically, the initial social code is used as the input of the attention mechanism, and the query matrix (Q), key matrix (K) and value matrix (V) are calculated based on the self-attention mechanism. Then, the dot product of the query matrix and the key matrix is ​​calculated to obtain the attention score, which is used to capture the dependencies between historical moments, that is, the relationship between the features of different historical moments. Then, the attention score is normalized using softmax to ensure that the attention distribution at each historical moment is a probability distribution and the sum is 1. The value matrix is ​​weighted and summed according to the calculated attention score to obtain the new historical moment feature. Finally, the results of different attention heads are merged to obtain the final social code. The expression is:

[0107]

[0108] Among them, Concat represents the concatenation operation, head1 represents the output of the first attention head, head2 represents the output of the second attention head, and head k represents the output of the kth attention head.

[0109] It is worth mentioning that the final social code of the target vehicle is calculated based on the hidden code of the historical trajectories of the neighbor vehicles, which fully analyzes the interaction information between the target vehicle and the neighbor vehicles, so that the final social code can accurately describe the interaction information between the target vehicle and all neighbor vehicles.

[0110] Step 14: Based on the historical trajectory hidden code and the final social code of the target vehicle, the trajectory of the target vehicle is predicted to obtain the future trajectory distribution.

[0111] The above future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuver mode at the future moment, as well as the probability of the target vehicle executing each maneuver mode at the future moment. Maneuver mode is used to describe the motion status of the target vehicle, such as changing lanes left, changing lanes right, keeping lanes, accelerating, maintaining speed, etc.

[0112] In some embodiments of the present application, the above-mentioned steps of performing trajectory prediction on the target vehicle based on the historical trajectory hidden code and the final social code of the target vehicle to obtain the future trajectory distribution are specifically as follows:

[0113] In the first step, the historical trajectory hidden code and the final social code of the target vehicle are added and normalized to obtain the interactive information vector.

[0114] Exemplarily, the normalization process may be performed using an existing normalization formula.

[0115] In the second step, the interactive information vector and the mapping matrices of all maneuvering modes are adaptively fused to obtain the fused vector.

[0116] Specifically, through the formula:

[0117]

[0118] Calculate the fusion vector

[0119] Among them, c t-t′ represents the element corresponding to the tt′th historical moment in the interaction information, Represents the combination vector of the mapping matrices of all maneuvering modes, Indicates the first T in the mapping matrix containing all maneuver modes h The elements corresponding to each historical moment, represents the elements corresponding to the first two historical moments in the mapping matrix containing all maneuver modes, represents the element corresponding to the first historical moment in the mapping matrix containing all maneuver modes, represents the mutual information vector, Indicates the first tT h The interaction information corresponding to the historical moments, c t-2 represents the interaction information corresponding to the t-2th historical moment, c t-1 represents the interaction information corresponding to the t-1th historical moment, and t represents the current moment.

[0120] It should be noted that the mapping matrix of the maneuvering mode is preset according to the maneuvering status of the maneuvering mode, and has a plurality of elements, and the plurality of elements correspond one-to-one to a plurality of historical moments.

[0121] The third step is to multiply the fusion vector and the interaction information vector to obtain the final vector, and then calculate the final vector to obtain the future trajectory distribution in, represents the mean position of the target vehicle at the future time. represents the variance of the target vehicle’s position at the future moment, Represents the correlation coefficient.

[0122] For example, the final vector can be calculated using a long short-term memory network to obtain the future trajectory distribution, which is a Gaussian distribution.

[0123] It is worth mentioning that the future trajectory distribution calculated based on the accurate final social coding is highly accurate and can accurately describe the probability of the target vehicle executing each maneuver mode at the future moment.

[0124] Step 15, sampling the future trajectory distribution to obtain the noisy future trajectory, and using all historical trajectories to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0125] In some embodiments of the present application, the steps of sampling the future trajectory distribution to obtain the noisy future trajectory, and denoising the noisy future trajectory using all historical trajectories to obtain the final future trajectory of the target vehicle are specifically as follows:

[0126] In the first step, the future trajectory distribution is sampled to obtain the noisy future trajectory.

[0127] Specifically, the maneuvering mode corresponding to the maximum probability of the median value of the future trajectory distribution is taken as the final maneuvering mode, and the future trajectory generated by the target vehicle executing the final maneuvering mode is taken as the noise future trajectory.

[0128] In the second step, all historical trajectories are used to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0129] Specifically, through the formula:

[0130]

[0131]

[0132]

[0133] Calculate the future trajectory of the noise after denoising in step r

[0134] Among them, α r , are the parameters of the diffusion process. represents the future trajectory of the noise after denoising in the r+1th step, represents the estimated noise, z represents the noise, z~N(z;0,I), I represents the unit matrix, f ∈ () represents the noise estimation model, represents spatiotemporal embedding, X tar represents the historical trajectory of the target vehicle, represents the historical trajectories of all neighboring vehicles, f context () represents the information encoder, r = 1, 2, ..., R, R represents the number of steps in the denoising process, when r = 1, the future trajectory of the noise after the first step of denoising is as the final future trajectory of the target vehicle.

[0135] It should be noted that in the above formula, when r=R, is the future trajectory of the input noise. The above denoising process is a reverse process, from the last step to the first step (similar to the reverse diffusion process of the diffusion model). The above information encoder can be a transformer-based encoder.

[0136] Exemplarily, the method of the present application can be implemented on an NVIDIA 3090 GPU server using the Pytorch framework. The parameters are set as follows: a 13×5 grid is used, which is defined with the target vehicle as the center, where each column corresponds to a lane and each row is 15 feet apart. A multilayer perceptron and a long short-term memory network are used to calculate the hidden encoding of the historical trajectory. The hidden features of the multilayer perceptron are set to 32, and the activation function is ReL u. In order to train the framework corresponding to the method of the present application, a two-stage training strategy is considered, in which the first stage trains the denoising module (i.e., the formula involved in step 15 of the present application), and the second stage focuses on training the spatiotemporal interaction module (i.e., the formulas, models, etc. involved in steps 11 to 14 of the present application). Each historical trajectory is segmented into segments over a sensing range (i.e., 8s), which contains the past (3s) and future (5s) positions of 5Hz. The data set consisting of all historical trajectories after segmentation is divided into a training set, a validation set, and a test set with a segmentation ratio of 7:2:1, which is used to execute the method of the present application, and to perform training, validation, and testing.

[0137] It is worth mentioning that the final social code of the target vehicle is calculated based on the hidden codes of the historical trajectories of the neighbor vehicles, so that the interaction information between the target vehicle and the neighbor vehicles can be fully analyzed, so that the final social code can accurately describe the interaction information between the target vehicle and all neighbor vehicles. The future trajectory distribution calculated based on the accurate final social code is highly accurate, and then the future trajectory distribution is sampled and denoised, which can reduce the uncertainty of the trajectory and further improve the accuracy of trajectory prediction.

[0138] In addition, the process of first obtaining the future trajectory distribution and then obtaining the final future trajectory through sampling and denoising realizes the trajectory prediction from coarse to fine stages, which can well capture the interaction between vehicles and model the multimodal trajectory of vehicles. Most importantly, the uncertainty of the future trajectory can be gradually reduced by sampling and gradually denoising.

[0139] The following is an exemplary description of the trajectory prediction device based on denoising provided by the present application.

[0140] like Figure 2 As shown, the embodiment of the present application provides a trajectory prediction device based on denoising, and the trajectory prediction device based on denoising 200 includes:

[0141] An acquisition module 201 acquires a historical track of a target vehicle and acquires a historical track of each neighboring vehicle of the target vehicle;

[0142] The encoding module 203 encodes the historical trajectory of the target vehicle to obtain a hidden historical trajectory code of the target vehicle, and encodes the historical trajectory of each neighboring vehicle to obtain a hidden historical trajectory code of each neighboring vehicle; the hidden historical trajectory code is used to describe the dynamic information of the historical trajectory;

[0143] The calculation module 204 calculates the final social code of the target vehicle according to the historical trajectory hidden codes of all neighboring vehicles; the final social code is used to describe the interaction information between the target vehicle and all neighboring vehicles;

[0144] The trajectory prediction module 205 predicts the trajectory of the target vehicle based on the historical trajectory hidden code and the final social code of the target vehicle to obtain a future trajectory distribution; the future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuvering mode at a future moment, and the probability of the target vehicle executing each maneuvering mode at a future moment;

[0145] The sampling module 206 samples the future trajectory distribution to obtain the noisy future trajectory, and uses all the historical trajectories to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle.

[0146] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0147] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0148] like Figure 3 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0149] Specifically, when the processor D100 executes the computer program D102, it obtains the historical trajectory of the target vehicle and the historical trajectory of each neighbor vehicle of the target vehicle, then encodes the historical trajectory of the target vehicle to obtain the hidden code of the historical trajectory of the target vehicle, and encodes the historical trajectory of each neighbor vehicle to obtain the hidden code of the historical trajectory of each neighbor vehicle, and then calculates the final social code of the target vehicle based on the hidden code of the historical trajectory of all neighbor vehicles, and then predicts the trajectory of the target vehicle based on the hidden code of the historical trajectory of the target vehicle and the final social code to obtain the future trajectory distribution, and finally samples the future trajectory distribution to obtain the noisy future trajectory, and uses all historical trajectories to denoise the noisy future trajectory to obtain the final future trajectory of the target vehicle. Among them, the final social code of the target vehicle is calculated based on the hidden code of the historical trajectory of the neighbor vehicles, so as to fully analyze the interactive information between the target vehicle and the neighbor vehicles, so that the final social code can accurately describe the interactive information between the target vehicle and all neighbor vehicles, and the future trajectory distribution calculated based on the accurate final social code is highly accurate, and then the future trajectory distribution is sampled and denoised, which can reduce the uncertainty of the trajectory and further improve the accuracy of trajectory prediction.

[0150] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0151] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0152] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0153] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the trajectory prediction method device / terminal device based on denoising, recording medium, computer memory, read-only memory (ROM, Read-OnlyMemory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0156] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0157] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A trajectory prediction method based on denoising, characterized in that: include: Obtaining a historical trajectory of a target vehicle and obtaining a historical trajectory of each neighboring vehicle of the target vehicle; Encoding the historical trajectory of the target vehicle to obtain a hidden code of the historical trajectory of the target vehicle, and encoding the historical trajectory of each of the neighboring vehicles to obtain a hidden code of the historical trajectory of each of the neighboring vehicles; The historical track hidden code is used to describe the dynamic information of the historical track; Calculate the final social code of the target vehicle based on the hidden codes of the historical trajectories of all neighboring vehicles; The final social code is used to describe the interaction information between the target vehicle and all neighboring vehicles; Based on the historical trajectory hidden code and the final social code of the target vehicle, the trajectory of the target vehicle is predicted to obtain a future trajectory distribution; the future trajectory distribution is used to describe the future trajectory generated by the target vehicle executing each maneuvering mode at a future moment, and the probability of the target vehicle executing each maneuvering mode at a future moment; The future trajectory distribution is sampled to obtain a noisy future trajectory, and all historical trajectories are used to denoise the noisy future trajectory to obtain a final future trajectory of the target vehicle.

2. The trajectory prediction method according to claim 1, characterized in that: The encoding of the historical track of the target vehicle to obtain the hidden code of the historical track of the target vehicle includes: Encoding the historical trajectory of the target vehicle to obtain the historical trajectory code of the target vehicle; The historical track code is re-encoded to obtain the historical track hidden code.

3. The trajectory prediction method according to claim 1, characterized in that: The step of calculating the final social code of the target vehicle based on the hidden codes of the historical trajectories of all neighboring vehicles includes: For each of the neighbor vehicles, based on the hidden code of the historical trajectory of the neighbor vehicle, generate a social code corresponding to the neighbor vehicle, and use the social code to calculate the amplitude embedding and phase embedding of the neighbor vehicle; Compute the spatial representation of each neighbor vehicle based on all amplitude embeddings and all phase embeddings; constructing an initial social encoding of the target vehicle based on all spatial representations; The initial social code is calculated using an attention mechanism to obtain a final social code of the target vehicle.

4. The trajectory prediction method according to claim 3, characterized in that: The calculating the amplitude embedding and phase embedding of the neighbor vehicle by using the social coding includes: By formula: z j =Plain-FC(h j ,W z ) θ j =Plain-FC(h j ,W θ ) Calculate the amplitude embedding z of the jth neighbor vehicle j and phase embedding θ j ; Among them, h j represents the social encoding of the jth neighbor vehicle, Plain-FC ( ) represents the naive full connection, W z represents the learnable weight matrix for computing amplitude embedding, W θ represents the learnable weight matrix for computing phase embedding, j = 1, 2, …, n, where n represents the number of neighbor vehicles.

5. The trajectory prediction method according to claim 4, characterized in that: The spatial representation of each neighbor vehicle is calculated based on all amplitude embeddings and all phase embeddings, including: By formula: Calculate the spatial representation o of the jth neighbor vehicle j ; in, represents a complex value, represents the amplitude embedding and phase embedding of all neighbor vehicles, represents the combination of amplitude embedding and phase embedding of the first neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the second neighbor vehicle, represents the combination of amplitude embedding and phase embedding of the nth neighbor vehicle, M pos represents the neighbor vehicle position mask matrix, W t , and Both represent learnable weight matrices, z k represents the amplitude embedding of the kth neighbor vehicle, θ k represents the phase embedding of the kth neighbor vehicle, z k ⊙cosθ k Represents complex value The true value of z k ⊙sinθ k Represents complex value imaginary value, k∈(1,2,…,n), k≠j.

6. The trajectory prediction method according to claim 1, characterized in that: The method of predicting the trajectory of the target vehicle based on the historical trajectory hidden code and the final social code of the target vehicle to obtain the future trajectory distribution includes: The historical trajectory hidden code and the final social code of the target vehicle are added and normalized to obtain an interactive information vector; Adaptively fusing the interaction information vector and the mapping matrices of all maneuvering modes to obtain a fusion vector; The fusion vector and the mutual information vector are multiplied to obtain a final vector, and the final vector is calculated to obtain the future trajectory distribution in, represents the mean value of the target vehicle's position at the future time, represents the variance of the target vehicle's position at the future time, Represents the correlation coefficient.

7. The trajectory prediction method according to claim 6, characterized in that: The step of adaptively fusing the interactive information and the mapping matrices of all motor modes to obtain a fusion vector includes: By formula: Calculate the fusion vector Among them, c t-t' represents the element corresponding to the t-t'th historical moment in the interaction information, Represents the combination vector of the mapping matrices of all maneuvering modes, Represents the -Tth in the mapping matrix containing all maneuver modes h The elements corresponding to each historical moment, represents the element corresponding to the -2th historical moment in the mapping matrix containing all maneuver modes, represents the element corresponding to the -1th historical moment in the mapping matrix containing all maneuver modes, represents the mutual information vector, Indicates the first h The interaction information corresponding to the historical moments, c t-2 represents the interaction information corresponding to the t-2th historical moment, c t-1 represents the interaction information corresponding to the t-1th historical moment, and t represents the current moment.

8. The trajectory prediction method according to claim 1, characterized in that: The sampling of the future trajectory distribution to obtain the noisy future trajectory includes: The maneuvering mode corresponding to the maximum probability of the median value of the future trajectory distribution is used as the final maneuvering mode, and the future trajectory generated by the target vehicle executing the final maneuvering mode is used as the noise future trajectory.

9. The trajectory prediction method according to claim 1, characterized in that: The method of denoising the noisy future trajectory by using all historical trajectories to obtain the final future trajectory of the target vehicle includes: By formula: Calculate the future trajectory of the noise after denoising in step r Among them, α r , are the parameters of the diffusion process. represents the future trajectory of the noise after denoising in the r+1th step, represents the estimated noise, z represents the noise, z~N(z;0,I), I represents the unit matrix, f ∈ ( ) represents the noise estimation model, represents spatiotemporal embedding, X tar represents the historical trajectory of the target vehicle, represents the historical trajectories of all neighboring vehicles, f context ( ) represents the information encoder, r = 1, 2, ..., R, R represents the number of steps in the denoising process, when r = 1, the future trajectory of the noise after the first step of denoising is as the final future trajectory of the target vehicle.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the trajectory prediction method based on denoising is implemented as described in any one of claims 1 to 9.

Images