Vehicle trajectory prediction methods, autonomous driving systems, vehicles and storage media

Abstract

This invention relates to the field of autonomous driving technology, providing a vehicle trajectory prediction method, an autonomous driving system, a vehicle, and a storage medium. First, vehicle image data is acquired, and a bird's-eye view of the vehicle is generated based on this data. Then, a stochastic time residual update model algorithm is used to predict the vehicle's trajectory. When generating the bird's-eye view feature map, an encoder is first used to process the images from each camera at time t to obtain image features and depth probabilities. Then, the image features are combined with the depth probabilities to form three-dimensional features. Finally, the three-dimensional features are projected vertically onto a plane of a certain area to form a bird's-eye view feature map of the vehicle's surroundings. This invention can process images from each camera more compactly, making the connections between the images closer, and it does not rely on expensive high-precision maps, resulting in lower costs.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a vehicle trajectory prediction method, an autonomous driving system, a vehicle, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous driving technology has gradually attracted much attention, and vehicles equipped with autonomous driving functions are becoming increasingly popular. Lane changes occur constantly during autonomous driving, including changes by the vehicle itself and those by surrounding vehicles. If the trajectories of surrounding vehicles can be predicted in advance, precise control signals can be provided for the vehicle's lane changes, and accurate information can be provided for the vehicle to take contingency measures in advance when other vehicles suddenly change lanes from other lanes to its own. Therefore, accurate vehicle trajectory prediction is one of the key steps in achieving fully autonomous driving.

[0003] In the prior art, patent CN114021080A discloses a trajectory prediction model training and trajectory prediction method, device, equipment, and medium. The method involves processing historical trajectory data of a sample vehicle and historical trajectory data of surrounding vehicles using a trajectory prediction model to obtain predicted trajectory data and predicted driving behavior data of the sample vehicle. A loss function is determined based on the predicted trajectory data, the actual trajectory data, and the predicted driving behavior data of the sample vehicle. The trajectory prediction model is then trained using the loss function.

[0004] While the aforementioned technical solution achieves higher accuracy in training trajectory prediction models compared to existing technologies, thus enabling more precise trajectory prediction for autonomous vehicles, it suffers from several drawbacks. The input data for the trajectory prediction model consists of images captured by individual cameras, without integrating these images. This results in weak connections between the camera images and reliance on costly high-precision maps. Furthermore, the solution uses the traditional LSTM algorithm to encode the input data, employs pooling layers and convolutional neural networks for intermediate data processing, and finally uses a decoder network to decode and obtain the target trajectory prediction. Due to various uncertainties in real-world scenarios, the accuracy of the predicted trajectory decreases over time. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a vehicle trajectory prediction method, an autonomous driving system, a vehicle and a storage medium, which can process images from various cameras more compactly, making the images from various cameras more closely related, and without relying on expensive high-precision maps, thus reducing costs.

[0006] To achieve the above technical objectives, the technical solution adopted in this application is as follows:

[0007] In the first aspect, this application provides a vehicle trajectory prediction method, which first acquires vehicle image data, generates a bird's-eye view of the vehicle based on the vehicle image data, and then uses a stochastic time residual update model algorithm to predict the vehicle trajectory.

[0008] Furthermore, the vehicle's image data is acquired by several cameras mounted on the vehicle body.

[0009] Furthermore, the number of cameras is six, and the six cameras are respectively installed at the front, front left, front right, rear, rear left, and rear right positions of the vehicle.

[0010] Furthermore, the method for generating the bird's-eye view of the vehicle involves using a standard convolutional encoder E to obtain a set of usable features and a set of discrete depth probabilities from six different images taken at the same time. Specifically, this includes the following steps:

[0011] A1. Definition Images from six different cameras at time t;

[0012] A2. Using an encoder: For each image Encoding is performed; where C is the number of channels per image, D is the number of discrete depth values ​​per image, (H e W e The spatial resolution size is characterized by ), and the value of D is equal to the value of D. min To D max Press D size =The number of equally spaced divisions made of 1.0m, D min and D max Code the minimum and maximum depth values ​​respectively;

[0013] A3, will Divided into two parts, namely in Perform an outer product operation between the features and the depth probabilities to obtain a vector. The expression is (1), thus obtaining 3D features;

[0014] In this step, depth probability, as a form of self-attention mechanism, can adjust the features of the plane based on the predicted depth. Using known camera internal and external parameters and referencing the same frame of data, these camera vectors are... Upgraded to three-dimensional space (3D).

[0015] A4. Obtaining Bird's-Eye View Features: Using the vehicle as the center, divide the surrounding 100m*100m space into a grid with 0.5m*0.5m intervals to create a bird's-eye view. Then, project the 3D features obtained in step A3 onto the bird's-eye view in the vertical dimension using a weighted average method to form a feature map x from the bird's-eye view perspective. t ∈R C*H*W Where (H, W) = (200, 200), which is the final bird's-eye view.

[0016] Furthermore, the method for predicting vehicle trajectory using the stochastic time residual update model algorithm includes the following steps:

[0017] B1. Use encoder h0 to process the high-resolution BEV (bird's-eye view) state to obtain a low-resolution feature space with a resolution of 50*50.

[0018] B2. Use a convolutional neural network to infer the first latent variable y1 from the first three encoded states;

[0019] B3. Using a recurrent neural network composed of ConvGRU and convolutional blocks, the random latent variable z is processed. t Inferred from the respective encoded state within each time period;

[0020] B4, Based on state dynamics y t and random latent variable z t+1 , use f θ The function predicts the residual changes in the dynamics and adds them to y. t To obtain y t+1 ;

[0021] B5, from each y t state With g θ Predict at the original resolution;

[0022] B6. Predicting from the state Decoding and predicting trajectories

[0023] Furthermore, the acquisition of y t+1 The process includes the following steps:

[0024] C1. Sample from the normal distribution learned from the latent variables of the previous state, using the random latent variable z. t+1 Introducing randomness, the formula is expressed as:

[0025] z t+1 ~N( μ θ(y t ), σ θ (y t(I) (2)

[0026] C2, given z t+1 The latent variable y is determined through residual update. t and y t+1 The dependency relationship between them can be expressed by the formula:

[0027] y t+1 =y t +f θ (y t , z t+1 (3)

[0028] Among them, f θ It is a small CNN (Convolutional Neural Network) used to learn y t The residuals are updated; the distribution of future trajectories is learned from the corresponding latent variables as a normal distribution with constant diagonal variance: The initial latent variables are inferred from the assumption of standard Gaussian prior conditions: y1 ~ N(0, I).

[0029] Furthermore, in the decoded prediction trajectory At that time, the bird's-eye view status s 1：T Output mode o t Latent variable z 1：T and y 1：T The joint probability is:

[0030]

[0031] p(z t y t |y t-1 )=p(y t |y t-1 , z t )p(z t |y t-1 (5)

[0032] In the formula, p(y t |y t-1 , z t y in ) t and y t-1 The relationship is determined by the random latent residuals described in formula (3), and p(o) in formula (4) t |s t Output mode o t It is a supervised approach using a deterministic decoder to obtain from s t Learned;

[0033] From p(s) 1：T o 1：TIn the corresponding output mode, maximize the probability of the bird's-eye view state, learn the deep variational inference model q, and the inference model q is composed of Parameterization, decomposed as follows:

[0034]

[0035] In the formula, k represents the number of conditional prior frames, which is used as the residual update q(y). t |y t-1 , z t ) and p(y t |y t-1 , z t Equal to z, by keeping or removing z t For o 1：t The dependency allows for the acquisition of two versions of the model.

[0036] Secondly, the present invention also discloses an autonomous driving system that uses the above-mentioned vehicle trajectory prediction method.

[0037] Thirdly, the present invention also discloses a vehicle, the vehicle comprising a vehicle body and the aforementioned autonomous driving system, the autonomous driving system being mounted on the vehicle body.

[0038] Fourthly, the present invention also discloses a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the above-described method.

[0039] The invention employing the above technical solution has the following advantages:

[0040] 1. In generating the bird's-eye view feature map, this invention first uses an encoder to process the images from each camera at time t to obtain image features and depth probabilities; then, the image features are combined with the depth probabilities to form three-dimensional features; finally, the three-dimensional features are projected vertically onto a plane of a certain area to form a bird's-eye view feature map around the vehicle. This method processes images from various cameras around the vehicle to generate a bird's-eye view feature map, making the connections between the images from different cameras closer and improving the accuracy of vehicle trajectory prediction.

[0041] 2. This invention uses a stochastic temporal residual update model to address uncertainties in real-world scenarios, thereby improving trajectory prediction accuracy. This model learns the temporal dynamics in the latent space by performing stochastic residual updates for each time period. It samples from the learned distribution in each time period, achieving more accurate trajectory predictions over a wider spatial area and a longer time range. Furthermore, by decoupling dynamic learning and trajectory prediction generation, the model can perform inference calculations quickly without relying on costly high-precision maps, resulting in lower vehicle trajectory prediction costs. Attached Figure Description

[0042] This application can be further illustrated by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0043] Figure 1 This is an example of the original camera image used in this invention.

[0044] Figure 2 This is an example of a bird's-eye view generated by the present invention.

[0045] Figure 3 This is a flowchart illustrating the steps involved in generating a bird's-eye view of a vehicle in this invention.

[0046] Figure 4 This is a flowchart of the trajectory prediction algorithm model in this invention. Detailed Implementation

[0047] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0048] Example 1

[0049] This example demonstrates a vehicle trajectory prediction method using a stochastic time residual update model. It comprises two steps: generating a bird's-eye view feature map and performing trajectory prediction using the stochastic residual update model. The details are as follows:

[0050] I. Generation of Bird's-eye View Feature Map

[0051] First, six cameras need to be installed around the vehicle, with the installation directions being: front, front left, front right, rear, rear left, and rear right. After acquiring video images from the six cameras, for each past time period, the features of each camera image are extracted and fused into a bird's-eye view (BEV) to form a bird's-eye view image.

[0052] Specifically, the generation method is as follows: six different images taken at the same time are used with a standard convolutional encoder E to obtain a set of usable features and a set of discrete depth probabilities.

[0053] 1. Definition Images from six different cameras at time t;

[0054] 2. Use an encoder: For each image Encoding is performed. Here, C represents the number of channels per image, D represents the number of discrete depth values ​​per image, and (H... e W e The spatial resolution size is characterized by ) . The value of D is equal to the spatial resolution size from D min To D max Press D size =The number of equally spaced divisions made of 1.0m, D min and D max Code the minimum and maximum depth values ​​respectively.

[0055] 3. Divided into two parts, namely in The vector can be obtained by performing an outer product operation between the features and the depth probability. As shown in the following formula:

[0056]

[0057] Depth probability, as a form of self-attention mechanism, can adjust the features of a plane based on the predicted depth. Using known camera internal and external parameters, and referencing the same frame of data, these camera vectors... Upgraded to three-dimensional space (3D).

[0058] 4. Obtaining Bird's-Eye View Features: Using the vehicle as the center, divide the surrounding 100m*100m space into a grid with 0.5m*0.5m intervals to create a bird's-eye view. Then, project the 3D features obtained in the previous step onto the bird's-eye view in a weighted average manner in the vertical dimension to form a feature map x representing the bird's-eye view perspective. t ∈R C*H*W Where (H, W) = (200, 200), which is the final bird's-eye view.

[0059] II. Trajectory Prediction Using a Stochastic Residual Update Model

[0060] The overall steps of trajectory prediction include the following:

[0061] 1. The BEV state remains high resolution, where s t ∈R C*H*W Since (H, W) = (200, 200), we first use an encoder. The high-resolution BEV state is processed to obtain a low-resolution feature space with a resolution of 50*50.

[0062] 2. Use a convolutional neural network to infer the first latent variable y1 from the first three encoded states;

[0063] 3. Using a recurrent neural network composed of ConvGRU and convolutional blocks, the random latent variable z is processed. t Inferred from the respective encoded state within each time period;

[0064] 4. Based on the previous state dynamics y t and random latent variable z t+1 , use f θ The function predicts the residual changes in the dynamics and adds them to y. t To obtain y t+1 .

[0065] 5. From each y t ,state With g θ The original resolution is used for prediction.

[0066] 6. Finally, predict from the state Decoding Predicted Trajectory

[0067] In this embodiment, k=3 is used as a condition to predict the trajectory for the next 4 to 12 steps.

[0068] First use s 1：T This represents a series of BEV (Bird's-eye View) feature maps, used to represent the state of the vehicle and its environment over time period T. In trajectory prediction, the goal is to predict the future trajectory based on the vehicle and environment state information from the initial k time periods.

[0069] BEV states input to the stochastic prediction framework t It is an intermediate representation in a high-dimensional space, not a video frame in pixel space; the frame output Predict trajectories in the same high-dimensional space, and combine the high-dimensional... This indicates the output mode of decoding into trajectory prediction.

[0070] The state changes over time by updating the random residuals of a series of latent variables. For each state s... t Each has a corresponding latent variable y t To generate it, regardless of the previous state (e.g.) Figure 1 (As shown). Each y t+1 Only depends on the previous y t and a random variable z t+1 Randomness is determined by the random latent variable z. t+1 It is introduced by sampling from the normal distribution learned from the latent variables of the previous state, as shown in Equation (2):

[0071] z t+1 ~N(μ) θ (y t ), σ θ (y t (I) (2)

[0072] Given z t+1 Latent variable y t and y t+1 The dependencies between them are determined through residual updates, as shown in formula (3):

[0073] y t+1 =y t +f θ (y t , z t+1 (3)

[0074] Among them, f θ It is a small CNN (Convolutional Neural Network, Deep Learning) used to learn about y t The residuals are updated. The distribution of future trajectories is learned from the corresponding latent variables as a normal distribution with constant diagonal variance: The initial latent variables were inferred from the assumption of standard Gaussian prior conditions: y1 ~ N(0, 1).

[0075] BEV Status 1：T Output mode o t and latent variable z 1：T and y 1：T The joint probabilities are as follows:

[0076]

[0077] p(z t y t |y t-1 )=p(y t |y t-1 , zt )p(z t |y t-1 (5)

[0078] In formula (5), p(y) t |y t-1 , z t y in ) t and y t-1 The relationship is determined by the random latent residuals described in formula (3), and p(o) in formula (4) t |s t Output mode o t It is a supervised approach using a deterministic decoder to obtain from s t Learned.

[0079] Our goal is to obtain p(s) 1：T o 1：T To maximize the BEV state probability in the corresponding output mode, we learned a deep variational inference model q, which is composed of... Parameterization, decomposed as follows:

[0080]

[0081] In the formula, k = 3, which is the number of conditional prior frames, used as the residual update q(y). t |y t-1 , z t ) and p(y t |y t-1 , z t Equal to z, by keeping or removing z t For o 1：t Due to the dependency, two versions of the model can be obtained.

[0082] Example 2

[0083] This embodiment is an autonomous driving system that uses the vehicle trajectory prediction method of Embodiment 1 described above. The autonomous driving system in this embodiment employs a stochastic temporal residual update model to address uncertainties in real-world scenarios, thereby improving trajectory prediction accuracy. This model learns the temporal dynamics of the latent space by performing stochastic residual updates for each time period. It samples from the learned distribution in each time period, achieving more accurate trajectory predictions over a wider spatial area and a longer time span. Furthermore, by decoupling dynamic learning from trajectory prediction generation, the model can perform rapid inference calculations without relying on costly high-precision maps, resulting in lower vehicle trajectory prediction costs.

[0084] Example 3

[0085] This embodiment is a vehicle, which includes a vehicle body and the autonomous driving system of Embodiment 2, and the autonomous driving system is mounted on the vehicle.

[0086] Example 4

[0087] This embodiment is a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it causes the computer to perform the methods described above. Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or with the aid of software and necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), including several instructions to cause a computer device (such as a personal computer, braking device, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0088] In the embodiments provided in this application, it should be understood that the disclosed apparatus, systems, and methods can also be implemented in other ways. The apparatus, systems, and methods embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0089] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A vehicle trajectory prediction method, characterized in that: First, image data of the vehicle is acquired, and a bird's eye view of the vehicle is generated according to the image data of the vehicle; then, a random time residual update model algorithm is used for trajectory prediction of the vehicle; The method for trajectory prediction of the vehicle by using the random time residual update model algorithm comprises the following steps: B1, with an encoder The high-resolution bird's eye view state is processed to obtain a 50*50 resolution low-resolution feature space; B2, using a convolutional neural network to infer the first latent variable y1 from the first three encoding states; B3. Using a recurrent neural network composed of a ConvGRU and a convolutional block, the stochastic latent variable z t is inferred from the respective encoding state within each time period; B4, based on state dynamics y t and random latent variables z t+1 , with f θ predicting the residual variation in dynamics and adding it to y t to obtain y t+1 ; B5. From each y t Status In g θ at the original resolution B6, predicting from state decoding a predicted trajectory obtaining the y t+1 In the process of obtaining the y comprising the following steps: C1, a normal distribution learned from the latent variables of the previous state, from the random latent variable z t+1 Introducing randomness, the formula is expressed as: z t+1 ~N(μ θ (y t ),σ θ (y t )I) (2) C2, given z t+1 The dependence between y and y t is determined by the residual update t+1 and is formulated as y t+1 = y t + f θ (y t , z t+1 ) (3) where f θ is a small CNN that learns to update the residual of y t ; learns the distribution of future trajectories from the corresponding latent variable as a normal distribution with constant diagonal variance: The initial latent variable is inferred by assuming a standard Gaussian prior condition: y1 ~ N(0, I); When decoding the predicted trajectory , the joint probability of the bird's eye view state s 1：T , the output mode o t , the latent variable z 1:T , and y 1:T is: p(z t ,y t |y t-1 )=p(y t |y t-1 ,z t )p(z t |y t-1 ) (5) In the formula, p(y t |y t-1 ,z t y in ) t and y t-1 The relationship is determined by the random latent residual in formula (3), and p(o) in formula (4) t |s t Output mode o t It is a supervised approach using a deterministic decoder to obtain from s t Learned; maximize the posterior probability p(s 1：T , o 1：T ) over the corresponding output mode, learn a deep variational inference model q, parameterized by , decomposed as follows: where k is the number of conditional prior frames, as residual update q(y t |y t-1 ,z t ) and p(y t |y t-1 ,z t ) are equal, two versions of the model are obtained by keeping or removing the dependence of o t on z 1：t .

2. The vehicle trajectory prediction method of claim 1, wherein: The image data of the vehicle is acquired by a plurality of cameras installed on the vehicle body.

3. The vehicle trajectory prediction method of claim 2, wherein: The number of the cameras is six, and the six cameras are respectively installed at the front, left front, right front, rear, left rear and right rear positions of the vehicle.

4. The vehicle trajectory prediction method of claim 3, wherein: The method for generating the bird's eye view of the vehicle is that six different images at the same time are used to obtain a set of available features and a set of discrete depth probabilities by using a standard convolutional encoder E, and specifically comprises the following steps, A1, definition I6(t) is the image of the six different cameras at time t; A2. Using an encoder For each image Encoding is performed; where C is the number of channels per image, D is the number of discrete depth values ​​per image, (H e W e The spatial resolution size is characterized by ), and the value of D is equal to the value of D. min To D max Press D size =The number of equally spaced divisions made of 1.0m, D min and D max Code the minimum and maximum depth values ​​respectively; A3, to into two parts, i.e. wherein with the feature and depth probability, to obtain a vector The expression is (1), thereby obtaining 3D features; A4, obtain the bird's eye view feature: take the ego vehicle as the center, divide the space around 100m*100m into a grid with an interval of 0.5m*0.5m to establish a bird's eye view, and then project the 3D feature obtained in step A3 in the vertical dimension into the bird's eye view in a weighted average manner to form a feature map x of the bird's eye view t ∈R C*H*W where (H, W) = (200, 200), i.e. the last obtained bird's eye view.

5. An autonomous driving system, characterized by: The automatic driving system uses the vehicle trajectory prediction method of claim 1.

6. A vehicle characterized by: The vehicle comprises a vehicle body and the automatic driving system of claim 5, and the automatic driving system is mounted on the vehicle body.

7. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and when the computer program runs on the computer, the computer executes the vehicle trajectory prediction method of claim 1.

Images