A vehicle multi-modal trajectory prediction method based on a semi-supervised model

Abstract

The present application relates to a kind of vehicle multimodal trajectory prediction method based on semi-supervised model, comprising: based on road map information acquisition vehicle's future possible intention and possible road and center line of travel, and the road map information around target vehicle and the historical trajectory of other vehicles around target vehicle are encoded, subsequently using semi-supervised model, for future possible target vehicle trajectory mode Generation pseudo label, and using pseudo label under different modal prediction trajectory is supervised learning, realize the multimodal prediction of vehicle trajectory.Compared with prior art, the present application converts multimodal problem into semi-supervised problem, and uses mean-teacher semi-supervised model to train model, can effectively improve the prediction accuracy of vehicle trajectory, and can solve the data imbalance problem existing in vehicle trajectory prediction, can effectively improve the modal distribution in vehicle prediction result, significantly improve trajectory prediction quality.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving control technology, and in particular to a method for predicting vehicle multimodal trajectories based on a semi-supervised model. Background Technology

[0002] From the first fully autonomous vehicle, the "American Wonder," in 1925 to the current commercialization of Level 3 autonomous driving with the Tesla Model S, vehicle intelligence has increased significantly. However, accurately modeling and predicting the future intentions and trajectories of surrounding road users remains a major challenge for autonomous driving technology. During autonomous driving, vehicles not only need to acquire the historical and current states of surrounding vehicles but also predict their future intentions and trajectories based on this information and map data. This allows for the timely detection of potential hazards in the surrounding environment, supporting effective decision-making and planning for timely hazard avoidance. Therefore, predicting the intentions and future trajectories of surrounding road users is crucial for improving the safety of autonomous vehicles.

[0003] Trajectory prediction is typically located at the back end of the perception module and the front end of the planning and control module. Inputs include historical trajectory information and road structure information provided by the perception module. It comprehensively considers high-precision map information and the interactions between predicted targets to predict the intentions (cut in / out, straight ahead, etc.) and future trajectories of various perceived targets (pedestrians, vehicles). Vehicle behavior is constrained by inertia, driving rules, and road geometry. Therefore, considering these characteristics of autonomous vehicles, typical vehicle trajectory prediction methods generally include a vehicle historical trajectory encoder, a vehicle interaction modeling encoder, and a road feature extractor, used to obtain the historical motion features of the vehicle trajectory, model the interaction features between vehicles, and extract surrounding road map information, respectively.

[0004] Single-modal trajectory prediction does not account for the uncertainties of driving behavior, and the predicted results may differ significantly from the actual situation. Furthermore, during training, single-modal vehicle trajectory prediction networks, in order to minimize error loss, may learn a predicted trajectory that is an average of motion trajectories from different modalities.

[0005] In early research, researchers used manual methods or clustering algorithms to segment modalities for multimodal prediction. Since vehicles move on highly structured roads, they can be pre-segmented into modes such as lane changing, overtaking, deceleration, and acceleration. The trajectories of different modes are then input into different decoders, or different one-hot vectors are concatenated and input into the decoder. Alternatively, regions can be pre-defined, and modalities can be distinguished based on the region where the vehicle's trajectory is located; however, manual segmentation is time-consuming and labor-intensive. Generative models can learn the multimodal characteristics of target motion by adding multidimensional Gaussian-distributed latent variables to the input of the trajectory prediction network. During prediction, multimodal trajectories are generated by sampling the latent variables multiple times. Social GAN ​​networks use generative models to generate multimodal trajectory predictions. By introducing a new diversity loss function, diverse predictions are encouraged, resulting in a large and socially reasonable trajectory sample space. However, these methods may result in high-probability modes being repeatedly sampled while low-probability modes are not sampled. Furthermore, the multimodal data generated by these methods does not have a clear modeling relationship with road information. Target-driven methods can explicitly utilize road map information to generate multimodal trajectories. HOME uses a decoder to generate a heatmap of target point distribution, then samples the heatmap to generate K target points, and finally generates a complete trajectory based on these target points. DenseTNT discretizes the road and uses an attention mechanism to obtain a probabilistic heatmap of target point distribution, then selects K target points through offline optimization and generates the entire trajectory based on these target points. However, target-point-driven trajectory prediction generally requires extensive sampling and filtering of target points, which increases the computational load of the model. The above methods also suffer from significant sample imbalance, meaning the amount of data for straight-ahead travel is much greater than the amount for turning travel. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a vehicle multimodal trajectory prediction method based on a semi-supervised model. This method can accurately, quickly and effectively determine the vehicle modes that may occur in the future, alleviate the data imbalance problem in trajectory prediction, and better predict the trajectory mode distribution and improve the trajectory prediction quality of the steering mode without increasing the computational load during inference.

[0007] The objective of this invention can be achieved through the following technical solution: a vehicle multimodal trajectory prediction method based on a semi-supervised model, comprising the following steps:

[0008] S1. Based on the historical trajectory of the target vehicle, obtain the lane the vehicle is traveling in and the corresponding lane centerline information, and obtain the possible future motion modes of the vehicle according to the direction of the lane centerline.

[0009] The vehicle's future real-world modality is obtained by matching its future trajectory with the lane centerline.

[0010] S2. Using the encoders in the student and teacher networks, encode the historical trajectory of the target vehicle, the historical trajectory of other vehicles within a set range around the target vehicle, and the environmental map within a set range around the target vehicle, respectively, to obtain the corresponding feature vector of the target vehicle trajectory, the feature vector of the surrounding vehicle trajectory, and the feature vector of the surrounding environment.

[0011] S3. In the student network of the mean_teacher model, all possible lane centerlines for the future driving of the vehicle are divided into N points. These points are encoded and concatenated with the output vector and modal vector of the encoder, and then input into the decoder.

[0012] Using the same approach, in the teacher network of the mean_teacher model, except for the lane centerline where the future true mode is located, all lane centerlines that the vehicle may travel in the future are divided into N points. These points are encoded and concatenated with the encoder's output vector, mode vector, etc., and then input into the decoder.

[0013] S4. Filter the trajectories output in the teacher network and use the filtered trajectories as pseudo-labels in the student network.

[0014] The mean-teacher model is trained by adding a constraint loss to the regression and classification losses during the training process to generate trajectories that conform to road constraints, thus obtaining a multimodal trajectory prediction model.

[0015] S5. Input the historical trajectory of the target vehicle and surrounding vehicles, surrounding map information, and the centerline information of all lanes to be driven in the future into the multimodal trajectory prediction model, and output the corresponding trajectory prediction results.

[0016] Furthermore, step S1 specifically includes the following steps:

[0017] S1.1 Obtain the trajectory of the target vehicle and other surrounding vehicles. Based on the historical trajectories of all vehicles and the current position of the target vehicle, search the map for the center lines of all lanes within a set range near the current location of the target vehicle.

[0018] S1.2. Based on the historical trajectory of the target vehicle, filter out the center line of the lane where the vehicle is currently located, and obtain all possible lane center lines for future travel using a breadth-first approach based on the current lane line. Mark each lane center line as a different mode: left turn, right turn, or straight ahead, according to its direction.

[0019] The complete trajectory, including historical and future trajectories, is matched with the obtained lane centerline to obtain the lane centerline for the vehicle's future travel.

[0020] S1.3. Classify all lane centerlines that the vehicle may travel in the future according to modality. Select the lane centerline closest to the current vehicle position in each class. If there is no lane centerline in a class, fill it with 0 as the lane centerline coordinates. Further, the specific process of searching for all lane centerlines within a set range near the current location of the target vehicle in step S1.1 is as follows:

[0021] The system searches for all lane lines within a set range of the target vehicle's current location, then filters out lane lines that are less than 2 meters away from the vehicle's historical trajectory points, as well as lane lines where the difference between the vehicle's direction and the lane direction is less than 45°. Finally, it selects the lane with the longest historical trajectory along the lane centerline.

[0022] Furthermore, the encoder in step S2 includes a vehicle and surrounding vehicle historical trajectory encoder and a road map encoder, specifically:

[0023]

[0024] Where, φ h (·) represents a fully connected layer used for initial extraction of historical trajectory features, s∈R n×t×2 This provides the location coordinates of the target vehicle and surrounding vehicles' historical trajectories, where n is the number of vehicles including the target vehicle and surrounding vehicles, t is the time length of the vehicle's historical trajectory, and h is the location coordinates of the historical trajectories. e For encoding the vehicle's historical trajectory, f e Encoding the grid map around the vehicle, l e For vector map encoding, r e For the final encoding of road map information, self_attention e (·) represents a self-attention encoder, φ l (·) represents a fully connected layer used for initial extraction of lane centerline features, GNN e (·) represents the GNN (graph neural network) encoder, where G(V,E) is a graph consisting of vector maps surrounding the target vehicle, with nodes V∈R. m×l×2 Let m be the number of lane centerlines around the target vehicle, l be the number of lane centerlines included in each lane centerline, and E be the number of lane segments included in each lane centerline. CNN represents the connection between nodes. e (·) represents the CNN (convolutional neural network) encoder, f∈R n×3×128×128A gridded map of the area around the vehicle. This indicates concatenation computation, cross_attention e (·) represents a cross-attention encoder.

[0025] Furthermore, step S3 specifically includes the following steps:

[0026] S3.1 Divide the center lines of all possible lanes for future vehicle travel into N points, and encode the points on the lane center lines using an MLP (multi-layer perceptron); S3.2 Concatenate the lane center line encoding obtained in step S3.1 with the context vector and the one-hot vector representing the mode obtained from the student network encoder, and then input it into the decoder of the student network. Use two MLPs to decode it and output the predicted trajectory and trajectory confidence respectively;

[0027] S3.3 Divide all possible lane centerlines, except for the lane centerline where the future real mode is located, into N points, and use MLP to encode the points on the lane centerlines;

[0028] S3.4. The lane centerline code obtained in step S3.3 is concatenated with the context vector and the one-hot vector representing the mode obtained by the encoder of the teacher network. Then, it is input into the decoder of the teacher network to output the predicted trajectory and trajectory confidence respectively.

[0029] Furthermore, the decoder specifically comprises:

[0030]

[0031] Among them, s future To predict the trajectory, p future To predict the confidence level of the trajectory, h e ∈R n×h This represents the feature vector of the vehicle's historical trajectory. c represents the feature vector of the surrounding environment. e τ is the feature vector of the lane centerline point. m ∈R 3 MLP is a feature vector representing a mode. s MLP is a vehicle trajectory decoder. p For trajectory confidence decoder, This indicates a splicing calculation.

[0032] Furthermore, step S4 specifically includes the following steps:

[0033] S4.1 Filter the trajectories initially predicted in the teacher network;

[0034] S4.2. Use the true value of the future trajectory as the label of the future true modal trajectory in the student network, and use the trajectory selected in step S4.1 as the label of the other corresponding modal trajectories in the student network. Calculate the network loss, perform backpropagation on the student network to update the parameters, and finally use the EMA (exponential moving average) method to update the parameters of the teacher network.

[0035] Furthermore, the specific process of step S4.1 is as follows:

[0036] First, remove trajectories where the predicted vehicle direction deviates from the direction of the road the vehicle is traveling by more than 45 degrees, and trajectories where the predicted trajectory endpoint deviates from the center line of the lane the vehicle is traveling by more than 2 meters.

[0037] If multiple trajectories still exist after filtering, select the trajectory with the highest confidence level.

[0038] Furthermore, the network loss in step S4.2 includes regression loss, classification loss, and constraint loss, specifically:

[0039] L=λ(L reg (s p-label ,s student )+L cla (p p-label ,p student ))+(1-λ)(L reg (s future ,s student )

[0040] +L cla (p future ,p student ))+L cons

[0041] Among them, s p-label For the pseudo-label trajectories selected from the teacher network, s student For the trajectory generated in the student network, s future p is the true value of the future trajectory on the ground. p-label ∈R N The confidence score is the output of the teacher network, where N is the number of road points on the centerline of the lane in this mode, and the distance s is... p-label The confidence level of the trajectory corresponding to the road point closest to the destination is set to 1, and the confidence level of the rest is 0. student L represents the confidence level output by the student network. reg For regression loss, L claFor classification loss, L cons The constraint loss is used to enable the teacher network to generate trajectories that meet the constraints. When the vehicle's direction of travel deviates from the direction of the road it is traveling on by more than 45 degrees, a cosine loss is added. When the predicted point of the vehicle deviates from the center line of the lane it is traveling on by more than 2m, an MSE (mean squared errors) loss is added.

[0042] Furthermore, step S4.2 specifically involves updating the weights of the teacher network using the EMA method:

[0043]

[0044] in, Let be the parameters of the teacher network at time t. These are the parameters of the teacher network at the previous time step. Let β be the parameters of the student network at time t, and β∈[0,1) be the hyperparameters.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] I. This invention obtains a vehicle's potential future intentions, possible routes, and centerlines based on road map information. It encodes the road map information surrounding the target vehicle and the historical trajectories of other vehicles in the vicinity. Then, a semi-supervised model is used to generate pseudo-labels for the possible future trajectory modes of the target vehicle. These pseudo-labels are then used to supervise the learning of predicted trajectories under different modes, achieving multimodal prediction of vehicle trajectories. Thus, the current true trajectory mode of the target vehicle is treated as labeled data, while all possible motion modes other than the target vehicle's future trajectory mode are treated as unlabeled data. This transforms the multimodal problem into a semi-supervised problem. The mean-teacher semi-supervised model is used to train the model, effectively improving the prediction accuracy of vehicle trajectories and resolving the data imbalance problem in vehicle trajectory prediction algorithms. This effectively improves the modal distribution in the prediction results and significantly enhances the quality of trajectory prediction.

[0047] Second, this invention filters out the center line of the lane where the target vehicle is currently located based on the historical trajectory of the target vehicle, and obtains all possible lane center lines for future travel using a breadth-first approach based on the current lane line. Each lane center line is labeled as a different mode: left turn, right turn, and straight ahead, based on its direction, which helps to improve the trajectory prediction quality of subsequent turning modes.

[0048] Third, when training the mean-teacher model, this invention adds constraint loss to the regression loss and classification loss. The constraint loss includes cosine loss and MSE loss, which are used to constrain the deviation between the vehicle's driving direction and the direction of the road the vehicle is traveling on, and the deviation distance between the vehicle's predicted point and the center line of the lane it is traveling on, thereby ensuring that a trajectory that conforms to the road constraints is generated. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0050] Figure 2 This is a schematic diagram of the application framework for an example.

[0051] Figure 3 This is a schematic diagram of the student network framework;

[0052] Figure 4 This is a schematic diagram of the teacher network framework;

[0053] Figure 5 A schematic diagram of the attention mechanism;

[0054] Figure 6 To convert vector graphics into schematic diagrams of graph structures;

[0055] Figure 7 This is a schematic diagram of a CNN encoder. Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0057] Example

[0058] like Figure 1 As shown, a vehicle multimodal trajectory prediction method based on a semi-supervised model includes the following steps:

[0059] S1. Based on the historical trajectory of the target vehicle, obtain the lane the vehicle is traveling in and the corresponding lane centerline information, and obtain the possible future motion modes of the vehicle according to the direction of the lane centerline.

[0060] The vehicle's future real-world modality is obtained by matching its future trajectory with the lane centerline.

[0061] S2. Using the encoders in the student and teacher networks, encode the historical trajectory of the target vehicle, the historical trajectory of other vehicles within a set range around the target vehicle, and the environmental map within a set range around the target vehicle, respectively, to obtain the corresponding feature vector of the target vehicle trajectory, the feature vector of the surrounding vehicle trajectory, and the feature vector of the surrounding environment.

[0062] S3. In the student network of the mean_teacher model, all possible lane centerlines for the future driving of the vehicle are divided into N points. These points are encoded and concatenated with the output vector and modal vector of the encoder, and then input into the decoder.

[0063] Using the same approach, in the teacher network of the mean_teacher model, except for the lane centerline where the future true mode is located, all lane centerlines that the vehicle may travel in the future are divided into N points. These points are encoded and concatenated with the encoder's output vector, mode vector, etc., and then input into the decoder.

[0064] S4. Filter the trajectories output in the teacher network and use the filtered trajectories as pseudo-labels in the student network.

[0065] The mean-teacher model is trained by adding a constraint loss to the regression and classification losses during the training process to generate trajectories that conform to road constraints, thus obtaining a multimodal trajectory prediction model.

[0066] S5. Input the historical trajectory of the target vehicle and surrounding vehicles, surrounding map information, and the centerline information of all lanes to be driven in the future into the multimodal trajectory prediction model, and output the corresponding trajectory prediction results.

[0067] This embodiment applies the above-described technical solution, such as Figure 2 As shown, a semi-supervised model framework is constructed, including a student network and a teacher network. The main contents are as follows:

[0068] 1. When performing step S1:

[0069] S1.1: Obtain the trajectory of the target vehicle and other surrounding vehicles. Based on the historical trajectories of all vehicles and the current position of the target vehicle, search the map for the center lines of all lanes within a certain range of the target vehicle's current location.

[0070] S1.2: Based on the target vehicle's historical trajectory, filter out the lane centerline that the vehicle is most likely to be currently in. Then, using a breadth-first search approach, obtain all possible future lane centerlines. Each lane centerline is labeled with one of three different modes: left turn, right turn, or straight ahead. The complete trajectory, including both historical and future trajectories, is matched against the obtained lane centerlines to determine the lane centerline the vehicle will travel in the future.

[0071] S1.3: Classify the obtained lane center lines according to their modes. Select the lane center line closest to the current vehicle position in each class. If there is no lane center line in a class, fill it with 0 as the lane center line coordinates.

[0072] II. When executing step S2:

[0073] This technical solution employs a universal approach that is not limited to a single encoder or decoder. For example... Figure 3 and Figure 4 As shown, the network uses GNN and CNN networks to extract road map features, and uses an attention mechanism to extract vehicle trajectory features, vehicle interaction features, and vehicle-road features.

[0074] In the encoder, attention mechanisms (such as...) are used on the one hand. Figure 5 As shown, the attention mechanism (for processing vehicle historical trajectories) offers significant advantages over traditional RNN structures and their variants (LSTM, GRU) in that it focuses on relevant information while ignoring irrelevant information, directly establishing the dependency between input and output without loops, exhibiting strong parallelism, and greatly improving processing speed. It also overcomes some limitations of traditional neural networks, such as performance degradation with increasing input length, low computational efficiency due to unreasonable input order, and a lack of feature extraction and enhancement. The attention mechanism can be formulated as follows:

[0075]

[0076] Q = W q q,K=W k k,V=W v v

[0077] Among them, W q W k W v Here, q represents the learnable parameters; k represents the query vector; and v represents the key vector. In the self-attention mechanism, q, k, and v are the same.

[0078] On the other hand, GNNs are used to process vector road map information, such as Figure 6 As shown, a graph G(V,E) is formed by treating lane segments as nodes and the relationships between lane segments as edges, where nodes V∈R. m×l×2 Let m be the lane centerlines surrounding the target vehicle, where m is the number of lane centerlines, l is the number of lane segments contained in each lane centerline, and E represents whether nodes are connected. Using a GNN network can effectively preserve the topology and connectivity between lane segments. (Reference) Figure 7As shown, compared to GNN networks, CNN networks that use RGB images as input can represent richer map information. Therefore, this technical solution uses both networks simultaneously, employing two layers of GCN (Graph Convolutional Networks) to extract vector map information, as formally expressed below:

[0079]

[0080] In the formula, I is the identity matrix; yes The degree matrix; It represents the features of each node in each layer; σ is the non-linear activation function; GNN e Using a two-layer GNN network, the final lane centerline encoding result is obtained. e .

[0081] To better utilize road map information, this embodiment uses a 4-layer CNN network to extract information such as drivable areas and lane centerlines. Compared to GNN networks, CNN networks that use RGB images as input can represent richer map information. This CNN network can be represented as follows:

[0082] f e =CNN(f)

[0083] Finally, the obtained feature map is unfolded into a one-dimensional vector to obtain the feature vector of the grid map around the vehicle.

[0084] An attention mechanism is used in the encoder to extract historical trajectory features, interaction relationships, and vehicle-road interactions. e This includes algorithms that apply attention mechanisms in both the temporal and spatial dimensions. The temporal attention mechanism can extract the vehicle's dynamic features. (Self-attention) e First, two attention layers are used to obtain the vehicle trajectory code, where the time-dimension attention layer can be formulated as follows:

[0085] s e =φ h (s)

[0086] h e =Attention(s) e ,s e ,s e )

[0087] Where φh(·) is a fully connected layer used for preliminary extraction of historical trajectory features, s∈R n×t×2Given the location coordinates of the target and surrounding vehicles' historical trajectories, we obtain... Time-dimensional vehicle trajectory encoding;

[0088] e h Transpose the time and space dimensions to obtain self-attention e Two more attention layers are used to obtain spatial interaction. The spatial dimension attention layer can be formulated as follows:

[0089]

[0090] in, This is the vehicle trajectory encoding after transposition.

[0091] In cross-attention e The encoder uses two cross-attention layers to capture the interaction between the vehicle and the road. The cross-attention layers can be formulated as follows:

[0092]

[0093] in, yes Transposition of vehicle historical trajectory encoding; f e Encode the grid map around the vehicle; e Encoding for vector maps.

[0094] III. When executing step S3:

[0095] S3.1: Divide the center lines of all possible lanes the vehicle will travel in the future into N points. Use MLP (multi-layer perceptron) to encode the points on the lane center lines.

[0096] S3.2: The lane centerline code obtained in step S3.1 is concatenated with the context vector and the one-hot vector representing the mode obtained by the student network encoder, and input into the decoder of the student network. Two MLPs are used to decode it, and the predicted trajectory and trajectory confidence are output respectively.

[0097] S3.3: Divide all possible lane centerlines, except for the lane centerline where the future real mode is located, into N points. Use MLP to encode the points on the lane centerlines.

[0098] S3.4: The lane centerline code obtained in step S3.3 is concatenated with the context vector and the one-hot vector representing the mode obtained by the encoder of the teacher network, and input into the decoder of the teacher network to output the predicted trajectory and trajectory confidence respectively.

[0099] In the decoder, the historical trajectory encoding, road map encoding, lane centerline point encoding, and one-hot vector representing the motion mode obtained from the encoder are concatenated together. Two MLP networks are then used for decoding to obtain the complete trajectory and its confidence level. The decoder can be formally represented as follows:

[0100]

[0101] In the formula, s future For predicting the trajectory; p future h is the confidence level for predicting the trajectory. e ∈R n×h This is the feature vector of the vehicle's historical trajectory. c is the feature vector of the surrounding environment; e τ is the feature vector of the lane centerline point; m ∈R 3 The feature vector representing the mode; MLP s MLP is a vehicle trajectory decoder. p For trajectory confidence decoder; This indicates a splicing operation.

[0102] IV. When executing step S4:

[0103] S4.1: Filter the trajectories initially predicted in the teacher network to obtain trajectories that meet road constraints and have high prediction quality.

[0104] The specific method for filtering predicted trajectories in the teacher network is as follows: First, remove trajectories whose predicted vehicle driving direction deviates from the direction of the road the vehicle is traveling by more than 45 degrees, as well as trajectories whose predicted trajectory endpoint deviates from the center line of the lane the vehicle is traveling by more than 2 meters; if multiple trajectories still exist after filtering, select the trajectory with the highest confidence as the pseudo-label of the student network.

[0105] S4.2: The ground truth value of the future trajectory is used as the label of the future true modal trajectory in the student network, and the trajectory obtained by the teacher network is used as the label of the other corresponding modal trajectories in the student network. The loss of the network is calculated, and backpropagation is performed on the student network to update the parameters. Finally, the EMA (exponential moving average) method is used to update the parameters of the teacher network.

[0106] The loss function includes regression loss, classification loss, and constraint loss, and is formulated as follows:

[0107] L=λ(L reg (s p-label ,s student )+L cla (p p-label ,p student ))+(1-λ)(L reg (s future ,s student )

[0108] +L cla (p future ,p student ))+L cons

[0109] Among them, s p-label The pseudo-label trajectories selected from the teacher network; s student Trajectories generated in the student network; s future p represents the true value of the future trajectory on the ground. p-label ∈R N The confidence score is the output of the teacher network, where N is the number of road points on the centerline of the lane in this modality, and the distance s is... p-label The confidence level of the trajectory corresponding to the road point closest to the destination is set to 1, and the confidence level of the rest is 0; p student L represents the confidence level output by the student network. reg For regression loss, this embodiment uses MSE (mean squared errors) loss, L cla For classification loss, cross-entropy loss is used in this embodiment. cons The constraint loss is used to enable the teacher network to generate trajectories that meet the constraints. When the vehicle's direction of travel deviates from the direction of the road it is traveling on by more than 45 degrees, a cosine loss is added; when the predicted point of the vehicle deviates from the center line of the lane it is traveling on by more than 2m, an MSE loss is added.

[0110] The weights of the teacher network are updated using the EMA method, which is formulated as follows:

[0111]

[0112] in Let be the parameters of the teacher network at time t. These are the parameters of the teacher network at the previous time step. Let β be the parameters of the student network at time t, and β∈[0,1) be the hyperparameters defined by the user.

[0113] In summary, this technical solution provides a vehicle multimodal trajectory prediction method based on a semi-supervised model. Determining potential future vehicle modes using map information offers advantages such as accuracy, speed, and effectiveness. Using a semi-supervised model for training alleviates the data imbalance problem in trajectory prediction and, without increasing computational load during inference, better predicts trajectory mode distribution, improving the trajectory prediction quality for steering modes. Furthermore, this method is universal; slight modifications to the model's training method can effectively increase the prediction accuracy of other models and improve the trajectory mode distribution.

[0114] This technical solution treats the multimodal prediction problem of vehicles as a semi-supervised problem by considering the true modalities of future trajectories as labeled data and the remaining possible modalities as unlabeled data. It then trains a semi-supervised model to achieve this, effectively improving the prediction accuracy and modal distribution of the predicted trajectory. Furthermore, by creating pseudo-labels, this solution also overcomes the data imbalance problem in trajectory prediction to some extent. This solution also has strong universality; only slight modifications to the training process of existing methods are needed to improve prediction accuracy and modal distribution.

Claims

1. A vehicle multi-modal trajectory prediction method based on a semi-supervised model, characterized in that, The method comprises the following steps: S1, based on the historical trajectory of the target vehicle, obtaining the lane and the corresponding lane center line information of the vehicle driving, and obtaining the possible motion mode of the vehicle in the future according to the direction of the lane center line; The real mode of the vehicle in the future is obtained by matching the future trajectory and the lane center line; S2, using the encoder in the student and teacher network, respectively encoding the historical trajectory of the target vehicle, the historical trajectory of other vehicles within the set range around the target vehicle, and the environment map within the set range of the target vehicle, to obtain the corresponding feature vector of the target vehicle trajectory, the surrounding vehicle trajectory feature vector and the surrounding environment feature vector; S3, in the student network of the mean_teacher model, all possible lane center lines of the vehicle in the future are divided into N points respectively, the points are encoded, and the output vector of the encoder, the mode vector and the like are spliced to input the decoder; In the same way, in the teacher network of the mean_teacher model, all lane center lines that the vehicle may drive in the future are divided into N points respectively except the lane center line where the real mode in the future is located, the points are encoded, and the output vector of the encoder, the mode vector and the like are spliced to input the decoder; S4, screening the trajectory output in the teacher network, and taking the screened trajectory as the pseudo label in the student network; The mean-teacher model is trained, and in the training process, constraint loss is added on the basis of regression loss and classification loss to generate a trajectory conforming to road constraints, and a multi-modal trajectory prediction model is trained; S5, inputting the historical trajectory of the target vehicle and the surrounding vehicles, the surrounding map information and all lane center line information of future driving into the multi-modal trajectory prediction model to output the corresponding trajectory prediction result.

2. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 1, characterized in that, The step S1 specifically comprises the following steps: S1.1, obtaining the trajectories of the target vehicle and other surrounding vehicles, searching all lane center lines within a set range near the current position of the target vehicle from the map according to the historical trajectories of all vehicles and the current position of the target vehicle; S1.2, according to the historical trajectory of the target vehicle, screening out the lane center line where the vehicle is currently located, and using a breadth-first method to obtain all possible lane center lines to be driven in the future according to the direction of each lane center line, and labeling each lane center line as left turn, right turn and straight ahead; The complete trajectory including the historical trajectory and the future trajectory is matched with the obtained lane center line to obtain the lane center line to be driven by the vehicle in the future; S1.3, all lane center lines that the vehicle may drive in the future are classified according to the mode, and a lane center line closest to the current vehicle position is selected in each category, and if there is no lane center line in the category, 0 is filled as the lane center line coordinate.

3. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 2, characterized in that, The specific process of searching all lane center lines within a set range near the current position of the target vehicle from the map in the step S1.1 is as follows: All lane lines in the set range of the current position of the target vehicle are searched, then lane lines with a distance less than 2m from the historical trajectory points of the vehicle and a difference in direction less than 45° between the vehicle and the lane are screened, and finally the lane with the largest length along the center line of the lane where the vehicle historically travels is selected.

4. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 1, characterized in that, The encoder in the step S2 comprises a vehicle and surrounding vehicle historical trajectory encoder and a road map encoder, specifically: where φ h (·) is a fully connected layer for preliminary extraction of historical trajectory features, s ∈ R n×t×2 is the position coordinate information of the historical trajectory of the target vehicle and surrounding vehicles, n is the number of vehicles including the target vehicle and surrounding vehicles, t is the time length of the historical trajectory of the vehicles, h e is the vehicle historical trajectory encoding, f e is the vehicle surrounding grid map encoding, l e is the vector map encoding, r e is the final encoding of the road map information, self_attention e (·) is a self-attention encoder, φ l (·) is a fully connected layer for preliminary extraction of lane center line features, GNN e (·) is a GNN encoder, G(V, E) is a graph composed of the vector map around the target vehicle, node V ∈ R m×l×2 is the lane center line around the target vehicle, m is the number of lane center lines, l is the number of lane segments included in each lane center line, edge E represents whether the nodes are connected, CNN e (·) is a CNN encoder, f ∈ R n×3×128×128 is the rasterized map around the vehicle, represents splicing calculation, cross_attention e (·) is a cross-attention encoder.

5. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 1, characterized in that, The step S3 specifically comprises the following steps, S3.1, dividing the center line of each lane where the vehicle can possibly travel in the future into N points, and using an MLP to encode the points on the center line of the lane; S3.2, splicing the lane center line encoding obtained in the step S3.1, the context vector obtained by the student network encoder and the one-hot vector representing the mode, and then inputting the decoder of the student network, and using two MLPs to decode, and respectively outputting the predicted trajectory and the trajectory confidence; S3.3, dividing the center line of each lane where the vehicle can possibly travel except the lane where the future real mode is located into N points, and using an MLP to encode the points on the center line of the lane; S3.4, splicing the lane center line encoding obtained in the step S3.3, the context vector obtained by the teacher network encoder and the one-hot vector representing the mode, and then inputting the decoder of the teacher network, and respectively outputting the predicted trajectory and the trajectory confidence.

6. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 5, characterized in that, The decoder specifically comprises: where s future is the predicted trajectory, p future is the predicted trajectory confidence, h e ∈ R n×h is the vehicle history trajectory feature vector, is the surrounding environment feature vector, c e is the lane centerline point feature vector, τ m ∈ R 3 is the feature vector representing the modality, MLP s is the vehicle trajectory decoder, MLP p is the trajectory confidence decoder, denotes concatenation computation.

7. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 1, characterized in that, The step S4 specifically comprises the following steps: S4.1, screening the trajectory preliminarily predicted in the teacher network; S4.2, taking the true value of the future trajectory as the label of the future real mode trajectory in the student network, taking the trajectory screened in the step S4.1 as the label of the corresponding mode trajectory in the student network, calculating the loss of the network, and performing back propagation and parameter updating on the student network, and finally updating the parameters of the teacher network by using the EMA method.

8. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 7, characterized in that, The specific process of the step S4.1 is as follows: First, remove the trajectory with a deviation greater than 45 degrees between the predicted driving direction of the vehicle and the direction of the road where the vehicle travels, and the trajectory with a deviation of more than 2m from the center line of the lane where the vehicle travels; If there are still multiple trajectories after screening, the trajectory with the highest confidence is selected.

9. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 8, characterized in that, The loss of the network in the step S4.2 comprises a regression loss, a classification loss and a constraint loss, specifically: L = λ(L reg (s p-label ,s student )+L cla (p p-label ,p student ))+(1 - λ)(L reg (s future ,s student )+L cla (p future ,p student ))+L cons Among them, s p-label For the pseudo-label trajectories selected from the teacher network, s student For the trajectory generated in the student network, s future p is the true value of the future trajectory on the ground. p-label ∈R N The confidence score is the output of the teacher network, where N is the number of road points on the centerline of the lane in this mode, and the distance s is... p-label The confidence level of the trajectory corresponding to the road point closest to the destination is set to 1, and the confidence level of the rest is 0. student L represents the confidence level output by the student network. reg For regression loss, L cla For classification loss, L cons The constraint loss is used to enable the teacher network to generate trajectories that meet the constraints. When the vehicle's direction of travel deviates from the direction of the road it is traveling on by more than 45 degrees, a cosine loss is added. When the predicted point of the vehicle deviates from the center line of the lane it is traveling on by more than 2m, an MSE loss is added.

10. The vehicle multi-modal trajectory prediction method based on a semi-supervised model according to claim 7, characterized in that, The weight of the teacher network is updated by using the EMA method in the step S4.2: wherein, are parameters of the teacher network at time t, are parameters of the teacher network at the previous time, are parameters of the student network at time t, and β ∈ [0, 1) is a hyperparameter.

Images