Vehicle trajectory prediction method based on lane point future trajectory offset amount auxiliary supervision

Abstract

The application discloses a kind of vehicle trajectory prediction methods based on lane point future trajectory offset amount auxiliary supervision, including the following steps: obtaining each surrounding intelligent agent historical feature and lane information;Scene coding and feature fusion;Decoding multimodal trajectory and offset amount, construct auxiliary supervision branch.The application establishes multi-scale lane topological relationship by sampling and neighborhood extraction, encodes the topological structure information and semantic information of lane using feature along sparse aggregation and multi-scale lane point encoder along anti-sparse direction diffusion, realizes the understanding of the topological structure and semantic attribute of high-precision map under different feature scales, and then improves the representation ability of encoder to scene, can support higher precision vehicle trajectory prediction.The application can assist lane point encoder to learn the features having position correlation with future trajectory by lane auxiliary offset loss, so that the encoder parameters accept more direct supervision signal, realize more efficient network training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, and in particular to a vehicle trajectory prediction method based on lane point future trajectory offset auxiliary supervision. BACKGROUND

[0002] In the driving process of an autonomous vehicle, predicting the intention or motion trajectory of surrounding intelligent agents in advance is an important basis for the autonomous vehicle to make correct decisions in various traffic environments. In the traditional decision-making process, a state transition model is usually formulated based on rules to update the future state of surrounding intelligent agents. However, this approach may fail in highly uncertain environments. Therefore, by taking advantage of the powerful representation capability of a deep encoder-decoder, the motion state and the associated scene information are encoded, and the motion trajectory is decoded, which has a significant positive effect on making correct decisions. How to efficiently encode scene information, associate it with the semantic information of the agent to be predicted, and effectively supervise these associations is the key to improving the performance of the encoder-decoder.

[0003] In the method of encoding scene information, a classic approach is to render the lane polygons in the scene to be predicted and the historical bounding boxes of surrounding intelligent agents into a grid image, and then use a convolutional neural network (CNN) to extract multi-scale semantic features. However, this method does not fully utilize sparse scene information and the semantic association between elements. Therefore, the current mainstream solution processes scene information as vectorized data and uses graph networks or point cloud networks to aggregate and propagate features under specific semantic relationships. For example, the Lane Graph Convolutional Network (LaneGCN) establishes a directed graph between lane nodes and between agents and lane nodes, and uses a graph network for encoding. The Temporal Point Cloud Networks (TPCN) combines point cloud and voxel networks to encode the scene, using a PointNet point cloud deep network at the point cloud scale and a sparse convolution at the voxel scale. Although the above methods effectively encode the sparsity of scene information and achieve significant performance improvement, there is still room for improvement. LaneGCN only considers the case where the adjacency relationship strictly corresponds to the actual drivable direction, and does not consider the case where the driving direction of some agents may not correspond to the road adjacency relationship, which may result in incorrect feature mapping when fusing node features into agent motion features. The encoding model of TPCN lacks modeling of semantic relationships and does not fully utilize the semantic feature association between elements. The above encoding methods do not fully represent the scene information, which may affect the prediction performance. SUMMARY

[0004] To solve the above problems existing in the prior art, the present application proposes a vehicle trajectory prediction method based on lane point future trajectory offset auxiliary supervision, which further improves the scene representation ability of the encoder.

[0005] The basic idea of the present application is as follows: a coding and decoding structure including an encoder module and a decoder module is designed, and the structure is as shown in Figure 3 The encoder module is composed of a hierarchical Transformer encoder, a lane point encoder and an interaction fusioner, the hierarchical Transformer encoder is composed of three Axial-Transformer blocks, and the hierarchical Transformer encoder is cascaded with the lane point encoder and the interaction fusioner as the main part of the encoder module. The decoder module is composed of three groups of multi-layer perceptron (MLP). First, the historical motion trajectory information of the surrounding agents is obtained according to the perception and positioning system of the autonomous vehicle, and the lane information in the scene is obtained through the high-precision map data; secondly, the state information of the agent is encoded by the hierarchical Transformer encoder to obtain the embedding features of the agent, the lane point information is aggregated and propagated by using the lane point encoder and the fusioner, and the embedding features of the lane point are output, and the embedding features of the lane point are fused with the embedding features of the agent output by the hierarchical Transformer encoder, the interaction information at the semantic level is fused through the interaction fusioner, and the final semantic embedding of the agent is obtained; then the predicted trajectory and the corresponding confidence, as well as the offset between the future lane point and the future trajectory, are obtained by decoding the agent embedding through the decoder module, the trajectory prediction of the surrounding agents is completed, and the lane point future trajectory offset auxiliary supervision model is trained. The whole coding and decoding structure can understand the internal topological structure and semantic attributes of the high-precision map at different feature scales through the proposed lane point coding method, thereby improving the scene representation ability of the encoder, fully extracting more effective environmental information, supporting higher precision vehicle trajectory prediction, and the proposed lane auxiliary offset loss can assist the road encoder to learn the features having position correlation with the future trajectory, so that the encoder parameters receive more direct supervision signals, and more efficient network training is realized.

[0006] The technical scheme of the present application is as follows: a vehicle trajectory prediction method based on lane point future trajectory offset auxiliary supervision, comprising the following steps:

[0007] A、obtaining the historical features and lane information of each surrounding agent

[0008] A1, obtaining the historical trajectory of each surrounding agent through the vehicle-mounted perception and positioning system of the autonomous vehicle Each s i,jis composed of two-dimensional coordinate points; subscript i = 0, 1, …, n represents the number of agents, wherein 0 represents the agent number to be predicted, n represents the maximum number of agents in the scene, subscript j = -t h , -t h + 1, …, 0 represents the time step, wherein j = 0 is the current time, t h represents the length of the historical observation interval, j = -t h is the last past historical time in the historical observation interval; the local position of the agent is represented as a D-dimensional position embedding In order to represent the incremental information of motion, the coordinate increment of the current coordinate and the coordinate of the last time are calculated A D-dimensional time embedding is constructed to make the attention mechanism better perceive the time sequence features; finally, the historical features of the agent are obtained by splicing as follows:

[0009]

[0010] wherein, is a Boolean quantity indicating whether there is a historical observation value at the current time. In order to facilitate parallel operation of a single scene, for a scene with an agent number greater than n, the agents with numbers i = 0, 1, …, n are intercepted, otherwise 0 is filled. And a mask m is constructed to indicate which number i corresponds to an agent obtained by filling, so that the attention mechanism focuses on the valid data of the scene frame.

[0011] A2, obtain the lane information in the scene through the high-precision map, construct the lane points on the lane center line in the high-precision map as a structured vector form for sparse coding, and the constructed lane point vector feature is wherein L is the number of lane points, c 0:L is the coordinate of the lane point, is the offset of the lane point in the road direction relative to the previous lane point, is a J-dimensional Boolean vector, and J different dimensions respectively represent different semantic attributes. In order to aggregate and propagate the lane point vector features in the entire scene, first, the lane point cloud is processed by grid sampling to obtain relatively uniform key lane points Then the nearest L jk neighbor points are selected according to the distance wherein L j represents the number of key points at the j-th scale, and L jk represents the number of corresponding neighbor points around each key point.

[0012] A3, taking the current road segment where the agent is located as the starting point, breadth-first search along the adjacent direction of each drivable road segment to extract candidate future lane points wherein Lg This represents the number of future candidate lane points. Since some future lane points have a low correlation with the agent's trajectory, a Boolean vector is constructed for this purpose. This indicates whether future lane points are within the range of the true trajectory and whether their deviation from the true trajectory does not exceed a threshold Tho. During training, these points with low correlation that exceed the threshold range are considered invalid points and do not participate in the backpropagation of errors.

[0013] B. Scene Coding and Feature Fusion

[0014] B1. Two Axial-Transformer blocks are used to encode the agent information in the scene. First, the agent's historical features are encoded. Perform a linear transformation to obtain the agent embedding Next The first Axial-Transformer block is input, and self-attention is calculated and encoded along the time axis, extracting the low-level motion characteristics of a single agent. Next, a second Axial-Transformer block is used to calculate and encode self-attention along the scene axis, extracting the joint state of the agent at a single time step. This joint state is updated based on the immediate decisions of each agent, which are influenced by traffic rules and interaction relationships at that moment, thus indirectly encoding this implicit information. Concatenating the two Axial-Transformer blocks with self-attention along the two axes fuses immediate intent decisions and low-level motion characteristics. After completing the concatenated encoding along the two axes, a new agent embedding is obtained. Then, downsampling is performed on the time axis using a one-dimensional convolutional neural network. Downsampling This is to achieve feature scaling, thereby enhancing the Transformer architecture's ability to learn multi-scale features:

[0015]

[0016]

[0017] B2. Encode the lane point cloud in the predicted scene using a lane point encoder. The lane point encoder first aggregates the neighboring points p through feature aggregation. n Features are aggregated to keypoint p k In the features, the input to the aggregation process is the lane point coordinates c after linear transformation. 0:L Offset relative to the previous lane point And a Boolean vector representing the semantic attributes of lane points Initial features g of lane points formed by splicing and combining 0:L; then the key point features are propagated back to the neighborhood point features in the opposite direction, while the relative information between them is fused. After multi-layer feature propagation, the embedded features g' of all lane points are obtained 0:L As follows:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] where g 0:L is the initial vector feature of all lane points, W is a linear transformation matrix, aggregatopm(·) and propagation(·) are the feature aggregation function and the feature propagation function respectively, layerfusion(·) is the feature fusion function between layers, Concatenate(·) is the feature concatenation function, is the initial neighborhood point feature, is the new key point feature, is the new neighborhood point feature. is the lane point fusion feature of each layer after layer fusion, g' 0:L is the embedded feature of all lane points.

[0024] B3, the environment features and interaction features associated with the agent motion are fused, the lane point features within a certain range around the agent are fused in an attention-based manner, the range is a circular area with the agent center as the center and the distance between adjacent lane lines as the radius. And through the last Axial-Transformer block for encoding, the fused environment features are fully propagated in the time dimension and the scene dimension, and then the agent interaction encoder FusionNetA 2A in LaneGCN is used to encode the agent interaction in the scene, propagate the semantic features along the agent associated path, and generate the final agent semantic embedding e 0:n As follows:

[0025]

[0026] C, decode multi-modal trajectory and offset, and construct auxiliary supervision branch

[0027] C1, a set of multi-layer perceptron (MLP) is used to combine the low-level motion semantic behaviors fully encoded in the encoding stage to decode different behavior modalities and obtain a set of future trajectories of the agent And on this basis, the regression supervision branch of the trajectory is constructed, that is, the regression loss loss is calculated r The specific calculation formula is as follows:

[0028]

[0029]

[0030] In the formula, t f is the future prediction interval length, k is the number of modalities, corresponding to k possible future situations, MLP(·) is a multi-layer perceptron, e i is the semantic embedding of the agent to be predicted, where i=0,1,…n is the agent number, NLL(·) is a negative log-likelihood loss function, is the true value of the future trajectory, is the optimal modality k * corresponding to the predicted trajectory, k * corresponding to the average displacement deviation between the predicted trajectory and the true trajectory, that is,

[0031] C2, another set of MLPs with the same structure is used to decode and calculate the confidence conf k of different modality trajectories c And on this basis, the classification supervision branch is constructed, that is, the classification loss loss is calculated k as follows:

[0032] conf i =Softmax(MLP(e * ))

[0033]

[0034] In the formula, Softmax(·) is a softmax function, NLL(·) is a negative log-likelihood loss function, is the optimal modality k * corresponding to the confidence.

[0035] C3, by predicting the position offset between the future lane point and the trajectory point, an auxiliary supervision branch is constructed. The position offset between the future true trajectory and the candidate future lane point found by searching is calculated, that is, the shortest distance between the real trajectory point. The k MLPs are used to predict the offset The modality k *Compute the regression loss loss with the true value o , construct a lane point future trajectory offset auxiliary supervision branch, use the auxiliary offset supervision loss to assist in decoding the predicted trajectory and encoding the lane point features, so that these future possible lane points provide prior guidance for as many behavior patterns as possible.

[0036]

[0037]

[0038] C4, through the above three supervision branches, only the optimal mode k * The corresponding error is backpropagated to avoid the parameters of the k MLPs from being assimilated and causing mode collapse, and a final loss function composed of the agent multi-modal trajectory loss, the trajectory corresponding confidence loss and the lane point future trajectory offset loss is constructed to supervise the end-to-end training of the model, and the calculation formula is as follows:

[0039] loss total =λ1loss r +λ2loss c +λ3loss o

[0040] In the formula, λ1, λ2 and λ3 are weights for balancing each term, loss total is the final total loss.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1. The present application establishes a multi-scale lane topology by sampling and neighborhood extraction, encodes the topological structure information and semantic information of the lane by using a multi-scale lane point encoder which aggregates along the sparsification and diffuses along the anti-sparsification direction of the features, realizes the understanding of the topological structure and semantic attributes of the high-precision map under different feature scales, and further improves the representation ability of the encoder to the scene, which can support higher precision vehicle trajectory prediction.

[0043] 2. The lane auxiliary offset loss proposed by the present application can assist the lane point encoder to learn the features having a position correlation with the future trajectory, so that the encoder parameters accept more direct supervision signals and realize more efficient network training. DETAILED DESCRIPTION

[0044] The present application has a total of Figure 5 Zhang, wherein:

[0045] Figure 1 is the flowchart of the present application.

[0046] Figure 2is a schematic diagram of the framework of the present application.

[0047] Figure 3 is a schematic diagram of the coding structure of the present application.

[0048] Figure 4 is a schematic diagram of the lane point encoding method.

[0049] Figure 5 is a schematic diagram of the lane auxiliary offset loss. DETAILED DESCRIPTION

[0050] The embodiments of the present application will be described in detail by specific embodiments, and those skilled in the art can easily understand the advantages of the present application from the disclosure. The technical solutions of the present application will be described clearly and completely in combination with the drawings. In the following description, many specific details will be included. In addition, in order to avoid confusion or obscuring the focus of the present application, some specific details will be omitted in the description.

[0051] In order to make the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application will be further described in detail below. As shown in Figure 1 A lane point future trajectory offset auxiliary supervision based vehicle trajectory prediction method flow chart, comprising the following steps:

[0052] A, obtaining the historical features of each surrounding agent and lane information

[0053] A1, obtaining the historical trajectory of each surrounding agent through the vehicle-mounted perception and positioning system of the autonomous vehicle Each s i,j is composed of two-dimensional coordinate points, subscript i=0,1,…,n represents the number of agents, subscript j=-t h , -t h +1,…,0 represents the time step, such as j=0 for the current time, j=-t h for the last historical time in the historical observation interval, t h represents the length of the historical observation interval; next, the local position of the agent is represented as D-dimensional position embedding In order to represent the incremental information of motion, the coordinate increment of the current coordinate and the coordinate of the last time is calculated D-dimensional time embedding is constructed to make the attention mechanism better perceive the time series features. Finally, the historical features of the agent are obtained by splicing wherein This is a Boolean value indicating whether historical observations exist at the current moment. To facilitate parallel computation on a single scene, for scenes with more than n agents, agents i = 0 to n are truncated; otherwise, they are padded with 0s. A mask m is constructed to indicate which agents corresponding to number i are obtained through padding, ensuring the attention mechanism focuses on valid data in the scene frames.

[0054] A2. Obtain lane information from the scene using a high-precision map. Construct lane points on the lane centerline from the high-precision map into structured vectors for sparse encoding. The features of the constructed lane point vectors are: Where L is the number of lane points, c 0:L The coordinates of the lane points The offset of the lane point relative to the previous lane point in the road direction. Let J be a J-dimensional Boolean vector, where J different dimensions represent different semantic attributes, such as the lane category of point l, including left turn, right turn, and straight ahead, whether the lane is controlled by traffic signals, and whether it is located at an intersection. To aggregate and propagate lane point features across the entire scene, the lane point cloud is first processed using raster sampling to obtain relatively uniform key lane points. Then filter by distance to... Recent L jk Neighboring points Where L j L represents the number of keypoints at the j-th scale. jk This represents the number of neighboring points around each keypoint.

[0055] A3. To further represent the relative relationship between keypoints and their neighboring points, a relative relationship vector is constructed. This represents the relative relationship between keypoints and their neighboring points, with each On... n,k =[oc n,k ,oa n,k ],oc n,k For key point p k Pointing to the neighboring point p n vector p k p n ,oa n,k For vector p k p n With key point p k The local direction vector p of the road segment kpre With p ksuc The angle between them, p kpre p ksuc Then they represent p respectively k The vector formed by the preceding and following points of the road segment.

[0056] A4, taking the current road segment where the agent is located as the starting point, breadth-first search along the adjacent direction of each drivable road segment to extract candidate future lane points where L g represents the number of future candidate lane points. Since there are points with low correlation to the agent's motion trajectory in the future candidate lane points, a Boolean vector is constructed to indicate whether the future lane point is within the range of the real trajectory and the offset from the real trajectory does not exceed the threshold Th o , Th o may be determined according to the distance between adjacent lane centerlines. In training, these out-of-range points with low correlation are considered as invalid points and do not participate in the backpropagation of errors. The selected future candidate lane points are shown in Figure 5 , and the extracted future candidate lane points are divided into valid inner points and invalid outer points according to the future real trajectory of the agent, where the points within the grid area are out-of-range outer points.

[0057] B, scene encoding and feature fusion

[0058] B1, two Axial-Transformer blocks are used to encode the agent information in the scene. First, the agent historical features are linearly transformed to obtain agent embedding Next, the is input into the first Axial-Transformer block, the self-attention is calculated along the time axis and encoded, and e 0:n,j is converted into corresponding query, key, value, and the corresponding attention coefficients are calculated as weights to weight the sum of values, completing the fusion of key / value information to query, thereby realizing the self-attention calculation process in one axis. The encoding along the time axis extracts the low-level motion characteristics of a single agent. Next, the second Axial-Transformer block calculates the self-attention along the scene axis and encodes it. The way of calculating self-attention is consistent with the way of calculating self-attention along the time axis. The encoding along the scene axis extracts the joint state of the agent at a single time step, which is updated with the real-time decision results of each agent, and the decision is affected by the traffic rules and interaction relationships at this moment, thus indirectly encoding these implicit information. The two self-attention Axial-Transformer blocks in the two axes are cascaded, i.e. the real-time intention decision and the low-level motion characteristics are fused. After completing the cascaded encoding of the two axes, a new agent embedding is obtained. is down-sampled to To achieve the scale transformation of the feature to enhance the ability of the Transformer structure to learn multi-scale features:

[0059]

[0060]

[0061] B2, input the extracted lane point information into the lane point encoder, encode the lane point cloud in the predicted scene through the lane point encoder, as shown in Figure 2 In the process of encoding lane points, first, the features of the neighborhood points p n are aggregated into the features of the key points p k The input of the aggregation process is the lane point initial feature g 0:L which is composed of the lane point coordinate c after linear transformation, the offset relative to the previous lane point, and the Boolean vector indicating the semantic attributes of the lane point. 0:L Where W is a linear transformation matrix, and the output of the feature aggregation is the new key point feature Then propagate the updated key point feature back to the neighborhood point feature in the opposite direction to get the new neighborhood point feature Then fuse their relative information to get the layer fusion feature For the propagation between layers, the features of the lane points in the next layer are spliced into the corresponding lane point features of the previous layer, and after multi-layer feature propagation, the encoded lane point embedding feature g' 0:L is obtained.

[0062] Figure 4 The process of encoding two layers of road key points by the lane point encoder is shown in the figure, where the solid circle points are multi-scale road key points obtained by layer-by-layer sampling, and the hollow circle points are the neighborhood points around the key points. In the first layer feature aggregation process (aggregation 1), the features of the neighborhood points around are aggregated into the corresponding key point features, and then enter the second layer aggregation (aggregation 2), which reduces the number of key points. Among the key points in the first layer, some are still reserved as key points, while the rest are reserved as neighborhood points of the key points. The neighborhood point features are aggregated into the features of the reserved key points. With the deepening of aggregation, the receptive field of a single lane node will expand layer by layer. The features of the low layer represent the local lane point cloud structure, while the features of the high layer learn the road structure information under the scene scale. Through the propagation from the high layer to the low layer, the relative relationship of the lane points is further fused, so as to extract multi-scale road structure information.

[0063]

[0064]

[0065]

[0066]

[0067] B3、Fusion of environment features and interaction features associated with agent motion, fusion of lane point features within a certain range around the agent in a attention-based manner, the range being a circular area with the agent center position as the center and the distance between adjacent lane centerlines as the radius, as shown in Figure 3 , and encoding through the last Axial-Transformer block to fully propagate the fused environment features in the time dimension and scene dimension, and then using the agent interaction encoder FusionNet A2A in LaneGCN to encode the agent interaction in the scene, propagate semantic features along the agent associated path, and generate the final agent semantic embedding e 0:n .

[0068]

[0069] C、Decoding multi-modal trajectories and offsets, constructing auxiliary supervision branch

[0070] C1、respectively using two sets of MLP to decode the agent semantic embedding to obtain the future trajectory of the agent to be predicted and the corresponding confidence conf k , where tf represents the future prediction interval length, k is the number of modalities, corresponding to k possible future situations. On this basis, the regression supervision branch of the trajectory is constructed, i.e. the loss loos r is calculated, the classification supervision branch of the trajectory probability is constructed, i.e. the loss loos c

[0071]

[0072] cinf k =Softmax(MLP(e i ))

[0073]

[0074]

[0075] where tf is the future prediction interval length, k is the number of modalities, corresponding to k possible future situations, MLP(·) is a multilayer perceptron, e iwhere i=0,1,…n is the agent number, Softmax(·) is the softmax function, is the ground truth of future trajectory, * 0:tf is the optimal modality k * is the corresponding predicted trajectory, * is the average displacement error between the corresponding predicted trajectory and the ground truth trajectory, i.e. NLL(·) is the negative log-likelihood loss function, is the optimal modality k * is the corresponding confidence.

[0076] C2, by predicting the position offset between the future candidate lane point and the real trajectory point, a lane point future trajectory offset auxiliary supervision branch is constructed to calculate the position offset between the future real trajectory and the candidate future lane point searched i.e. the shortest distance between the real trajectory point. The offset is predicted by k MLPs The modality k * with the smallest average displacement error is selected to calculate the regression loss loss o with the ground truth. The future candidate point offset in the lane auxiliary offset loss is shown in Figure 5 , where the black line is the real trajectory of the vehicle, and the shortest distance between the boxed lane point (inner point) and the trajectory point on the real trajectory is the real future candidate point offset. The auxiliary offset supervision loss is used to assist the decoding of the predicted trajectory and the encoding of the lane point features, so that these future possible lane points provide prior guidance for as many behavior patterns as possible.

[0077]

[0078]

[0079] C3, by the above three supervision branches, only the corresponding error of k * is back-propagated to avoid the parameters of the k MLPs being assimilated and causing mode collapse, and a final loss function composed of the agent multi-modal trajectory loss, the confidence loss corresponding to the trajectory, and the lane point future trajectory offset loss is constructed to supervise the end-to-end training of the model.

[0080] loss total =λ1loss r +λ2loss c +λ3loss o

[0081] where λ1, λ2, λ3are weights balancing the terms, loss total is the final total loss.

[0082] The foregoing describes the basic principles, main features and advantages of the present application. The present application is not limited to the above-described embodiments, which are merely illustrative of the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and all such changes and improvements fall within the scope of the present application.

Claims

1. A vehicle trajectory prediction method based on lane point future trajectory offset amount auxiliary supervision, characterized by: Comprising the following steps: A、Obtain the historical features of each surrounding agent and lane information A1. Obtain the historical trajectories of each surrounding agent by the on-board perception and localization system of the autonomous vehicle where each s i,j is composed of two-dimensional coordinate points; subscript i = 0, 1, …, n represents the number of agents, where 0 represents the agent number to be predicted, n represents the maximum number of agents in the scene, and subscript j = -t h , -t h + 1, …, 0 represents the time step, where j = 0 is the current time, t h represents the length of the historical observation interval, j = -t h is the last past historical time in the historical observation interval; the local position of the agent is represented as a D-dimensional position embedding In order to represent the incremental information of motion, the coordinate increment of the current coordinate and the coordinate of the last time are calculated A D-dimensional time embedding is constructed so that the attention mechanism can better perceive the time series features; finally, the historical features of the agent are obtained by splicing as follows: wherein, is a Boolean quantity indicating whether there is a historical observation value at the current moment; in order to facilitate parallel operation of a single scene, for a scene with the number of agents exceeding n, the agents numbered i=0, 1, …, n are intercepted, otherwise, 0 is filled; and a mask m is constructed to indicate which numbered agent i is obtained by filling, so that the attention mechanism focuses on the valid data of the scene frame; A2, lane information in the scene is acquired through a high-precision map, lane points on a lane center line in the high-precision map are constructed into a structured vector form to perform sparse coding, and the lane point vector feature constructed is wherein L is the number of lane points, c 0:L is a coordinate of the lane point, is an offset of the lane point in a road direction relative to a previous lane point, is a J-dimensional Boolean vector, and J different dimensions respectively represent different semantic attributes; in order to aggregate and propagate the lane point vector feature in the entire scene, first, the lane point cloud is processed by using grid sampling to obtain relatively uniform key lane points Then, L jk nearest neighbor points are screened out according to the distance wherein L j represents the number of key points in the j-th scale, and L jk represents the number of corresponding neighbor points around each key point; A3, taking the current road segment where the agent is located as the starting point, breadth-first search along the adjacent direction of each drivable road segment to extract candidate future lane points where L g represents the number of future candidate lane points; since there are points in the future lane points that are associated with the agent's motion trajectory with low correlation, a Boolean vector is constructed indicates whether the future lane point is within the range of the real trajectory and the offset from the real trajectory does not exceed the threshold Th o, in training, these points that exceed the threshold range with low correlation are considered as invalid points and do not participate in the back propagation of the error; B、Scene encoding and feature fusion B1, two Axial-Transformer blocks are used to encode the agent information in the scene, first, the linear transformation is performed on the agent historical features to obtain the agent embedding Next, the input the first Axial-Transformer block, calculate the self-attention along the time axis and encode it, that is, extract the low-level motion characteristics of a single agent; Next, through the second Axial-Transformer block, the self-attention is calculated along the scene axis and encoded, that is, the joint state of the agent at a single time step is extracted, which is updated with the real-time decision results of each agent, and the decision is affected by the traffic rules and interaction relationship at this moment, Therefore, it indirectly encodes these implicit information; The two self-attention Axial-Transformer blocks along the two axes are cascaded together, that is, the real-time intention decision and the low-level motion characteristics are fused; After completing the cascaded encoding of the two axes, a new agent embedding is obtained Then, a one-dimensional convolutional neural network is used to downsample along the time axis to achieve the scale transformation of the features to enhance the ability of the Transformer structure to learn multi-scale features: B2. Encode the lane point cloud in the predicted scene using a lane point encoder; the lane point encoder first aggregates the neighboring points p n Features are aggregated to keypoint p k In the features, the input to the aggregation process is the lane point coordinates c after linear transformation. 0:L Offset relative to the previous lane point And a Boolean vector representing the semantic attributes of lane points Initial features g of lane points formed by splicing and combining 0:L Then, the keypoint features are propagated back to the neighboring point features in the opposite direction, while fusing the relative information between them; after multi-layer feature propagation, the embedded features g' of all lane points are obtained. 0:L as follows: where g 0:L is the initial vector feature of all lane points, W is a linear transformation matrix, aggregatopm(·) and propagation(·) are the feature aggregation function and the feature propagation function respectively, layerfusion(·) is the feature fusion function between layers, and Concatenate(·) is the feature concatenation function, is the initial neighbor point feature, is the new key point feature, is the new neighbor point feature; is the lane point fusion feature of each layer after layer fusion, g' 0:L is the embedding feature of all lane points. B3, fuse the environment features associated with the agent motion and the interaction features, fuse the lane point features within a certain range around the agent in a attention-based manner, the range is a circular area with the agent center as the center and the distance between adjacent lane lines as the radius; and encode through the last Axial-Transformer block, so that the fused environment features are fully propagated in the time dimension and the scene dimension, and then the agent interaction encoder FusionNet in LaneGCN is used to encode the agent interaction in the scene, propagate the semantic features along the agent associated path, and generate the final agent semantic embedding e A2A as follows: 0:n B3, fuse the environment features associated with the agent motion and the interaction features, fuse the lane point features within a certain range around the agent in a attention-based manner, the range is a circular area with the agent center as the center and the distance between adjacent lane lines as the radius; and encode through the last Axial-Transformer block, so that the fused environment features are fully propagated in the time dimension and the scene dimension, and then the agent interaction encoder FusionNet in LaneGCN is used to encode the agent interaction in the scene, propagate the semantic features along the agent associated path, and generate the final agent semantic embedding e A2A as follows: 0:n B3, fuse the environment features associated with the agent motion and the interaction features, fuse the lane point features within a certain range around the agent in a attention-based manner, the range is a circular area with the agent center as the center and the distance between adjacent lane C、Decode multi-modal trajectory and offset, build auxiliary supervision branch C1, a set of multi-layer perceptron (MLP) is used to combine the low-level motion semantic behaviors that are fully encoded in the encoding stage to decode different behavior modalities and obtain a set of future trajectories of the agent On this basis, a regression supervision branch of trajectory is constructed, i.e., a regression loss is calculated r The specific calculation formula is as follows: where t f is the future prediction horizon length, k is the number of modes, corresponding to k possible future scenarios, MLP(·) is a multi-layer perceptron, e i is the semantic embedding of the agent to be predicted, where i = 0, 1, … n is the agent number, NLL(·) is the negative log-likelihood loss function, is the true value of the future trajectory, is the optimal mode k * corresponding predicted trajectory, k * corresponding predicted trajectory and the true trajectory, i.e. C2, another set of MLPs with the same structure is adopted to decode the confidence conf corresponding to the different modal trajectories k On this basis, a classification supervision branch is constructed, that is, the classification loss loss is calculated c As follows: conf k = Softmax(MLP(e i )) In the formula, Softmax(·) is a softmax function, NLL(·) is a negative log-likelihood loss function, is the optimal mode k * corresponding confidence C3, construct an auxiliary supervision branch by predicting the position offset between the future lane points and the trajectory points; calculate the shortest distance between the future real trajectory and the candidate future lane points searched of the position offset i.e. to the real trajectory point; use k MLPs to predict the offset Select the mode k with the smallest average displacement deviation * to calculate the regression loss loss with the true value o , construct a lane point future trajectory offset auxiliary supervision branch, use auxiliary offset supervision loss to assist the decoding of the predicted trajectory and the encoding of the lane point features, so that these future possible lane points provide prior guidance for as many behavior patterns as possible; C4. Through the above three supervision branches, only the optimal mode k * The corresponding error is back-propagated to avoid the assimilation of the parameters of the k MLPs to cause mode collapse. An end-to-end training of the model is supervised by constructing a final loss function composed of three parts, i.e., an agent multi-modal trajectory loss, a confidence loss corresponding to the trajectory, and a lane point future trajectory offset loss. The calculation formula is as follows: loss total = λ1loss r + λ2loss c + λ3loss o where λ1, λ2, λ3are the weights balancing the terms, and loss total is the final total loss.

Images