Trajectory prediction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a trajectory prediction method and device, electronic equipment and a storage medium. The trajectory prediction method comprises the steps that three-dimensional point cloud data and two-dimensional image data collected by a target vehicle are acquired; determining historical track information, map data and image data of the target object based on the three-dimensional point cloud data and the two-dimensional image data; extracting features for the historical trajectory data, the map data and the image data, and generating first feature vector data, second feature vector data and third feature vector data; fusing the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data; and determining predicted trajectory information of the target object based on the target fusion feature vector data. According to the embodiment of the invention, the method achieves the fusion of the feature data obtained based on the three-dimensional point cloud data and the two-dimensional image data, and achieves the prediction of the trajectory, so as to improve the accuracy of trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to an intelligent driving trajectory prediction method and device, electronic equipment, and storage medium. Background Art

[0002] In intelligent driving systems, trajectory prediction refers to predicting the future behavior intentions of targets in the environment. Mobile target trajectory prediction methods based on recurrent neural networks have become mainstream. Traditional trajectory prediction methods can only handle some simple interaction scenarios. Compared with vehicles, pedestrians and cyclists are less constrained and regulated by traffic rules and have strong subjectivity and randomness. Autonomous vehicles rely on predicting the intentions of surrounding targets when making driving decisions and trajectory planning. Therefore, how to improve the accuracy of prediction has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above problems, a trajectory prediction method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A trajectory prediction method, the method comprising:

[0005] Acquire three-dimensional point cloud data and two-dimensional image data collected by the target vehicle;

[0006] Determine historical trajectory information, map data, and image data of a target object based on the three-dimensional point cloud data and the two-dimensional image data;

[0007] Extracting features from the historical trajectory data, the map data, and the image data to generate first feature vector data, second feature vector data, and third feature vector data;

[0008] Fusing the first feature vector data, the second feature vector data, and the third feature vector data to obtain target fused feature vector data;

[0009] The predicted trajectory information of the target object is determined based on the target fused feature vector data.

[0010] Optionally, determining the historical trajectory information, map data and image data of the target object based on the three-dimensional point cloud data and the two-dimensional image data includes:

[0011] Annotating the three-dimensional point cloud data and the two-dimensional image data to obtain annotation information;

[0012] According to the annotation information, historical track information, map data and image data of the target object are determined.

[0013] Optionally, extracting features from the historical trajectory data, the map data, and the image data to generate first feature vector data, second feature vector data, and third feature vector data includes:

[0014] Determine a first encoder corresponding to the historical trajectory data, a second encoder corresponding to the map data, and a third encoder corresponding to the image data according to the data types of the historical trajectory data, the map data, and the image data;

[0015] Using the first encoder to encode the historical trajectory data to extract features and generate first feature vector data;

[0016] Using the second encoder to encode the map data to extract features and generate second feature vector data;

[0017] The third encoder is used to extract encoding features from the image data to generate third feature vector data.

[0018] Optionally, fusing the first feature vector data, the second feature vector data, and the third feature vector data to obtain target fused feature vector data includes:

[0019] splicing the first feature vector data, the second feature vector data, and the third feature vector data according to a preset vector dimension to obtain initial fused feature vector data;

[0020] Inputting the initial fused feature vector data into multiple attention layers in parallel to obtain associated feature vector data output by each attention layer, wherein the associated feature vector data is used to represent the association between different target objects, between target objects and environmental elements, and between target objects in different motion time periods;

[0021] The multiple associated feature vector data are subjected to feature fusion to obtain target fused feature vector data.

[0022] Optionally, determining the predicted trajectory information of the target object based on the target fusion feature vector data includes:

[0023] Obtaining an initial motion query pair for capturing the long-term intent and short-term motion details of the target object;

[0024] Performing linear layer processing based on the target fused feature vector data to obtain first fused feature vector data and second fused feature vector data;

[0025] performing query optimization on the initialized motion query pair according to the first fused feature vector data and the second fused feature vector data to obtain updated query data;

[0026] The predicted trajectory information of the target object is determined according to the query data.

[0027] Optionally, determining the predicted trajectory information of the target object according to the query data includes:

[0028] Inputting the query data into a multi-layer perceptron layer for trajectory prediction to obtain a prediction probability and initial trajectory data for each query data;

[0029] generating Gaussian distribution data based on the predicted probability and the initial trajectory data;

[0030] The predicted trajectory information of the target object is generated according to the predicted center of the Gaussian distribution data.

[0031] Optionally, generating Gaussian distribution data based on the predicted probability and initial trajectory data includes:

[0032] The predicted probability and the initial trajectory data are input into a Gaussian mixture model for Gaussian distribution processing, and Gaussian distribution data corresponding to the predicted probability and the initial trajectory data are output.

[0033] A trajectory prediction device, comprising:

[0034] A data acquisition module, used to acquire three-dimensional point cloud data and two-dimensional image data collected by the target vehicle;

[0035] A data processing module, used to determine historical trajectory information, map data and image data of a target object based on the three-dimensional point cloud data and the two-dimensional image data;

[0036] A feature extraction module, used for extracting features from the historical trajectory data, the map data and the image data to generate first feature vector data, second feature vector data and third feature vector data;

[0037] A feature fusion module, used for fusing the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data;

[0038] The trajectory prediction module is used to determine the predicted trajectory information of the target object based on the target fusion feature vector data.

[0039] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the trajectory prediction method as described above when executed by the processor.

[0040] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the trajectory prediction method described above is implemented.

[0041] The embodiments of the present invention have the following advantages:

[0042] The embodiment of the present invention obtains three-dimensional point cloud data and two-dimensional image data for data processing and feature extraction, thereby obtaining target fused feature vector data of multiple feature vectors, and then determines the predicted trajectory information of the target object based on the fused feature vector. In the trajectory prediction process, a variety of related data are integrated for analysis, thereby improving the accuracy of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0044] Figure 1 is a flow chart of steps of a trajectory prediction method provided by one embodiment of the present invention;

[0045] Figure 2 is a flowchart of another trajectory prediction method provided by an embodiment of the present invention;

[0046] Figure 3 It is a schematic diagram of a model structure provided by an embodiment of the present invention;

[0047] Figure 4 It is a schematic diagram of the structure of a trajectory prediction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In practical applications, future trajectories can be directly predicted based on latent features. This motion prediction technology does not rely on explicit target points or destination predictions, but directly learns and predicts possible future trajectories from the current state and historical behavior of traffic participants. This method usually uses direct regression to predict future trajectories, that is, the model directly maps from input features to points in the future trajectory. Since it does not rely on specific target points, this method can flexibly adapt to different traffic scenarios and participant behavior patterns.

[0050] In addition, there is currently a method for identifying the agent's destination using dense target candidates, which includes the following steps: the system first generates a set of possible destination candidate points that cover the area where the agent may reach. These candidate points are usually determined based on map information, traffic rules, and the current state of the agent; for each target candidate, the method predicts the probability that the agent will eventually reach the destination. This involves weighting each possibility, and the weights are learned by the model based on the current situation and historical data; once the probability of each target candidate is determined, the system generates a complete trajectory for each possible destination. This usually involves path planning from the agent's current location to the predicted destination.

[0051] However, despite its flexibility, the first method of predicting future trajectories directly based on latent features may face the challenge of slow convergence because all motion patterns are regressed from the same features without utilizing spatial prior knowledge. The performance of the second method depends heavily on the density of target candidates. If there are too few candidate points, the agent's true destination may be missed; if there are too many candidate points, the computational cost will increase and noise may be introduced. Generating dense target candidates can increase the computational and memory burden because probabilities need to be calculated and trajectories generated for each candidate point.

[0052] In an embodiment of the present invention, three-dimensional point cloud data and two-dimensional image data collected by the vehicle for the surrounding environment are obtained for data processing, feature extraction, and feature fusion, thereby obtaining target fusion feature vector data, and the predicted trajectory information of the target object can be determined based on the fused feature vector. In the trajectory prediction process, a variety of related data are integrated for analysis, thereby improving the accuracy of trajectory prediction.

[0053] Reference Figure 1 , shows a flow chart of a trajectory prediction method provided by an embodiment of the present invention, which may specifically include the following steps:

[0054] Step S101, acquiring three-dimensional point cloud data and two-dimensional image data collected by the target vehicle;

[0055] In practical applications, a data acquisition device for collecting three-dimensional point cloud data and two-dimensional image data may be installed on the target vehicle, and the three-dimensional point cloud data and two-dimensional image data may be collected through the data acquisition device. The data acquisition device used to realize data acquisition may be a laser radar sensor and a camera. The laser point cloud sensor can collect three-dimensional point cloud data, and the camera can collect two-dimensional image data.

[0056] The core scenario of the application scenario of the present invention is self-driving cars, especially in the context of predicting the future behavior of other vehicles, pedestrians, bicycles and other traffic participants. This is crucial for self-driving vehicles to make safe driving decisions. In these scenarios, pedestrians and bicycles are less constrained and regulated by traffic rules than vehicles, and have strong subjectivity and randomness. Therefore, when making driving decisions and trajectory planning, self-driving vehicles rely on predictions of the intentions of surrounding targets. In an embodiment of the present invention, in order to improve the accuracy of trajectory prediction, three-dimensional point cloud data and two-dimensional image data can be introduced to comprehensively analyze the characteristics of the target object, thereby achieving accurate prediction of the trajectory of the target object around the vehicle and providing effective guidance for the vehicle's driving.

[0057] In one example, the LiDAR laser radar system equipped on the vehicle, including the front, right front side, left front side and two side radar devices, can collect laser 3D point cloud data. At the same time, 2D image data is captured by five cameras - front, rear, left and right sides and top cameras. The LiDAR laser radar and camera in the embodiment of the present invention jointly record the real-time status and historical behavior of traffic participants, forming a series of continuous frame data sets, providing comprehensive and accurate environmental perception information for the autonomous driving system.

[0058] Step S102, determining historical trajectory information, map data and image data of the target object based on the three-dimensional point cloud data and the two-dimensional image data;

[0059] After acquiring the three-dimensional point cloud data and the two-dimensional image data, the historical trajectory information, map data and image data of the target object can be obtained by processing the three-dimensional point cloud data and the two-dimensional image data. By processing the data, it is convenient to perform subsequent operations such as feature extraction on the data.

[0060] Among them, the historical trajectory information can be used to reflect the previous trajectory information of the target object, the map data can reflect the surrounding environment of the target vehicle, and the image data can reflect the information of the processed image frame.

[0061] In one embodiment of the present invention, historical trajectory information, map data and image data of a target object are determined based on three-dimensional point cloud data and two-dimensional image data, including: annotating the three-dimensional point cloud data and two-dimensional image data to obtain annotation information; and determining the historical trajectory information, map data and image data of the target object according to the annotation information.

[0062] In practical applications, the collected three-dimensional point cloud data and two-dimensional image data can be annotated to obtain annotation information, and then the historical trajectory information, map data and image data of the target object can be obtained according to the annotation information.

[0063] Among them, the 3D point cloud data annotation can include target object annotation: annotating the location, category and other information of targets such as vehicles, pedestrians, obstacles, etc. Trajectory annotation: annotating the historical trajectory information of the target object, including location, speed, acceleration, etc. Map annotation: annotating the location and category of map elements such as roads, traffic signs, and buildings.

[0064] Two-dimensional image data annotation can include target object annotation: annotating the location, category and other information of targets such as vehicles, pedestrians, and traffic signs. Trajectory annotation: annotating the historical trajectory information of target objects, including location, speed, acceleration, etc. Map annotation: annotating the location and category of map elements such as roads, traffic signs, and buildings.

[0065] In an embodiment of the present invention, the annotation method can be automatic annotation, and specifically, a trained annotation model can be used to implement automatic annotation. For example, for three-dimensional point cloud data, a target detection model (such as PointRCNN, PointPillars) can be used to detect target objects; a trajectory prediction model (such as LSTM, GRU) can be used to predict trajectories, and a map generation model (such as DeepScene) can be used to generate maps.

[0066] For two-dimensional image data, target detection models (such as YOLO and Faster R-CNN) can be used for target object detection. Trajectory prediction models (such as LSTM and GRU) can be used for trajectory prediction, and maps can be generated.

[0067] For example, for the collected 3D point cloud data, you need to use the 3D point cloud annotation tool to annotate the traffic participants and road information in the point cloud, including pedestrians, motor vehicles, bicycles, and traffic lights. The specific annotation steps are to first import the collected point cloud data. Each data format is a pcd file, and the number of internal point clouds is 34k points. After the original data is imported, you need to use the box function of the annotation tool to automatically annotate the target point cloud data. Here, a 3D box is annotated. Because the point cloud itself is a 3D data, the length, width, and height of the target need to be annotated. You can see that pedestrians are marked as red boxes, motor vehicles are marked as blue boxes, bicycles are marked as brown boxes, and traffic lights are marked as yellow boxes.

[0068] After the above annotation of point cloud data, multiple annotation information can be generated. The format of each annotation information is a 7-degree-of-freedom 3D bbox: (cx, cy, cz, l, w, h, θ, cls), where cx, cy, cz represent the center coordinates of the bbox, l, w, h represent the length, width and height of the object, θ represents the yaw angle of the object, and cls represents the category of the target box.

[0069] Similarly, two-dimensional image data can be labeled.

[0070] Specifically, for the 2D image data collected by the camera, you also need to use the annotation tool to annotate the target frame. This includes pedestrians, motor vehicles, bicycles, and traffic lights. The specific annotation steps are to first import the collected 2D image data. The data format is jpg file, and the size of each image is 1920x1280. Figure 3 Right, next you need to use the box function of the annotation tool to annotate the target image. Here, a 2D box is annotated, and the length and width of the target need to be annotated. The specific annotation is as follows Figure 4 On the right, you can see that pedestrians are marked with red boxes, motor vehicles are marked with blue boxes, bicycles are marked with brown boxes, and traffic lights are marked with yellow boxes.

[0071] After the above annotation of 2D image data, multiple annotation information will be generated. The format of each annotation information is: cx, cy, w, h, cls, where cx, cy, cz represent the center coordinates of the bbox, and cls represents the category of the target box.

[0072] In an embodiment of the present invention, historical trajectory information can be obtained based on the obtained labeled data, wherein the historical trajectory information may include but is not limited to batch size, the number of target objects, the number of historical frames, and the state of each frame of the target object, including position, angle, and speed.

[0073] In an embodiment of the present invention, map data may also be obtained based on the annotation data, wherein the map data may include but is not limited to batch size, number of map elements, number of points in each element, attributes of the points including location, and map type.

[0074] In the embodiment of the present invention, image data may also be obtained according to the annotation data, wherein the image data may include but is not limited to batch size, number of historical frames, height and width of the image.

[0075] Step S103, extracting features from the historical trajectory data, the map data, and the image data to generate first feature vector data, second feature vector data, and third feature vector data;

[0076] After obtaining the historical trajectory data, map data, and image data, features can be extracted for each of the three types of data to obtain their respective feature vector data. Among them, the first feature vector data can be used for further processing and analysis of the subsequent network. It contains rich information about the agent's motion state and historical behavior, and provides a basis for predicting the agent's future trajectory. The second feature vector data contains the geometric shape of the road, the curvature of the road, the length of the road segment, the geometric information of the layout of the intersection, and topological features involving the connectivity of the road network. It can also include relative position features, the position of road elements (such as lane lines, intersections, traffic signs) relative to the Agent (vehicles, pedestrians, etc.), and finally dynamic features, environmental features, map element features, and distance feature information. The third feature vector data can include learning the correlation between different channels,

[0077] In one embodiment of the present invention, extracting features from historical trajectory data, map data, and image data to generate first feature vector data, second feature vector data, and third feature vector data includes the following sub-steps:

[0078] Sub-step S11, determining a first encoder corresponding to the historical trajectory data, a second encoder corresponding to the map data, and a third encoder corresponding to the image data according to the data types of the historical trajectory data, the map data, and the image data;

[0079] In practical applications, the first encoder may be an encoder for extracting features from historical trajectory data, such as an Obj encoder. The second encoder may be an encoder for extracting features from map data, such as a Map encoder. The third encoder may be an encoder for extracting features from image data, such as an Img encoder.

[0080] Sub-step S12, using a first encoder to encode the historical trajectory data to extract features and generate first feature vector data;

[0081] The input data of the first encoder is the historical trajectory data, and the output data is the first feature vector data. Specifically, when the Obj encoder encodes the historical trajectory of the target, the specific input dimension is [B, No, To, Co], where B is the batch size, No is the number of agents, To is the number of historical frames, and Co is the state of each frame, including position, angle, and speed. At this stage, the encoder can input the agent's historical trajectory into the polygon encoder based on the state information position, angle, and speed of each frame, and then use the polygon encoder to generate a series of numerical feature vectors, which are then used for further processing and analysis by the subsequent network. These feature vectors contain rich information about the agent's motion state and historical behavior, providing a basis for predicting the agent's future trajectory.

[0082] Sub-step S13, using a second encoder to encode the map data to extract features and generate second feature vector data;

[0083] The input of the second encoder is image data, and the output data is second feature vector data.

[0084] Among them, the Map encoder needs to encode the map data. The specific input dimensions are [B, Nm, n, Cm], B is the batch size, Nm is the number of map elements, n is the number of points in each element, and Cm is the attributes of the points, including the location and map type. The map data is input to the polygon encoder, and then the polygon encoder is used to generate a series of numerical feature vectors. This part of the feature vector contains the geometric shape of the road, the curvature of the road, the length of the road segment, the geometric information of the layout of the intersection, and the topological features related to the connectivity of the road network. It also includes relative position features, the position of road elements (such as lane lines, intersections, traffic signs) relative to the Agent (vehicles, pedestrians, etc.), and finally includes dynamic features, environmental features, map element features, and distance feature information.

[0085] The polygon encoder mentioned above can be a neural network architecture similar to PointNet, so that it can directly operate on the original 3D point cloud data without relying on a regular grid structure or a predefined shape. Therefore, the above-mentioned historical trajectory data and map data are both 3D point cloud data types. Through the multi-layer perceptron structure, features can be extracted for each point separately. These features may include the point's position information, normal information, and color information.

[0086] Sub-step S14, using a third encoder to extract encoding features from the image data to generate third feature vector data.

[0087] The input of the third encoder is image data, and the output data is third feature vector data.

[0088] Among them, the Img encoder can encode image data, and the specific input dimensions are [B, To, C, H, W], where B is the batch size, To is the number of historical frames, C is 3, and H and W are the height and width of the image. Here, the encoder uses a combination of multi-layer convolution and residual structure to extract image features. It can extract features in the channel dimension and learn the correlation between different channels. At the same time, the residual connection can maintain the position information in the spatial dimension, which is helpful for learning the features of different regions in the image. It can also capture the multi-scale features of the image through convolution kernels of different sizes.

[0089] Step S104, fusing the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data;

[0090] In an embodiment of the present invention, the first feature vector data, the second feature vector data and the third feature vector data are used to represent different features. In order to improve the accuracy of trajectory prediction, in practical applications, the first feature vector data, the second feature vector data and the third feature vector data can be fused to obtain fused feature data. The fused feature data integrates features from different aspects and can more accurately describe the features of the target object of the trajectory to be predicted.

[0091] In one embodiment of the present invention, the first feature vector data, the second feature vector data and the third feature vector data can be spliced ​​according to a preset vector dimension to obtain initial fused feature vector data; the initial fused feature vector data is input into multiple attention layers in parallel to obtain associated feature vector data output by each attention layer, and the associated feature vector data is used to represent the correlation between different target objects, between target objects and environmental elements, and between target objects in different motion time periods; multiple associated feature vector data are feature fused to obtain target fused feature vector data.

[0092] In practical applications, the first feature vector data, the second feature vector data, and the third feature vector data have the same feature dimensions, which can be represented by [B, N, Dim], where N, Dim are the number of hidden layer dimensions. Furthermore, the first feature vector data, the second feature vector data, and the third feature vector data can be vector-joined to obtain preliminary fused feature vector data. Specifically, the second vector dimension can be joined, that is, 3 [B, N, Dim] can be joined to obtain the feature dimension [B, 3xN, Dim].

[0093] The initial fusion feature vector data only focuses on its own features, while ignoring the correlation between target objects, between target objects and the environment, and between target objects in different motion time periods.

[0094] Therefore, after obtaining the preliminary fused feature vector data, it can be input into multiple attention layers in parallel to analyze the data feature correlation relationship, and the output multiple correlated feature vector data can be feature fused to obtain target fused feature vector data. The target fused feature vector data can reflect the correlation between different target objects, between target objects and environmental elements, and between target objects in different motion time periods.

[0095] For example, there are six attention layers, and the preliminary fused feature vector data is input in parallel. Finally, the outputs of these six layers will be concat together. Each layer of attention will capture different connections between agents, the relationship characteristics between agents and surrounding environmental elements (such as roads, traffic signs, and other vehicles), and the input is continuous frames, which can capture the characteristics of agent movement changing over time.

[0096] Step S105: determining the predicted trajectory information of the target object based on the target fusion feature vector data.

[0097] After the target fused feature vector data is obtained, the predicted trajectory information of the target object can be determined according to the target fused feature vector data.

[0098] In an embodiment of the present invention, by adopting a method of combining 3D point cloud and 2D image, the input data contains more information, and thus a more accurate prediction result can be obtained; at the same time, more than two feature extraction networks can be used for feature extraction, so that the model can better fuse spatial information and image feature information.

[0099] The embodiment of the present invention obtains three-dimensional point cloud data and two-dimensional image data for data processing and feature extraction, thereby obtaining target fused feature vector data of multiple feature vectors, and then determines the predicted trajectory information of the target object based on the fused feature vector. In the trajectory prediction process, a variety of related data are integrated for analysis, thereby improving the accuracy of trajectory prediction.

[0100] Reference Figure 2 , shows a flowchart of another trajectory prediction method provided by an embodiment of the present invention, which may specifically include the following steps:

[0101] Step S201, acquiring three-dimensional point cloud data and two-dimensional image data collected by the target vehicle;

[0102] Step S202, determining historical trajectory information, map data and image data of the target object based on the three-dimensional point cloud data and the two-dimensional image data;

[0103] Step S203, extracting features from the historical trajectory data, the map data, and the image data to generate first feature vector data, second feature vector data, and third feature vector data;

[0104] Step S204, fusing the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data;

[0105] Step S205, obtaining an initialization motion query pair for capturing the long-term intention and short-term motion details of the target object;

[0106] Among them, the initialized motion query pairs used to capture the long-term intention and short-term motion details of the target object can be divided into static intent queries and dynamic search queries, wherein the static intent query is a learnable position embedding associated with a specific intent point, usually initialized as a vector of fixed dimension, [B, K, Dim], where K is the number of motion query pairs. Dynamic search queries can be used to capture the motion details of the target in a short time (such as within a few seconds). The initialization methods of the two can include but are not limited to any of the following:

[0107] (1) Random initialization: Generate initial values ​​through normal distribution or uniform distribution.

[0108] (2) Fixed value initialization: use an all-zero vector or an all-one vector.

[0109] In an embodiment of the present invention, through static intention query, the model can efficiently identify the potential movement intentions of traffic participants, which helps to reduce the uncertainty of future trajectories; dynamic search query allows the model to be dynamically updated according to the predicted trajectory to collect fine-grained local features, thereby achieving iterative refinement of the trajectory.

[0110] Step S206, performing linear layer processing based on the target fused feature vector data to obtain first fused feature vector data and second fused feature vector data;

[0111] In practical applications, the linear layer can be used to map input features to output features. Specifically, the linear layer can achieve this mapping through matrix multiplication and addition of bias terms. Thus, the linear layer can be set to convert the target fused diagnostic vector data into a feature vector of the expected dimension.

[0112] For example, the fused feature vector data of dimension [B, 3xN, Dim] is input into the first linear layer to obtain the first fused feature vector data of dimension [B, No, Dim], where No is the target number.

[0113] The fused feature vector data of dimension [B, 3xN, Dim] is input into the second linear layer to obtain the second fused feature vector data of dimension [B, Nm, Dim], where Nm is the number of map features.

[0114] Step S207, performing query optimization on the initialized motion query pair according to the first fused feature vector data and the second fused feature vector data to obtain updated query data;

[0115] After the linear layer conversion, the input [B, K, Dim] can be optimized, which is called query here. Specifically, the query can be optimized by defining a learnable position embedding to obtain position information. The formula is as follows:

[0116]

[0117] Among them, MLP is a multi-layer convolutional layer, PE is a sinusoidal positional encoding, i∈{1...K}, and due to its learnability and adaptability, only a minimum number of queries are needed to make effective predictions, rather than using densely placed target candidates to query.

[0118] Get query (ie ), the value can be input into the self-attention layer, and the attention module is applied to propagate information among K intent queries to determine the query content features. The specific formula is as follows:

[0119]

[0120] where i∈{1...K}, is the query content feature from the previous decoder layer, initialized to zero in the first decoder layer, It is the updated query content feature.

[0121] In order to aggregate the scene context features from the encoder network, the content features and position embeddings of the query and key are concatenated to decouple their contributions to the attention weights. A cross-attention module can be introduced, which is formulated as follows:

[0122]

[0123] Among them, i∈{1...K}, MHCA(Q, K, V), Q, K and V in the formula represent Query, Key and Value respectively, MHCA(Q, K, V) represents a multi-head cross attention layer, the symbol “[·, ·]” represents feature concatenation, and “∪” combines agent token∈[B, No, Dim] and map token∈[B, Nm, Dim] as the key and value of the cross attention module. Finally, is the query content feature finally updated in the decoder layer of this transformer (i.e., query data, vector dimension is [B, K, Dim], where B is the batch size, K is the number of queries, and Dim is the feature dimension).

[0124] Step S208: determining predicted trajectory information of the target object according to the query data.

[0125] Specifically, the query data can be input into a multi-layer perceptron layer for trajectory prediction to obtain the prediction probability and initial trajectory data of each query data; Gaussian distribution data is generated based on the prediction probability and the initial trajectory data; and the predicted trajectory information of the target object is generated according to the prediction center of the Gaussian distribution data.

[0126] In practical applications, the optimized multiple query data can be input into the multi-layer perceptron layer for prediction, and the output data is the prediction probability and initial trajectory data corresponding to each query data. Then, the prediction probability and the initial trajectory data are subjected to Gaussian distribution processing to obtain Gaussian distribution data. For example, the prediction probability and the initial trajectory data can be input into a Gaussian mixture model for Gaussian distribution processing, and the prediction probability and the Gaussian distribution data corresponding to the initial trajectory data are output.

[0127] After obtaining the Gaussian distribution data, the prediction center of the Gaussian distribution data can be determined, and the prediction trajectory information obtained by the final prediction can be obtained according to the prediction center.

[0128] For example, after obtaining multiple query data, the query data can be input into the prediction head. In order to obtain the rough prediction probability and detailed future trajectory (i.e., preliminary trajectory data) of each query, it needs to be processed through a layer of MLP layer to obtain the probabilities of K queries and the results of [K, T, 5] respectively, where the value of 5 represents the parameters of the two-dimensional Gaussian distribution (μx, σx; μy, σy; ρ).

[0129] The rough prediction probability and detailed future estimate are input into the GMM module (Gaussian mixture module), and the predicted Gaussian distribution is combined into a mixed Gaussian distribution. The formula is as follows:

[0130]

[0131] Where f(k) is the probability density function of each Gaussian distribution. x , O y represents a specific xy coordinate, pk represents O x , O y The probabilistic information of the position can then be used to generate the predicted trajectory by simply extracting the predicted center of the Gaussian component.

[0132] The embodiment of the present invention obtains three-dimensional point cloud data and two-dimensional image data for data processing and feature extraction, thereby obtaining target fused feature vector data of multiple feature vectors, and then performs query optimization based on the fused feature vector to determine the predicted trajectory information of the target object. In the trajectory prediction process, a variety of related data are integrated for analysis, thereby improving the accuracy of trajectory prediction.

[0133] Reference Figure 3 , shows a schematic diagram of a model structure in an embodiment of the present invention, the model may include a coding layer, a decoding layer and a prediction head, wherein the coding layer may be divided into an Obj coding layer, a Map coding layer, an Img coding layer, and an attention layer. The decoding layer includes a linear layer and a decoding layer.

[0134] Corresponding encoders can be used for encoding in the Obj coding layer, Map coding layer, and Img coding layer to obtain the first feature vector data, the second feature vector data, and the third feature vector data, respectively. The three data are then fused together using the concat layer and input into the six attention layers in parallel for feature association. The data output by the attention layer is fused again to obtain the target feature fusion data.

[0135] The target feature fusion data is input into the decoding layer, and multiple query data are obtained by processing through the linear layer and the decoding layer in the decoding layer.

[0136] The query data is input into the prediction head and processed by the MLP layer and the GMM module in sequence to obtain Gaussian distribution data, and the predicted trajectory data is generated based on the prediction center of the obtained Gaussian distribution data.

[0137] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0138] Reference Figure 4, shows a schematic diagram of the structure of a trajectory prediction device provided by an embodiment of the present invention, which may specifically include the following modules:

[0139] The data acquisition module 401 is used to acquire the three-dimensional point cloud data and two-dimensional image data collected by the target vehicle;

[0140] A data processing module 402, for determining historical trajectory information, map data and image data of a target object based on the three-dimensional point cloud data and the two-dimensional image data;

[0141] A feature extraction module 403 is used to extract features from the historical trajectory data, the map data and the image data to generate first feature vector data, second feature vector data and third feature vector data;

[0142] A feature fusion module 404 is used to fuse the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data;

[0143] The trajectory prediction module 405 is used to determine the predicted trajectory information of the target object based on the target fused feature vector data.

[0144] In an embodiment of the present invention, the data processing module 402 may include the following submodules:

[0145] A labeling information determination submodule, used to label the three-dimensional point cloud data and the two-dimensional image data to obtain labeling information;

[0146] The data processing submodule is used to determine the historical trajectory information, map data and image data of the target object according to the annotation information.

[0147] In one embodiment of the present invention, the feature extraction module 403 may include the following submodules:

[0148] an encoder determination submodule, configured to determine a first encoder corresponding to the historical trajectory data, a second encoder corresponding to the map data, and a third encoder corresponding to the image data according to data types of the historical trajectory data, the map data, and the image data;

[0149] A first feature vector extraction submodule, configured to use the first encoder to encode the historical trajectory data to extract features, and generate first feature vector data;

[0150] A second feature vector extraction submodule, used to use the second encoder to encode the map data to extract features and generate second feature vector data;

[0151] The third feature vector extraction submodule is used to extract encoding features from the image data using the third encoder to generate third feature vector data.

[0152] In an embodiment of the present invention, the feature fusion module 404 may include the following submodules:

[0153] A vector splicing submodule, used for splicing the first feature vector data, the second feature vector data and the third feature vector data according to a preset vector dimension to obtain initial fused feature vector data;

[0154] A vector association submodule, used to input the initial fused feature vector data into multiple attention layers in parallel to obtain associated feature vector data output by each attention layer, wherein the associated feature vector data is used to represent the association between different target objects, between target objects and environmental elements, and between target objects in different motion time periods;

[0155] The multiple associated feature vector data are subjected to feature fusion to obtain target fused feature vector data.

[0156] In one embodiment of the present invention, the trajectory prediction module 405 may include the following submodules:

[0157] An initial motion query pair acquisition submodule, used to acquire an initial motion query pair for capturing the long-term intention and short-term motion details of the target object;

[0158] A linear layer processing submodule, used for performing linear layer processing based on the target fused feature vector data to obtain first fused feature vector data and second fused feature vector data;

[0159] A query optimization submodule, configured to perform query optimization on the initialized motion query pair according to the first fused feature vector data and the second fused feature vector data to obtain updated query data;

[0160] The trajectory prediction submodule is used to determine the predicted trajectory information of the target object according to the query data.

[0161] In one embodiment of the present invention, the trajectory prediction submodule may include the following units:

[0162] A trajectory prediction unit, used for inputting the query data into a multi-layer perceptron layer for trajectory prediction, to obtain a prediction probability and initial trajectory data for each query data;

[0163] A Gaussian distribution data generating unit, configured to generate Gaussian distribution data based on the predicted probability and the initial trajectory data;

[0164] A predicted trajectory generating unit is used to generate predicted trajectory information of the target object according to the predicted center of the Gaussian distribution data.

[0165] In one embodiment of the present invention, when the Gaussian distribution data generating unit is used to generate Gaussian distribution data based on the predicted probability and the initial trajectory data, it is specifically used to:

[0166] The predicted probability and the initial trajectory data are input into a Gaussian mixture model for Gaussian distribution processing, and Gaussian distribution data corresponding to the predicted probability and the initial trajectory data are output.

[0167] The embodiment of the present invention obtains three-dimensional point cloud data and two-dimensional image data for data processing and feature extraction, thereby obtaining target fused feature vector data of multiple feature vectors, and then determines the predicted trajectory information of the target object based on the fused feature vector. In the trajectory prediction process, a variety of related data are integrated for analysis, thereby improving the accuracy of trajectory prediction.

[0168] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the trajectory prediction method as described above is implemented.

[0169] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the trajectory prediction method as described above is implemented.

[0170] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0172] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0173] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0177] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0178] The above provides a detailed introduction to a trajectory prediction method and device, an electronic device, and a storage medium. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A trajectory prediction method, characterized in that: The method comprises: Acquire three-dimensional point cloud data and two-dimensional image data collected by the target vehicle; Determine historical trajectory information, map data, and image data of a target object based on the three-dimensional point cloud data and the two-dimensional image data; Extracting features from the historical trajectory data, the map data, and the image data to generate first feature vector data, second feature vector data, and third feature vector data; Fusing the first feature vector data, the second feature vector data, and the third feature vector data to obtain target fused feature vector data; The predicted trajectory information of the target object is determined based on the target fused feature vector data.

2. The method according to claim 1, characterized in that The determining of historical trajectory information, map data, and image data of a target object based on the three-dimensional point cloud data and the two-dimensional image data includes: Annotating the three-dimensional point cloud data and the two-dimensional image data to obtain annotation information; According to the annotation information, historical track information, map data and image data of the target object are determined.

3. The method according to claim 1, characterized in that The extracting features from the historical trajectory data, the map data, and the image data to generate first feature vector data, second feature vector data, and third feature vector data includes: Determine a first encoder corresponding to the historical trajectory data, a second encoder corresponding to the map data, and a third encoder corresponding to the image data according to the data types of the historical trajectory data, the map data, and the image data; Using the first encoder to encode the historical trajectory data to extract features and generate first feature vector data; Using the second encoder to encode the map data to extract features and generate second feature vector data; The third encoder is used to extract encoding features from the image data to generate third feature vector data.

4. The method according to claim 1, characterized in that The fusing the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data includes: splicing the first feature vector data, the second feature vector data, and the third feature vector data according to a preset vector dimension to obtain initial fused feature vector data; Inputting the initial fused feature vector data into multiple attention layers in parallel to obtain associated feature vector data output by each attention layer, wherein the associated feature vector data is used to represent the association between different target objects, between target objects and environmental elements, and between target objects in different motion time periods; The multiple associated feature vector data are subjected to feature fusion to obtain target fused feature vector data.

5. The method according to claim 1, characterized in that The determining the predicted trajectory information of the target object based on the target fusion feature vector data includes: Obtaining an initial motion query pair for capturing the long-term intent and short-term motion details of the target object; Performing linear layer processing based on the target fused feature vector data to obtain first fused feature vector data and second fused feature vector data; performing query optimization on the initialized motion query pair according to the first fused feature vector data and the second fused feature vector data to obtain updated query data; The predicted trajectory information of the target object is determined according to the query data.

6. The method according to claim 5, characterized in that The determining the predicted trajectory information of the target object according to the query data includes: Inputting the query data into a multi-layer perceptron layer for trajectory prediction to obtain a prediction probability and initial trajectory data for each query data; generating Gaussian distribution data based on the predicted probability and the initial trajectory data; The predicted trajectory information of the target object is generated according to the predicted center of the Gaussian distribution data.

7. The method according to claim 6, characterized in that The generating Gaussian distribution data based on the predicted probability and the initial trajectory data comprises: The predicted probability and the initial trajectory data are input into a Gaussian mixture model for Gaussian distribution processing, and Gaussian distribution data corresponding to the predicted probability and the initial trajectory data are output.

8. A trajectory prediction device, characterized in that: The device comprises: A data acquisition module, used to acquire three-dimensional point cloud data and two-dimensional image data collected by the target vehicle; A data processing module, used to determine historical trajectory information, map data and image data of a target object based on the three-dimensional point cloud data and the two-dimensional image data; A feature extraction module, used for extracting features from the historical trajectory data, the map data and the image data to generate first feature vector data, second feature vector data and third feature vector data; A feature fusion module, used for fusing the first feature vector data, the second feature vector data and the third feature vector data to obtain target fused feature vector data; The trajectory prediction module is used to determine the predicted trajectory information of the target object based on the target fusion feature vector data.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the trajectory prediction method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the trajectory prediction method according to any one of claims 1 to 7 is implemented.

Images