Trajectory prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a trajectory prediction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring map information of a driving scene, motion information of each agent and relative position information between the agents; the motion information is constructed on the basis of a local coordinate system with each agent as the center; performing feature extraction on the map information, the motion information and the relative position information, and performing feature fusion based on extracted map features, motion features and relative position features to obtain fused motion features of each agent; scene modeling is carried out in a scene representation mode with each agent as the center, the defects that in a traditional scheme, the calculation cost is large, and view angle invariance loss is caused are well overcome, track prediction is carried out on the basis, accurate prediction can be achieved, and meanwhile the scene modeling accuracy is improved. The generalization ability is improved, the requirement for hardware resources is reduced, and the scene adaptability is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and particularly relates to a trajectory prediction method, apparatus, electronic device, and storage medium. Background Art

[0002] In the field of autonomous driving, trajectory prediction is a crucial link. It is located at the back end of the perception module and the front end of the planning and control module, playing a transitional role between the two. It can predict the trajectories of targets in the surrounding environment based on the state information of the targets provided by the perception module, so as to provide a decision-making basis for the planning and control module and ensure the safe and efficient driving of autonomous vehicles.

[0003] Currently, the methods for scene representation in trajectory prediction tasks are mainly divided into two categories. The first category is centered on a certain intelligent agent in the scene, and represents the information of other road participants in the scene to achieve accurate trajectory prediction. The other category is to select fixed coordinate points and perform coordinate normalization on the entire scene.

[0004] However, both of the above two methods have obvious limitations. Although the first method can achieve good results in trajectory prediction tasks, it needs to standardize the entire scene with a target intelligent agent as the center of the scene, thus bringing huge computational overhead. Although the other method alleviates the above computational overhead problem, it loses perspective invariance and has poor accuracy and generalization ability. Summary of the Invention

[0005] The present invention provides a trajectory prediction method, apparatus, electronic device, and storage medium to solve the problems of huge computational overhead in scene representation in existing trajectory prediction tasks, as well as poor trajectory prediction accuracy and generalization ability. While achieving continuous and accurate trajectory prediction, the amount of calculation is reduced, the requirements for hardware resources are lowered, and the practicability is improved.

[0006] The present invention provides a trajectory prediction method, including: Obtaining map information of the driving scene, motion information of each intelligent agent in the driving scene, and relative position information between intelligent agents; the motion information is constructed based on a local coordinate system centered on each intelligent agent; Performing feature extraction on the map information, the motion information, and the relative position information respectively, and performing feature fusion based on the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion feature of each intelligent agent; Based on the fused motion feature, performing trajectory prediction on each intelligent agent to obtain the predicted motion trajectory of each intelligent agent.

[0007] A trajectory prediction method provided by the present invention, wherein the motion information of each agent is determined based on the following steps: Taking the current position of each agent as the center, a local coordinate system of each agent is established; In the local coordinate system of each agent, based on the historical motion trajectory of each agent, the motion information of each agent is constructed.

[0008] A trajectory prediction method provided by the present invention, wherein the map features, motion features, and relative position features obtained based on feature extraction are subjected to feature fusion to obtain the fused motion feature of each agent, including: Based on the map features and the motion features of each agent, feature fusion is performed to obtain the scene feature corresponding to each agent; The relative position feature associated with each agent is determined, and feature interaction is performed based on the relative position feature associated with each agent and the corresponding scene feature to obtain the fused motion feature of each agent.

[0009] A trajectory prediction method provided by the present invention, wherein feature interaction is performed based on the relative position feature associated with each agent and the corresponding scene feature to obtain the fused motion feature of each agent, including: Taking the scene feature corresponding to each agent as a node and the associated relative position feature as an edge connecting the nodes, a complete directed graph with self-loops is constructed; In the complete directed graph, each node is updated based on the edges connected to each node to obtain each updated node; Based on each updated node, the fused motion feature of each agent is determined.

[0010] A trajectory prediction method provided by the present invention, wherein based on the fused motion feature, trajectory prediction is performed on each agent to obtain the predicted motion trajectory of each agent, including: Based on the fused motion feature of each agent, trajectory prediction is performed on each agent to obtain the initial motion trajectory of each agent; Based on the initial motion trajectory, trajectory encoding is performed to obtain the trajectory motion feature of each agent; Based on the trajectory motion feature and the fused motion feature, trajectory prediction is performed on each agent to obtain the predicted motion trajectory of each agent.

[0011] A trajectory prediction method provided by the present invention, which predicts the trajectory of each agent based on the trajectory motion features and the fused motion features to obtain the predicted motion trajectory of each agent, includes: Perform feature interaction based on the trajectory motion features and the fused motion features to obtain the predicted motion features of each agent; Based on the predicted motion features, predict the trajectory of each agent to obtain the predicted motion trajectory of each agent.

[0012] A trajectory prediction method provided by the present invention, which predicts the initial motion trajectory of each agent based on the fused motion features of each agent to obtain the initial motion trajectory of each agent, includes: Based on the fused motion features, use a motion decoder to perform continuous parametric trajectory representation on each agent to obtain the continuous motion trajectory of each agent, and use the continuous motion trajectory as the initial motion trajectory of each agent; Wherein, the motion decoder is trained based on a Bezier curve constructed based on Bernstein polynomials.

[0013] The present invention also provides a trajectory prediction device, including: An information acquisition unit, configured to acquire map information of a driving scenario, motion information of each agent in the driving scenario, and relative position information between agents; the motion information is constructed based on a local coordinate system centered on each agent; A feature fusion unit, configured to respectively perform feature extraction on the map information, the motion information, and the relative position information, and perform feature fusion based on the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion features of each agent; A trajectory prediction unit, configured to predict the trajectory of each agent based on the fused motion features to obtain the predicted motion trajectory of each agent.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor, wherein when the processor executes the computer program, the trajectory prediction method described in any one of the above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the trajectory prediction method described in any one of the above is implemented.

[0016] The trajectory prediction method, device, electronic device, and storage medium provided by the present invention acquire map information of a driving scenario, motion information of each intelligent agent, and relative position information between intelligent agents; the motion information is constructed based on a local coordinate system centered on each intelligent agent; feature extraction is respectively performed on the map information, motion information, and relative position information, and feature fusion is performed according to the extracted map features, motion features, and relative position features to obtain the fused motion features of each intelligent agent, and based on this, trajectory prediction is performed on each intelligent agent to obtain the predicted motion trajectory of each intelligent agent. By performing scene modeling in a scene representation manner centered on each intelligent agent, the defects of large computational overhead and loss of view invariance in the traditional solution are well overcome. Based on this, trajectory prediction can improve its generalization ability while achieving accurate prediction, reduce the requirements for hardware resources, and optimize scene adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 is a flowchart of the trajectory prediction method provided by the present invention; Figure 2 is an overall framework diagram of the trajectory prediction method provided by the present invention; Figure 3 is a structural diagram of the trajectory prediction device provided by the present invention; Figure 4 is a structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention fall within the protection scope of the present invention.

[0020] In autonomous driving technology, trajectory prediction is a crucial link. It is located at the back end of the perception module and the front end of the planning and control module, playing a key role in connecting the upper and lower levels. It can estimate the future trajectories of dynamic objects in the surrounding environment based on the target state information provided by the perception module, thereby providing a decision-making basis for the planning and control module to ensure the safe and efficient driving of autonomous vehicles.

[0021] Currently, the methods for scene representation in trajectory prediction tasks are mainly divided into two categories. The first category of methods centers around a certain intelligent agent (such as a vehicle, pedestrian, etc.) in the scene to represent the information of other road participants in the scene. By focusing on a specific target intelligent agent, this method can more accurately predict its future movement trajectory and is suitable for complex and changeable traffic environments. However, it needs to perform scene normalization processing for each target intelligent agent under the current center respectively, which not only significantly increases the computational complexity but also raises the requirements for hardware resources, making it difficult to meet the driving scenarios with high real-time requirements.

[0022] The other category of methods uses fixed coordinate points as references to perform coordinate normalization processing on the entire scene. By unifying the coordinate system, this method simplifies the complexity of scene representation and effectively reduces the computational cost. However, due to the loss of view invariance (view invariance refers to the consistency and stability of the autonomous driving system of a vehicle in perceiving and understanding the environment from different perspectives), this method performs poorly in terms of the accuracy and generalization ability of trajectory prediction and is difficult to adapt to diverse driving scenarios and complex and changeable traffic conditions.

[0023] In response to this, the present invention provides a trajectory prediction method, aiming to construct a local coordinate system centered on each intelligent agent in the driving scene, perform scene information representation based on this, not only greatly reduce the computational complexity but also ensure view invariance, and perform trajectory prediction according to the obtained information of the driving scene to improve its generalization ability, as well as scene adaptability and practicality while achieving accurate trajectory prediction.

[0024] Figure 1 is a schematic flowchart of the trajectory prediction method provided by the present invention. As Figure 1 shown, this method can be applied to the autonomous driving system of a vehicle and has the advantages of simplicity, high efficiency, and deployability. The method includes: Step 110, obtaining the map information of the driving scene, the movement information of each intelligent agent in the driving scene, and the relative position information between the intelligent agents; the movement information is constructed based on the local coordinate system centered on each intelligent agent; Step 120: Extract features from the map information, motion information, and relative position information respectively, and perform feature fusion based on the map features, motion features, and relative position features obtained from the feature extraction to obtain the fused motion feature of each agent. Step 130: Based on the fused motion feature, perform trajectory prediction for each agent to obtain the predicted motion trajectory of each agent.

[0025] Considering that in the current trajectory prediction task, the scene representation brings huge computational overhead, resulting in a significant increase in the requirements for hardware resources, and the accuracy, stability, and generalization ability of trajectory prediction are poor. In the embodiments of the present invention, it is proposed that the scene information can be represented centered on each instance in the scene, so as to reduce the amount of calculation and computational complexity and reduce the computational overhead while ensuring view invariance.

[0026] Specifically, in order to improve the accuracy of trajectory prediction, in the embodiments of the present invention, in the early stage of trajectory prediction, a scene representation method centered on instances is used for scene modeling. Specifically, a local coordinate system is established centered on each instance in the driving scene, and on this basis, the dynamic information of the entire scene is constructed. Here, the instance is the dynamic target / agent in the scene, which can be a vehicle, a person, or other organisms, such as cats, dogs, etc. The embodiments of the present invention do not make specific limitations on this. By representing the scene information centered on each agent in the driving scene, the motion information of each agent in the entire driving scene can be obtained.

[0027] However, it should be noted that in the entire process of trajectory prediction, if it is necessary to better ensure the accuracy and precision of the finally predicted trajectory, in the information representation stage, in addition to considering the dynamic targets, the static targets in the scene need to be considered, that is, the static elements in the entire driving scene, including lane dividing lines, road boundaries, crosswalks, street lights, fences, indicator lights, etc. That is, it is also necessary to obtain the static information in the driving scene, and this static information can also be called map information. Specifically, for the static elements in the road structure, such as lane dividing lines, here, the midpoint of the dividing line can be used as the anchor point, and the angle of the vector formed between the endpoints can be used as the direction of the line, and based on this, the accurate map information in the driving scene can be constructed.

[0028] Further, during the scene representation process, in addition to the above-mentioned dynamic motion information and static map information, in order to further improve the comprehensiveness and integrity of the scene representation and the accuracy of the final trajectory prediction, in the embodiments of the present invention, it is also necessary to consider the relative positions between dynamic objects to clarify the relative position relationships between all dynamic objects, so as to take this into account during the subsequent trajectory prediction process and obtain an accurate and reliable predicted motion trajectory based on comprehensive and accurate information. That is, it is also necessary to obtain the relative position information between agents in the driving scene, and this relative position information can be determined based on the position information of the corresponding agents.

[0029] After that, in order to more accurately capture various key information in the driving scene, in the embodiments of the present invention, the various information obtained can also be encoded to encode it in the form of feature vectors, so as to obtain the features corresponding to the various information. That is, feature extraction can be performed on the map information of the driving scene, the motion information of each agent, and the relative position information between agents respectively, so as to represent the map information, motion information, and relative position information in the driving scene in the form of feature vectors by using the Vectorized Scene Representation (VSR) method, rather than the traditional rasterized expression. In this way, it can better capture the key information in the driving scene, reduce the consumption of computing resources, and improve the accuracy of prediction.

[0030] Figure 2 is the overall framework diagram of the trajectory prediction method provided by the present invention, as Figure 2 shown. Here, corresponding encoders, namely Actor Encoder, Map Encoder, and Relative Pose Encoder (RPE), can be used to encode the motion information, map information, and relative position information respectively to convert the above information into feature vectors. Specifically, a one-dimensional convolutional neural network is used in the Actor Encoder to process the motion information of the agent, PointNet-based is used in the Map Encoder to process the map information, and a multilayer perceptron (MLP) is used in the Relative Pose Encoder to encode the relative position information between agents, respectively obtaining the map features corresponding to the map information, the motion features corresponding to the motion information, and the relative position features corresponding to the relative position information.

[0031] In the embodiments of the present invention, by constructing a local coordinate system centered on the agent (local reference frame), the characteristics of the agent itself and the relative position characteristics between instances are decoupled accordingly. By encoding the information represented in the instance-centered scene representation manner, the system can learn the view invariance of the trajectory prediction task, thereby improving the accuracy and precision of the trajectory prediction.

[0032] After obtaining the characteristics corresponding to various types of information, in the embodiments of the present invention, it is also necessary to fuse them to achieve feature update in a view-invariant manner. Here, specifically, it can be based on the motion characteristics of each agent, combined with map characteristics and relative position characteristics for feature fusion, so as to achieve information supplementation through interaction during the feature fusion process, thereby obtaining the fused motion characteristics of each agent.

[0033] Specifically, it can be through the map characteristics representing the key static information in the driving scene and the relative position characteristics representing the relative position relationship between agents to supplement the motion characteristics of each agent, so that it can contain more and more accurate key information in the driving scene, thus being able to gain an advantage in the subsequent trajectory prediction process, making the finally output predicted motion trajectory more accurate and reliable.

[0034] After that, the trajectory prediction can be performed according to the fused motion characteristics of each agent to output the predicted motion trajectory of each agent. That is, based on the fused motion characteristics of each agent, the trajectory of each agent in the next period of time, such as 3 - 5 seconds, is estimated to predict its possible direction and output the final trajectory, that is, the predicted motion trajectory. The finally obtained predicted motion trajectory here can be one or multiple. The embodiments of the present invention do not make specific limitations on this. In the case of multiple, usually the probability of each predicted motion trajectory, that is, the confidence level, will also be output. The higher the confidence level, the closer the trajectory is to the true trajectory in the next period of time. On the contrary, the lower the confidence level, the less likely it is to be close to the true trajectory. Through this method, the multi-possibility prediction of the trajectories of all agents in the driving scene can be realized, and finally a continuous, reliable predicted motion trajectory close to the true trajectory can be obtained.

[0035] The trajectory prediction method provided by the present invention obtains the map information of the driving scene, the motion information of each agent, and the relative position information between agents; the motion information is constructed based on a local coordinate system centered on each agent; feature extraction is respectively performed on the map information, motion information, and relative position information, and feature fusion is performed according to the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion feature of each agent. According to the fused motion feature, trajectory prediction is performed on each agent to obtain the predicted motion trajectory of each agent. By performing scene modeling in a scene representation manner centered on each agent, it well overcomes the defects of large computational overhead and loss of view invariance in the traditional scheme. Based on this, trajectory prediction can improve its generalization ability while achieving accurate prediction, reduce the requirements for hardware resources, and optimize scene adaptability.

[0036] Based on the above embodiments, the motion information of each agent is determined based on the following steps: Taking the current position of each agent as the center, a local coordinate system of each agent is established; In the local coordinate system of each agent, based on the historical motion trajectory of each agent, the motion information of each agent is constructed.

[0037] Specifically, the process of representing scene information centered on each agent to obtain the motion information of each agent may specifically include: For each agent in the driving scene, its current position can be obtained first, and a local coordinate system is constructed with the current position as the center. Then, in this local coordinate system, the historical motion trajectory of the agent can be used to construct scene information, thereby obtaining the motion information of the agent. This motion information includes the historical motion trajectory of the agent at historical moments, that is, the historical motion trajectory, and the position at the current moment, that is, the current position.

[0038] In detail, in the actual operation process, the local coordinate system of the historical motion trajectory of the agent can be established at the position of the agent at the current moment, so that the historical motion trajectory of the agent can be reflected in the local coordinate system established with the current position of the agent as the center, and then the motion information of the agent can be constructed. In this way, while realizing scene information representation, view invariance can be ensured.

[0039] Based on the above embodiments, in step 120, feature fusion is performed according to the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion feature of each agent, including: Based on the map features and the motion features of each agent, feature fusion is performed to obtain the scene feature corresponding to each agent; Determine the relative position features associated with each agent, and perform feature interaction based on the relative position features associated with each agent and the corresponding scene features to obtain the fused motion features of each agent.

[0040] Specifically, in the above process, in the process of performing feature fusion based on the map features, motion features, and relative position features to obtain the fused motion features of each agent, the following steps may specifically be included: After obtaining the scene features corresponding to the instance-centered scene information (including the motion features of the agent and the map features) and the relative position features corresponding to the relative position information, the symmetric fusion strategy based on Global Transformer can be used to perform feature fusion on the above features, so as to achieve feature update in a perspective-invariant manner.

[0041] In detail, for each agent, here it may be to first perform feature fusion on the map features and its motion features to obtain the scene features corresponding to each agent; here, the way of feature fusion can be weighted, concatenated, cascaded, etc., and the embodiments of the present invention do not make specific limitations on this. Preferably, the concatenation method is selected for feature fusion here.

[0042] At the same time, it is necessary to determine the relative position features associated with each agent. That is, since the relative position information can reflect the association relationship between agents, for each agent, its associated relative position features can be determined through its associated relative position information. Here, specifically, the relative position features between agents may be processed first, and transformed and enhanced through a feedforward neural network FFN (Feedforward Neural Network) to obtain the processed relative position features. Immediately afterwards, the processed relative position features corresponding to its associated relative position information can be used as the relative position features associated with this agent. It should be noted that there are often more than one relative position feature associated with each agent.

[0043] After that, feature interaction can be performed based on the relative position features associated with each agent and the corresponding scene features to obtain the fused motion features of each agent. Here, specifically, through the way of feature interaction, the associated relative position features are used to supplement the information of the scene features of the corresponding agent, so that it contains more and more accurate key information in the driving scene, thereby obtaining the fused motion features of each agent.

[0044] Based on the above embodiments, performing feature interaction based on the relative position features associated with each agent and the corresponding scene features to obtain the fused motion features of each agent includes: Construct a complete directed graph with self-loops, where the scene features corresponding to each agent are nodes and the associated relative position features are the edges connecting the nodes. In the complete directed graph, update each node based on the edges connected to each node to obtain the updated nodes. Based on the updated nodes, determine the fused motion features of each agent.

[0045] Specifically, the process of feature interaction according to the relative position features associated with each agent and the corresponding scene features can actually model the driving scene as a complete directed graph with self-loops. In this complete directed graph, feature updates are performed in a perspective-invariant manner, so that the predicted motion trajectories output during subsequent trajectory prediction are more accurate and stable.

[0046] In detail, here the driving scene can be regarded as a complete directed graph with self-loops. The input scene features are regarded as nodes, and the associated relative position features are used to describe the edges. That is, a complete directed graph with self-loops (self-closed loops) is constructed, where the scene features corresponding to each agent are nodes and the relative position features associated with it are the edges connecting the nodes.

[0047] Subsequently, the nodes can be updated in this complete directed graph. Specifically, here, in the complete directed graph, each node can be updated using the edges connected to each node to obtain the updated nodes. It should be noted that during the process of interactive update of the nodes, each node is only affected by the edges connected to it, so that the updated nodes can still have perspective invariance. And after each interaction, the nodes and edges are updated.

[0048] After that, the fused motion features of each agent can be determined based on the updated nodes. Since only dynamic targets need to be predicted during the final trajectory prediction process and the required input is only the updated motion features, after feature interaction to achieve feature update, the output is the updated motion features and map features of each agent, rather than the complete scene features. The updated motion features are used as the fused motion features of the corresponding agents.

[0049] Based on the above embodiments, step 130 includes: Perform trajectory prediction on each agent based on the fused motion features of each agent to obtain the initial motion trajectory of each agent; Perform trajectory encoding based on the initial motion trajectory to obtain the trajectory motion features of each agent; Perform trajectory prediction on each agent based on the trajectory motion features and the fused motion features to obtain the predicted motion trajectory of each agent.

[0050] Specifically, when predicting the trajectory based on the fusion motion characteristics of each agent, considering that most single-stage trajectory prediction schemes directly output the prediction results of the agent, the stability of the predicted trajectory is usually poor. In view of this, in the embodiments of the present invention, a multi-stage trajectory prediction scheme is proposed, that is, first use a single-stage trajectory prediction scheme to predict the trajectory, and use the output trajectory as the initial motion trajectory to clarify the possible solution space of the final result, and then use a two-stage trajectory prediction scheme to further encode and update the initial motion trajectory, so that the finally obtained predicted motion trajectory is more accurate and stable.

[0051] In detail, here it can be to first predict the trajectory of each agent according to the fusion motion characteristics of each agent, so as to predict the trajectory trend of each agent in the future for a period of time through the Motion Decoder and output it. At this time, the obtained trajectory is the initial motion trajectory. There are generally multiple initial motion trajectories, and each initial motion trajectory corresponds to a probability, that is, a confidence level, which can reflect the distribution of the initial motion trajectory in the driving scenario.

[0052] Then, the initial motion trajectory obtained in the previous step can be encoded for trajectory to extract its motion and trend characteristics, so as to obtain the trajectory motion characteristics of each agent. After that, based on this trajectory motion characteristic, combined with the fusion motion characteristic of the corresponding agent, the final trajectory prediction can be carried out to obtain the final predicted motion trajectory. Here, interacting the trajectory motion characteristic with the fusion motion characteristic can better clarify the possibility and trend of the final predicted motion trajectory, so as to output a more accurate and reliable predicted motion trajectory.

[0053] Based on the above embodiments, based on the trajectory motion characteristic and the fusion motion characteristic, the trajectory of each agent is predicted to obtain the predicted motion trajectory of each agent, including: Performing feature interaction based on the trajectory motion characteristic and the fusion motion characteristic to obtain the predicted motion characteristic of each agent; Based on the predicted motion characteristic, predicting the trajectory of each agent to obtain the predicted motion trajectory of each agent.

[0054] Specifically, the process of predicting the trajectory of each agent based on the trajectory motion characteristic and the fusion motion characteristic to obtain the predicted motion trajectory of each agent may specifically include the following steps: First, feature interaction can be performed based on the trajectory motion features and the fused motion features to obtain the predicted motion features of each agent. Here, specifically, in a Transformer-based manner, the trajectory motion features and the fused motion features of each agent are interacted to further clarify the trajectory trend of the corresponding agent in a future period of time by integrating the motion information of the agent, the relative position information between agents, and the map information included in the fused motion features. That is, the trajectory motion features and the fused motion features of each agent are input into the Transformer Decoder for feature interaction, and the updated features, namely the predicted motion features, are output.

[0055] Next, based on this predicted motion feature, trajectory prediction can be performed on each agent to determine the final trajectory from the preliminarily determined solution space, and thus the predicted motion trajectory of each agent can be obtained. Here, specifically, the predicted motion feature output after passing through the Transformer Decoder is input into the MLP-based Decoder to decode the predicted motion feature of each agent through this decoder and output the final predicted motion trajectory.

[0056] In the embodiment of the present invention, first, a single-stage trajectory prediction scheme (Motion Decoder) is used for trajectory prediction, and the output trajectory is used as the initial motion trajectory to clarify the possible solution space of the final result. Then, a two-stage trajectory prediction scheme (Transformer Decoder and MLP-based Decoder) is adopted to further encode and update the initial motion trajectory, so that the finally obtained predicted motion trajectory is more accurate and stable.

[0057] Based on the above embodiments, trajectory prediction is performed on each agent based on the fused motion feature of each agent to obtain the initial motion trajectory of each agent, including: Based on the fused motion feature, the motion decoder is used to perform continuous parametric trajectory representation on each agent to obtain the continuous motion trajectory of each agent, and the continuous motion trajectory is used as the initial motion trajectory of each agent; Among them, the motion decoder is trained based on the B-spline curve constructed based on Bernstein polynomials.

[0058] Specifically, considering the current single-stage trajectory prediction process, most of which are for predicting trajectory points and determining multiple prediction results of the agent based on the trajectory points. At this time, the continuity and stability of the predicted trajectory are usually poor. Moreover, specifically when performing trajectory prediction, the training of the trajectory prediction model is mostly based on a single ground truth label. During the training process, the model needs to continuously learn the distribution of trajectory points in order to reasonably output the motion trajectory. However, learning the trajectory points requires a long time and a large amount of data as a basis, resulting in more time and cost.

[0059] In response to this, in the embodiments of the present invention, it is proposed that continuous trajectory prediction can be performed based on Bezier control points to ensure that the predicted initial motion trajectory is more continuous and stable. At the same time, during the training process, the model (Motion Decoder) does not need to learn trajectory points, but only needs to learn the control points of the Bezier curve. In this way, the model can converge faster, the training is more efficient, and the performance is more excellent.

[0060] In detail, here, the fused motion features of each agent can be input into the Motion Decoder to predict the continuous motion trajectory of each agent in the future for a period of time through the Motion Decoder, and the predicted continuous motion trajectory is output, so as to obtain the initial motion trajectory of each agent.

[0061] It should be noted that the Motion Decoder includes a regression head, a classification head, and a softmax. The former is used to predict the trajectory, and the latter two are used to classify and output probabilities. For the regression head, different from directly predicting the two-dimensional coordinate points of the trajectory in the traditional single-stage trajectory prediction scheme, in the embodiments of the present invention, the Bernstein basis polynomial, that is, the Bernstein polynomial, is used to perform continuous parameterization of the trajectory. The parameterized polynomial has a continuous representation, which can make the motion of the trajectory in the future for a period of time smooth and all have high-order derivatives, so as to obtain a continuous and smooth initial motion trajectory.

[0062] In addition, it is worth noting that compared with the case where using a single polynomial to parameterize the trajectory will cause a significant decline in the performance of the model (the reason for the performance decline is that the predicted coefficients have numerical instability, making the regression task very difficult), in the embodiments of the present invention, the Bernstein basis polynomial is introduced, and its coefficients are control points with specific spatial meanings, which can make the Motion Decoder converge faster and better, thus achieving performance improvement.

[0063] In addition, it should be noted that, in addition to the Bernstein basis polynomials introduced above, polynomials with specific spatial meanings for control points of other coefficients can also be applied to the embodiments of the present invention. Correspondingly, the constructed curve equation is not limited to B-spline curves.

[0064] In the embodiments of the present invention, in order to make full use of the advantages of the parameterized trajectory, Bernstein basis polynomials are introduced to continuously parameterize the trajectory. By using the characteristic that its coefficients are control points with specific spatial meanings, the motion decoder can converge better and maintain the continuity and accuracy of the trajectory.

[0065] The trajectory prediction device provided by the present invention will be described below. The trajectory prediction device described below can be correspondingly referred to the trajectory prediction method described above.

[0066] Figure 3 is a schematic structural diagram of the trajectory prediction device provided by the present invention, as Figure 3 shown. The device includes: An information acquisition unit 310, configured to acquire map information of a driving scenario, motion information of each agent in the driving scenario, and relative position information between agents; the motion information is constructed based on a local coordinate system centered on each agent; A feature fusion unit 320, configured to respectively extract features from the map information, the motion information, and the relative position information, and perform feature fusion based on the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion feature of each agent; A trajectory prediction unit 330, configured to perform trajectory prediction on each agent based on the fused motion feature to obtain the predicted motion trajectory of each agent.

[0067] The trajectory prediction device provided by the present invention acquires map information of a driving scenario, motion information of each agent, and relative position information between agents; the motion information is constructed based on a local coordinate system centered on each agent; respectively extracts features from the map information, the motion information, and the relative position information, performs feature fusion according to the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion feature of each agent, and performs trajectory prediction on each agent according to the fused motion feature to obtain the predicted motion trajectory of each agent. By performing scene modeling in a scene representation manner centered on each agent, it well overcomes the defects of large computational overhead and loss of view invariance in the traditional scheme. Based on this, trajectory prediction can be performed while achieving accurate prediction, improving its generalization ability, reducing the requirements for hardware resources, and optimizing scene adaptability.

[0068] Based on the above embodiments, the information acquisition unit 310 is configured to: Taking the current position of each agent as the center, establish a local coordinate system for each agent; In the local coordinate system of each agent, based on the historical movement trajectory of each agent, construct the movement information of each agent.

[0069] Based on the above embodiments, the feature fusion unit 320 is configured to: Perform feature fusion based on the map features and the movement features of each agent to obtain the scene features corresponding to each agent; Determine the relative position features associated with each agent, and perform feature interaction based on the relative position features associated with each agent and the corresponding scene features to obtain the fused movement features of each agent.

[0070] Based on the above embodiments, the feature fusion unit 320 is configured to: Taking the scene features corresponding to each agent as nodes and the associated relative position features as edges connecting the nodes, construct a complete directed graph with self-loops; In the complete directed graph, update each node based on the edges connected to each node to obtain each updated node; Based on each updated node, determine the fused movement features of each agent.

[0071] Based on the above embodiments, the trajectory prediction unit 320 is configured to: Based on the fused movement features of each agent, perform trajectory prediction on each agent to obtain the initial movement trajectory of each agent; Perform trajectory encoding based on the initial movement trajectory to obtain the trajectory movement features of each agent; Based on the trajectory movement features and the fused movement features, perform trajectory prediction on each agent to obtain the predicted movement trajectory of each agent.

[0072] Based on the above embodiments, the trajectory prediction unit 320 is configured to: Perform feature interaction based on the trajectory movement features and the fused movement features to obtain the predicted movement features of each agent; Based on the predicted movement features, perform trajectory prediction on each agent to obtain the predicted movement trajectory of each agent.

[0073] Based on the above embodiments, the trajectory prediction unit 320 is configured to: Based on the fused motion features, a motion decoder is applied to perform continuous parametric trajectory representation on each agent, obtaining the continuous motion trajectory of each agent, and using the continuous motion trajectory as the initial motion trajectory of each agent; Among them, the motion decoder is trained based on a Bezier curve constructed based on Bernstein polynomials.

[0074] Figure 4 An example of the physical structure diagram of an electronic device is shown as Figure 4 shown. The electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 complete mutual communication through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute a trajectory prediction method, which includes: obtaining map information of a driving scenario, motion information of each agent in the driving scenario, and relative position information between agents; the motion information is constructed based on a local coordinate system centered on each agent; respectively performing feature extraction on the map information, the motion information, and the relative position information, and performing feature fusion based on the map features, motion features, and relative position features obtained by feature extraction to obtain the fused motion feature of each agent; based on the fused motion feature, performing trajectory prediction on each agent to obtain the predicted motion trajectory of each agent.

[0075] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is capable of executing the trajectory prediction method provided by each of the above methods. The method includes: obtaining map information of a driving scenario, motion information of each agent in the driving scenario, and relative position information between the agents; the motion information is constructed based on a local coordinate system centered on each agent; respectively performing feature extraction on the map information, the motion information, and the relative position information, and performing feature fusion based on the map features, motion features, and relative position features obtained by the feature extraction to obtain the fused motion feature of each agent; based on the fused motion feature, performing trajectory prediction on each agent to obtain the predicted motion trajectory of each agent.

[0077] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the trajectory prediction method provided by each of the above methods. The method includes: obtaining map information of a driving scenario, motion information of each agent in the driving scenario, and relative position information between the agents; the motion information is constructed based on a local coordinate system centered on each agent; respectively performing feature extraction on the map information, the motion information, and the relative position information, and performing feature fusion based on the map features, motion features, and relative position features obtained by the feature extraction to obtain the fused motion feature of each agent; based on the fused motion feature, performing trajectory prediction on each agent to obtain the predicted motion trajectory of each agent.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0079] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A trajectory prediction method, characterized in that, Including: Obtaining map information of the driving scenario, motion information of each agent in the driving scenario, and relative position information between agents; The motion information is constructed based on a local coordinate system centered on each agent; Feature extraction is respectively performed on the map information, the motion information, and the relative position information, and feature fusion is performed based on the map features, motion features, and relative position features obtained by the feature extraction to obtain the fused motion feature of each agent; Based on the fused motion feature, trajectory prediction is performed on each agent to obtain the predicted motion trajectory of each agent.

2. The trajectory prediction method according to claim 1, wherein The motion information of each agent is determined based on the following steps: Taking the current position of each agent as the center, a local coordinate system of each agent is established; In the local coordinate system of each agent, the motion information of each agent is constructed based on the historical motion trajectory of each agent.

3. The trajectory prediction method according to claim 1, wherein The feature fusion based on the map features, motion features, and relative position features obtained by the feature extraction to obtain the fused motion feature of each agent includes: Performing feature fusion based on the map features and the motion features of each agent to obtain the scenario feature corresponding to each agent; Determining the relative position feature associated with each agent, and performing feature interaction based on the relative position feature associated with each agent and the corresponding scenario feature to obtain the fused motion feature of each agent.

4. The trajectory prediction method according to claim 3, wherein The performing feature interaction based on the relative position feature associated with each agent and the corresponding scenario feature to obtain the fused motion feature of each agent includes: Constructing a fully directed graph with self-loops with the scenario feature corresponding to each agent as the node and the associated relative position feature as the edge connecting the nodes; Updating each node in the fully directed graph based on the edges connected to each node to obtain each updated node; Based on each updated node, determining the fused motion feature of each agent.

5. The trajectory prediction method according to any one of claims 1 to 4, characterized in that The performing trajectory prediction on each agent based on the fused motion feature to obtain the predicted motion trajectory of each agent includes: Performing trajectory prediction on each agent based on the fused motion feature of each agent to obtain the initial motion trajectory of each agent; Performing trajectory encoding based on the initial motion trajectory to obtain the trajectory motion feature of each agent; Performing trajectory prediction on each agent based on the trajectory motion feature and the fused motion feature to obtain the predicted motion trajectory of each agent.

6. The trajectory prediction method according to claim 5, characterized in that The performing trajectory prediction on each agent based on the trajectory motion feature and the fused motion feature to obtain the predicted motion trajectory of each agent includes: Performing feature interaction based on the trajectory motion feature and the fused motion feature to obtain the predicted motion feature of each agent; Performing trajectory prediction on each agent based on the predicted motion feature to obtain the predicted motion trajectory of each agent.

7. The trajectory prediction method according to claim 5, wherein Performing trajectory prediction on each agent based on the fused motion feature of each agent to obtain the initial motion trajectory of each agent, including: Based on the fused motion feature, using a motion decoder to perform continuous parametric trajectory representation on each agent to obtain the continuous motion trajectory of each agent, and using the continuous motion trajectory as the initial motion trajectory of each agent; wherein, the motion decoder is trained based on a Bezier curve constructed based on Bernstein polynomials.

8. A trajectory prediction device, characterized in that, Including: An information acquisition unit, configured to acquire map information of a driving scene, motion information of each agent in the driving scene, and relative position information between agents; The motion information is constructed based on a local coordinate system centered on each agent; A feature fusion unit, configured to respectively perform feature extraction on the map information, the motion information, and the relative position information, and perform feature fusion based on the map feature, motion feature, and relative position feature obtained by the feature extraction to obtain the fused motion feature of each agent; A trajectory prediction unit, configured to perform trajectory prediction on each agent based on the fused motion feature to obtain the predicted motion trajectory of each agent.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, the trajectory prediction method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the trajectory prediction method according to any one of claims 1 to 7 is implemented.

Images