Trajectory prediction method and related device

Abstract

The invention discloses a trajectory prediction method and a related device, and the method comprises the steps: obtaining historical trajectory data of a target object and surrounding objects and related data of surrounding roads, and the related data of the surrounding roads comprises topological structures and reference line data of the surrounding roads; determining the current road of the target object according to the current position of the target object and the reference line data of the surrounding roads; according to the topological structure of the current road of the target object, determining the next selectable road of the target object to obtain a plurality of candidate roads, and determining the subsequent road of each candidate road to obtain a plurality of candidate road reference lines; and removing repeated road section reference lines from the road section reference lines forming the candidate road reference lines, and predicting the future trajectory of the target object according to the historical trajectory data of the target object and the surrounding objects and the reference line data of the remaining road section reference lines. The trajectory prediction method disclosed by the invention has a relatively good prediction effect and relatively low computing resource consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of trajectory prediction, and in particular to a trajectory prediction method and related device. BACKGROUND

[0002] In some fields, it is necessary to predict the future trajectory of a movable object. For example, in the field of autonomous driving, it is necessary to predict the future trajectory of a vehicle around the ego vehicle to provide a basis for decision-making for the ego vehicle.

[0003] Most of the current trajectory prediction schemes are rule-based trajectory prediction schemes, that is, according to the historical trajectory data of a target object, the future trajectory of the target object is predicted according to a preset prediction rule.

[0004] Although the rule-based trajectory prediction scheme can complete trajectory prediction with relatively low computing resources, the prediction effect is not good. SUMMARY

[0005] Therefore, the present application provides a trajectory prediction method and related device to solve the problem of poor prediction effect of the existing trajectory prediction scheme, and the technical solutions are as follows:

[0006] The first aspect of the present application provides a trajectory prediction method, comprising:

[0007] obtaining historical trajectory data of a target object, historical trajectory data of objects around the target object, and related data of roads around the target object, wherein the related data of the surrounding roads includes the topological structure of the surrounding roads and reference line data;

[0008] determining the current road where the target object is located from the surrounding roads according to the current position of the target object and the reference line data of the surrounding roads;

[0009] determining the next selectable road of the target object according to the topological structure of the current road where the target object is located, obtaining a plurality of candidate roads, and determining the successor road of each candidate road to obtain a plurality of candidate road reference lines, wherein any candidate road reference line is composed of the reference line of a candidate road and the reference line of the successor road of the candidate road;

[0010] predicting the future trajectory of the target object according to the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of the plurality of candidate road reference lines.

[0011] In a possible implementation, any candidate road reference line is composed of a plurality of road segment reference lines.

[0012] The future trajectory of the target object is predicted according to the historical trajectory data of the target object, the historical trajectory data of the surrounding object, and the reference line data of the candidate road reference lines.

[0013] The repeated road segment reference lines are removed from the road segment reference lines constituting each candidate road reference line, and a candidate road segment reference line set of the target object is composed of the remaining road segment reference lines.

[0014] The future trajectory of the target object is predicted according to the historical trajectory data of the target object, the historical trajectory data of the surrounding object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object.

[0015] In a possible implementation, the reference line data of the surrounding road includes reference line coordinate data of the surrounding road.

[0016] The current road of the target object is determined from the surrounding road according to the current position of the target object and the reference line data of the surrounding road, including:

[0017] The roads with a distance less than a preset distance from the target object are filtered out from the surrounding road according to the current position of the target object and the reference line coordinate data of the surrounding road.

[0018] The roads that do not match the motion direction of the target object are filtered out from the filtered roads.

[0019] If the remaining roads are one, the remaining road is determined as the current road of the target object.

[0020] If the remaining roads are multiple, the road closest to the target object is determined as the current road of the target object.

[0021] In a possible implementation, the future trajectory of the target object is predicted according to the historical trajectory data of the target object, the historical trajectory data of the surrounding object, and the reference line data of the candidate road reference lines, including:

[0022] Features are extracted from the historical trajectory data of the target object and the historical trajectory data of the surrounding object to obtain trajectory features of the target object and the surrounding object, and features are extracted from the reference line data of each road segment reference line in the candidate road segment reference line set of the target object to obtain a plurality of road segment features.

[0023] The trajectory feature of the target object, the trajectory feature of the surrounding object, and the road segment features are processed into a target feature, wherein the target feature can reflect the influence degree of the surrounding object on the target object and the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object;

[0024] The future trajectory of the target object is predicted according to the trajectory feature of the target object and the target feature.

[0025] In a possible implementation, the future trajectory of the target object is predicted according to the historical trajectory data of the target object, the historical trajectory data of the surrounding object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, and the prediction comprises:

[0026] The future trajectory of the target object is predicted according to the historical trajectory data of the target object, the historical trajectory data of the surrounding object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object by using a pre-trained trajectory prediction model;

[0027] The trajectory prediction model is trained by using training samples labeled with real future trajectories, and the training samples include historical trajectory data of a sample object, historical trajectory data of surrounding objects of the sample object, and reference line data of each road segment reference line in a candidate road segment reference line set of the sample object.

[0028] In a possible implementation, the trajectory prediction model comprises a trajectory encoder, a road encoder, a first attention module, a second attention module, and a trajectory decoder.

[0029] The future trajectory of the target object is predicted according to the historical trajectory data of the target object, the historical trajectory data of the surrounding object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object by using a pre-trained trajectory prediction model, and the prediction comprises:

[0030] The historical trajectory data of the target object and the historical trajectory data of the surrounding object are encoded by using the trajectory encoder to obtain the trajectory feature of the target object and the trajectory feature of the surrounding object.

[0031] The reference line data of each road segment reference line in the candidate road segment reference line set of the target object is encoded by using the road encoder to obtain road segment features.

[0032] Using the first attention module, attention is calculated on the trajectory features of the target object and the trajectory features of the surrounding objects to obtain a first attention feature, wherein the first attention feature can reflect the degree of influence of the surrounding objects on the target object;

[0033] Using the second attention module, attention calculation is performed on the first attention feature and the several road segment features to obtain the second attention feature, wherein the second attention feature can reflect the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object;

[0034] Using the trajectory decoder, the future trajectory of the target object is predicted based on the second attention feature and the trajectory feature of the target object.

[0035] In one possible implementation, the trajectory encoder is a long short-term memory network, the road encoder is a first multilayer perceptron, and the trajectory decoder is a second multilayer perceptron.

[0036] A second aspect of this application provides a trajectory prediction device, comprising: a data acquisition module, a current road determination module, a candidate road reference line acquisition module, and a trajectory prediction module;

[0037] The data acquisition module is used to acquire historical trajectory data of the target object, historical trajectory data of objects surrounding the target object, and relevant data of roads surrounding the target object. The relevant data of the surrounding roads includes the topology and reference line data of the surrounding roads.

[0038] The current road determination module is used to determine the current road of the target object from the surrounding roads based on the current position of the target object and the reference line data of the surrounding roads.

[0039] The candidate road reference line acquisition module is used to determine the next selectable road for the target object based on the topology of the road where the target object is currently located, to obtain a number of candidate roads, and to determine the successor road of each candidate road, so as to obtain a number of candidate road reference lines. Each candidate road reference line consists of a reference line of a candidate road and a reference line of a successor road of that candidate road.

[0040] The trajectory prediction module is used to predict the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of the several candidate road reference lines.

[0041] A third aspect of this application provides an electronic device, comprising: at least one processor;

[0042] and a memory communicatively connected to the at least one processor;

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the computer program so that the electronic device can implement the steps of any of the above trajectory prediction methods.

[0045] The fourth aspect of this application provides an autonomous driving vehicle, including the aforementioned electronic devices.

[0046] Considering that the movement of a target object is influenced by the movement of surrounding objects and the surrounding road conditions, the trajectory prediction method provided in this application, when predicting the future trajectory of the target object, considers not only the historical trajectory data of the target object itself, but also the historical trajectory data of surrounding objects and the relevant data of the surrounding roads. Based on the target object's own historical trajectory data, combined with the historical trajectory data of surrounding objects and the relevant data of the surrounding roads, a more accurate trajectory prediction can be achieved. Considering that some data in the relevant data of the surrounding roads is irrelevant to the target object, in order to reduce the computational load of subsequent trajectory prediction and avoid the influence of irrelevant data on the prediction results, this application determines the road the target object is currently on from the surrounding roads, and then, based on the topology of the road the target object is currently on, determines the next selectable road for the target object, obtaining several candidate roads. Then, it determines the successor road of each candidate road, obtaining several candidate road reference lines. Finally, based on the historical trajectory data of the target object, the historical trajectory data of surrounding objects, and the reference line data of the candidate road reference lines, the future trajectory of the target object is predicted. In summary, the trajectory prediction method provided in this application has good prediction performance and relatively low computational resource consumption. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A schematic flowchart illustrating the trajectory prediction method provided in this application embodiment;

[0049] Figure 2 A flowchart illustrating the trajectory prediction method based on a trajectory prediction model provided in this application embodiment;

[0050] Figure 3 A schematic diagram illustrating the trajectory prediction implementation of each module based on the trajectory prediction model provided in the embodiments of this application;

[0051] Figure 4 A schematic diagram illustrating the input of a first attention feature and several road segment features into a second attention module for calculation, and the fusion of the second attention feature with the trajectory features of the target object, provided for embodiments of this application;

[0052] Figure 5 A schematic diagram illustrating the prediction effect of the trajectory prediction method provided in this application embodiment in a straight-line scenario;

[0053] Figure 6 A schematic diagram illustrating the prediction effect of the trajectory prediction method provided in this application embodiment in a turning scenario;

[0054] Figure 7 A schematic diagram illustrating the prediction effect of the trajectory prediction method provided in this application embodiment in a lane change scenario;

[0055] Figure 8 This is a schematic diagram of the trajectory prediction device provided in an embodiment of this application. Detailed Implementation

[0056] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0057] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0058] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0059] Most current trajectory prediction schemes are rule-based. Rule-based trajectory prediction schemes take into account the state of the target object itself and use simple trajectories that satisfy kinematics and dynamics as the predicted trajectory.

[0060] Since rule-based trajectory prediction schemes only consider the target object's own state (i.e., historical trajectory data) and do not consider the state of surrounding objects and the surrounding road conditions, they are only suitable for short-term prediction in relatively simple scenarios with few targets, and are not suitable for trajectory prediction in complex scenarios. If applied to complex scenarios, the prediction effect is poor.

[0061] For rule-based trajectory prediction schemes, one current strategy is to use a deep learning-based trajectory prediction scheme. This involves pre-training a trajectory prediction model and inputting historical trajectory data of the target object, historical trajectory data of objects surrounding the target object, and relevant data of all roads around the target object (such as the entire reference line data of all roads) into the trajectory prediction model. The trajectory prediction model then predicts the future trajectory of the target object based on the input data.

[0062] The study of the trajectory prediction scheme based on the above trajectory prediction model found that the input data of the trajectory prediction model includes relevant data of all roads around the target object. However, the relevant data of all roads around the target object contains some irrelevant and redundant data. This data increases the amount of computation and is not conducive to the model extracting useful features, thus affecting the prediction effect.

[0063] To address the problems existing in the above solutions, this application provides a trajectory prediction method that can be applied to electronic devices such as mobile phones, tablets, wearable electronic devices (such as smartwatches), augmented reality (AR) / virtual reality (VR) devices, and in-vehicle devices. The trajectory prediction method provided by this application will be described in the following embodiments.

[0064] Please see Figure 1 The diagram illustrates a flowchart of a trajectory prediction method provided in an embodiment of this application, which may include:

[0065] Step S101: Obtain historical trajectory data of the target object, historical trajectory data of objects surrounding the target object, and relevant data of roads surrounding the target object.

[0066] In this embodiment, the target object is a movable object, which can be a vehicle or a pedestrian.

[0067] For example, in an autonomous driving scenario, the target object can be any vehicle or pedestrian around the vehicle.

[0068] In this embodiment, the historical trajectory data consists of trajectory point data up to and including the target time. The trajectory point data may include trajectory point coordinates. Assuming the target time is time t, the historical trajectory data of the target object can be represented as follows: ,in, The plane representing the target object at time t coordinate.

[0069] In this embodiment, the relevant data of the roads around the target object may include the topology and reference line data of the roads around the target object, wherein the reference line data of the roads around the target object may include the reference line coordinate data of the roads around the target object.

[0070] Step S102: Based on the current location of the target object and the reference line data of the roads around the target object, determine the road where the target object is currently located from the roads around the target object.

[0071] The current position of the target object is the position of the target object at the target time. ).

[0072] In one possible implementation, the process of determining the current road location of the target object from the surrounding roads, based on the target object's current location and reference line data of the roads around the target object, may include:

[0073] Step S1021: Based on the current location of the target object and the reference line coordinate data of the roads around the target object, filter out the roads that are less than a preset distance from the target object.

[0074] Specifically, for each road around the target object, the distance between the target object and the road is calculated based on the target object's current position and the road's reference line coordinate data. This allows us to obtain the distance between the target object and all surrounding roads, and then filter out roads from the surrounding roads that are less than a preset distance from the target object.

[0075] The reference line coordinates of any road surrounding the target object are a coordinate sequence. When calculating the distance between the target object and any surrounding road, the current position of the target object can be calculated. The distance between the target object and each coordinate in the reference line coordinate sequence of the road is calculated, and the minimum distance among the calculated distances is taken as the distance between the target object and the road.

[0076] It should be noted that if no roads are found based on the preset distance, the preset distance can be increased, and then the search can be performed again based on the adjusted distance.

[0077] Step S1022: Filter out the roads that do not match the movement direction of the target object from the selected roads.

[0078] For example, in an autonomous driving scenario, the target is vehicle A, and vehicle A travels in the direction of a. If a road only allows vehicles to travel in the direction of b, then the road does not match the travel direction of vehicle A.

[0079] Step S1023: If there is only one remaining road, then the remaining road is determined as the road where the target object is currently located. If there are multiple remaining roads, then the road closest to the target object is determined as the road where the target object is currently located.

[0080] After filtering out roads that do not match the target object's direction of movement from the selected roads, if there is only one remaining road, then that remaining road is directly determined as the target object's current road. If there are multiple remaining roads, then the target object's current road is determined based on the distance between the target object and the road, that is, the road closest to the target object is determined as the target object's current road.

[0081] Step S103: Based on the topology of the road where the target object is currently located, determine the next road that the target object can choose, obtain several candidate roads, and determine the successor road of each candidate road to obtain several candidate road reference lines.

[0082] Each candidate road reference line consists of a reference line of a candidate road and a reference line of a subsequent road of that candidate road.

[0083] For example, in an autonomous driving scenario, the target is vehicle A. After determining the lane currently occupied by vehicle A, the next lane that vehicle A can drive in can be determined based on the topology of the lane currently occupied by vehicle A, so as to obtain several candidate lanes. After determining several candidate lanes, a depth search is further performed on each candidate lane based on a depth search function to obtain the successor lane of each candidate lane. In this way, several candidate lane reference lines (i.e. several drivable lane reference lines) can be obtained. Any candidate lane reference line consists of the reference line of a candidate lane and the reference line of a successor lane of that candidate lane.

[0084] Step S104: Based on the historical trajectory data of the target object, the historical trajectory data of objects surrounding the target object, and the reference line data of several candidate road reference lines, predict the future trajectory of the target object.

[0085] Any candidate road reference line consists of several road segment reference lines. In one possible implementation, the road segment reference lines that make up each candidate road reference line can form a set of candidate road segment reference lines for the target object. Then, based on the historical trajectory data of the target object, the historical trajectory data of the objects surrounding the target object, and the reference line data of each road segment reference line in the set of candidate road segment reference lines for the target object, the future trajectory of the target object (the trajectory after a period of time after the target time) can be predicted.

[0086] Considering that there are duplicate road segment reference lines among the road segment reference lines that make up the candidate road reference lines, in order to reduce the computational load of subsequent trajectory prediction and avoid redundant information affecting the prediction effect, in another possible implementation, duplicate road segment reference lines can be removed from the road segment reference lines that make up the candidate road reference lines, and the remaining road segment reference lines can form a candidate road segment reference line set for the target object. Then, based on the historical trajectory data of the target object, the historical trajectory data of the objects surrounding the target object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, the future trajectory of the target object can be predicted.

[0087] For example, there are two candidate road reference lines, one of which is... Another candidate road reference line is Reference lines for road segments with the same envelope and ,Right now and For repeated road segment reference lines, therefore, and Remove, and form a set of candidate road segment reference lines for the target object from the remaining road segment reference lines. .

[0088] Assuming the candidate road segment reference line set of the target object includes N road segment reference lines, and each road segment reference line is represented by k (e.g., 10) two-dimensional coordinates, then the reference line data of each road segment reference line in the candidate road segment reference line set of the target object can be represented as a matrix E. ref ∈ℝ N×k×2 .

[0089] Considering that the movement of a target object is affected by the movement of surrounding objects and the surrounding road conditions, the trajectory prediction method provided in this application, when predicting the future trajectory of the target object, not only considers the historical trajectory data of the target object itself, but also the historical trajectory data of surrounding objects and the relevant data of the surrounding roads. Based on the target object's own historical trajectory data, combined with the historical trajectory data of surrounding objects and the relevant data of the surrounding roads, a more accurate trajectory prediction can be achieved. Considering that the relevant data of the surrounding roads contains some data unrelated to the target object and redundant data, in order to reduce the computational load of subsequent trajectory prediction and avoid the influence of irrelevant and redundant data on the prediction results, this application embodiment determines the current road of the target object from the surrounding roads, and then, based on the target... The topology of the road currently in which the target object is located is used to determine the next selectable road for the target object, resulting in several candidate roads. Then, the successor road for each candidate road is determined, resulting in several candidate road reference lines (road reference lines related to the target object). Considering the existence of duplicate road segment reference lines (redundant data) among the road segment reference lines that make up the candidate road reference lines, in order to reduce the computational load of subsequent trajectory prediction and avoid the influence of redundant data on the prediction results, this embodiment removes duplicate road segment reference lines from the road segment reference lines that make up the candidate road reference lines, and forms a set of candidate road segment reference lines for the target object. Then, based on the historical trajectory data of the target object, the historical trajectory data of surrounding objects, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, the future trajectory of the target object is predicted. In summary, the trajectory prediction method provided by this embodiment has good prediction performance and relatively low computational resource consumption.

[0090] As mentioned in the above embodiments, after obtaining the set of candidate road segment reference lines for the target object, the future trajectory of the target object can be predicted based on the historical trajectory data of the target object, the historical trajectory data of the objects surrounding the target object, and the reference line data of each road segment reference line in the set of candidate road segment reference lines for the target object.

[0091] In one possible implementation, the process of predicting the future trajectory of the target object based on its historical trajectory data, the historical trajectory data of objects surrounding the target object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object may include:

[0092] Step a1: Extract features from the historical trajectory data of the target object and the historical trajectory data of surrounding objects to obtain the trajectory features of the target object and the trajectory features of surrounding objects. Extract features from the reference line data of each road segment reference line in the candidate road segment reference line set of the target object to obtain several road segment features.

[0093] This embodiment extracts features from the historical trajectory data of the target object and surrounding objects on the one hand, and extracts features from the reference line data of each road segment reference line in the candidate road segment reference line set on the other hand.

[0094] Step a2: Process the trajectory features of the target object, the trajectory features of surrounding objects, and the features of several road segments into target features.

[0095] Among them, the target features can reflect the degree of influence of the surrounding objects on the target object and the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object.

[0096] Step a3: Based on the trajectory characteristics of the target object and the target characteristics, predict the future trajectory of the target object.

[0097] In one possible implementation, steps a1 to a3 above can be implemented based on a pre-trained trajectory prediction model (deep learning model). This trajectory prediction model is trained using training samples labeled with real future trajectories. The training samples include historical trajectory data of the sample object, historical trajectory data of objects surrounding the sample object, and reference line data of each road segment reference line in the candidate road segment reference line set of the sample object. The method of obtaining the candidate road segment reference line set of the sample object is the same as the method of obtaining the candidate road segment reference line set of the target object.

[0098] The following section introduces the process of trajectory prediction based on a pre-trained trajectory prediction model.

[0099] like Figure 2 As shown, the process of trajectory prediction based on a pre-trained trajectory prediction model can include:

[0100] Step S201a: Using the trajectory prediction model, extract features from the historical trajectory data of the target object and the historical trajectory data of surrounding objects to obtain the trajectory features of the target object and the trajectory features of surrounding objects.

[0101] Trajectory prediction models can include trajectory encoders, such as Figure 3 As shown, the historical trajectory data of the target object and the historical trajectory data of the objects surrounding the target object can be input into the trajectory encoder for encoding. The trajectory encoder outputs the trajectory features of the target object and the trajectory features of the objects surrounding the target object.

[0102] Historical trajectory data is a time series. Considering that Long Short-Term Memory Networks (LSTM) can effectively capture the long-short-term dependencies in the time series, the trajectory encoder in this embodiment can adopt LSTM.

[0103] The encoding process of a trajectory encoder can be represented as follows:

[0104] (1).

[0105] Assuming there are n-1 objects surrounding the target object, the historical trajectory data of the i-th object among the n objects (the target object and the n-1 surrounding objects) is... During encoding, in the first time step, according to and get (The output value of the output gate at the first time step) and (Hidden state at the first time step), at the second time step, according to and get (The output value of the output gate at the second time step) and (Hidden state at the second time step), ..., at the t-th time step, according to and get (The output value of the output gate at time step t) and (The hidden state at time step t) will eventually output the value of the gate at the last time step, which is the output value at time step t. As the trajectory feature of the i-th object.

[0106] The trajectory encoder processes historical trajectory data of n objects ( Encoding this yields:

[0107] (2).

[0108] In one possible implementation, the trajectory data of n objects (the target object and the surrounding n-1 objects) can be directly used. The extracted trajectory features are encoded by the trajectory encoder. In order to standardize the extracted trajectory features, in another possible implementation, for each of the n objects, the last trajectory point in the historical trajectory data of that object (i.e., the trajectory point at time t) can be used as the origin to transform the overall trajectory from the global coordinate system to the local coordinate system, so as to obtain the historical trajectory data of the n objects after transformation. Then, the trajectory data of the n objects after transformation is input into the trajectory encoder for encoding.

[0109] It should be noted that the historical trajectory data input to the trajectory encoder includes not only the coordinates of each trajectory point, but also one or more of the following: velocity, acceleration, heading angle, and angular velocity corresponding to each trajectory point.

[0110] Step S201b: Using the trajectory prediction model, extract features from the reference line data of each road segment reference line in the candidate road segment reference line set of the target object to obtain several road segment features.

[0111] Trajectory prediction models can include road encoders, such as Figure 3 As shown, the reference line data of each road segment reference line in the candidate road segment reference line set of the target object can be input into the road encoder for encoding. The road encoder outputs several road segment features. It should be noted that encoding the reference line data of a road segment reference line can yield a road segment feature.

[0112] In one possible implementation, the road encoder can be a first multilayer perceptron (MLP), which may include a first linear layer, a normalization layer, an activation layer, and a second linear layer.

[0113] If we represent the reference line data of each road segment reference line in the candidate road segment reference line set of the target object as E ref The characteristics of several road segments are represented by E. Lane ,but:

[0114] E Lane =MLP1(E ref (3).

[0115] E Lane For one The matrix is ​​N, where N represents the number of candidate road segment reference lines in the candidate road segment reference line set of the target object, and dim represents the encoding dimension.

[0116] In addition to inputting the reference line data of each road segment into the road encoder, the road type corresponding to each road segment reference line can also be input into the road encoder for encoding.

[0117] Step S202: Using the trajectory prediction model, perform attention calculation on the trajectory features of the target object and the trajectory features of surrounding objects to obtain the first attention feature.

[0118] Among them, the primary attention feature can reflect the degree of influence of surrounding objects on the target object.

[0119] like Figure 3 As shown, the trajectory prediction model may include a first attention module, which inputs the trajectory features of the target object and the trajectory features of the objects around the target object into the first attention module. The first attention module performs attention calculation on the trajectory features of the target object and the trajectory features of the objects around the target object, and outputs the first attention feature.

[0120] Step S203: Using the trajectory prediction model, perform attention calculation on the first attention feature and several road segment features to obtain the second attention feature.

[0121] In trajectory prediction tasks, the movement of a target object is influenced by surrounding road reference lines. The influence of road reference lines in different directions and at different distances on trajectory prediction varies. For example, in the field of autonomous driving, when there are straight and right-turn lanes ahead, if the target vehicle tends to move to the right, more attention should be paid to the right-turn lane reference line. Therefore, this embodiment, after obtaining the first attention feature, performs attention calculations on the first attention feature and several road segment features to determine the correlation between the reference lines of each road segment and the target object.

[0122] like Figure 3 As shown, the trajectory prediction model may include a second attention module. Figure 4 As shown, the first attention feature is combined with several road segment features E Lane Input the second attention module, and the second attention model applies the first attention features and several road segment features E. Lane Attention calculations are performed, and a second attention feature is output. This second attention feature reflects the correlation between the reference lines of each road segment in the candidate road segment reference line set and the target object.

[0123] In the analysis of the first attention feature and several road segment features E Lane When performing attention calculation, the first attention feature is used as the query Q, and several road segment features E are used as the query E. Lane The second attention feature is calculated using either a single-head attention mechanism or a multi-head attention mechanism, with K as the key and V as the value.

[0124] The formula for the attention mechanism is as follows:

[0125] (4)

[0126] Where Q is the query, K is the key, V is the value, and d k Let K represent the dimension. The above formula means that the similarity between Q and K is calculated by taking the dot product of Q and K. The dot product result of Q and K is scaled and passed through Softmax to obtain a probability value from 0 to 1. The probability value is then taken as a dot product with V to obtain the attention feature.

[0127] When using a single-head attention mechanism for computation, the calculation is performed directly based on the attention mechanism formula above.

[0128] To focus on features across different dimensions, a multi-head attention mechanism can be employed. Assuming there are h attention heads, for the i-th attention head among the h attention heads: first, Q, K, and V are processed through different linear projections to obtain Q.i K i and V i :

[0129] Q i =QW i Q K i =KW i K V i =VW i V (5).

[0130] Among them, W i Q W i K W i V Let be the weight matrix of the i-th attention head.

[0131] In obtaining Q i K i and V i Then, based on the above attention mechanism formula, the head is obtained. i :

[0132] head i =Attention(Q i ,K i V i (6).

[0133] After obtaining head1, head2, ..., head h Then, head1, head2, ..., head h After concatenation and linear transformation, the final attention features are obtained:

[0134] MutiHead(Q,K,V)=Concat(head1, head2,…, head h W O (7).

[0135] Among them, W O This is the weight matrix.

[0136] Step S204: Using a trajectory prediction model, predict the future trajectory of the target object based on its trajectory features and second attention features.

[0137] Trajectory prediction models can include trajectory decoders, which, after obtaining second attention features, such as... Figure 3 and Figure 4As shown, the trajectory features of the target object are fused with the second attention features (for example, the trajectory features of the target object are concatenated with the second attention features) to obtain fused features. The fused features are then input into the trajectory decoder for decoding. The trajectory decoder outputs several trajectories and the prediction probabilities corresponding to the several trajectories.

[0138] In one possible implementation, several trajectories output by the trajectory decoder can be used as the final trajectory prediction result. In another possible implementation, the trajectory with the highest prediction probability among several trajectories can be used as the final trajectory prediction result.

[0139] In one possible implementation, the trajectory decoder can be a second multilayer perceptron. If the fused features described above (i.e., the features obtained by fusing the trajectory features of the target object with the second attention features) are represented as E... f If the second multilayer perceptron is represented as MLP2, then the trajectory F output by the trajectory decoder is represented as:

[0140] F=MLP2(E f (8)

[0141] Where, F∈ M is the number of trajectories output by the trajectory decoder, P is the number of trajectory points contained in each trajectory, that is, F is M trajectories containing the coordinates of P two-dimensional trajectory points.

[0142] The prediction method based on the trajectory prediction model provided in the above embodiments was validated using the Argoverse dataset. The Argoverse dataset, released by Argo AI, Carnegie Mellon University, and others, is used to support research on 3D Tracking and Motion Forecasting for autonomous vehicles. It includes LiDAR data, RGB video data, and forward-facing binocular data. Most importantly, it is the first dataset to include high-precision map information such as road lines, road boundaries, and traffic lights. Three scenarios (straight ahead, turning, and lane changing) from the Argoverse dataset were tested, and the results were visualized. Figure 6 , Figure 7 and Figure 8 . Figure 6 , Figure 7 and Figure 8In the diagram, light-colored boxes represent drivable road areas, and dashed lines represent road reference lines. Light gray curves with dots represent observed trajectories (i.e., historical trajectories), with the dots indicating the position at the last observation time. Dark gray curves with dots represent actual trajectories, with the dots indicating the position at the final observation time. Six black curves with dots represent predicted trajectories, with the dots indicating the position at the final observation time. The diagram shows that the predicted trajectories are generally consistent with the actual trajectories in direction, and the overlap is good, indicating that the prediction results are relatively accurate.

[0143] Furthermore, the trajectory prediction method based on the trajectory prediction model provided in this application was compared with existing trajectory prediction methods, as shown in the table below:

[0144] Trajectory prediction method MinADE (M=6) MinFDE (M=6) MR Parameters Existing method 2.08 meters 4.19 meters 0.67 3434 Method of the present application 0.94 meters 1.53 meters 0.18 790,000

[0145] Among them, MinADE, MinFDE and MR are three indicators for evaluating the trajectory prediction effect. MinADE is the minimum average displacement error, MinFDE is the minimum final displacement error, MR is the false negative rate, and Parameters is the parameter scale of the trajectory prediction model.

[0146] Existing trajectory prediction methods based on trajectory prediction models input historical trajectory data of the target object, historical trajectory data of surrounding objects, and the entire road reference lines of all surrounding roads into the model for trajectory prediction. The trajectory prediction is implemented based on LSTM.

[0147] As can be seen from the table above, compared with existing trajectory prediction methods, the trajectory prediction method provided in this application has higher prediction accuracy and significantly improved prediction performance. In addition, the trajectory prediction model in this application has a small number of model parameters, and the inference time of the model with a low number of parameters is shorter, which can directly improve the timeliness of the system for autonomous driving scenarios.

[0148] This application also provides an apparatus for performing the trajectory prediction method provided in the above embodiments. Please refer to... Figure 8 , Figure 8 This is a schematic diagram of the structure of a trajectory prediction device provided in an embodiment of this application. The trajectory prediction device may include: a data acquisition module 801, a current road determination module 802, a candidate road reference line acquisition module 803, and a trajectory prediction module 804.

[0149] The data acquisition module 801 is used to acquire historical trajectory data of the target object, historical trajectory data of objects surrounding the target object, and relevant data of roads surrounding the target object.

[0150] The relevant data for the roads surrounding the target object includes the topology and reference line data of the roads surrounding the target object.

[0151] The current road determination module 802 is used to determine the current road of the target object from the roads around the target object based on the current position of the target object and the reference line data of the roads around the target object.

[0152] The candidate road reference line acquisition module 803 is used to determine the next selectable road for the target object based on the topology of the road where the target object is currently located, obtain a number of candidate roads, and determine the successor road of each candidate road to obtain a number of candidate road reference lines.

[0153] Each candidate road reference line consists of a reference line of a candidate road and a reference line of a subsequent road of that candidate road.

[0154] The trajectory prediction module 804 is used to predict the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of the objects surrounding the target object, and the reference line data of several candidate road reference lines.

[0155] In one possible implementation, when the trajectory prediction module 804 predicts the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of objects surrounding the target object, and the reference line data of several candidate road reference lines, it is specifically used for:

[0156] Remove duplicate road segment reference lines from the road segment reference lines that make up each candidate road reference line, and form a set of candidate road segment reference lines for the target object from the remaining road segment reference lines.

[0157] Based on the historical trajectory data of the target object, the historical trajectory data of the objects surrounding the target object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, predict the future trajectory of the target object.

[0158] In one possible implementation, the reference line data for the roads surrounding the target object includes the reference line coordinate data for the roads surrounding the target object.

[0159] When the candidate road reference line acquisition module 803 determines the road where the target object is currently located from the roads surrounding the target object based on the target object's current position and the reference line data of the roads surrounding the target object, it is specifically used for:

[0160] Based on the current location of the target object and the reference line coordinate data of the roads around the target object, roads that are less than a preset distance from the target object are selected from the roads around the target object.

[0161] Filter out roads that do not match the direction of movement of the target object from the selected roads;

[0162] If there is only one remaining road, then the remaining road is determined as the road where the target object is currently located;

[0163] If there are multiple remaining roads, the road closest to the target object will be determined as the road where the target object is currently located.

[0164] In one possible implementation, when the trajectory prediction module 804 predicts the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of objects surrounding the target object, and the reference line data of several candidate road reference lines, it is specifically used for:

[0165] Features are extracted from the historical trajectory data of the target object and the historical trajectory data of the objects surrounding the target object to obtain the trajectory features of the target object and the trajectory features of the objects surrounding the target object. Features are also extracted from the reference line data of each road segment reference line in the candidate road segment reference line set of the target object to obtain several road segment features.

[0166] The trajectory features of the target object, the trajectory features of the objects surrounding the target object, and several road segment features are processed into target features. The target features can reflect the degree of influence of the surrounding objects on the target object and the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object.

[0167] Predict the future trajectory of the target object based on its trajectory characteristics and target features.

[0168] In one possible implementation, when the trajectory prediction module 804 predicts the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of objects surrounding the target object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, it is specifically used for:

[0169] Using a pre-trained trajectory prediction model, the future trajectory of the target object is predicted based on the historical trajectory data of the target object, the historical trajectory data of the objects surrounding the target object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object.

[0170] The trajectory prediction model is trained using training samples labeled with real future trajectories. The training samples include historical trajectory data of the sample object, historical trajectory data of objects surrounding the sample object, and reference line data of each road segment reference line in the candidate road segment reference line set of the sample object.

[0171] In one possible implementation, the trajectory prediction model includes: a trajectory encoder, a road encoder, a first attention module, a second attention module, and a trajectory decoder.

[0172] The trajectory prediction module 804, when using a pre-trained trajectory prediction model to predict the future trajectory of the target object based on its historical trajectory data, the historical trajectory data of objects surrounding the target object, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, specifically performs the following:

[0173] Using a trajectory encoder, the historical trajectory data of the target object and the historical trajectory data of the surrounding objects are encoded to obtain the trajectory features of the target object and the trajectory features of the surrounding objects;

[0174] Using a road encoder, the reference line data of each road segment reference line in the candidate road segment reference line set of the target object is encoded to obtain several road segment features;

[0175] Using the first attention module, attention is calculated on the trajectory features of the target object and the trajectory features of surrounding objects to obtain the first attention feature, which can reflect the degree of influence of surrounding objects on the target object;

[0176] Using a second attention module, attention calculation is performed on the first attention feature and the several road segment features to obtain a second attention feature, wherein the second attention feature can reflect the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object;

[0177] Using a trajectory decoder, the future trajectory of the target object is predicted based on the second attention features and the trajectory features of the target object.

[0178] In one possible implementation, the trajectory encoder is a long short-term memory network, the road encoder is a first multilayer perceptron, and the trajectory decoder is a second multilayer perceptron.

[0179] The trajectory prediction device provided in this application has good prediction performance and relatively low computational resource consumption.

[0180] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor.

[0181] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0182] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the steps of the trajectory prediction method provided in the above embodiments.

[0183] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the trajectory prediction method provided in the above embodiments.

[0184] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the steps of the trajectory prediction method provided in the above embodiments.

[0185] This application also provides an autonomous driving vehicle, which includes the aforementioned electronic equipment.

[0186] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0189] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A trajectory prediction method, characterized in that, include: The historical trajectory data of the target object, the historical trajectory data of objects surrounding the target object, and the relevant data of roads surrounding the target object are obtained. The relevant data of the surrounding roads includes the topology and reference line data of the surrounding roads. Based on the current location of the target object and the reference line data of the surrounding roads, determine the road where the target object is currently located from the surrounding roads; Based on the topology of the road where the target object is currently located, the next selectable road for the target object is determined, resulting in several candidate roads. Then, the successor road of each candidate road is determined to obtain several candidate road reference lines. Each candidate road reference line consists of a reference line of a candidate road and a reference line of a successor road of that candidate road. Based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of the several candidate road reference lines, the future trajectory of the target object is predicted.

2. The trajectory prediction method according to claim 1, characterized in that, Any candidate road reference line consists of several road segment reference lines; The step of predicting the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of the several candidate road reference lines includes: Remove duplicate road segment reference lines from the road segment reference lines that make up each candidate road reference line, and form a set of candidate road segment reference lines for the target object from the remaining road segment reference lines. Based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object, the future trajectory of the target object is predicted.

3. The trajectory prediction method according to claim 1, characterized in that, The reference line data for the surrounding roads includes the reference line coordinate data for the surrounding roads; Determining the current road location of the target object from the surrounding roads based on the target object's current location and reference line data of the surrounding roads includes: Based on the current location of the target object and the reference line coordinate data of the surrounding roads, roads that are less than a preset distance from the target object are selected from the surrounding roads. Filter out roads from the selected roads that do not match the movement direction of the target object; If there is only one remaining road, then the remaining road is determined as the current road where the target object is located; If there are multiple remaining roads, the road closest to the target object will be determined as the road where the target object is currently located.

4. The trajectory prediction method according to claim 1, characterized in that, The step of predicting the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of the several candidate road reference lines includes: Features are extracted from the historical trajectory data of the target object and the historical trajectory data of the surrounding objects to obtain the trajectory features of the target object and the trajectory features of the surrounding objects. Features are also extracted from the reference line data of each road segment reference line in the candidate road segment reference line set of the target object to obtain several road segment features. The trajectory features of the target object, the trajectory features of the surrounding objects, and the features of the road segments are processed into target features, wherein the target features can reflect the degree of influence of the surrounding objects on the target object and the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object; Based on the trajectory features of the target object and the target features, predict the future trajectory of the target object.

5. The trajectory prediction method according to claim 2, characterized in that, The step of predicting the future trajectory of the target object based on its historical trajectory data, the historical trajectory data of surrounding objects, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object includes: Using a pre-trained trajectory prediction model, the future trajectory of the target object is predicted based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object. The trajectory prediction model is trained using training samples labeled with real future trajectories. The training samples include historical trajectory data of the sample object, historical trajectory data of objects surrounding the sample object, and reference line data of each road segment reference line in the candidate road segment reference line set of the sample object.

6. The trajectory prediction method according to claim 5, characterized in that, The trajectory prediction model includes: a trajectory encoder, a road encoder, a first attention module, a second attention module, and a trajectory decoder; The method of using a pre-trained trajectory prediction model to predict the future trajectory of the target object based on its historical trajectory data, the historical trajectory data of surrounding objects, and the reference line data of each road segment reference line in the candidate road segment reference line set of the target object includes: The trajectory encoder is used to encode the historical trajectory data of the target object and the historical trajectory data of the surrounding objects to obtain the trajectory features of the target object and the trajectory features of the surrounding objects. Using the road encoder, the reference line data of each road segment reference line in the candidate road segment reference line set of the target object is encoded to obtain several road segment features; Using the first attention module, attention is calculated on the trajectory features of the target object and the trajectory features of the surrounding objects to obtain a first attention feature, wherein the first attention feature can reflect the degree of influence of the surrounding objects on the target object; Using the second attention module, attention calculation is performed on the first attention feature and the several road segment features to obtain the second attention feature, wherein the second attention feature can reflect the correlation between each road segment reference line in the candidate road segment reference line set of the target object and the target object; Using the trajectory decoder, the future trajectory of the target object is predicted based on the second attention feature and the trajectory feature of the target object.

7. The trajectory prediction method according to claim 6, characterized in that, The trajectory encoder is a long short-term memory network, the road encoder is a first multilayer perceptron, and the trajectory decoder is a second multilayer perceptron.

8. A trajectory prediction device, characterized in that, include: The module includes a data acquisition module, a current road determination module, a candidate road reference line acquisition module, and a trajectory prediction module. The data acquisition module is used to acquire historical trajectory data of the target object, historical trajectory data of objects surrounding the target object, and relevant data of roads surrounding the target object. The relevant data of the surrounding roads includes the topology and reference line data of the surrounding roads. The current road determination module is used to determine the current road of the target object from the surrounding roads based on the current position of the target object and the reference line data of the surrounding roads. The candidate road reference line acquisition module is used to determine the next selectable road for the target object based on the topology of the road where the target object is currently located, to obtain a number of candidate roads, and to determine the successor road of each candidate road, so as to obtain a number of candidate road reference lines. Each candidate road reference line consists of a reference line of a candidate road and a reference line of a successor road of that candidate road. The trajectory prediction module is used to predict the future trajectory of the target object based on the historical trajectory data of the target object, the historical trajectory data of the surrounding objects, and the reference line data of the several candidate road reference lines.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the steps of the trajectory prediction method as described in any one of claims 1 to 7.

10. An autonomous vehicle, characterized in that, Including the electronic device as described in claim 9.

Images