Trajectory prediction method and apparatus for traffic intersection scenario, and terminal and storage medium

Abstract

A trajectory prediction method and apparatus for a traffic intersection scenario, and a terminal and a storage medium. The method comprises: acquiring a road scenario, statuses of traffic signal lights, and historical motion trajectories (S100); on the basis of the road scenario, constructing several directed graphs, wherein the directed graphs are used for simulating an interaction relationship between moving entities in the road scenario (S200); on the basis of the historical motion trajectories and the directed graphs, obtaining spatiotemporal features of the moving entities (S300); on the basis of the statuses of the traffic signal lights and the historical motion trajectories, determining several candidate moving action probabilities of a vehicle (S400); and on the basis of the spatiotemporal features and the candidate moving action probabilities, determining a motion trajectory to be followed by the vehicle (S500). Thus, significantly different moving patterns can be distinguished and predicted more accurately, thereby effectively solving the problem in the prior art of it being difficult to achieve high accuracy in predicting the behaviors of moving entities within traffic intersections.

Description

Trajectory prediction method and device in intersection scene, terminal and storage medium TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, in particular to a trajectory prediction method and device in intersection scene, terminal and storage medium. BACKGROUND

[0002] As one of the most important and common scenes in the traffic system, the intersection carries extremely frequent vehicle and pedestrian flow. In this complex environment, numerous traffic accidents occur frequently. Therefore, accurately predicting the future trajectory of the moving subject in the intersection scene plays a crucial role in preventing accidents and can significantly reduce the probability of traffic accidents. However, compared with other road scenes, the complexity of the intersection is more prominent. The intersection scene not only involves various traffic participants such as pedestrians, bicycles, electric vehicles and motor vehicles, but also contains various traffic rules and signal control elements. This multi-element and dynamic detection environment makes it difficult for the prior art to achieve high accuracy in predicting the behavior of the moving subject in the intersection.

[0003] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0004] The technical problem solved by the present application is to provide a trajectory prediction method and device in intersection scene, terminal and storage medium to solve the problem that the prior art is difficult to achieve high accuracy in predicting the behavior of the moving subject in the intersection.

[0005] The technical solution adopted by the present application to solve the problem is as follows:

[0006] In a first aspect, the present application provides a trajectory prediction method in intersection scene, wherein the method comprises:

[0007] obtaining a road scene, a traffic signal state and a historical motion trajectory;

[0008] constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between the moving subjects in the road scene;

[0009] obtaining the space-time features of each moving subject according to the historical motion trajectory and each directed graph;

[0010] determining a plurality of candidate moving action probabilities of the vehicle according to the traffic signal state and the historical motion trajectory;

[0011] determining the to-be-moved trajectory of the vehicle according to each space-time feature and each candidate moving action probability.

[0012] In an implementation method, the constructing a plurality of directed graphs according to the road scene comprises:

[0013] Obtaining a motion influence factor corresponding to each motion body, wherein the motion influence factor comprises one or more of a field of view, a speed, a motion direction, and a lane rule;

[0014] Constructing a directed graph corresponding to each motion influence factor according to the road scene and the motion influence factor, respectively.

[0015] In an implementation method, the constructing a directed graph corresponding to each motion influence factor according to the road scene and the motion influence factor, respectively, comprises:

[0016] Determining an interaction relationship between each motion body corresponding to each motion influence factor based on the motion influence factor and the road scene, respectively, wherein the interaction relationship comprises one or more of an equal interaction relationship, an unequal interaction relationship, and no interaction relationship;

[0017] Constructing a directed graph corresponding to each motion influence factor according to each interaction relationship.

[0018] In an implementation method, the obtaining a spatiotemporal feature of each motion body according to the historical motion trajectory and each directed graph comprises:

[0019] Using a spatiotemporal graph convolution network to extract a temporal feature according to the historical trajectory feature and a spatial feature according to each directed graph;

[0020] Determining the spatiotemporal feature of each motion body according to the temporal feature and the spatial feature.

[0021] In an implementation method, the determining a plurality of candidate moving action probabilities of a vehicle according to the traffic signal light state and the historical motion trajectory comprises:

[0022] Determining a plurality of candidate actions according to the traffic signal light state;

[0023] Determining a plurality of candidate moving actions according to the historical motion trajectory;

[0024] Using a gated recurrent unit algorithm and a multilayer perceptron to calculate the candidate moving action probability corresponding to each candidate moving action according to each candidate action and each candidate moving action.

[0025] In an implementation method, the determining a to-be-moved trajectory of a vehicle according to each spatiotemporal feature and each candidate moving action probability comprises:

[0026] obtain a real action code corresponding to the vehicle, and determine a first loss function according to the candidate moving action probability and the real action code by using a minimum cross-entropy function;

[0027] determine a second loss function according to the spatio-temporal feature and the candidate moving action probability by using a multi-distribution position estimation;

[0028] determine the moving trajectory of the vehicle by minimizing the sum of the first loss function and the second loss function.

[0029] In an implementation method, the multi-distribution position estimation is a t-distribution position estimation, and the determination of the second loss function according to the spatio-temporal feature and the candidate moving action probability by using the multi-distribution position estimation includes:

[0030] determine a plurality of groups of t-distribution parameters according to the spatio-temporal feature and the candidate moving action probability;

[0031] construct a t-distribution corresponding to each group of t-distribution parameters based on each group of t-distribution parameters;

[0032] fit each t-distribution by using a maximum likelihood estimation, and determine the second loss function.

[0033] In a second aspect, an embodiment of the present application further provides a trajectory prediction device in a traffic intersection scenario, wherein the trajectory prediction device in the traffic intersection scenario includes:

[0034] a data acquisition module configured to acquire a road scene, a traffic signal state, and a historical motion trajectory;

[0035] a directed graph construction module configured to construct a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate an interaction relationship between motion subjects in the road scene;

[0036] a feature extraction module configured to obtain spatio-temporal features of the motion subjects according to the historical motion trajectory and the directed graphs;

[0037] a probability calculation module configured to determine a plurality of candidate moving action probabilities of a vehicle according to the traffic signal state and the historical motion trajectory;

[0038] a trajectory determination module configured to determine a moving trajectory of the vehicle according to the spatio-temporal features and the candidate moving action probabilities.

[0039] In a third aspect, an embodiment of the present application further provides a terminal, which includes a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the trajectory prediction method in a traffic intersection scenario as described above; and the processor is configured to execute the programs.

[0040] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the trajectory prediction method in the traffic intersection scenario as described above.

[0041] The present application has the following beneficial effects: The embodiments of the present application obtain a road scene, a traffic signal state and a historical motion trajectory, construct a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between the motion subjects in the road scene, obtain the space-time features of the motion subjects according to the historical motion trajectory and the directed graphs, determine a plurality of candidate moving action probabilities of the vehicle according to the traffic signal state and the historical motion trajectory, and determine the moving trajectory of the vehicle to be moved according to the space-time features and the candidate moving action probabilities. Since the embodiments of the present application construct the directed graphs and predict the candidate actions according to the interaction relationship between the motion subjects in the road scene, the embodiments of the present application can more accurately distinguish and predict various significantly different moving modes, and effectively solve the problem that the prior art is difficult to achieve a high accuracy when predicting the behaviors of the motion subjects in the traffic intersection. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0043] FIG. 1 is a flowchart of the trajectory prediction method in the traffic intersection scenario provided by the embodiments of the present application.

[0044] FIG. 2 is a framework diagram of the trajectory prediction model in the traffic intersection scenario provided by the embodiments of the present application.

[0045] FIG. 3 is a diagram showing the difference between the undirected graph and the directed graph in representation provided by the embodiments of the present application.

[0046] FIG. 4 is a diagram showing the influence of the field of view on the motion subject provided by the embodiments of the present application.

[0047] FIG. 5 is a diagram showing the influence of the motion direction on the motion subject provided by the embodiments of the present application.

[0048] FIG. 6 is a diagram showing the influence of different lanes on the attention of the vehicle provided by the embodiments of the present application.

[0049] FIG. 7 is a diagram showing the approximate motion direction of the motion subject in different lanes provided by the embodiments of the present application.

[0050] FIG. 8 is a schematic diagram of candidate moving action generation according to an embodiment of the present application.

[0051] FIG. 9 is a schematic diagram of similarity between historical trajectory and input candidate action according to an embodiment of the present application.

[0052] FIG. 10 is a schematic diagram of predicted trajectory generated based on different distributions according to an embodiment of the present application.

[0053] FIG. 11 is a schematic diagram of prediction of future action and motion rate of a motorcycle according to an embodiment of the present application.

[0054] FIG. 12 is a schematic diagram of prediction of non-obvious turning action of a car according to an embodiment of the present application.

[0055] FIG. 13 is a schematic diagram of rate prediction in a red light turning green light scenario according to an embodiment of the present application.

[0056] FIG. 14 is a schematic diagram of effectiveness of multi-distribution position estimation in multi-modal trajectory short-term prediction results according to an embodiment of the present application.

[0057] FIG. 15 is a schematic diagram of prediction of non-obvious turning action of a bicycle and a motorcycle according to an embodiment of the present application.

[0058] FIG. 16 is a schematic diagram of effectiveness of multi-distribution position estimation in long-term prediction according to an embodiment of the present application.

[0059] FIG. 17 is a schematic diagram of influence of lane rule map on interaction according to an embodiment of the present application.

[0060] FIG. 18 is a schematic diagram of influence of candidate action prediction method on prediction results according to an embodiment of the present application.

[0061] FIG. 19 is a schematic diagram of influence of multi-distribution position estimation method on multi-modal prediction results according to an embodiment of the present application.

[0062] FIG. 20 is a schematic diagram of internal modules of a trajectory prediction device in a traffic intersection scenario according to an embodiment of the present application.

[0063] FIG. 21 is a schematic diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] The present application discloses a trajectory prediction method and device in a traffic intersection scenario, a terminal and a storage medium. In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0065] As will be understood by persons skilled in the art of the present technology, the singular forms "a," "an," "said," and "the" include plural referents unless the context clearly dictates otherwise. It should be further understood that the word "comprise" or "comprises," when used in the specification, means that the stated features, integers, steps, operations, elements, and / or components are present, but not excluding the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or intervening elements can be present. Also, as used herein, "connected" or "coupled" can include wirelessly connected or wirelessly coupled. As used herein, the word "and / or" includes all combinations of one or more of the associated listed items.

[0066] As will be understood by persons skilled in the art of the present technology, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art in the field of the present technology. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0067] To solve the problem that the prior art is difficult to achieve high accuracy in predicting the behavior of a moving body in a traffic intersection, the present application provides a trajectory prediction method in a traffic intersection scenario, which comprises the following steps: acquiring a road scene, a traffic signal light state, and a historical motion trajectory; constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between each moving body in the road scene; obtaining the space-time features of each moving body according to the historical motion trajectory and each directed graph; determining the candidate moving action probabilities of a vehicle according to the traffic signal light state and the historical motion trajectory; and determining the to-be-moved trajectory of the vehicle according to each space-time feature and each candidate moving action probability. Since the present application constructs directed graphs and predicts candidate actions for the interaction relationship between each moving body in the road scene, it can more accurately distinguish and predict various significantly different moving modes, effectively solving the problem that the prior art is difficult to achieve high accuracy in predicting the behavior of a moving body in a traffic intersection.

[0068] Exemplary method

[0069] As shown in FIG. 1, the method comprises the following steps:

[0070] In step S100, a road scene, a traffic signal light state, and a historical motion trajectory are acquired.

[0071] In brief, since the traffic intersection scenario involves multiple traffic participants, such as pedestrians, bicycles, electric vehicles, and motor vehicles, and contains various traffic rules and signal controls, etc., the embodiment first acquires the road scene, traffic signal state, and historical motion trajectory, so as to analyze the influencing factors of the traffic intersection scenario through these data, and then predict the future trajectory of the vehicle.

[0072] In step S200, a plurality of directed graphs are constructed according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between the motion subjects in the road scene.

[0073] As shown in FIG. 2, in the road scene of a non-traffic intersection, the driver mainly focuses on the motion state of the vehicle in the adjacent lane. However, at a traffic intersection, since the vehicles of multiple lanes converge and cross, the driver not only needs to closely observe the vehicles in the adjacent lane, but also needs to pay attention to those vehicles or pedestrians far away according to the complex lane rules. Therefore, in order to accurately depict the strong interaction relationship between the motion subjects far away, the embodiment constructs a plurality of directed graphs according to the road scene, and simulates the interaction relationship between the motion subjects in the road scene through the directed graphs, so that the method can pay more attention to the motion behavior of the relevant vehicles or motion subjects far away, to provide more accurate prediction.

[0074] As shown in FIG. 3, in an implementation manner, the construction of the plurality of directed graphs according to the road scene comprises:

[0075] In step S201, a motion influencing factor corresponding to each motion subject is acquired, wherein the motion influencing factor comprises one or more of the field of view, the speed, the motion direction, and the lane rule.

[0076] In step S202, a directed graph corresponding to each motion influencing factor is constructed according to the road scene and the motion influencing factor.

[0077] The distance-based undirected graph topology is used to simulate the interaction relationship between the motion subjects, which can only express the symmetric interaction behavior, and is difficult to accurately reflect the common asymmetric interaction relationship in the traffic scene. For example, when one pedestrian follows another pedestrian, the behavior of the former may be influenced by the behavior of the latter, especially when the latter stops; on the contrary, the latter is less likely to change its trajectory due to the behavior of the former. It can be seen that the undirected graph topology is not competent to describe such interaction behavior.

[0078] In view of the limitation of the distance-based undirected graph topology, the embodiment determines a plurality of influence factors affecting the vehicle trajectories by analyzing the road scene; and constructs a directed graph corresponding to each of the motion influence factors according to the road scene and the motion influence factor, specifically including: determining the interaction relationship between each motion subject corresponding to each motion influence factor based on each motion influence factor and the road scene respectively, wherein the interaction relationship includes one or more of the peer interaction relationship, the non-peer interaction relationship and the no interaction relationship; and constructing a directed graph corresponding to each motion influence factor according to each interaction relationship.

[0079] The specific process of constructing an undirected graph corresponding to each influence factor is as follows:

[0080] (1) Vision-based directed graph. As shown in FIG. 4, it is considered that the motion decision of a pedestrian or a driver is largely affected by other motion subjects within the range of vision. The behavior of the surrounding motion subjects directly determines the path selection, speed adjustment and avoidance measures of the pedestrian or the driver. Through observation and interpretation of the motion patterns and behavior intentions of the surrounding motion subjects, the pedestrian or the driver can make corresponding decisions to ensure safety. Therefore, the observation of the motion subjects within the range of vision is crucial for future decisions, while the motion subjects outside the range of vision have little influence on the future motion.

[0081] Taking FIG. 3(a) as an example, the cyclist C cannot see other moving subjects except himself (assuming the visual angle is 180 degrees straight vision angle), so in theory he will not be affected by them. However, the cyclist C is in the line of sight of the pedestrian A, the pedestrian D and the vehicle E, which means that the actions of the cyclist C can affect the future movement behavior of these motion subjects. In order to accurately describe the influence of the range of vision on the motion behavior, the embodiment adopts a directed graph topology based on the range of vision. According to the comparison between FIG. 3(a) and FIG. 3(c), the embodiment defines three types of interaction behavior in the vision graph:

[0082] 1) Peer interaction relationship, i.e. both motion subjects appear in each other's range of vision. For example, node A and node D, this peer influence is represented by a bidirectional arrow;

[0083] 2) Non-peer interaction relationship, i.e. one motion subject appears in the range of vision of another motion subject, and vice versa does not hold. For example, node A and node E, this one-way influence is represented by a one-way arrow, and the motion subject pointed by the arrow represents that it is covered by the other's range of vision;

[0084] 3) No interaction relationship, i.e. two motion subjects do not appear in each other's range of vision. For example, node A and B.

[0085] The visual field directed graph is constructed based on the above rules. It is assumed that the motion direction of the motion body evenly divides the visual angle range, and the visual field range of the pedestrian and the non-motor vehicle is 180 degrees, and the visual field range of the motor vehicle is 360 degrees. At any historical moment, the mathematical model of the visual field graph (VG) is defined as follows: VG=(U, VE), U=α∪β, VE={ve ij |i,j=1,...,N} (1)

[0086] The set α includes all nodes of the pedestrians and the non-motor vehicles in the scene, and the set β includes all nodes of the motor vehicles in the scene, U represents the set of the positions of all motion bodies in the scene, and N represents the number of the motion bodies in the scene. It is assumed that t is the current moment, and t-1 is the last moment, and the directed edge ve ij from the ith node to the jth node is defined as follows:

[0087] The set α includes all nodes of the pedestrians and the non-motor vehicles in the scene, and the set β includes all nodes of the motor vehicles in the scene, U represents the set of the positions of all motion bodies in the scene, and N represents the number of the motion bodies in the scene. It is assumed that t is the current moment, and t-1 is the last moment, and the directed edge ve i (t) = (x i (t), y i (t)) represents the real position of the ith motion body at the moment t, wherein x and y are the coordinates of the key point.

[0088] The set α includes all nodes of the pedestrians and the non-motor vehicles in the scene, and the set β includes all nodes of the motor vehicles in the scene, U represents the set of the positions of all motion bodies in the scene, and N represents the number of the motion bodies in the scene. It is assumed that t is the current moment, and t-1 is the last moment, and the directed edge ve i represents the motion direction of the ith motion body, and v ij represents the direction vector from the position of the ith motion body to the position of the jth motion body. According to formula (2), 0≤ve ij ≤1. When ve ij =0, it indicates that there is no interaction between the ith motion body and the jth motion body, and therefore the features of the jth motion body do not need to be considered. When i=j, ve ij =1, which indicates that the interaction intensity is the highest, and all features of the target motion body need to be extracted. When the jth motion body appears in the visual field of the ith motion body and the positions of the two are very close, ve ij →1 indicates that the interaction intensity is high, and the features of the jth motion body need to be extracted. On the contrary, if the interaction intensity is low, only a small amount of features needs to be extracted.

[0089] Figure 4 visually describes the interaction of the view directional graph construction. In the figure, the red dot represents the position of the target moving body A, and the black dots represent the positions of the other two moving bodies. It can be seen that moving body B is behind the target moving body A. According to formula (2), moving body B does not affect the movement of the target moving body A, i.e., there is no interaction between the two. Another moving body C is within the view of the target moving body A, so its interaction strength is inversely proportional to the distance between the two. If the target moving body A appears in the view of moving body C, then there is an equal interaction between them; otherwise, it is an unequal interaction.

[0090] (2) Directional graph based on moving direction. As shown in Figure 5, the moving direction of pedestrians or vehicles has a significant impact on the decision of surrounding moving bodies. When a pedestrian or vehicle adjusts its moving direction, the surrounding moving bodies may need to adjust their own behavior accordingly. For example, if a pedestrian suddenly changes the direction of travel, the surrounding pedestrians may take evasive action or pause. Similarly, when a vehicle changes lanes or turns, the surrounding vehicles may need to change their original path to avoid collision.

[0091] According to Figures 3(a) and 3(d), the definitions of the three types of interaction behavior of the directional graph are given:

[0092] 1) Equal interaction, i.e., the moving directions of the two moving bodies have intersection points, and both are within the view of the other. For example, nodes A and D are represented by double arrows;

[0093] 2) Unequal interaction, i.e., the moving directions of the two moving bodies have intersection points, but one of them is not within the view of the other. For example, nodes A and C are represented by single arrows, and the moving body indicated by the arrow represents that it is within the view of the other;

[0094] 3) No interaction, i.e., there is no interaction in the moving direction between the two moving bodies. For example, nodes C and E.

[0095] Based on the above rules, the directional graph is constructed. At any historical moment, the mathematical model of the directional graph (DG) is defined as follows: DG = (U, DE), U = {u i |i = 1,..., N}, DE = {de ij |i, j = 1,..., N} (4)

[0096] where d i = u i (t) - u i (t - 1), d j = u j (t) - u j(t-1),v ij = u j (t)-u i (t) (9)

[0097] where t and t-1 represent the current time and the last time respectively, d i , d j , v ij represent the moving direction of the ith moving body, the moving direction of the jth moving body, the direction vector from the ith moving body to the jth moving body respectively. When de ij →1, it means the interaction intensity is large, at this time more features of the jth moving body need to be extracted to present its influence on the ith moving body. Conversely, the interaction intensity is small.

[0098] Figure 5 shows how the direction directed graph depicts the interaction relationship by an example. In Figure 4, the intersection of the moving direction d A of the target moving body A and the moving direction d B of the moving body B is located behind the target moving body A, which is equivalent to θ1+θ2≠θ3, so the collision possibility of the two is small, therefore, they can be considered as having no interaction in direction. On the contrary, if the intersection of the moving direction of the target moving body A and the moving direction of the moving body B is located in front of the two moving bodies, which is equivalent to θ1+θ2=θ3, so the collision possibility of the two is large, that is, there is interaction in direction.

[0099] (3) Rate-based directed graph. The moving rate of an individual also has a non-negligible influence on the behavior of surrounding moving bodies. Taking the pedestrian crossing the road as an example, more attention is paid to the vehicles with faster moving speed rather than the surrounding pedestrians for safety consideration, especially the vehicles approaching quickly, therefore, the moving rate is an important influencing factor.

[0100] Two types of interaction relationships are defined in the rate graph:

[0101] 1) Unequal interaction relationship, that is, there is interaction in the moving direction between the two moving bodies, but due to the difference in moving rate, the influence intensity of the two is different. For example, nodes A and D in Figure 3(a) and Figure 3(e), the solid arrow and the dashed arrow represent different influence intensities, and the node pointed by the arrow represents the node affected;

[0102] 2) No interaction relationship, that is, there is no interaction in the moving direction between the moving bodies. For example, nodes A and B in Figure 3(a) and Figure 3(e).

[0103] Based on the above interaction relationship, the rate directed graph (Rate Graph, RG) is defined as RG=(U, RE), U={u ii = 1,..., N}, RE = {re ij i,j = 1,..., N} (10)

[0104] where t and t-1 represent the current time and the last time, respectively. If there is a conflict in the moving direction of the two moving subjects, the faster moving subject has a greater influence on the slower moving subject, and vice versa.

[0105] (4) Lane rule based directed graph. In the complex scenario of a traffic intersection, the lane rule specifies the target lane that a vehicle can possibly enter under normal circumstances. This indicates that the vehicle will pay more attention to other vehicles in the target lane and adjacent lanes. As shown in FIG. 6, vehicles Al and A2 move from left to right, while vehicles A3 and A4 move from top to bottom. If the attention of vehicle Al to other vehicles is considered, it is obvious that vehicle A2 is the key attention object of vehicle Al because it is closest to vehicle Al. Although vehicle A3 is closer to vehicle Al than vehicle A4, according to the road rule, vehicle A3 is almost impossible to affect the future path of vehicle Al, while there is a potential collision risk between vehicle Al and vehicle A4. Therefore, compared with vehicle A3, vehicle Al pays more attention to vehicle A4. Since vehicles have significantly different attention levels to moving subjects in different lanes, this embodiment explains the influence of lane rules on vehicle interaction by constructing the connection between related lanes.

[0106] For a typical four-way intersection scenario, the road network is usually carefully divided into nine areas, as shown in FIG. 7. This scenario consists of eight lanes (labeled lanes A to H) and a central intersection area (labeled area Z). Among them, lanes B, D, F and H are controlled by traffic lights, while lanes A, C, E and G are not directly controlled by traffic lights. Based on such analysis, this embodiment divides the scenario of a four-way intersection into three types of areas: outflow area (e.g., lane A), inflow area (e.g., lane B) and central intersection area Z. Accordingly, the lane rule (LR) can be accurately summarized as follows.

[0107] LR1: When a vehicle is in a certain outflow area (e.g., lane A), it can be affected by moving subjects in adjacent areas (e.g., lane B). Therefore, the attention of the vehicle to the adjacent area (lane B) and the area where it is located (lane A) should be significantly higher than other areas.

[0108] LR2: When the vehicle is in a certain merging area (e.g., lane B), it will focus on the moving behavior of the moving body in the intersection area Z, while considering the illegal lane changing behavior that may occur in the adjacent area (e.g., lane A). Therefore, these vehicles will focus on the moving behavior of the moving body in the area (lane B), the adjacent area (lane A), and the intersection area Z.

[0109] LR3: Generally, all vehicles from the merging area will pass through the intersection area Z. Therefore, when the vehicle is in the intersection area Z, it needs to closely monitor the moving behavior of the moving body in all merging areas (i.e., lanes B, D, F, and H) and the intersection area.

[0110] According to the above analysis, this embodiment proposes a directed graph topology based on lane rules, aiming to effectively depict the interaction relationship between vehicles within and between lanes. According to LR1 to LR3, this embodiment defines a lane rule graph (LG), which can accurately represent the interaction dynamics between traffic entities within and between different lanes. LG = (U, LE), U = {u i |i = 1, 2,..., N}, LE = {le ij |i, j = 1, 2,..., N}, (12)

[0111] wherein

[0112] wherein, t and t-1 represent the current time and the last time, respectively, u i (t) = (x i (t), y i (t)) represents the real position of the i-th moving body at time t, where x and y are the coordinates of the key point. U represents the set of all moving body positions in the scene, N represents the number of moving bodies in the scene, and tanh is the hyperbolic tangent function. When any lane rule LR1, LR2, LR3 is satisfied, the influence strength of the j-th vehicle on the i-th vehicle is defined based on the distance and speed as le ijAs shown in equation (13). The lane rule graph focuses on the relative position and speed relationship of the moving body. On the one hand, the lane rule graph can quickly synchronize the moving speed of the vehicle in the same lane. For example, when the traffic light changes, the two vehicles in front of and behind each other in the same lane need to coordinate to accelerate or decelerate to avoid traffic congestion or collision. Therefore, efficient interaction of moving speed is very important in the traffic intersection scene. On the other hand, the lane rule graph also makes the vehicle focus on the moving body with faster moving speed in the relevant lane, thereby avoiding possible collision. When i=j, equation (13) makes the model directly obtain the historical moving speed of the target moving body, which is beneficial to more accurately predict the future moving speed of the moving body.

[0113] By the proposed lane rule graph, the embodiment can depict the mutual influence of vehicles in the relevant lane. On the one hand, the proposed interaction method can describe the interaction behavior between distant vehicles according to the lane relationship. On the other hand, the method can effectively extract and interact the motion speed relationship between vehicles, so that the front and rear vehicles in the same lane can coordinate to accelerate or decelerate, thereby avoiding traffic congestion or collision.

[0114] Step S300, obtaining the space-time features of each moving body according to the historical motion trajectory and each directed graph.

[0115] In order to more accurately depict the influence of the interaction behavior between moving bodies on the moving body, in the embodiment, obtaining the space-time features of each moving body according to the historical motion trajectory and each directed graph includes: adopting a space-time graph convolution network, extracting a time feature according to the historical trajectory feature, and extracting a space feature according to each directed graph; and determining the space-time features of each moving body according to the time feature and the space feature.

[0116] The specific implementation manner is as follows:

[0117] 1) Space feature: considering that the graph convolution network can only update the node feature through a single adjacency matrix, the embodiment combines the adjacency matrices of the three types of directed graphs, i.e., the view directed graph, the direction directed graph and the speed directed graph, into one adjacency matrix.

[0118] wherein tanh is a hyperbolic tangent function, concat(·) represents a concatenation operation, DE, RE and VE are in equation (4), (10) and equation (1) respectively. t is the combined adjacency matrix at the tth historical moment, A t The value of each element in A

[0119] 2) Temporal Features: This embodiment uses a Temporal Convolution Network (TCN) to extract the historical trajectory features of the moving subject. V = TCN(concat(U1,U2,...,U...) a (15)

[0120] Among them, U t Let represent the set of node features at the t-th historical moment, a represent the number of historical moments, and concat(·) represent the concatenation operation.

[0121] After obtaining the lane rule directed graph, view directed graph, direction directed graph, and velocity directed graph, this embodiment uses a spatiotemporal graph convolutional network to integrate the spatial features of these graphs with the temporal series features of the moving subject. H=MLP(concat(VW1σ(norm(A)),σ(VW2LE))) (16)

[0122] Where W1 and W2 represent the learning parameters, the temporal feature V comes from formula (15), and LE (the influence intensity of each moving subject) comes from formula (12). For the spatial features of a historical moments, this embodiment uses A = concat(A1,...,A a Let ) represent A here. t The adjacency matrix obtained through formula (14) represents the spatial relationships after merging at the t-th historical moment. It is worth noting that since the adjacency matrix of a directed graph cannot be quickly normalized using methods for undirected graphs, this embodiment uses norm(·) to normalize the elements of the adjacency matrix A row by row; that is, the sum of the elements in each row is used as the denominator to normalize each element. Next, the spatiotemporal characteristics of the moving subject are obtained through formula (16).

[0123] Step S400: Determine the probability of several candidate movement actions of the vehicle based on the traffic light status and the historical movement trajectory.

[0124] The state of traffic lights can be represented as binary features (assuming that yellow and green lights have the same meaning), that is, by inputting an additional character information to represent the current state of traffic lights. This method is difficult to capture spatial or temporal information and cannot reveal the impact of traffic lights on vehicle motion, thus resulting in low accuracy of trajectory prediction.

[0125] Considering that traffic signal only specifies the legal motion behavior of vehicles, the future movement of vehicles mainly depends on their historical behavior and the intention of drivers. However, it can be determined that the legal movement behavior specified by the red traffic signal is to stop and wait or turn right, while the green traffic signal does not limit the movement of vehicles. As shown in FIG. 8, the embodiment first determines a plurality of candidate actions according to the state of the traffic signal. In the traffic intersection scenario, the motion behavior of the vehicle includes straight driving, stopping and waiting, left turning and right turning. When the traffic signal is green, all the above motion behaviors are possible future actions. When the traffic signal is red, stopping and waiting and right turning are the legal motion behaviors. Specifically, the candidate actions are embodied in the form of time sequence features.

[0126] After generating the candidate actions, a plurality of candidate movement actions are generated according to the historical motion trajectory: stop = [l a , l a ,..., l a , l a ] (17)

[0127] wherein

[0128] L stop , L straight , L left , L right represent the candidate movement actions of stopping and waiting, straight driving, left turning and right turning, respectively. In formula (17), L stop contains a same coordinate position l a , and l a is a two-dimensional column vector, and l a represents the position of the last observation time of the historical trajectory L past . In formula (18), d i = (l a -l1) / (a-i). represents the true position of the vehicle at the initial time. Formulas (21) and (22) rotate the candidate action of straight driving with as the center point, thereby obtaining the turning behavior of the vehicle, wherein θ i = θ / (a-i+1), and θ is a preset value. FIG. 7 presents the candidate actions of straight driving, left turning and right turning. In this example, the observation time of the historical trajectory is 4, i.e., a = 4.

[0129] Finally, the candidate moving action probabilities corresponding to each candidate moving action are calculated using a gated recurrent unit algorithm and a multi-layer perceptron based on each candidate action and each candidate moving action. This embodiment takes into account that the candidate action is expressed in the form of time series features, and the interaction between the historical trajectory and different candidate actions is established through the attention mechanism of the Transformer model. Specifically, different candidate actions are used as input according to different states of the traffic light. When the traffic light is red: C=softmax(MLP(act)) (23)

[0130] where C is a four-dimensional vector representing the probabilities of stop and wait, right turn, straight, and left turn actions. Softmax is the output class probability, and MLP is a multi-layer perceptron. act=concat(GRU(L past ),ATTN1(L past ,L stop ),ATTN2(L past ,L right ),padding(2n)) (24)

[0131] concat(·) represents the concatenation operation, and GRU is a gated recurrent unit algorithm. padding(2n) represents padding 2n 0 elements to represent the straight and left turn action candidate actions that do not belong to the red traffic light scenario, and n is the feature length output by ATTN i (L past ,L # ).

[0132] where i=1,...,4 corresponds to four candidate moving actions, L # ∈{L left ,L right ,L stop ,L straight}, d represents the feature length output by GRU(L past ) and GRU(L # ), W1, W2, and W3 represent learnable parameters, and L past is the historical trajectory. When the traffic light is green, the future action of the vehicle is not restricted, and padding(2n) in formula (24) is replaced by ATTN3(L past ,L straight ) and ATTN4(L past ,L left ).

[0133] FIG. 9 provides an example to explain why the proposed method can accurately predict the future action of the vehicle. In this example, the vehicle violates the indication of the red traffic signal to make a left-turning motion behavior. The proposed method in this embodiment estimates the future action of the vehicle by converting the historical trajectory into candidate motion actions, taking the stop-and-wait and right-turning actions as input candidate actions, and calculating the similarity of the historical features and the time series features of these two types of actions. It can be seen that the historical trajectory is completely different from the stop-and-wait action, and the motion trend of the historical trajectory is also not similar to the right-turning action. Therefore, the proposed method in this embodiment realizes the effective interaction of the historical trajectory and the traffic signal light by converting the state of the traffic signal light into time series features. According to the different states of the traffic signal light, the time series features of the corresponding candidate actions are input, and the similarity of these features and the historical features is calculated through the attention mechanism, so that the future action of the vehicle can be accurately predicted, which is difficult to achieve by directly inputting the binary state of the traffic signal light.

[0134] Step S500, determining the to-be-moved trajectory of the vehicle according to each spatio-temporal feature and each candidate motion action probability.

[0135] Specifically, first, the real action code corresponding to the vehicle is obtained, and a first loss function is determined according to each candidate motion action probability and the real action code by using a minimum cross-entropy function.

[0136] The prediction error is reduced by minimizing the cross-entropy loss function (CE): loss1 = CE(C, C gt ) (26)

[0137] where C is a four-dimensional vector obtained from the above formula (23). C gt represents the one-hot code of the real future action of the vehicle. For the stop-and-wait, right-turning, straight, and left-turning actions, their corresponding one-hot code distributions are [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], and [0, 0, 0, 1].

[0138] Then, a second loss function is determined according to each spatio-temporal feature and each candidate motion action probability by using multi-distribution position estimation, specifically including: determining a plurality of groups of t-distribution parameters according to each spatio-temporal feature and each candidate motion action probability; constructing a t-distribution corresponding to each group of t-distribution parameters based on each group of t-distribution parameters; fitting each t-distribution by using maximum likelihood estimation to determine the second loss function. The implementation is as follows:

[0139] In non-intersection road scenes, the movements of most vehicles usually obey the motion direction specified by the lanes. However, in intersection scenes, the motion patterns of vehicles can vary significantly under different situations, which means that the future trajectories of vehicles obey different probability distributions under different motion patterns. To this end, the embodiments adopt a probability model of t-distribution to characterize diversified motion patterns. The probability density function of t-distribution is:

[0140] where z is a random variable, v is a degree of freedom parameter, and m is a component of a two-dimensional position coordinate,

[0141] When v = 1, there is Γ(1) = 1,

[0142] At this time, the t-distribution is equal to the standard Cauchy distribution. When v = ∞,

[0143] At this time, the t-distribution is equal to the standard Gaussian distribution.

[0144] As can be seen from formula (27), by adjusting the degree of freedom parameter of the t-distribution, the characteristics of the t-distribution can be flexibly changed. On the one hand, the t-distribution can obtain samples closer to the mean, which is conducive to the model to more effectively estimate the future trajectory of a vehicle such as a stop-and-wait or a vehicle with a very small motion rate. On the other hand, when predicting the future trajectory of a pedestrian, the t-distribution can also exhibit a long-tail feature similar to the Cauchy distribution to characterize the diverse motion directions of pedestrians. Therefore, using the t-distribution can better characterize the motion behaviors with significant differences in intersection scenes.

[0145] The method proposed in the embodiments adopts maximum likelihood estimation to fit the parameters of the t-distribution, and determines a second loss function:

[0146] where b and N represent the number of future time points and the total number of moving subjects in the scene, respectively. In the stage of training the model, m x and m y represent the two-dimensional position coordinates of the moving subject at the tthfuture time point, z x , z y , v x , and v y represent the related parameters learned. In the prediction stage, the values of z x , z y , v x , and v y are determined, that is, the specific t-distribution is obtained, and at this time, m x and m y are random variables, and the multi-pattern prediction of the moving trajectory can be obtained.

[0147] To characterize the diverse and distinct motion patterns of vehicles and pedestrians more precisely, the proposed method employs multiple t-distributions to characterize the motion behaviors of vehicles and pedestrians in different situations:

[0148] H is obtained by equation (16), TCN represents the output of the spatio-temporal graph convolutional network, i.e., the spatio-temporal features. C is obtained by equation (23), which represents the estimated future motion probabilities of vehicles. i represents one of the straight, stop-and-wait, left-turn, or right-turn actions. Therefore, equation (32) includes five TCNs, four of which estimate the t-distribution parameters of the four different actions of vehicles, respectively, while TCN P The t-distribution parameters of pedestrian motion are estimated. Figure 10 provides an example to show the importance of independently characterizing the motion behaviors of different actions. In this embodiment, it is difficult to simultaneously characterize the left-turn and straight motion behaviors of vehicles using a Cauchy distribution or a Gaussian distribution alone. Although the predicted trajectories generated by the Cauchy distribution have diverse motion directions, they still cannot exhibit the left-turn motion behavior of vehicles. Unlike the position estimation method described above, the method proposed in this section can separately characterize the left-turn and straight motion behaviors of vehicles, and since the t-distribution can fit the Cauchy distribution or the Gaussian distribution, the trajectories of the straight action generated based on the t-distribution are more concentrated than the trajectories of the turning action, which is beneficial to characterize the differences between different actions.

[0149] The multi-distribution position estimation method can effectively characterize the different motion behaviors of vehicles in different situations and the motion patterns of pedestrians. On the one hand, the t-distribution has the characteristics of the Gaussian distribution and the Cauchy distribution, making the proposed method more flexible in characterizing different motion behaviors. On the other hand, according to the different possible actions of vehicles in the future, the proposed method employs multiple t-distributions to independently characterize the motion trajectories of different actions of vehicles, thereby more precisely describing the distinct motion patterns of vehicles in different situations.

[0150] Finally, the to-be-moved trajectory of the vehicle is determined by minimizing the sum of the first loss function and the second loss function.

[0151] According to the corresponding first loss function in equation (26) and the second loss function in equation (31), the training objective of the method proposed in this embodiment is to minimize the following function to determine the to-be-moved trajectory of the vehicle. Minimize(loss = loss1 + loss2) (33)

[0152] This example uses the Signalized INtersection Dataset (SIND) dataset to test the trajectory prediction accuracy of the proposed method IATP in traffic intersection environment. SIND is based on a city urban traffic intersection scenario, and a total of 11596 motion data of various vehicles are collected, including cars (such as trucks, buses), motorcycles, bicycles, tricycles, and 1652 pedestrian motion trajectories. Among them, the training set, validation set, and test set contain 7948, 2650, and 2650 trajectories, respectively. The experiment uses short-term prediction and long-term prediction settings to evaluate the prediction performance of the trajectory prediction algorithm under different prediction lengths. The observation time of the historical trajectory and the future trajectory for short-term prediction is 8 and 12, respectively, while the observation time of the historical trajectory and the future trajectory for long-term prediction is 4 and 16, respectively. Eight recently proposed trajectory prediction algorithms are selected as comparison objects in the experiment, each with its own characteristics and covering different technical paths. Specifically, they include: 1) Goal-GAN, 2) TPNMS, 3) AMEnet, 4) Causal-STGAT, 5) TPNSTA, 6) SGCN, 7) Social-STGCNN, and 8) MDGI. For detailed information about these comparison algorithms, please refer to Table 1.

[0153] Table 1 Comparison algorithms for trajectory prediction task in traffic scenario

[0154] The experiment uses the commonly used performance evaluation indicators in the trajectory prediction task, minimal Average Displacement Error (mADE) and minimal Final Displacement Error (mFDE), to verify the prediction error rate of the algorithm.

[0155] 1) mADE calculates the minimal average error of the multi-mode prediction trajectory and the future trajectory at each prediction time:

[0156] where N represents the number of motion subjects, and ade i represents the minimal average error of the K multi-mode prediction trajectories of the i-th motion subject and its future trajectory at each prediction time:

[0157] represents the true position of the i-th motion subject at the j-th prediction time (a is the number of historical time), and represents the predicted position of the k-th multi-mode prediction trajectory of the i-th motion subject at the j-th prediction time, and dist calculates the Euclidean distance between the two. b and K represent the number of prediction times and the number of multi-mode prediction trajectories, respectively. The smaller the mADE, the smaller the prediction error.

[0158] 2) mFDE calculates the minimum error between the last predicted moment of multi-mode prediction position and the true position:

[0159] where fde i represents the error between the K multi-mode prediction trajectories of the ith moving subject and its future trajectory at the last prediction moment:

[0160] The smaller the mFDE, the smaller the prediction error at the end point.

[0161] This experiment is implemented on the PyTorch platform, and the running hardware environment is as follows: AMD R9 3900X@4.3Ghz (CPU), 4x16GB DDR4 3600 (RAM), NVIDIA GeFore RTX 2070 (GPU). In the training phase, we set the batch size and epoch to 64 and 1000, respectively. When optimizing the model parameters, we use the Adam optimizer, and set the learning rate of the candidate action prediction network to 1e -6 , and the learning rates of the remaining parts are set to 1e -2 -4, respectively. In addition, we set the size z of the local environment perception window to 32. In order to realize the adaptive adjustment of the learning rate, we update the learning rate to 90% of the current learning rate every 10 epochs. In order to evaluate the prediction performance of all participating algorithms, we use mADE and mFDE as evaluation indicators. For each historical trajectory, all comparison algorithms generate 5 multi-mode prediction trajectories. It should be noted that the experimental settings of the proposed method are consistent with those of all comparison algorithms to ensure fairness.

[0162] Tables 2 and 3 show the short-term prediction results of each comparison algorithm for different types of moving subjects on the SIND dataset. From Tables 2 and 3, it can be observed that, except for bicycles, IATP exhibits the best prediction accuracy for each type of moving subject. It is worth noting that, compared to other types of vehicles, bicycles have more flexible movement directions, which makes it difficult for candidate movement action prediction methods to accurately predict their movement actions, thus making IATP not significantly advantageous in predicting the future trajectories of bicycles. However, in terms of the average results of mADE and mFDE, IATP still exhibits the best prediction performance. This experimental result verifies the effectiveness of IATP in predicting the short-term future trajectories of multiple types of moving subjects in traffic intersection scenarios.

[0163] Table 2 mADE results of short-term prediction in traffic intersection scenarios

[0164] Table 3 mFDE results of short-term prediction in traffic intersection scenarios

[0165] To more intuitively illustrate the advantage of IATP in short-term prediction, Figures 11 to 14 visually compare the prediction results of MDGI, Social-STGCNN and IATP. In these comparisons, the circles, triangles and broken lines respectively represent the historical positions, future positions and future trajectories of moving subjects. The orange, blue (or pink, yellow), green and red trajectories respectively represent the moving subjects of motorcycles, cars, bicycles and pedestrians. In Figures 11 to 13, the squares represent the predicted positions output by the algorithm, and the displayed predicted trajectory is the one closest to the future trajectory selected from the multi-modal predicted trajectory. In Figure 14, the black broken line shows the multi-modal predicted trajectory output by the algorithm.

[0166] In Figure 11, it can be clearly seen that the prediction accuracy of MDGI and Social-STGCNN is not as good as that of IATP, while the predicted trajectory of IATP almost completely coincides with the true trajectory. This is because the lane rule graph not only depicts lane-based interaction rules, but also enables the model to directly use historical moving speed as prior information, thereby more accurately predicting the interval between future positions.

[0167] Figure 12 shows a scenario in which car A and car B perform right and left turning movements, respectively. Compared with the obvious turning action of motorcycle A in Figure 11, the turning actions of the two cars in this scenario are more subtle, thus putting higher requirements on the algorithm's ability to accurately predict turning actions. Unlike Social-STGCNN and the prediction method based on Cauchy distribution, IATP can better predict the future actions of car A and car B even when the turning action is not obvious, which further proves the effectiveness of the candidate action prediction method.

[0168] In traffic intersection scenarios, the change of traffic signal lights is one of the key factors that affect the motion state of vehicles. For example, when the red traffic signal light changes to green, the vehicles that were waiting in the outflow lane begin to enter the motion state. Ideally, these vehicles in the lane should be orderly driven out one by one at this time, that is, the subsequent vehicles follow the previous vehicles. This means that the motion rates of the front and rear vehicles should be similar, so the trajectory prediction algorithm must be able to effectively synchronize the motion rates of the related vehicles to effectively avoid vehicle collisions or road traffic congestion. Figure 13 further demonstrates the effectiveness of IATP in predicting future motion rates. In order to examine the prediction results of different algorithms in more detail, the area in the dashed box in Figure 13 is enlarged for observation. In this scenario, as the traffic signal lights on the left and right sides change from red to green, car B and car E begin to accelerate to prepare for driving, and their historical rates are relatively low, which leads to the fact that Social-STGCNN and MDGI do not predict the position changes of car B and car E obviously. According to the proposed lane rule directed graph, IATP can effectively interact the historical rate information of vehicles in the same lane, thereby accurately predicting the future rates of car B and car E. At the same time, observing the prediction results of motorcycle A and motorcycle C, although their historical rate changes are more significant than those of car B and car E, the prediction effect of IATP is still obviously better than that of Social-STGCNN and MDGI. This further proves the effectiveness of the proposed lane rule graph in interacting the motion states of related vehicles, thereby accurately predicting the future trajectories of vehicles in such scenarios.

[0169] To verify the effectiveness of the proposed multi-distribution position estimation method, Figure 14 shows the comparison of multi-mode trajectory prediction results of three trajectory prediction algorithms. In this particular scenario, motion subjects A, B, and C perform straight driving, while motion subjects D, E, and F perform turning operations. By observing the prediction results in Figures 14(a) and 14(b), it can be found that both Social-STGCNN and MDGI output multi-mode prediction trajectories that exhibit similar dispersion levels, because both algorithms rely on a single probability distribution to estimate the future position of a motion subject. For straight driving motion subjects A, B, and C, it is crucial to predict multi-mode trajectories that contain different rates, while for motion subjects D, E, and F performing turning operations, different motion directions need to be predicted.

[0170] The IATP with the multi-distribution position estimation method can more accurately depict the multi-mode trajectory of the moving body in different states. For the straight-line moving bodies A, B, and C, the multi-mode trajectories predicted by the IATP algorithm are consistent in direction, and the main difference lies in the speed, while the other two algorithms exhibit multi-mode characteristics in direction, which reduces the accuracy of prediction (as can be seen from the predicted trajectory of moving body C). For the turning moving bodies D, E, and F, IATP can effectively depict the multi-mode characteristics in direction, making the predicted trajectory more diverse and better approximating the real trajectory. This result shows that the multi-distribution position estimation method proposed in this embodiment can effectively predict the future trajectory of the moving body in different states, making the prediction more flexible and accurate.

[0171] Tables 4 and 5 respectively show the long-term prediction results of all comparative algorithms on the SIND dataset for different types of moving bodies. As can be clearly seen from Tables 4 and 5, IATP exhibits the best prediction performance for moving bodies of any type. Although in short-term prediction, the prediction results of IATP for bicycle trajectories do not achieve the smallest prediction error, in long-term prediction, the prediction results of IATP exhibit certain advantages compared with MDGI, and these advantages are more significant compared with other models. In order to more intuitively show the advantages of IATP in long-term prediction, Figures 15 to 16 compare the prediction results of MDGI, Social-STGCNN, and IATP.

[0172] Table 4 mADE results of long-term prediction in traffic intersection scenarios

[0173] Table 5 mFDE results of long-term prediction in traffic intersection scenarios

[0174] Figure 15 provides a visual example to illustrate the importance of the candidate moving action prediction method in long-term prediction. In this figure, the circles, triangles and broken lines represent the historical positions, future positions and future trajectories of the moving subjects, respectively. The orange and green colors represent the motorcycle and bicycle types of moving subjects, respectively. The squares represent the predicted positions output by the algorithm, and the displayed predicted trajectories are the ones closest to the actual future trajectories from the multi-modal predicted trajectories. As can be clearly seen from Figure 15, the prediction accuracy of MDGI and Social-STGCNN is significantly lower than that of IATP. The predicted trajectory of IATP is almost consistent with the actual trajectory. For bicycle A and bicycle B, which are performing right-turn operations, MDGI and Social-STGCNN have difficulty in effectively describing their right-turn actions, while IATP can accurately depict the right-turn operations of both. When considering the future trajectory of motorcycle C, although its turning action is not as obvious as that of bicycle A and bicycle B, IATP can still accurately predict its left-turn action. This shows that even in long-term prediction, the candidate action prediction method of IATP can effectively judge the future actions of moving subjects.

[0175] To demonstrate the effectiveness of the proposed multi-distribution position estimation method in long-term prediction, Figure 16 shows the multi-modal prediction results of Social-STGCNN, MDGI and IATP in long-term prediction. The orange, blue, green and red trajectories in the figure represent moving subjects of different categories: motorcycles, cars, bicycles and pedestrians, respectively, while the black broken line represents the multi-modal predicted trajectory output by the algorithm. In this scenario, moving subjects B, C and D are merging into the same lane, while other moving subjects are moving relatively independently. For moving subjects B, C and D, the multi-modal trajectory generated by IATP can better approach the future trajectory (occluded future trajectory). For moving subjects F, E and G, IATP produces more dispersed predicted positions at later prediction times. Since the prediction error is usually larger at later times, the dispersed predicted positions generated by the t-distribution can better simulate diverse motion behaviors, thereby compensating for the performance decline caused by the increasing prediction error. For low-speed moving car A and pedestrian H, IATP uses independent distributions to describe the moving behaviors of vehicles and pedestrians, respectively, enabling IATP to more accurately and effectively model the diverse moving patterns of pedestrians.

[0176] To further verify the effectiveness of the proposed multi-distribution position estimation, lane rule graph and candidate moving action prediction method, this embodiment will show the experimental results and analysis of prediction models based on a single Gaussian distribution, a single Cauchy distribution, a single t-distribution, no lane rule graph and no candidate moving action prediction. It is particularly noteworthy that the model without candidate moving action prediction directly uses the state of the traffic light as input and predicts whether the future behavior of the vehicle violates the control of the traffic light. The quantitative results of the prediction are shown in Tables 6 to 9.

[0177] Table 6 mADE results of different prediction models for short-term prediction in intersection scenarios

[0178] Table 7 mFDE results of different prediction models for short-term prediction in intersection scenarios

[0179] IATP exhibits the best prediction performance in both short-term and long-term prediction. When comparing the short-term prediction results with a single probability distribution, IATP leads other models by at least 18% and 13% in terms of mADE and mFDE evaluation metrics. From the short-term prediction results based on single t-distribution, Cauchy distribution, and Gaussian distribution, the performance of t-distribution is better than Cauchy distribution and Gaussian distribution because t-distribution can fit the characteristics of Cauchy distribution and Gaussian distribution. T-distribution can obtain samples closer to the mean, which helps the model to more effectively estimate the future trajectory of vehicles such as stop-and-wait or slow-moving vehicles. When predicting the future trajectory of pedestrians, t-distribution also presents the long-tail characteristics similar to Cauchy distribution, which characterizes the diverse moving directions of pedestrians. Compared with the average results of short-term prediction without using lane graph method, IATP achieves 25% and 11% performance improvement in terms of mADE and mFDE evaluation metrics. If the candidate movement action prediction method is not used, the prediction performance decreases by 31% and 25%. These experimental results fully demonstrate that the proposed multi-distribution position estimation, lane graph, and candidate action prediction methods can significantly improve the short-term prediction accuracy of the model in intersection scenarios. The experimental results of long-term prediction (Tables 8 and 9) also support this conclusion.

[0180] Table 8 mADE results of different prediction models for long-term prediction in intersection scenarios

[0181] Table 9 mFDE results of different prediction models for long-term prediction in intersection scenarios

[0182] Table 10 shows the computation time of the above comparative models, Goal-GAN, AMEnet, TPNSTA, MDGI and IATP when predicting the multi-modal trajectory of a single moving agent. From the comparison of different distribution models, it can be seen that the computation cost of Gaussian distribution, Cauchy distribution and t-distribution is the same, and IATP only needs an additional 0.0007 seconds of computation time when generating the trajectory, which is due to the multi-distribution position estimation method using 5 independent TCN modules to describe multiple distributions. In contrast, the methods without using lane rule graph and without using candidate action prediction make the computation time of IATP increase by 0.0003 seconds and 0.0008 seconds, respectively. Although the proposed method increases the prediction time, the increase is still within an acceptable range. Compared with AMEnet, Goal-GAN and TPNSTA, the prediction speed of IATP has a clear advantage, which is at least 20 times faster than these methods. This is mainly because these methods rely on LSTM model to generate the predicted trajectory step by step, and use more complex multi-modal trajectory generation technology. Compared with MDGI based on multi-class directed graph interaction, IATP only increases the computation time by 0.0016 seconds when predicting the trajectory, which is the additional computation cost brought by the lane rule graph, the candidate action prediction method and the multi-distribution position estimation method. From the trajectory prediction results in Tables 2 to 5, it can be seen that the prediction performance of IATP is significantly better than other comparison algorithms, and at the same time, the results in Table 10 show that IATP increases the computation time while maintaining high-speed prediction, and the increase is completely within an acceptable range.

[0183] Table 10 prediction speed comparison

[0184] Figure 17 shows the comparison of the prediction results of the model without using the lane rule graph and the model using the lane rule graph. In this example, for moving agent A, it can be observed that there is no significant difference in prediction accuracy between the two models. However, when predicting the future trajectory of moving agent B, the model without using the lane rule graph only interacts the moving speed of the two moving agents through the speed graph, and this method is difficult to effectively interact the moving speed of the two moving agents, resulting in the predicted trajectory of moving agent B gradually deviating from the true trajectory. In contrast, the model using the lane rule graph can effectively solve the above problem. By interacting the moving speed of the two moving agents, the model can accurately perceive the difference in moving speed between the two moving agents, thereby efficiently synchronizing their moving speed and achieving accurate prediction of the future trajectory. This result shows that the use of lane rule graph significantly improves the prediction accuracy of the model in complex traffic scenarios.

[0185] FIG. 18 shows the difference between the model without the candidate action prediction method and the IATP in the prediction results. From the prediction results of the three moving subjects A, B and C, it can be seen that the complete model IATP integrated with the candidate action prediction method is significantly better than the model without the method.

[0186] FIG. 19 provides a visual example showing the advantage of the multi-distribution position estimation method in the traffic intersection scene. In this example, moving subject A and moving subject B are pedestrians, and the remaining moving subjects are vehicles. In FIG. 19(a), the multi-modal prediction trajectories of all moving subjects show a certain diversity in the non-motion direction. However, when the vehicles in the scene mainly move straight, too much diversity in the prediction in the non-motion direction can actually reduce the accuracy of the prediction. In contrast, the model using the multi-distribution position estimation method in FIG. 19(b) effectively avoids generating too dispersed prediction positions in the non-motion direction when predicting the future trajectories of the vehicles, thereby ensuring the accuracy of the prediction of the straight motion of the vehicles. For pedestrians A and B, it is difficult for the model using a single distribution to accurately depict the diverse movement patterns of the pedestrians, because the same distribution is also used to describe the multi-modal motion characteristics of the vehicles, which results in the characteristics of the multi-modal behavior of the pedestrians not being significant enough. The multi-distribution position estimation method proposed in this embodiment can independently depict the significantly different motion patterns of the pedestrians and the vehicles through multiple probability distributions, thereby ensuring that the model can generate diverse prediction trajectories of the pedestrians.

[0187] Based on the above embodiments, the application also provides a trajectory prediction device in a traffic intersection scene, as shown in FIG. 20, the device comprises:

[0188] a data acquisition module 01 for acquiring a road scene, a traffic signal lamp state and a historical motion trajectory;

[0189] a directed graph construction module 02 for constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between the moving subjects in the road scene;

[0190] a feature extraction module 03 for obtaining the spatiotemporal features of the moving subjects according to the historical motion trajectory and the directed graphs;

[0191] a probability calculation module 04 for determining a plurality of candidate movement action probabilities of the vehicles according to the traffic signal lamp state and the historical motion trajectory;

[0192] a trajectory determination module 05 for determining the to-be-moved trajectory of the vehicles according to the spatiotemporal features and the candidate movement action probabilities.

[0193] Based on the above embodiments, the application further provides a terminal, and a principle block diagram of the terminal can be shown in FIG. 21. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the trajectory prediction method in the traffic intersection scenario. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0194] Those skilled in the art can understand that the principle block diagram shown in FIG. 21 is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0195] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, which includes instructions for performing the trajectory prediction method in the traffic intersection scenario.

[0196] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0197] To sum up, the application discloses a trajectory prediction method and device in a traffic intersection scene, a terminal and a storage medium. The method obtains a road scene, a traffic signal state and a historical motion trajectory. A plurality of directed graphs are constructed according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between each motion subject in the road scene. The spatiotemporal features of each motion subject are obtained according to the historical motion trajectory and each directed graph. The probabilities of a plurality of candidate moving actions of a vehicle are determined according to the traffic signal state and the historical motion trajectory. The moving trajectory to be moved of the vehicle is determined according to each spatiotemporal feature and each candidate moving action probability. Since the directed graph of the interaction relationship between each motion subject in the road scene and the candidate action prediction are constructed in the application, various significantly different moving modes can be more accurately distinguished and predicted, and the problem that the behavior of the motion subject in the traffic intersection cannot be predicted with high accuracy in the prior art is effectively solved.

[0198] It should be understood that the application is not limited to the above examples, and can be improved or changed according to the above description for those skilled in the art, and all these improvements and changes shall belong to the protection scope of the appended claims of the application.

Claims

1. A trajectory prediction method in a traffic intersection scene, characterized in that, The method comprises: acquiring a road scene, a traffic signal state, and a historical motion trajectory; constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate interaction relationships between motion subjects in the road scene; obtaining spatiotemporal features of the motion subjects according to the historical motion trajectory and the directed graphs; determining a plurality of candidate moving action probabilities of a vehicle according to the traffic signal state and the historical motion trajectory; determining a to-be-moved trajectory of the vehicle according to the spatiotemporal features and the candidate moving action probabilities.

2. The trajectory prediction method for traffic intersection scenarios according to claim 1, characterized in that, The method comprises: acquiring a plurality of motion influence factors corresponding to the motion subjects, wherein the motion influence factors comprise one or more of a field of view, a speed, a motion direction, and a lane rule; constructing a directed graph corresponding to each of the motion influence factors according to the road scene and the motion influence factor.

3. The trajectory prediction method for traffic intersection scenarios according to claim 2, characterized in that, The method comprises: determining interaction relationships between the motion subjects corresponding to each of the motion influence factors based on the motion influence factor and the road scene, wherein the interaction relationships comprise one or more of an equal interaction relationship, an unequal interaction relationship, and no interaction relationship; constructing a directed graph corresponding to each of the motion influence factors according to the interaction relationships.

4. The trajectory prediction method for traffic intersection scenarios according to claim 1, characterized in that, The method comprises: adopting a spatiotemporal graph convolution network to extract a temporal feature according to a historical trajectory feature and to extract a spatial feature according to the directed graphs; determining the spatiotemporal features of the motion subjects according to the temporal feature and the spatial feature.

5. The trajectory prediction method for traffic intersection scenarios according to claim 1, characterized in that, The method comprises: determining a plurality of candidate actions according to the traffic signal state; determining a plurality of candidate moving actions according to the historical motion trajectory; adopting a gated recurrent unit algorithm and a multilayer perceptron to calculate the candidate moving action probabilities corresponding to the candidate moving actions according to the candidate actions and the candidate moving actions.

6. The trajectory prediction method for traffic intersection scenarios according to claim 1, characterized in that, The method comprises: acquiring a real action encoding corresponding to the vehicle, adopting a minimum cross-entropy function to determine a first loss function according to the candidate moving action probabilities and the real action encoding; adopting a multi-distribution position estimation to determine a second loss function according to the spatiotemporal features and the candidate moving action probabilities; determining the to-be-moved trajectory of the vehicle by minimizing a sum of the first loss function and the second loss function.

7. The trajectory prediction method for traffic intersection scenarios according to claim 6, characterized in that, The multi-distribution position estimation is a t-distribution position estimation, and the method comprises: determining a plurality of groups of t-distribution parameters according to the spatiotemporal features and the candidate moving action probabilities; constructing a t-distribution corresponding to each group of t-distribution parameters based on the group of t-distribution parameters. The second loss function is determined by fitting each t-distribution using maximum likelihood estimation.

8. A trajectory prediction apparatus in a traffic intersection scenario, characterized in that, The device comprises: a data acquisition module configured to acquire a road scene, traffic signal states, and historical motion trajectories; a directed graph construction module configured to construct a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate interaction relationships between motion subjects in the road scene; a feature extraction module configured to obtain spatiotemporal features of the motion subjects according to the historical motion trajectories and the directed graphs; a probability calculation module configured to determine a plurality of candidate moving action probabilities of a vehicle according to the traffic signal states and the historical motion trajectories; a trajectory determination module configured to determine a to-be-moved trajectory of the vehicle according to the spatiotemporal features and the candidate moving action probabilities.

9. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the trajectory prediction method at a traffic intersection scene according to any one of claims 1-7; and the processors are configured to execute the programs.

10. A computer readable storage medium having stored thereon a plurality of instructions, the plurality of instructions comprising: The instructions are adapted to be loaded and executed by the processor to implement the steps of the trajectory prediction method at a traffic intersection scene according to any one of claims 1-7.

Images