Trajectory prediction method and device in traffic intersection scene, terminal and storage medium

Abstract

The invention discloses a trajectory prediction method and device in a traffic intersection scene, a terminal and a storage medium. The method comprises the steps of obtaining a road scene, a traffic signal lamp state and a historical motion trajectory; according to the road scene, a plurality of directed graphs are constructed, and the directed graphs are used for simulating interaction relations among all motion subjects in the road scene; obtaining spatio-temporal characteristics of the motion subjects according to the historical motion trails and the digraphs; determining a plurality of candidate movement action probabilities of the vehicle according to the traffic signal lamp state and the historical movement track; and determining a to-be-moved track of the vehicle according to each spatio-temporal feature and each candidate moving action probability. Due to the fact that the directed graph is constructed for the interaction relation among all the motion subjects in the road scene, candidate motion prediction and the like are carried out, various remarkably different moving modes can be distinguished and predicted more accurately, and the problem that high accuracy is difficult to achieve when behaviors of the motion subjects in a traffic intersection are predicted in the prior art is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a trajectory prediction method, device, terminal and storage medium in a traffic intersection scenario. Background Art

[0002] Traffic intersections are one of the most important and common scenes in the traffic system, carrying extremely frequent flows of vehicles and pedestrians. In this complex environment, many traffic accidents occur frequently. Therefore, accurately predicting the future trajectory of moving subjects in traffic intersection scenarios plays a vital role in preventing accidents and can significantly reduce the probability of traffic accidents. However, compared with other road scenes, traffic intersections are more complex. Traffic intersection scenes not only involve a variety of traffic participants, such as pedestrians, bicycles, electric vehicles, and motor vehicles, but also include various traffic rules and signal control elements. This diverse and dynamic detection environment makes it difficult for existing technologies to achieve a high accuracy rate when predicting the behavior of moving subjects in traffic intersections.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a trajectory prediction method, device, terminal and storage medium in a traffic intersection scenario are provided, aiming to solve the problem that the prior art is difficult to achieve a high accuracy when predicting the behavior of moving subjects in traffic intersections.

[0005] The technical solution adopted by the present invention to solve the problem is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a trajectory prediction method in a traffic intersection scenario, wherein the method comprises:

[0007] Obtain road scenes, traffic light status, and historical movement trajectories;

[0008] Constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene;

[0009] Obtaining the spatiotemporal characteristics of each moving subject according to the historical movement trajectory and each directed graph;

[0010] Determining several candidate moving action probabilities of the vehicle according to the traffic light state and the historical motion trajectory;

[0011] The vehicle's trajectory to be moved is determined according to each of the spatiotemporal features and each of the candidate movement action probabilities.

[0012] In one implementation method, constructing a plurality of directed graphs according to the road scene includes:

[0013] Obtaining a motion influence factor corresponding to each moving subject, wherein the motion influence factor includes one or more of a field of view, a speed, a motion direction, and a lane rule;

[0014] A directed graph corresponding to each motion impact factor is constructed according to the road scene and the motion impact factor.

[0015] In an implementation method, constructing directed graphs corresponding to the motion impact factors according to the road scene and the motion impact factors respectively includes:

[0016] Determining the interaction relationship between each moving subject corresponding to each moving influence factor based on each moving influence factor and the road scene, wherein the interaction relationship includes one or more of a peer-to-peer interaction relationship, an unequal interaction relationship, and a non-interaction relationship;

[0017] Directed graphs corresponding to the sports influencing factors are constructed according to the interaction relationships.

[0018] In an implementation method, obtaining the spatiotemporal characteristics of each moving subject according to the historical movement trajectory and each directed graph includes:

[0019] Using a spatiotemporal graph convolutional network, extracting temporal features based on historical trajectory features, and extracting spatial features based on each of the directed graphs;

[0020] The temporal and spatial characteristics of each moving subject are determined according to the temporal characteristics and the spatial characteristics.

[0021] In one implementation method, the determining of several candidate movement probabilities of the vehicle according to the traffic light state and the historical movement trajectory includes:

[0022] determining a number of candidate actions according to the state of the traffic light;

[0023] Determine a number of candidate movement actions according to the historical movement trajectory;

[0024] According to each of the candidate actions and each of the candidate moving actions, a gated recurrent unit algorithm and a multi-layer perceptron are used to calculate the candidate moving action probability corresponding to each of the candidate moving actions.

[0025] In an implementation method, determining the trajectory of the vehicle to be moved according to each of the spatiotemporal features and each of the candidate movement action probabilities includes:

[0026] Obtaining a real action code corresponding to the vehicle, and determining a first loss function according to each of the candidate moving action probabilities and the real action code by minimizing a cross entropy function;

[0027] Determine a second loss function according to each of the spatiotemporal features and each of the candidate movement action probabilities using multi-distribution position estimation;

[0028] The to-be-moved trajectory of the vehicle is determined by minimizing the sum of the first loss function and the second loss function.

[0029] In an implementation method, the multi-distribution position estimation is a t-distribution position estimation, and the multi-distribution position estimation is used to determine the second loss function according to each of the spatiotemporal features and each of the candidate movement action probabilities, including:

[0030] Determine a plurality of groups of t-distribution parameters according to each of the spatiotemporal features and each of the candidate movement action probabilities;

[0031] Based on each group of t distribution parameters, construct the t distribution corresponding to each group of t distribution parameters;

[0032] The maximum likelihood estimation is used to fit each t distribution and determine the second loss function.

[0033] In a second aspect, an embodiment of the present invention further provides a trajectory prediction device in a traffic intersection scenario, wherein the trajectory prediction device in the traffic intersection scenario includes:

[0034] Data acquisition module, used to obtain road scenes, traffic light status and historical movement trajectories;

[0035] A directed graph construction module, used to construct a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene;

[0036] A feature extraction module, used to obtain the spatiotemporal features of each moving subject according to the historical motion trajectory and each directed graph;

[0037] A probability calculation module, used to determine the probabilities of several candidate movement actions of the vehicle according to the state of the traffic light and the historical movement trajectory;

[0038] The trajectory determination module is used to determine the vehicle's trajectory to be moved according to each of the spatiotemporal features and each of the candidate movement action probabilities.

[0039] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing a trajectory prediction method in a traffic intersection scenario as described in any one of the above; and the processor is used to execute the program.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement any of the above-mentioned trajectory prediction methods in a traffic intersection scenario.

[0041] Beneficial effects of the present invention: The embodiments of the present invention obtain road scenes, traffic light states, and historical motion trajectories; construct several directed graphs based on the road scenes, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scenes; obtain the spatiotemporal characteristics of each moving subject based on the historical motion trajectories and the directed graphs; determine several candidate moving action probabilities of the vehicle based on the traffic light states and the historical motion trajectories; and determine the vehicle's trajectory to be moved based on the spatiotemporal characteristics and the probability of each candidate moving action. Since the present invention constructs directed graphs, candidate action predictions, etc. for the interaction relationship between various moving subjects in the road scenes, it is possible to more accurately distinguish and predict various significantly different movement modes, effectively solving the problem that the prior art is difficult to achieve a high accuracy rate when predicting the behavior of moving subjects in traffic intersections. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flow chart of a trajectory prediction method in a traffic intersection scenario provided by an embodiment of the present invention.

[0044] Figure 2 It is a framework diagram of a trajectory prediction model in a traffic intersection scenario provided by an embodiment of the present invention.

[0045] Figure 3 This is a diagram showing the difference in representation between an undirected graph and a directed graph provided by an embodiment of the present invention.

[0046] Figure 4 It is a schematic diagram of the influence of the field of view on a moving subject provided by an embodiment of the present invention.

[0047] Figure 5 It is a schematic diagram of the influence of the motion direction on the moving subject provided by an embodiment of the present invention.

[0048] Figure 6 It is a schematic diagram of the influence of different lanes on vehicle attention provided by an embodiment of the present invention.

[0049] Figure 7 It is a schematic diagram of the approximate movement direction of a moving subject in different lanes provided by an embodiment of the present invention.

[0050] Figure 8 This is a schematic diagram of generating candidate movement actions provided by an embodiment of the present invention.

[0051] Fig. 9 It is a schematic diagram of the similarity between the candidate actions considering the historical trajectory and the input provided by the embodiment of the present invention.

[0052] Fig.10 It is a schematic diagram of predicted trajectories generated based on different distributions provided in an embodiment of the present invention.

[0053] Fig.11 It is a schematic diagram of predicting the future action and movement speed of a motorcycle provided by an embodiment of the present invention.

[0054] Fig.12 It is a schematic diagram of predicting an unnoticeable turning action of a car provided by an embodiment of the present invention.

[0055] Fig.13 It is a schematic diagram of rate prediction in a red light to green light scenario provided by an embodiment of the present invention.

[0056] Fig.14 It is a schematic diagram of the short-term prediction results of multi-distribution position estimation in multi-mode trajectories provided by an embodiment of the present invention.

[0057] Fig.15 It is a schematic diagram of predicting the subtle turning action of a bicycle and a motorcycle provided by an embodiment of the present invention.

[0058] Fig.16 It is a schematic diagram of the effectiveness of multi-distribution position estimation in long-term prediction provided by an embodiment of the present invention.

[0059] Fig.17 It is a schematic diagram of the influence of the lane regulation diagram on the interaction provided by an embodiment of the present invention.

[0060] Fig.18 It is a schematic diagram of the influence of the candidate action prediction method provided by an embodiment of the present invention on the prediction result.

[0061] Fig.19 It is a schematic diagram of the influence of the multi-distribution position estimation method provided by an embodiment of the present invention on the multi-mode prediction results.

[0062] Fig. 20 It is a schematic diagram of the internal modules of the trajectory prediction device in the traffic intersection scenario provided by an embodiment of the present invention.

[0063] Fig.21 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention discloses a trajectory prediction method, device, terminal and storage medium in a traffic intersection scenario. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0065] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0066] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0067] In view of the problem that the prior art is difficult to achieve a high accuracy rate when predicting the behavior of a moving subject in a traffic intersection, the present invention provides a trajectory prediction method in a traffic intersection scenario, the method obtaining a road scene, a traffic light state, and a historical motion trajectory; constructing a number of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between the moving subjects in the road scene; obtaining the spatiotemporal characteristics of each moving subject according to the historical motion trajectory and each directed graph; determining a number of candidate moving action probabilities of the vehicle according to the traffic light state and the historical motion trajectory; and determining the vehicle's trajectory to be moved according to each spatiotemporal characteristic and each candidate moving action probability. Since the present invention constructs a directed graph, candidate action prediction, etc. for the interaction relationship between the moving subjects in the road scene, it is possible to more accurately distinguish and predict various significantly different movement modes, effectively solving the problem that the prior art is difficult to achieve a high accuracy rate when predicting the behavior of a moving subject in a traffic intersection.

[0068] Exemplary Methods

[0069] like Figure 1 As shown, the method includes:

[0070] Step S100: Acquire road scenes, traffic light status, and historical movement trajectories.

[0071] In simple terms, since traffic intersection scenes involve multiple traffic participants, such as pedestrians, bicycles, electric vehicles, and motor vehicles, and also include various traffic rules and signal control elements, this embodiment first obtains road scenes, traffic light status, and historical motion trajectories so that these data can be used to subsequently analyze the factors affecting the traffic intersection scene, thereby predicting the future trajectory of the vehicle.

[0072] Step S200: constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene.

[0073] like Figure 2 As shown, in a non-intersection road scene, the driver mainly pays attention to the movement state of vehicles in adjacent lanes. However, at a traffic intersection, since vehicles in multiple lanes converge and cross each other, the driver must not only pay close attention to the vehicles in the adjacent lanes, but also must pay attention to those vehicles or pedestrians that are far away according to complex lane rules. Therefore, in order to accurately depict the strong interactive relationship between such long-distance moving subjects, this embodiment constructs several directed graphs according to the road scene, and simulates the interactive relationship between the moving subjects in the road scene through the directed graphs, so that the method can pay more attention to the movement behavior of related vehicles or moving subjects in the distance to provide more accurate predictions.

[0074] like Figure 3 As shown, in one implementation, constructing a plurality of directed graphs according to the road scene includes:

[0075] Step S201: obtaining a motion influencing factor corresponding to each moving subject, wherein the motion influencing factor includes one or more of a field of view, a speed, a motion direction, and a lane rule;

[0076] Step S202: constructing directed graphs corresponding to the motion impact factors according to the road scene and the motion impact factors.

[0077] Using distance-based undirected graph topology to simulate the interaction between moving subjects can only express symmetrical interaction behaviors, and it is difficult to accurately reflect the asymmetric interaction relationships commonly seen in traffic scenes. For example, when a pedestrian follows another pedestrian, the former's behavior may be affected by the latter's behavior, especially when the latter stops; conversely, the latter is unlikely to change its trajectory due to the former's behavior. It can be seen that undirected graph topology is incapable of describing such interaction behaviors.

[0078] Taking into account the limitations of the distance-based undirected graph topology, this embodiment determines a number of influencing factors that affect the vehicle trajectory by analyzing the road scene; constructs a directed graph corresponding to each motion influence factor according to the road scene and the motion influence factor, specifically including: determining the interaction relationship between each motion subject corresponding to each motion influence factor based on each motion influence factor and the road scene, wherein the interaction relationship includes one or more of a peer-to-peer interaction relationship, an unequal interaction relationship, and a non-interaction relationship; and constructs a directed graph corresponding to each motion influence factor according to each interaction relationship.

[0079] For various impact factors, the specific process of constructing the undirected graph corresponding to each impact factor is as follows:

[0080] (1) Directed graph based on field of view. Figure 4 As shown in Figure 1, the movement decisions of pedestrians or drivers are largely affected by other moving subjects within the field of view. The behavior of surrounding moving subjects directly determines the path selection, speed adjustment and avoidance measures of pedestrians or drivers. By observing and interpreting the movement patterns and behavioral intentions of surrounding moving subjects, pedestrians or drivers can make corresponding decisions to ensure safety. Therefore, the observation of moving subjects within the field of view is crucial for future decision-making, while moving subjects outside the field of view have little impact on upcoming movements.

[0081] by Figure 3 Taking (a) as an example, it is difficult for rider C to see other moving subjects except himself (assuming that the visual angle is 180 degrees straight-ahead viewing angle), so in theory he will not be affected by them. However, rider C is in the sight of pedestrian A, pedestrian D and vehicle E, which means that rider C's actions may affect the future movement behavior of these moving subjects. In order to accurately describe the impact of the field of view on the movement behavior, this embodiment adopts a directed graph topology based on the field of view. According to Figure 3 (a) with Figure 3 (c), this embodiment defines three types of interactive behaviors in the visual field diagram:

[0082] 1) Peer-to-peer interaction, that is, both moving entities appear in each other's field of view. For example, node A and node D. This peer-to-peer influence is represented by a bidirectional arrow;

[0083] 2) Unequal interaction, that is, one moving subject appears in the field of vision of another moving subject, but not vice versa. For example, between nodes A and E, this one-way influence is represented by a one-way arrow, and the moving subject pointed by the arrow is covered by the other party's field of vision;

[0084] 3) No interaction, that is, the two moving entities are not within each other's field of view, such as nodes A and B.

[0085] Based on the above rules, a visual field graph is constructed. Assume that the moving direction of the moving subject divides its visual field angle range equally, and the visual field range of pedestrians and non-motor vehicles is 180 degrees, and the visual field range of motor vehicles is 360 degrees. At any historical moment, the mathematical model of the visual field graph (VG) is defined as follows:

[0086] VG=(U,VE),U=α∪β,VE={ve ij |i,j=1,...,N} (1)

[0087] The set α includes all the nodes of pedestrians and non-motor vehicles in the scene, the set β includes all the nodes of motor vehicles in the scene, U represents the set of positions of all moving subjects in the scene, and N represents the number of moving subjects in the scene. Assume that t is the current time and t-1 is the previous time, and define the directed edge ve from the i-th node to the j-th node ij for

[0088]

[0089] Among them, u i (t) = (x i (t),y i (t)) represents the real position of the i-th moving subject at time t, where x and y are the coordinates of the key point.

[0090]

[0091] where d i represents the direction of motion of the i-th moving subject, and v ij represents the direction vector from the position of the i-th moving subject to the position of the j-th moving subject. According to formula (2), 0≤ve ij ≤1. When ve ij = 0, which means that there is no interaction between ij, so there is no need to consider the characteristics of the jth moving subject. When i = j, ve ij=1, which means the interaction intensity is the highest and all the features of the target moving subject need to be extracted. When the jth moving subject appears in the field of view of the ith moving subject and the two are very close, there is ve ij →1 indicates that the interaction intensity is large, and it is necessary to extract some features of the jth moving subject. On the contrary, if the interaction intensity is low, only fewer features need to be extracted.

[0092] Figure 4 Intuitively describe the interaction relationship constructed by the vision directed graph. In the figure, the red dot indicates the position of the target moving subject A, and the black dot indicates the position of the other two moving subjects. It can be seen that the moving subject B is behind the target moving subject A. According to formula (2), the moving subject B does not affect the movement of the target moving subject A, which means that there is no interaction relationship between the two. Another moving subject C is within the vision of the target moving subject A, so its interaction strength is inversely proportional to the distance between the two. If the target moving subject A appears in the vision of the moving subject C, then there is an equal interaction relationship between them; otherwise, it is an unequal interaction relationship.

[0093] (2) Directed graph based on movement direction. Figure 5 The direction of motion of a pedestrian or vehicle shown has a significant impact on the decisions of surrounding moving entities. When a pedestrian or vehicle adjusts its direction of motion, the surrounding moving entities may need to adjust their behavior accordingly. For example, if a pedestrian suddenly changes direction, surrounding pedestrians may take evasive or pause actions. Similarly, when a vehicle changes lanes or turns, surrounding vehicles may also need to change their original paths to avoid collisions.

[0094] according to Figure 3 (a) with Figure 3 (d) Define three types of interaction behaviors of directed graphs:

[0095] 1) Equivalent interaction, that is, the movement directions of the two moving entities have an intersection, and both are in each other's field of vision. For example, nodes A and D are represented by double arrows;

[0096] 2) Unequal interaction, that is, there is an intersection between the movement directions of the two moving subjects, but one of them does not appear in the field of vision of the other. For example, nodes A and C are represented by a single arrow, and the moving subject pointed by the arrow appears in the field of vision of the other party;

[0097] 3) No interaction, that is, there is no interaction between the two moving entities in the direction of movement, such as nodes C and E.

[0098] Based on the above rules, a directional graph is constructed. At any historical moment, the mathematical model of the directional graph (DG) is defined as follows:

[0099] DG=(U,DE),U={u i |i=1,...,N},DE={de ij |i,j=1,...,N} (4)

[0100] in

[0101]

[0102]

[0103] d i =u i (t)-u i (t-1),d j =u j (t)-u j (t-1),v ij =u j (t)-u i (t) (9)

[0104] Where t and t-1 represent the current time and the previous time respectively, d i ,d j 、v ij They represent the motion direction of the i-th moving subject, the motion direction of the j-th moving subject, and the direction vector from the position of the i-th moving subject to the position of the j-th moving subject. ij →1, indicating that the interaction strength is large, and at this time, more features of the jth moving subject need to be extracted to show its influence on the ith moving subject. Conversely, the interaction strength is small.

[0105] Figure 5 The example shows how a directed graph can depict interactive relationships. Figure 4 The moving direction of the target moving subject A is d A The direction of motion of the moving subject B is d B The intersection of is located behind the target moving subject A, which is equivalent to θ1+θ2≠θ3, so the possibility of collision between the two is small, so it can be considered that they have no interaction in direction. On the contrary, if the intersection of the moving directions of the target moving subject A and the moving subject B is located in front of the two moving subjects, this is equivalent to θ1+θ2=θ3, so the possibility of collision between the two is high, that is, there is interaction in direction.

[0106] (3) Directed graph based on speed. The speed of an individual’s movement also has a significant impact on the behavior of surrounding moving entities. For example, when pedestrians cross the road, for safety reasons, they pay more attention to vehicles moving faster than to the movement of pedestrians around them, especially vehicles approaching quickly. Therefore, speed is an important influencing factor.

[0107] Two types of interactions exist to define the rate graph:

[0108] 1) Unequal interaction, that is, the two moving bodies interact in the direction of movement, but due to the different movement speeds, the influence intensity between the two is different. Figure 3 (a) with Figure 3 In (e), for nodes A and D, the solid arrows and dashed arrows represent different impact intensities, respectively, where the nodes pointed by the arrows represent the affected nodes;

[0109] 2) No interaction, that is, there is no interaction between the moving subjects in terms of movement direction. Figure 3 (a) with Figure 3 (e) Nodes A and B.

[0110] Based on the above interaction relationship, the rate graph (RG) is defined as

[0111] RG=(U,RE),U={u i |i=1,...,N},RE={re ij |i,j=1,...,N} (10)

[0112]

[0113] Where t and t-1 represent the current moment and the previous moment respectively. If there is a conflict in the moving directions of the two moving bodies, the faster moving body will have a greater influence on the slower moving body, otherwise, the influence will be smaller.

[0114] (4) Directed graph based on lane rules. In complex scenes at traffic intersections, lane rules specify the target lanes that vehicles may enter under normal circumstances. This means that vehicles will pay more attention to other vehicles in the target lane and adjacent lanes. Figure 6 As shown, vehicles A1 and A2 are traveling from left to right, while vehicles A3 and A4 are moving from top to bottom. If vehicle A1's attention to other vehicles is considered, it can be clearly seen that vehicle A2 is the focus of vehicle A1 because it is closest to vehicle A1. Although vehicle A3 is closer to vehicle A1 than vehicle A4, according to road rules, it is almost impossible for vehicle A3 to affect the future path of vehicle A1. On the contrary, there is a potential risk of collision between vehicle A1 and vehicle A4. Therefore, vehicle A1 pays more attention to vehicle A4 than vehicle A3. Given that vehicles have significantly different levels of attention to moving entities in different lanes, this embodiment illustrates the impact of lane rules on vehicle interactions by constructing connections between related lanes.

[0115] For a typical four-way intersection scenario, the road network is usually divided into nine areas, such as Figure 7 As shown. This scene consists of eight lanes (marked as lanes A to H) and a central intersection area (marked as area Z). Among them, lanes B, D, F and H are regulated by traffic lights, while lanes A, C, E and G are not directly controlled by traffic lights. Based on this type of analysis, this embodiment divides the scene of the four-way intersection into three types of areas: the confluence area (such as lane A), the confluence area (such as lane B) and the central intersection Z. Based on this, the lane rule (Lane Rule, LR) can be accurately summarized as follows.

[0116] LR1: When a vehicle is in a certain exit area (such as lane A), it may be affected by the moving objects in the adjacent area (such as lane B). Therefore, the vehicle should pay more attention to the adjacent area (lane B) and the area it is in (lane A) than other areas.

[0117] LR2: When the vehicle is in a certain merging area (such as lane B), it will focus on the movement of the moving subject in the intersection area Z, while taking into account the possible illegal lane change behavior of the moving subject in the adjacent area (such as lane A). Therefore, these vehicles will focus on the movement of the moving subject in the area they are in (lane B), the adjacent area (lane A), and the intersection area Z.

[0118] LR3: Usually, all vehicles coming from the merging zone will cross the merging zone Z. Therefore, when the vehicle is in the merging zone Z, it needs to pay close attention to the movement behaviors of all the merging zones (i.e., lanes B, D, F, and H) and the moving bodies in the merging zone.

[0119] Based on the above analysis, this embodiment proposes a directed graph topology based on lane rules, which aims to effectively characterize the interaction relationship between vehicles in and between lanes. Based on LR1 to LR3, this embodiment defines a lane rule graph (LG), which can accurately characterize the interaction dynamics between different lanes and between traffic entities in the lanes.

[0120] LG=(U,LE),U={u i |i=1,2,...,N},LE={le ij |i,j=1,2,...,N}, (12)

[0121] in

[0122]

[0123] Among them, t and t-1 represent the current time and the previous time respectively, u i (t) = (xi (t),y i (t)) represents the real position of the i-th moving subject at time t, where x and y are the coordinates of the key point. U represents the set of positions of all moving subjects in the scene, N represents the number of moving subjects in the scene, and tanh is the hyperbolic tangent function. When any lane rules LR1, LR2, and LR3 are established, the influence intensity of the j-th vehicle on the i-th vehicle is defined based on distance and speed. ij As shown in formula (13). The lane rule map focuses on the relationship between the relative position and speed of the moving subject. On the one hand, the lane rule map can quickly synchronize the moving speed of vehicles in the same lane. For example, when the traffic light changes, the two vehicles in front and behind in the same lane need to accelerate or decelerate in coordination to avoid traffic jams or collisions. Therefore, efficient interactive moving speed is very important in traffic intersection scenarios. On the other hand, the lane rule map also allows vehicles to focus on moving subjects with faster moving speeds in the relevant lanes, thereby avoiding possible collisions. When i=j, formula (13) allows the model to directly obtain the historical moving speed of the target moving subject, which is conducive to more accurate prediction of the future moving speed of the moving subject.

[0124] Through the proposed lane rule graph, this embodiment can depict the mutual influence of vehicles in related lanes. On the one hand, the proposed interaction method can describe the interaction behavior between long-distance vehicles based on the lane relationship. On the other hand, the method can effectively extract and interact the movement rate relationship between vehicles, so that the front and rear vehicles in the same lane can coordinate acceleration or deceleration, thereby avoiding traffic jams or collisions.

[0125] Step S300: obtaining the spatiotemporal characteristics of each moving subject according to the historical motion trajectory and each directed graph.

[0126] In order to more accurately characterize the impact of interactive behaviors between moving subjects on the mobile subject, in this embodiment, the spatiotemporal characteristics of each moving subject are obtained according to the historical motion trajectory and each directed graph, including: using a spatiotemporal graph convolutional network to extract time features according to historical trajectory features, and extracting spatial features according to each directed graph; determining the spatiotemporal characteristics of each moving subject according to the time features and the spatial features.

[0127] The specific implementation is as follows:

[0128] 1) Spatial features: Considering that the graph convolutional network can only use a single adjacency matrix when updating node features, this embodiment combines the view directed graph, direction directed graph, and rate directed graph into a spatial feature matrix. Figure 3 The adjacency matrices of the directed graph-like classes are merged into a single adjacency matrix.

[0129]

[0130] Where tanh is the hyperbolic tangent function, concat(·) represents the concatenation operation, DE, RE and VE are in formula (4), (10) and formula (1) respectively. t is the adjacency matrix after merging at the t-th historical moment, A t The value of each element in is between 0 and 1 (including 0 and 1). When the element value is 0, it means that there is no interaction between the corresponding entities; when the element value is 1, it indicates that the influence on the moving subject has reached the maximum.

[0131] 2) Temporal features: This embodiment uses a temporal convolution network (TCN) to extract the historical trajectory features of the moving subject.

[0132] V=TCN(concat(U1,U2,...,U a )) (15)

[0133] Among them, U t represents the node feature set at the t-th historical moment, a represents the number of historical moments, and concat(·) represents the concatenation operation.

[0134] After obtaining the lane rule directed graph, the field of view directed graph, the direction directed graph, and the speed directed graph, this embodiment uses a spatiotemporal graph convolutional network to integrate the spatial features in these graphs with the time series features of the moving subject.

[0135] H=MLP(concat(VW1σ(norm(A)),σ(VW2LE))) (16)

[0136] Where W1 and W2 represent learning parameters, the time feature V comes from formula (15), and LE (the influence intensity of each moving subject) comes from formula (12). For the spatial features of a historical moments, this embodiment uses A=concat(A1,...,A a ) to indicate that A t is the adjacency matrix obtained by formula (14), representing the spatial relationship after merging at the t-th historical moment. It is worth noting that since the adjacency matrix of a directed graph cannot be quickly normalized using the method of an undirected graph, this embodiment uses norm(·) to normalize the elements of the adjacency matrix A by row, that is, the sum of the elements of each row is used as the denominator to normalize each element. Next, the spatiotemporal characteristics of the moving subject are obtained by formula (16).

[0137] Step S400: determining several candidate moving action probabilities of the vehicle according to the traffic light state and the historical motion trajectory.

[0138] The state of a traffic light can be represented as a binary feature (assuming that a yellow light has the same meaning as a green light), that is, by inputting an additional character information to represent the current state of the traffic light. This method is difficult to capture information in space or time series and cannot reveal the impact of traffic lights on vehicle motion, resulting in low accuracy in trajectory prediction.

[0139] Considering that traffic lights only specify the movement behavior of vehicles, the future movement of vehicles mainly depends on their historical behavior and the intention of the driver. However, it is certain that the legal movement actions specified by red traffic lights are stopping and waiting or turning right, while green traffic lights do not limit the movement of vehicles. Figure 8 As shown, this embodiment first determines several candidate actions according to the state of the traffic light. In the traffic intersection scene, the vehicle's motion behaviors include going straight, stopping and waiting, turning left, and turning right. When the traffic light is green, all of the above motion behaviors are possible actions in the future. When the traffic light is red, stopping and waiting and turning right are legal motion behaviors. Specifically, the candidate actions are embodied in the form of time series features.

[0140] After generating candidate actions, generate several candidate moving actions based on the historical motion trajectory:

[0141] L stop =[l a ,l a ,...,l a ,l a ] (17)

[0142]

[0143] in

[0144]

[0145] L stop , L straight , L left , L right They represent the candidate movement actions of stop waiting, go straight, turn left, and turn right. In formula (17), L stop Contains a positions with the same coordinates l a , and l a is a two-dimensional column vector, l a Represents the historical trajectory L past The position of the last observation time. In formula (18), d i =(l a -l1) / (ai). represents the actual position of the vehicle at the initial moment. Formula (21) and Formula (22) convert the candidate actions of going straight into Rotate the center point to get the steering behavior of the vehicle, where θ i =θ / (a-i+1), and θ is a preset value. Figure 7 The candidate actions of going straight, turning left and turning right are presented. In this example, the observation time of the historical trajectory is 4, that is, a=4.

[0146] Finally, according to each candidate action and each candidate mobile action, the gated recurrent unit algorithm and the multi-layer perceptron are used to calculate the candidate mobile action probability corresponding to each candidate mobile action. This embodiment takes into account that the candidate action is expressed in the form of time series features, and uses the attention mechanism of the Transformer model to establish the interactive relationship between the historical trajectory and different candidate actions. Specifically, different candidate actions are used as input according to the different states of the traffic light. When the traffic light is red:

[0147] C = softmax(MLP(act)) (23)

[0148] Where C is a four-dimensional vector, which represents the probability of stop waiting, right turn, straight ahead, and left turn. Softmax is the output category probability, and MLP is a multi-layer perceptron.

[0149] act = concat(GRU(L past ),ATTN1(L past ,L stop ), ATTN2(L past ,L right ), padding(2n))(24)

[0150] concat(·) indicates concatenation operation, GRU is the gated recurrent unit algorithm. padding(2n) indicates padding 2n zero elements to represent the candidate actions of going straight and turning left that do not belong to the red traffic light scenario, n is ATTN i (L past ,L # ) output characteristic length.

[0151]

[0152] Among them, i=1,...,4 corresponds to four candidate movement actions, L # ∈{L left ,L right ,L stop ,L straight}, d represents GRU(L past ) and GRU(L# ) is the feature length of the output, W1, W2 and W3 represent the learnable parameters, L past is the historical trajectory. When the traffic light is green, the vehicle’s future actions are not constrained, and padding(2n) in formula (24) is replaced by ATTN3(L past ,L straight ) and ATTN4(L past ,L left ).

[0153] Fig. 9 An example is provided to explain why the proposed method can accurately predict the future action of the vehicle. In this example, the vehicle violated the instruction of the red traffic signal and made a left turn. The method proposed in this embodiment estimates the future action of the vehicle by converting the historical trajectory into candidate moving actions, and taking the stop waiting and right turn actions as input candidate actions, and calculating the similarity between the historical features and the time series features of these two types of actions. It can be seen that the historical trajectory is completely different from the stop waiting action, and the movement trend of the historical trajectory is also not similar to the right turn action. Therefore, the method proposed in this embodiment realizes the effective interaction between the historical trajectory and the traffic light by converting the state of the traffic light into a time series feature. According to the different states of the traffic light, the time series features of the corresponding candidate actions are input, and the similarity between these features and the historical features is calculated through the attention mechanism, so that the future action of the vehicle can be accurately predicted, which is difficult to achieve by directly inputting the binary state of the traffic light.

[0154] Step S500: determining the vehicle's trajectory to be moved according to each of the spatiotemporal features and each of the candidate movement action probabilities.

[0155] Specifically, firstly, the real action code corresponding to the vehicle is obtained, and a first loss function is determined according to each of the candidate moving action probabilities and the real action code by minimizing the cross entropy function.

[0156] Reduce the prediction error by minimizing the cross entropy loss function (CE):

[0157] loss1=CE(C,C gt ) (26)

[0158] Where C is a four-dimensional vector obtained by the previous formula (23). gt One-hot encoding of the vehicle’s future real action. For the four actions of stop and wait, right turn, straight ahead, and left turn, the corresponding one-hot encoding distribution is [1,0,0,0], [0,1,0,0], [0,0,1,0], [0,0,0,1].

[0159] Then, a second loss function is determined according to each of the spatiotemporal features and each of the candidate movement action probabilities using multi-distribution position estimation, specifically including: determining several groups of t distribution parameters according to each of the spatiotemporal features and each of the candidate movement action probabilities; constructing t distributions corresponding to each group of t distribution parameters based on each group of t distribution parameters; fitting each t distribution using maximum likelihood estimation to determine the second loss function. The implementation method is as follows:

[0160] In non-traffic intersection road scenes, most vehicles usually move in the direction of motion specified by the lane. However, in traffic intersection scenes, the vehicle's motion pattern can be significantly different in different situations, which means that the future trajectory of the vehicle in different motion modes obeys different probability distributions. To this end, this embodiment uses a t-distribution probability model to characterize the diverse motion patterns. The probability density function of the t-distribution is:

[0161]

[0162] Where z is a random variable, v is a degree of freedom parameter, and m is a component of a two-dimensional position coordinate.

[0163]

[0164] When v = 1, Γ(1) = 1,

[0165]

[0166] At this time, the t distribution is equal to the standard Cauchy distribution. When v = ∞,

[0167]

[0168] At this point, the t distribution is equal to the standard Gaussian distribution.

[0169] According to formula (27), by adjusting the degree of freedom parameter of the t-distribution, the characteristics of the t-distribution can be flexibly changed. On the one hand, the t-distribution can obtain samples that are closer to the mean, which helps the model to more effectively estimate the future trajectory of vehicles such as vehicles that are waiting or moving at a very low rate. On the other hand, when predicting the future trajectory of pedestrians, the t-distribution can also present a long-tail feature similar to the Cauchy distribution to characterize the diverse movement directions of pedestrians. Therefore, the use of the t-distribution can better characterize the significantly different movement behaviors in traffic intersection scenes.

[0170] The method proposed in this embodiment uses maximum likelihood estimation to fit the parameters of the t distribution and determines the second loss function:

[0171]

[0172] Where b and N represent the number of future moments and the total number of moving subjects in the scene, respectively. x With m y represents the two-dimensional position coordinates of the moving subject at the tth future moment, z x 、z y 、v x 、v y Indicates the relevant parameters obtained through learning. In the prediction stage, z x 、z y 、v x 、v y The value of is determined, that is, a specific t distribution is obtained, and m is set at this time x With m y As a random variable, the multi-mode predicted moving trajectory can be obtained.

[0173] In order to more finely characterize the diverse and significantly different motion patterns of vehicles and pedestrians, the proposed method uses multiple t-distributions to characterize the motion behaviors of vehicles and pedestrians in different situations:

[0174]

[0175] H is obtained by formula (16), TCN represents the output of the spatiotemporal graph convolutional network, that is, the spatiotemporal features. C is obtained by formula (23), which represents the estimated probability of the vehicle's future action. i represents one of the actions of going straight, stopping and waiting, turning left or turning right. Therefore, formula (32) includes five TCNs, four of which estimate the t distribution parameters of the four different actions of the vehicle respectively, and TCN P Estimate parameters of the t-distribution for pedestrian motion. Fig.10 An example is provided to show the importance of independently characterizing the movement behaviors of different actions. In this embodiment, it is difficult to simultaneously characterize the vehicle's left turn and straight-moving movement behaviors using only the Cauchy distribution or the Gaussian distribution. Although the predicted trajectory generated by the Cauchy distribution has a variety of movement directions, it still cannot show the vehicle's left turn movement behavior. Different from the above-mentioned position estimation method, the method proposed in this section can characterize the vehicle's left turn and straight-moving movement behaviors separately, and because the t-distribution can fit the Cauchy distribution or the Gaussian distribution, the trajectory of the straight-moving action generated based on the t-distribution is more concentrated than the trajectory of the turning action, which is conducive to characterizing the differences between different actions.

[0176] The multi-distribution position estimation method can effectively characterize different motion behaviors of vehicles and pedestrians in different situations. On the one hand, the t-distribution has the characteristics of Gaussian distribution and Cauchy distribution, which makes the proposed method more flexible to characterize different movement behaviors. On the other hand, according to the different possible actions of the vehicle in the future, the proposed method uses multiple t-distributions to independently characterize the motion trajectories of different vehicle actions, thereby more finely describing the significantly different movement patterns of the vehicle in different situations.

[0177] Finally, the to-be-moved trajectory of the vehicle is determined by minimizing the sum of the first loss function and the second loss function.

[0178] According to the corresponding first loss function in formula (26) and the second loss function in formula (31), the training objective of the method proposed in this embodiment is to minimize the following function to determine the vehicle's trajectory to be moved.

[0179] Minimize(loss=loss1+loss2) (33)

[0180] This embodiment uses the Signalized INtersection Dataset (SIND) data set to test the trajectory prediction accuracy of the proposed method IATP in a traffic intersection environment. SIND is based on a traffic intersection scene in a certain city, and a total of 11,596 motion data of various vehicles, including cars (such as trucks, buses), motorcycles, bicycles, and tricycles, are collected. The motion trajectories of 1,652 pedestrians are also collected. Among them, the training set, validation set, and test set contain 7,948, 2,650, and 2,650 trajectories, respectively. The experiment uses two settings, short-term prediction and long-term prediction, to evaluate the prediction performance of the trajectory prediction algorithm under different prediction durations. The observation moments of the historical trajectory and future trajectory of the short-term prediction are 8 and 12, respectively, while the observation moments of the historical trajectory and future trajectory of the long-term prediction are 4 and 16, respectively. The experiment selected eight recently proposed trajectory prediction algorithms as comparison objects. These algorithms have their own characteristics and cover different technical paths, including: 1) Goal-GAN, 2) TPNMS, 3) AMEnet, 4) Causal-STGAT, 5) TPNSTA, 6) SGCN, 7) Social-STGCNN, 8) MDGI. For details of these comparison algorithms, please refer to Table 1.

[0181] Table 1 Comparison of trajectory prediction algorithms in traffic scenarios

[0182]

[0183] This experiment uses the common performance rating indicators in trajectory prediction tasks, minimal Average Displacement Error (mADE) and minimal Final Displacement Error (mFDE), to verify the prediction error rate of the algorithm.

[0184] 1) mADE calculates the minimum average error between the multi-mode predicted trajectory and the future trajectory at each prediction moment:

[0185]

[0186] Where N represents the number of moving subjects, ade i It represents the minimum average error between the K multi-modal predicted trajectories of the i-th moving subject and its future trajectory at each prediction moment:

[0187]

[0188] represents the real position of the i-th moving subject at the j-th prediction moment (a is the number of historical moments), represents the predicted position of the kth multi-modal predicted trajectory of the i-th moving subject at the jth prediction moment, and dist calculates the Euclidean distance between the two. b and K represent the number of prediction moments and the number of multi-modal predicted trajectories, respectively. The smaller mADE is, the smaller the prediction error is.

[0189] 2) mFDE calculates the minimum error between the multi-mode predicted position and the true position at the last prediction moment:

[0190]

[0191] Among them, fde i Represents the error between the K multi-modal predicted trajectories of the i-th moving subject and its future trajectory at the last predicted moment:

[0192]

[0193] The smaller the mFDE, the smaller the prediction error of the endpoint.

[0194] This experiment was implemented on the PyTorch platform, and the operating hardware environment is as follows: AMD R93900X@4.3Ghz (CPU), 4×16GB DDR43600 (RAM), NVIDIAGeFore RTX 2070 (GPU). In the training phase, we set the batch size and epoch to 64 and 1000 respectively. When optimizing the model parameters, the Adam optimizer was used, and the learning rate of the candidate action prediction network was set to 1e -6, and the learning rates of the rest are set to 1e -2 . In addition, the size z of the local environment perception window is set to 32. In order to achieve adaptive adjustment of the learning rate, we update the learning rate to 90% of the current learning rate after every 10 epochs. In order to evaluate the prediction performance of all compared algorithms, this embodiment uses mADE and mFDE as evaluation indicators. For each historical trajectory, all compared algorithms generate 5 multi-modal prediction trajectories. It should be noted that the experimental settings of the proposed method are consistent with those of all compared algorithms to ensure fairness.

[0195] Tables 2 and 3 show the short-term prediction results of each compared algorithm for different types of moving subjects on the SIND dataset. It can be observed from Tables 2 and 3 that, except for bicycles, IATP shows the best prediction accuracy for all types of moving subjects. It is worth noting that compared with other types of vehicles, the movement direction of bicycles is more flexible, which makes it difficult for candidate mobile action prediction methods to accurately predict their movement, so that IATP does not have a significant advantage in predicting the future trajectory of bicycles. However, in the average results of mADE and mFDE, IATP still shows the best prediction performance. This experimental result verifies the effectiveness of IATP in predicting the short-term future trajectories of various types of moving subjects in traffic intersection scenarios.

[0196] Table 2 mADE results for short-term prediction in traffic intersection scenarios

[0197]

[0198] Table 3 mFDE results for short-term prediction in traffic intersection scenarios

[0199]

[0200] In order to more intuitively illustrate the advantages of IATP in short-term forecasting, Figures 11 to 14 The prediction results of MDGI, Social-STGCNN and IATP are compared in a visual way. In these comparison results, circles, triangles and broken lines represent the historical position, future position and future trajectory of the moving subject respectively. The orange, blue (or pink, yellow), green and red trajectories represent the types of moving subjects, namely motorcycles, cars, bicycles and pedestrians respectively. Figures 11 to 13 In , the square represents the predicted position output by the algorithm, and the predicted trajectory displayed is the one that is closest to the future trajectory selected from the multi-mode predicted trajectory. Fig.14 In the figure, the black line shows the multi-modal prediction trajectory output by the algorithm.

[0201] exist Fig.11It can be clearly seen that the prediction accuracy of MDGI and Social-STGCNN is not as good as that of IATP, while the predicted trajectory of IATP is almost completely consistent with the actual trajectory. This is because the lane rule graph can not only depict the lane-based interaction rules, but also enables the model to directly use the historical moving speed as prior information, thereby more accurately predicting the interval between future positions.

[0202] Fig.12 The following shows the scene where car A and car B perform right turn and left turn respectively. Fig.11 Compared with the obvious turning action of motorcycle A in the scene, the turning actions of the two cars in this scene are more subtle, which puts higher requirements on the algorithm's ability to accurately predict turning actions. Unlike Social-STGCNN and the prediction method based on Cauchy distribution, IATP can better predict the future actions of car A and car B even when the turning actions are not obvious, which further proves the effectiveness of the candidate action prediction method.

[0203] In traffic intersection scenarios, the change of traffic lights is one of the key factors affecting the movement state of vehicles. For example, when the red traffic light turns green, the vehicles waiting in the lanes of the merging area begin to move. Ideally, these vehicles in the lanes should move out in order, that is, the following vehicles follow the front vehicles. This means that the movement speed of the front and rear vehicles should be similar, so the trajectory prediction algorithm must be able to effectively synchronize the movement speed of related vehicles to effectively avoid vehicle collisions or road traffic congestion. Fig.13 This further demonstrates the effectiveness of IATP in predicting future motion rates. To examine the prediction results of different algorithms in more detail, Fig.13 The dotted boxed area in is enlarged for easier observation. In this scenario, as the traffic lights on the left and right sides change from red to green, cars B and E start to accelerate and prepare to drive. Their historical rates are relatively low, which makes Social-STGCNN and MDGI's predictions of the position changes of cars B and E not obvious enough. According to the proposed lane rule directed graph, IATP can effectively interact with the historical rate information of vehicles in the same lane, thereby accurately predicting the future rates of cars B and E. At the same time, observing the prediction results of motorcycles A and C, although their historical rate changes are more significant than those of cars B and E, the prediction effect of IATP is still significantly better than that of Social-STGCNN and MDGI. This further proves the effect of our proposed lane rule graph in interacting with the motion states of related vehicles, thereby accurately predicting the future trajectory of vehicles in such scenarios.

[0204] In order to verify the effectiveness of our proposed multi-distribution location estimation method, Fig.14The comparison of the three trajectory prediction algorithms in terms of multi-modal trajectory prediction results is shown. In this particular scenario, the moving bodies A, B, and C are performing straight-line driving, while the moving bodies D, E, and F are performing turning maneuvers. By observing Fig.14 (a) with Fig.14 From the prediction results in (b), we can see that the multi-modal prediction trajectories output by both Social-STGCNN and MDGI show similar dispersion, because they both rely on a single probability distribution to estimate the future position of the moving subject. For moving subjects A, B, and C traveling in a straight line, it is crucial to predict multi-modal trajectories with different rates, while for moving subjects D, E, and F performing turning operations, different directions of movement need to be predicted.

[0205] IATP, which uses a multi-distributed position estimation method, can more accurately characterize the multi-mode motion trajectories of moving subjects in different states. For moving subjects A, B, and C traveling in a straight line, the multi-mode trajectories predicted by the IATP algorithm remain consistent in direction, and the main difference is reflected in the speed, while the other two algorithms show multi-mode characteristics in direction, which reduces the accuracy of the prediction (as can be clearly seen from the predicted trajectory of moving subject C). For turning moving subjects D, E, and F, IATP can effectively depict the multi-mode characteristics in direction, making the predicted trajectory more diverse, thereby better approximating the real trajectory. This result shows that the multi-distributed position estimation method proposed in this embodiment can effectively predict the future trajectory of moving subjects in different states, making the prediction more flexible and accurate.

[0206] Tables 4 and 5 show the long-term prediction results of all the compared algorithms for different types of moving subjects on the SIND dataset. It can be clearly seen from Tables 4 and 5 that IATP shows the best prediction performance regardless of the type of moving subject. Although IATP's prediction results for bicycle trajectories did not achieve the minimum prediction error in short-term prediction, in terms of long-term prediction, IATP's prediction results show certain advantages over MDGI, and this advantage is more significant compared to other models. In order to more intuitively show the advantages of IATP in long-term prediction, Figure 15 to Figure 16 The prediction results of MDGI, Social-STGCNN and IATP are compared.

[0207] Table 4 mADE results for long-term prediction in traffic intersection scenarios

[0208]

[0209] Table 5 mFDE results for long-term prediction in traffic intersection scenarios

[0210]

[0211] Fig.15 A visualization example is provided to illustrate the importance of candidate mobile action prediction methods in long-term prediction. In this figure, circles, triangles, and lines represent the historical position, future position, and future trajectory of the moving subject, respectively. Orange and green represent moving subjects of the type of motorcycle and bicycle, respectively. The square represents the predicted position output by the algorithm, and the predicted trajectory displayed is the one that is closest to the actual future trajectory from the multi-modal predicted trajectory. Fig.15 It can be clearly seen that the prediction accuracy of MDGI and Social-STGCNN is significantly lower than that of IATP. The predicted trajectory of IATP is almost consistent with the actual trajectory. For bicycle A and bicycle B, which are performing right turns, MDGI and Social-STGCNN find it difficult to effectively describe their right turn actions, while IATP can accurately characterize their right turn operations. When considering the future trajectory of motorcycle C, although its turning action is not as obvious as bicycle A and bicycle B, IATP can still accurately predict its left turn action. This shows that even in long-term predictions, IATP's candidate action prediction method can effectively judge the future actions of the moving subject.

[0212] To verify the effectiveness of the proposed multi-distribution position estimation method in long-term prediction, Fig.16 The multimodal prediction results of Social-STGCNN, MDGI and IATP in long-term prediction are given. The orange, blue, green and red trajectories in the figure represent different categories of moving subjects: motorcycles, cars, bicycles and pedestrians, respectively, while the black broken line represents the multimodal prediction trajectory output by the algorithm. In this scene, moving subjects B, C and D are merging into the same lane, while other moving subjects are moving relatively independently. For the three moving subjects B, C and D, the multimodal trajectory generated by IATP can better approach the future trajectory (occluding the future trajectory). For moving subjects F, E and G, IATP produces more dispersed prediction positions at later prediction moments. Since the prediction error at later moments is usually larger, the dispersed prediction positions generated by t distribution can better simulate the diverse motion behaviors, thereby compensating for the performance degradation caused by the increase in prediction error. For the slow-moving car A and pedestrian H, IATP uses independent distribution to describe the movement behaviors of vehicles and pedestrians respectively, enabling IATP to effectively model the diverse movement patterns of pedestrians more accurately.

[0213] In order to further verify the effectiveness of the proposed multi-distribution position estimation, lane rule map and candidate mobile action prediction methods, this embodiment will show the experimental results and analysis of the prediction model based on a single Gaussian distribution, a single Cauchy distribution, a single t distribution, no lane rule map and no candidate mobile action prediction. It is particularly noteworthy that the model without candidate mobile action prediction directly uses the state of the traffic light as input and predicts whether the future behavior of the vehicle violates the control of the traffic light. The quantitative results of the prediction are detailed in Tables 6 to 9.

[0214] Table 6 mADE results of short-term prediction of different prediction models in traffic intersection scenarios

[0215]

[0216] Table 7 mFDE results of short-term prediction of different prediction models in traffic intersection scenarios

[0217]

[0218] IATP shows the best prediction performance in both short-term and long-term prediction. When comparing the short-term prediction results using a single probability distribution, the average prediction performance of IATP under the mADE and mFDE evaluation indicators is at least 18% and 13% ahead of other models. From the short-term prediction results based on a single t-distribution, Cauchy distribution and Gaussian distribution, the performance of the t-distribution is better than that of the Cauchy distribution and Gaussian distribution because the t-distribution can fit the characteristics of the Cauchy distribution and Gaussian distribution. The t-distribution can obtain samples closer to the mean, which helps the model to more effectively estimate the future trajectory of vehicles such as stopped waiting or slow-moving vehicles. When predicting the future trajectory of pedestrians, the t-distribution can also present a long-tail characteristic similar to the Cauchy distribution, thereby characterizing the diverse movement directions of pedestrians. Compared with the average short-term prediction results without the lane rule map method, IATP achieved a performance improvement of 25% and 11% under the mADE and mFDE evaluation indicators, respectively. If the candidate mobile action prediction method is not used, the prediction performance drops by 31% and 25%. These experimental results fully prove that the proposed multi-distribution position estimation, lane rule map and candidate action prediction methods can significantly improve the short-term prediction accuracy of the model in traffic intersection scenarios. The experimental results of long-term prediction (Tables 8 and 9) also support this conclusion.

[0219] Table 8 mADE results of long-term prediction of different prediction models in traffic intersection scenarios

[0220]

[0221] Table 9 mFDE results of long-term prediction of different prediction models in traffic intersection scenarios

[0222]

[0223] Table 10 shows the computation time required for the above comparison models, Goal-GAN, AMEnet, TPNSTA, MDGI and IATP to predict the multi-modal trajectory of a single moving subject. From the comparison of models with different distributions, it can be seen that the computational cost of Gaussian distribution, Cauchy distribution and t distribution is the same, while IATP only requires an additional 0.0007 seconds of computation time to generate trajectories, which is attributed to the fact that the multi-distribution position estimation method uses 5 independent TCN modules to describe multiple distributions. In comparison, the methods without using lane rule maps and without using candidate action prediction increase the computation time of IATP by 0.0003 seconds and 0.0008 seconds, respectively. Although the proposed method increases the prediction time, the increase is still within an acceptable range. Compared with AMEnet, Goal-GAN and TPNSTA, IATP has a significant advantage in prediction speed, which is at least 20 times faster than the above methods. This is mainly because these methods rely on LSTM models to gradually generate predicted trajectories and use more complex multi-modal trajectory generation techniques. Compared with MDGI based on multi-class directed graph interaction, IATP only adds 0.0016 seconds of computing time when predicting trajectories, which is the additional computing cost caused by the lane rule graph, candidate action prediction method and multi-distribution position estimation method. From the trajectory prediction results in Tables 2 to 5, it can be seen that the prediction performance of IATP is significantly better than other comparison algorithms. At the same time, the results in Table 10 show that while IATP maintains high-speed prediction, the increased computing time is completely within an acceptable range.

[0224] Table 10 Prediction speed comparison

[0225]

[0226] Fig.17 The comparison of prediction results between the model without using the lane rule map and the model with the lane rule map is shown. In this example, for the moving subject A, it can be observed that there is no significant difference in prediction accuracy between the two models. However, when predicting the future trajectory of the moving subject B, the model without the lane rule map only interacts the moving rates of the two moving subjects through the rate map. This method makes it difficult to achieve effective interaction of the moving rates of the two moving subjects, causing the predicted trajectory of the moving subject B to gradually deviate from the true trajectory. In contrast, the model using the lane rule map can effectively solve the above problem. By interacting the moving rates between the two moving subjects, the model can accurately perceive the difference in their moving rates, thereby efficiently synchronizing their moving rates and achieving accurate prediction of future trajectories. This result shows that the use of the lane rule map significantly improves the prediction accuracy of the model in complex traffic scenarios.

[0227] Fig.18The difference in prediction results between the model without the candidate action prediction method and IATP is shown. From the prediction results of the three moving subjects A, B, and C, it can be clearly seen that the complete model IATP that integrates the candidate action prediction method is significantly better than the model without the method.

[0228] Fig.19 A visualization example is provided to demonstrate the advantages of the multi-distribution position estimation method in traffic intersection scenarios. In this example, moving subjects A and B are pedestrians, and the remaining moving subjects are vehicles. Fig.19 In (a), the multi-modal predicted trajectories of all moving subjects show some diversity in the non-moving directions. However, when the vehicles in the scene mainly move straight, too much diversity in the non-moving directions may actually reduce the accuracy of the prediction. In contrast, Fig.19 The model using the multi-distribution position estimation method in (b) effectively avoids generating overly dispersed predicted positions in non-moving directions when predicting the future trajectory of the vehicle, thereby ensuring the accuracy of the prediction of the vehicle's linear motion. For pedestrians A and B, it is difficult to accurately characterize the diverse movement patterns of pedestrians using a single distribution model, because the same distribution is also used to describe the multi-modal motion characteristics of vehicles, which results in the characteristics of the pedestrian's multi-modal behavior not being significant enough. The multi-distribution position estimation method proposed in this embodiment can independently characterize the significantly different motion patterns of pedestrians and vehicles through multiple probability distributions, thereby ensuring that the model can generate diverse pedestrian prediction trajectories.

[0229] Based on the above embodiments, the present invention also provides a trajectory prediction device in a traffic intersection scenario, such as Fig. 20 As shown, the device comprises:

[0230] Data acquisition module 01, used to obtain road scenes, traffic light status and historical movement trajectories;

[0231] A directed graph construction module 02, used to construct a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene;

[0232] A feature extraction module 03, used to obtain the spatiotemporal features of each moving subject according to the historical motion trajectory and each directed graph;

[0233] A probability calculation module 04, used to determine the probability of several candidate movement actions of the vehicle according to the traffic light state and the historical movement trajectory;

[0234] The trajectory determination module 05 is used to determine the vehicle's pending moving trajectory according to each of the spatiotemporal features and each of the candidate moving action probabilities.

[0235] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Fig.21 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a trajectory prediction method in a traffic intersection scenario is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0236] Those skilled in the art will understand that Fig.21 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0237] In one implementation, the terminal stores one or more programs in its memory, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a trajectory prediction method in a traffic intersection scenario.

[0238] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0239] In summary, the present invention discloses a trajectory prediction method, device, terminal and storage medium in a traffic intersection scenario, wherein the method obtains a road scene, a traffic light state and a historical motion trajectory; constructs a number of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene; obtains the spatiotemporal characteristics of each moving subject according to the historical motion trajectory and each directed graph; determines a number of candidate moving action probabilities of the vehicle according to the traffic light state and the historical motion trajectory; and determines the vehicle's trajectory to be moved according to each spatiotemporal characteristic and each candidate moving action probability. Since the present invention constructs a directed graph, candidate action prediction and the like for the interaction relationship between various moving subjects in the road scene, it is possible to more accurately distinguish and predict various significantly different movement modes, effectively solving the problem that the prior art is difficult to achieve a high accuracy rate when predicting the behavior of moving subjects in traffic intersections.

[0240] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A trajectory prediction method in a traffic intersection scenario, characterized in that: The method comprises: Obtain road scenes, traffic light status, and historical movement trajectories; Constructing a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene; Obtaining the spatiotemporal characteristics of each moving subject according to the historical movement trajectory and each directed graph; Determining several candidate moving action probabilities of the vehicle according to the traffic light state and the historical motion trajectory; The vehicle's trajectory to be moved is determined according to each of the spatiotemporal features and each of the candidate movement action probabilities.

2. The trajectory prediction method in a traffic intersection scenario according to claim 1 is characterized in that: The step of constructing a plurality of directed graphs according to the road scene comprises: Obtaining a motion influence factor corresponding to each moving subject, wherein the motion influence factor includes one or more of a field of view, a speed, a motion direction, and a lane rule; A directed graph corresponding to each motion impact factor is constructed according to the road scene and the motion impact factor.

3. The trajectory prediction method in the traffic intersection scenario according to claim 2 is characterized in that: The step of constructing directed graphs corresponding to the motion impact factors according to the road scene and the motion impact factors respectively includes: Determining the interaction relationship between each moving subject corresponding to each moving influence factor based on each moving influence factor and the road scene, wherein the interaction relationship includes one or more of a peer-to-peer interaction relationship, an unequal interaction relationship, and a non-interaction relationship; Directed graphs corresponding to the sports influencing factors are constructed according to the interaction relationships.

4. The trajectory prediction method in a traffic intersection scenario according to claim 1, characterized in that: The step of obtaining the spatiotemporal characteristics of each moving subject according to the historical movement trajectory and each directed graph includes: Using a spatiotemporal graph convolutional network, extracting temporal features based on historical trajectory features, and extracting spatial features based on each of the directed graphs; The temporal and spatial characteristics of each moving subject are determined according to the temporal characteristics and the spatial characteristics.

5. The trajectory prediction method in a traffic intersection scenario according to claim 1, characterized in that: The determining of several candidate moving action probabilities of the vehicle according to the traffic light state and the historical motion trajectory includes: determining a number of candidate actions according to the state of the traffic light; Determine a number of candidate movement actions according to the historical movement trajectory; According to each of the candidate actions and each of the candidate moving actions, a gated recurrent unit algorithm and a multi-layer perceptron are used to calculate the candidate moving action probability corresponding to each of the candidate moving actions.

6. The trajectory prediction method in a traffic intersection scenario according to claim 1, characterized in that: The determining of the to-be-moved trajectory of the vehicle according to each of the spatiotemporal features and each of the candidate moving action probabilities includes: Obtaining a real action code corresponding to the vehicle, and determining a first loss function according to each of the candidate moving action probabilities and the real action code by minimizing a cross entropy function; Determine a second loss function according to each of the spatiotemporal features and each of the candidate movement action probabilities using multi-distribution position estimation; The to-be-moved trajectory of the vehicle is determined by minimizing the sum of the first loss function and the second loss function.

7. The trajectory prediction method in a traffic intersection scenario according to claim 6, characterized in that: The multi-distribution position estimation is a t-distribution position estimation, and the multi-distribution position estimation is used to determine the second loss function according to each of the spatiotemporal features and each of the candidate movement action probabilities, including: Determine a plurality of groups of t-distribution parameters according to each of the spatiotemporal features and each of the candidate movement action probabilities; Based on each group of t distribution parameters, construct the t distribution corresponding to each group of t distribution parameters; The maximum likelihood estimation is used to fit each t distribution and determine the second loss function.

8. A trajectory prediction device in a traffic intersection scenario, characterized in that: The device comprises: Data acquisition module, used to obtain road scenes, traffic light status and historical movement trajectories; A directed graph construction module, used to construct a plurality of directed graphs according to the road scene, wherein the directed graphs are used to simulate the interaction relationship between various moving subjects in the road scene; A feature extraction module, used to obtain the spatiotemporal features of each moving subject according to the historical motion trajectory and each directed graph; A probability calculation module, used to determine the probabilities of several candidate movement actions of the vehicle according to the state of the traffic light and the historical movement trajectory; The trajectory determination module is used to determine the vehicle's trajectory to be moved according to each of the spatiotemporal features and each of the candidate movement action probabilities.

9. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program includes instructions for executing the trajectory prediction method in a traffic intersection scenario as described in any one of claims 1-7; and the processor is used to execute the program.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the trajectory prediction method in a traffic intersection scenario described in any one of claims 1-7 above.

Images