Scene-level joint trajectory prediction method and system based on multi-level interaction relationship graph and storage medium

Abstract

The application discloses a scene-level joint trajectory prediction method and system based on a multi-level interaction relationship graph and a storage medium, and comprises the following steps: S1, constructing a multi-agent space-time relationship graph; S2, dividing a space interaction subset based on the space-time relationship graph, reasoning a future interaction relationship of a target agent, and dividing a multi-level interaction relationship graph; S3, realizing group information interaction in a single interaction subset unit, and guiding element state coding and information fusion in a trajectory prediction model; S4, embedding a characteristic value of the target agent to decode a predicted trajectory; in the decoding, a low-level agent will fuse a high-level agent characteristic to perform trajectory decoding; and S5, generating a scene-level multi-modal prediction result, and serving each group of interaction relationship graphs generated in S2 to a single scene-level mode. The application solves the scene consistency problem and simultaneously outputs reasonable multi-modal prediction results, infers the interaction relationship among multi-agents by using an agent historical state quantity and a driving scene environment element, and realizes the scene consistency of the trajectory prediction results.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of automatic driving, in particular to a scene-level multi-modal joint trajectory prediction method and system based on a multi-level interaction relationship graph and a storage medium. BACKGROUND

[0002] In the implementation of a driving scene, accurate and effective trajectory prediction is an important prerequisite for ensuring that an automatic driving vehicle makes a safe behavior decision and path planning. Existing mainstream algorithms are limited to obtaining interaction information from historical observed trajectories, which is difficult to ensure the rationality of the interaction relationship between the predicted trajectories, and the prediction results do not have scene consistency. At the same time, due to the multi-modal nature of trajectory prediction, it is a great challenge for the downstream decision module to generate prediction results for the target agent on the basis of explicitly considering all future trajectory modalities of other agents, and it is not feasible.

[0003] To solve the above problems, the application provides a scene-level joint trajectory prediction method based on a multi-level interaction relationship graph. First, an interaction relationship judgment model based on an interaction mask is designed. The model learns the implicit relationship in each agent in the driving scene by using a data-driven method, and represents the interaction behavior between each pair of agents through an interaction mask to determine the interaction relationship and generate training labels, thereby avoiding the defects of a manual design method. Secondly, a multi-level interaction relationship graph based on a graph neural network is constructed for interaction representation and evolution of the driving scene. The historical observed trajectory embedding code of the agent will be used to generate the interaction modalities in the prediction period of the agent, and will be explicitly used in the multi-level trajectory decoding at the scene level to realize the scene consistency of the trajectory prediction results at different levels. To improve the robustness of processing different scene scales, the application also provides a spatial interaction graph subset division module that can exchange information. The above invention contents realize the output of reasonable multi-modal prediction results while solving the scene consistency problem. SUMMARY

[0004] The application aims to solve the problems in the prior art, and provides a scene-level joint trajectory prediction method, system and storage medium based on a multi-level interaction relationship graph.

[0005] Technical scheme

[0006] According to a first aspect of the application, a scene-level joint trajectory prediction method based on a multi-level interaction relationship graph is provided, as shown in the accompanying drawings, comprising: Figure 1

[0007] S1, constructing a multi-agent space-time interaction relationship graph based on the driving scene to which the intelligent vehicle belongs, comprising:

[0008] ​S1.1, build an interaction relationship judgment model based on an interaction mask, used to generate an agent interaction true value label on a training data set, comprising:

[0009] First, a neural network model is constructed to realize trajectory prediction function, as follows:

[0010] The model will comprehensively process road map elements and agent elements in the driving scene, and output the trajectory prediction result of the target agent in a fixed period;

[0011] The links include agent and road map element data preprocessing link, agent-road map element interaction encoding link, agent-agent interaction encoding link, and predicted trajectory decoding link;

[0012] In order to ensure the integrity of agent-agent interaction, self-attention mechanism will be used to fully integrate the encoding information of all effective agents in the driving scene for each target agent;

[0013] On the training data set, the training of the above neural network trajectory prediction model is completed, and the training weight is obtained;

[0014] Secondly, an interaction relationship judgment model based on an interaction mask is constructed to analyze the interaction influence of adjacent agents on the target agent and determine the interaction relationship between them, and the interaction mask strategy is as follows:

[0015] The target agent will obtain the encoding information from the adjacent agents through the attention mechanism for subsequent trajectory decoding;

[0016] The interaction mask strategy will shield the adjacent agent elements of the target agent in turn, that is, an attention mask is constructed, so that the target agent cannot obtain the encoding information from the agents whose interaction relationship needs to be determined, but can interact normally with other agents;

[0017] Compare and analyze the trajectory results decoded after adding the mask to the target agent with the decoding results without adding the mask, so as to determine the interaction relationship between the target agent and the currently selected other agents, and generate an interaction true value label;

[0018] The label will be divided into two categories, the first category is that the target agent is affected by the currently selected agent, and the second category is that the target agent is not affected by the currently selected agent;

[0019] Finally, based on the above strategy, the interaction relationship true value label is generated for all agents in the training data set.

[0020] S1.2, train the interaction relationship judgment network based on the interaction true value label, and construct a multi-agent spatio-temporal interaction relationship graph, comprising

[0021] Obtain the state information of various elements in the driving scene where the autonomous vehicle is located, including the position, speed, and semantic attributes of the agent elements, and the position and semantic attributes of the road map elements;

[0022] Construct a road map element encoding module and an agent element encoding module to respectively encode and interactively fuse the information of the road map elements and the historical trajectories of the agents;

[0023] Construct a graph attention neural network to realize the fusion of the features of the agents and the features of the road map elements through a cross-attention mechanism, so as to obtain the agent embedding encoding of the fused driving scene environment information;

[0024] Construct a fully connected graph of agent elements, with each agent acting as a node of the interaction graph and the edges of the nodes being bidirectional edges. The node encoding is initialized using the agent embedding encoding;

[0025] The edge information is initialized by using the source node embedding encoding, the target node embedding encoding, and the state quantity encoding of the source node relative to the target node through a multi-layer perception machine;

[0026] For the target agent, the target node and the edge node are spliced, and the interaction relationship classification result is output through a multi-layer perception machine. The classification result has three types: the first type is that the source node influences the target node, the second type is that the target node influences the source node, and the third type is that there is no interaction relationship;

[0027] In the training process, the loss function is calculated using the above classification result and the true value label generated in S1.1. The loss function adopts Focal Loss to solve the problem of uneven distribution of classification results;

[0028] In the inference process, the interaction relationship graph under the current driving scene is constructed through the prediction result.

[0029] S2, based on the space-time interaction relationship graph, divide the spatial interaction subset, and based on the interaction level, divide the multi-level interaction relationship graph.

[0030] Including:

[0031] S2.1, obtain the relative relationship between the agents in the driving scene through S1, that is, including the source node influencing the target node, the target node influencing the source node, and the non-interaction relationship;

[0032] S2.2, divide the spatial interaction subset based on the relative relationship, including:

[0033] Cluster the agents with interaction relationship in the current driving scene. The agent with interaction relationship is the agent whose relationship between the source node and the target node is not the non-interaction relationship;

[0034] The clustering operation aggregates all mutually influencing agents in the driving scene and divides the driving scene into several subsets, each of which will be processed independently for internal interaction behavior, effectively reducing the size of the scene scale;

[0035] After the preliminary initialization of the subset is completed, for a larger subset in the current scene, a secondary segmentation will be performed, that is, the interaction relationship probability result in S1 is used to find the single interaction part of the agents in the current subset for segmentation, so as to further reduce the scene scale size of the subset without affecting the conditional prediction and improve the operation efficiency;

[0036] S2.3, constructing a multi-level interaction relationship graph based on the relative relationship between the nodes in the subset, including:

[0037] At this link, each subset in the current driving scene will be constructed as an independent scene to construct a multi-level interaction relationship graph;

[0038] For each subset, the internal hierarchical division is based on the yielding relationship between low-level and high-level, that is, the low-level will be affected by the high-level agent;

[0039] First, the agent with the highest interaction relationship in itself is screened, that is, the target agent not affected by other agents, as the highest level;

[0040] Secondly, on the basis of the highest level agent, the next level agent affected by the current level is searched in turn and used as the next level;

[0041] In turn, the search is ended when the current level agent has no next level agent that can be affected;

[0042] Based on the above operation, a multi-level interaction relationship graph is constructed inside each subset.

[0043] S3, realizing inter-group information interaction based on a single subset to guide element state coding and information fusion in the trajectory prediction model, including:

[0044] The multi-level interaction relationship graph inside the single subset is obtained through S2, and the graph attention neural network edge is constructed using the multi-level interaction relationship graph;

[0045] Based on the graph attention neural network edge, the information coding and interaction in the subset covering scene are guided, that is, the low-level agent will increase the weight of the embedding coding of the high-level agent connected thereto, and preferentially obtain the coding information from such agent, thereby improving the efficiency of information fusion and reducing the calculation demand;

[0046] In addition, in order to better fuse global information, each subset will generate global features for information fusion of each target agent in each subset. By obtaining global features in a single subset coverage scene, the embedding feature value of the target agent is updated based on the subset features and the features of the adjacent scene elements, so as to realize information interaction and fusion among multiple subsets in the overall driving scene.

[0047] S4, decoding and predicting the trajectory based on the embedding feature of the target agent, comprising:

[0048] In the decoding process, the low-level agent will decode the trajectory based on the fusion of the high-level agent features;

[0049] First, obtain the multi-level interaction relationship graph in S2 and the target agent encoding result of the adjacent level agent in S3;

[0050] Secondly, the current level is initialized as the highest level, and the trajectory prediction of the agent belonging to the current level is carried out through the trajectory decoding module, and the prediction result is output;

[0051] After synchronously completing the trajectory decoding of the agent belonging to the current level, the trajectory prediction result of each agent under the current level is encoded, and the encoded features are saved, and the multi-level interaction relationship graph is used to store the information;

[0052] Decode the trajectory prediction result of the next level, and consider the trajectory prediction result encoding features of the agents in the current level in the decoding process, to ensure the scene consistency between the prediction result of the next level and the prediction result of the current level;

[0053] Repeat the above steps until the lowest level is decoded.

[0054] S5, in order to generate a scene-level multi-modal prediction result, S2 will generate a specific number of group interaction relationship graphs, each group will serve a single scene-level mode, comprising:

[0055] In the inference link, a plurality of groups of interaction relationship graphs will be generated for each subset in S2, and each group of interaction relationship graphs represents a driving behavior mode.

[0056] According to the second aspect of the present application, a scene-level joint trajectory prediction system based on a multi-level interaction relationship graph is provided, comprising:

[0057] The space-time relationship graph acquisition module is used for acquiring the space-time interaction relationship among the agents around the autonomous vehicle in the real-time driving scene; and the content of the foregoing step S1 is realized.

[0058] The relationship reasoning module is used for reasoning the interaction relationship of the adjacent agents and dividing the spatial interaction subsets; and the content of the foregoing step S2 is realized.

[0059] An information fusion module is configured to guide driving scene information fusion by using the relationship reasoning result, and realize information transmission between spatial interaction subsets.

[0060] A trajectory prediction module is configured to realize conditional prediction trajectory decoding based on the target intelligent agent level relationship and fusion encoding obtained by the information fusion module; and the contents of steps S4 and S5 are realized.

[0061] According to a third aspect of the present application, a storage medium for scene-level joint trajectory prediction based on a multi-level interaction relationship graph is provided, and the storage medium internally stores a computer program for executing the above prediction method.

[0062] The present application has the following advantages:

[0063] (1) Compared with the prior art, the present application has the following advantages: based on the trajectory prediction method, system and storage medium provided by the disclosed embodiment of the present application, an interaction relationship determination model based on an interaction mask is realized, the model learns the implicit relationship in each intelligent agent in the driving scene by using a data-driven method, and the interaction behavior between two intelligent agents is represented by an interaction mask, the interaction relationship is determined, and a training label is generated, thereby avoiding the defects of a manual design method; at the same time, a multi-agent spatiotemporal relationship graph is constructed by acquiring a driving scene to which the intelligent vehicle belongs by using a spatiotemporal relationship graph acquisition module, the spatial interaction subsets are segmented based on the spatiotemporal relationship graph, the future interaction relationship of the target intelligent agent is reasoned, the level interaction relationship graph is constructed, the inter-group and intra-group information fusion interaction is guided, and the joint trajectory prediction in the driving scene is realized.

[0064] (2) In order to solve the scene consistency problem and output reasonable multi-modal prediction results at the same time, the present application reasons the interaction relationship between the intelligent agents by using the historical state quantity of the intelligent agent and the driving scene environment element, and applies it to the trajectory prediction, thereby realizing the scene consistency of the trajectory prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The figure is a schematic diagram of the algorithm flow disclosed by the present application.

[0066] Figure 2 The figure is a schematic diagram of the module flow provided by the example embodiment disclosed by the present application.

[0067] Figure 3 The figure is a schematic diagram of the scene provided by the example embodiment disclosed by the present application.

[0068] Figure 4 The figure is a schematic diagram of the multi-level interaction relationship provided by the example embodiment disclosed by the present application.

[0069] Figure 5A trajectory prediction process schematic diagram is provided for an example embodiment of the present application. DETAILED DESCRIPTION

[0070] The present application uses the historical state quantity of the agent and the driving scene environment element to infer the interaction relationship between multi-agents, and applies it to trajectory prediction, so as to realize the scene consistency of the trajectory prediction result. In the following, the example embodiments of the present application will be described in conjunction with the drawings. In the specification, all the features of the actual embodiments are not described for the sake of clarity and brevity. However, it should be understood that many embodiment-specific decisions can be made in the process of developing any such actual embodiment in order to achieve the specific goals of the developers, and these decisions can vary from embodiment to embodiment. The scope of protection of the present application is not limited to the described embodiments.

[0071] The present embodiment realizes a scene-level joint trajectory prediction method system based on a multi-level interaction relationship graph and a storage medium.

[0072] When the intelligent vehicle performs a trajectory prediction task in a cycle, the evolution from the space-time relationship graph to the multi-level interaction relationship graph will be realized, and the driving scene element coding and information fusion will be guided based on the multi-level interaction relationship graph, and finally the multi-modal scene-level joint trajectory prediction result output will be realized.

[0073] Figure 2 A module flow schematic diagram is shown. The multi-modal scene-level joint trajectory prediction model includes the following modules: a space-time relationship graph acquisition module, a relationship reasoning module, an information fusion module, and a trajectory prediction module.

[0074] The space-time relationship graph acquisition module is used to acquire the space-time interaction relationship between the agents around the autonomous vehicle in the real-time driving scene.

[0075] The relationship reasoning module is used to reason the interaction relationship between adjacent agents and divide the space interaction subset.

[0076] The information fusion module is used to guide the driving scene information fusion using the relationship reasoning result, and realize the information transmission between the space interaction subsets.

[0077] The trajectory prediction module is used to realize the conditional prediction trajectory decoding based on the target agent level relationship and the fusion coding obtained by the information fusion module.

[0078] It should be understood that the present application will not be limited to the described embodiments due to the description of the drawings. Where possible, different embodiments can combine, replace, borrow, and omit the modules or features in different embodiments without changing the ideas of the present application.

[0079] First, an interaction relationship determination model based on an interaction mask is constructed to generate the interaction true value label of the agent on the training data set, including:

[0080] A neural network model implementing trajectory prediction function is constructed, which will comprehensively process the road map elements and agent elements in the driving scene and output the trajectory prediction result of the target agent in a fixed period.

[0081] The intelligent vehicle obtains the driving scene information through but not limited to perception, road information, high-precision map and the like, including the adjacent agent historical state information such as position, speed, heading angle, semantic attribute and the like; road map information such as lane center line, lane line, pedestrian crossing and traffic light and the like.

[0082] The trajectory prediction based on neural network includes the following basic links: agent and road map element data preprocessing link, agent-road map element interaction encoding link, agent-agent interaction encoding link and prediction trajectory decoding link.

[0083] In the embodiment, the agent and road map element data preprocessing link will be implemented by using multi-layer perception and attention mechanism, specifically as follows:

[0084] For the agent element, the historical observation state quantity will be encoded, including position, speed, heading, semantic attribute and the like. A fixed historical observation period is set, and the state quantity of the agent at each historical observation time is encoded by MLP to obtain the encoding result in the time sequence. After the encoding of all historical observation times is completed, the self-attention mechanism with time mask is used to fuse the embedding encoding of each time with the embedding feature values of other times, so that it has time sequence information.

[0085] For the road map element, the spatial state quantity including position, semantic attribute and direction will be encoded. In the embodiment, the vectorized road map element encoding will be taken as an example, each road map element will be constructed as a multi-segment group, and a multi-segment group is composed of several points. First, the point state quantity in the segment group is encoded by MLP, and then the single embedding feature representing the multi-segment group is obtained by maximum pooling operation to save computing resources.

[0086] In the embodiment, the agent-road map element interaction encoding link will be implemented by using cross-attention mechanism, specifically as follows:

[0087] First, the agent encoding and road map element encoding are obtained based on the agent and road map element data preprocessing link, and then the cross-attention mechanism is constructed, in which the agent element encoding is taken as Query, and the road map element encoding is taken as Key and Value, so that the agent element can fuse the road map element information, and the specific formula is as follows:

[0088]

[0089] where AgentEmbeds is the updated agent feature of the fused road map element, Q, K, and V are Query, Key, and Value, respectively, d k is the dimension of the Query vector, and softmax is the normalization exponential function.

[0090] In this embodiment, the agent-agent interaction encoding link will be implemented using a self-attention mechanism. In order to ensure the integrity of the agent-agent interaction, the self-attention mechanism fully integrates the encoding information of all effective agents in the driving scene for each target agent. The specific formula is as follows:

[0091]

[0092] where Q, K, and V are initialized using AgentEmbeds, and the final result is used to update AgentEmbeds.

[0093] In this embodiment, the trajectory decoding link will use an MLP network to directly decode the agent trajectory results in the prediction period. The target agent feature value will be initialized using the AgentEmbeds feature value that fuses the road map element and the adjacent agent feature.

[0094] In the training process, the predicted trajectory is calculated with the trajectory true value on the training set to calculate the regression loss. The training of the above neural network trajectory prediction model is completed on the training data set, and the training weight is obtained.

[0095] After the construction of the neural network trajectory prediction model is completed, an interaction mask-based interaction relationship determination model is constructed based on the trained model to analyze the interaction influence of adjacent agents on the target agent and determine their interaction relationship. The interaction mask strategy constructed in this embodiment is as follows:

[0096] In the inference process, the target agent obtains the encoding information from the adjacent agents through the self-attention mechanism, which is used for subsequent trajectory decoding.

[0097] Under normal circumstances, the target agent will be allowed to obtain information from any adjacent agent and infer the most suitable trajectory result in the current driving scene.

[0098] To analyze the interaction degree between the target agent and the adjacent agent, the construction strategy of the interaction mask will evaluate the importance of the selected adjacent agent to the trajectory decoding of the target agent in turn, that is, the interaction mask will shield the adjacent agent elements of the target agent in turn, that is, to construct an attention mask, so that the target agent cannot obtain the encoding information from the agent that needs to determine the interaction relationship, but can interact normally with other agents.

[0099] At this time, the encoding features of the target agent fused with complete scene information and the encoding features of the incomplete scene information using the attention mask can be obtained. The above two kinds of encoding features are used to decode the predicted trajectory respectively.

[0100] The trajectory results decoded after adding the mask to the target agent are compared and analyzed with the decoding results without adding the mask, the trajectory similarity is calculated, and the interaction relationship between the target agent and the currently selected other agent is determined.

[0101] In this embodiment, if the difference between the predicted trajectory result after adding the mask and the predicted trajectory result without adding the mask is less than the set threshold, it is determined that the adjacent agent shielded by the interaction mask has less influence on the target agent, and it is determined that there is no interaction behavior between the two;

[0102] If the difference between the predicted trajectory result after adding the mask and the predicted trajectory result without adding the mask is greater than the set threshold, it is determined that the adjacent agent shielded by the interaction mask has a major influence on the future trajectory of the target agent, and it is determined that the target agent will be affected by the adjacent agent.

[0103] Based on the above judgment conditions, the interaction relationship of each adjacent agent will be analyzed for all target agents in turn, and the interaction true value label will be generated; the label will be divided into two categories, the first category is that the target agent is affected by the currently selected agent, and the second category is that the target agent is not affected by the currently selected agent.

[0104] Based on the above strategy, the interaction relationship true value label is generated for all agents in the training data set.

[0105] Secondly, the interaction relationship judgment network is trained using the interaction relationship true value label, and a multi-agent empty interaction relationship graph is constructed.

[0106] In the embodiment, the state information of various elements in the driving scene where the autonomous vehicle is located includes the position, speed, and semantic attribute of the agent element, and the position and semantic attribute of the road map element. A road map element information encoding module and an agent element information encoding module are constructed to respectively perform information encoding and interactive fusion of the road map element and the agent historical trajectory. The information encoding and interactive fusion method is similar to the method of obtaining the interactive true value label described above: by constructing a graph attention neural network, the self-attention mechanism is used to realize the internal information fusion of the agent feature and the road map feature; and the cross-attention mechanism is used to realize the information fusion of the agent feature to the road map feature, so as to obtain the agent embedding encoding of the fused driving scene environment information.

[0107] A multi-agent spatio-temporal interaction graph is constructed. First, the interaction graph is initialized as a fully connected graph, and bidirectional edges are used. The agents in the driving scene will be nodes. The node encoding will be initialized by the agent embedding encoding, the edge information will use the source node embedding encoding, the target node embedding encoding, and the state quantity encoding of the source node relative to the target node to realize the initialization of the edge embedding encoding through the multi-layer perception fusion.

[0108] The multi-layer perception (MLP) and the maximum pooling operation (Max Pooling) are used to process the node and edge state quantities. For the target agent, in order to obtain the relative interaction relationship between the two nodes, the target node and the edge node are spliced, and the multi-layer perception is used to output the interaction relationship classification result. There are three kinds of classification results: the first kind is that the source node affects the target node, the second kind is that the target node affects the source node, and the third kind is that there is no interaction relationship.

[0109] In the training process, the predicted result true value label p t The loss function is calculated, and the loss function uses the three-class extended Focal Loss to solve the problem of uneven distribution of classification results. The specific formula is as follows:

[0110] FL(p t )=-α(1-p t ) γ log(p t )

[0111] Where FL(p t ) is the loss function, γ is the adjustment factor for adjusting the weight of difficult and easy samples, and α is the balance factor for adjusting the weight of each class to prevent the loss of a certain class from being too prominent.

[0112] In the inference process, the interaction relationship graph under the current driving scene is constructed by the interaction relationship inference result.

[0113] Then based on the classification results of the driving scene interaction relationship, the space subsets are divided and the multi-level interaction relationship graph is constructed. Due to the constraints of driving rules and road maps in large-scale driving scenes, the scene range considered by different target intelligent agents during trajectory prediction is different, and the division of subsets can constrain the scene range and reduce the computational complexity. Among them, the driving scene example is shown in Figure 3 .

[0114] Specifically, taking the highest level agent with the largest influence as the base point, the agents with influence probability greater than the threshold are included in the next highest level node, and iteration is performed in turn until the influence probability of the layer is lower than the threshold. Then take the node with the largest influence from the remaining agents as the highest level of the new subset, repeat the above steps until all nodes are added to the relevant subset.

[0115] In order to guarantee the multi-modal of trajectory prediction, a certain number of interaction relationship graphs will be generated.

[0116] Specifically, when the yield and influence probability difference between the two nodes are not large in the interaction relationship determination, it is considered that the existing historical state information cannot accurately determine the interaction relationship between the two, at which time two behavior modes can be derived.

[0117] Considering the multi-modal situation in all subsets, the scene unit is generated by combining a certain number of modal results. The multi-modal scene level multi-level interaction relationship diagram is shown in Figure 4 .

[0118] After obtaining the multi-level interaction relationship graph, it will be used to guide the information interaction and fusion of each element in the driving scene. In this embodiment, the graph attention neural network will be used as the information fusion network. Specifically, the graph attention network edges are screened through the multi-level interaction relationship graph, and invalid edges are removed to simplify the fusion process.

[0119] Based on the graph attention network, the effective adjacent intelligent agent element information and road map element information are fused for each intelligent agent node. The graph attention network will be composed of multi-head attention module, linear module, residual module and gating mechanism. The node original code will be used as Query, and the adjacent node and edge code will be spliced and used as Key and Value vectors through the linear layer (Linear Layer).

[0120] The Query, Key and Value are input into the multi-head attention mechanism, and the Query fuses the features from the adjacent road map elements.

[0121] The updated Query value will be spliced with the original Query value through the residual network and restored to the set feature dimension through the linear layer.

[0122] Then, the updated Query value is processed by using a Sigmoid function to obtain a gating vector g. The gating vector will selectively update the original feature encoding of the target agent in the following manner:

[0123] Y = X + g · (Q - X)

[0124] where X is the original feature encoding of the target agent, Q is the updated Query value, and Y is the final feature vector result obtained by the target agent through the gating mechanism.

[0125] Based on the updated feature vector of the target agent, the trajectory decoding module will perform joint trajectory prediction through the following process. Figure 5

[0126] Specifically, first, the high-level target agent features are input into the trajectory decoder in parallel to obtain the predicted trajectory.

[0127] Second, the predicted trajectory is input into the trajectory encoder to obtain the future trajectory encoding feature information.

[0128] Then, the next level of target agent will fuse the future trajectory encoding feature information of the previous level through the attention mechanism before the trajectory decoding operation.

[0129] Finally, the above operations are repeated until the predicted trajectory of the lowest level of target agent is obtained.

[0130] In this embodiment, the trajectory decoder is composed of a multi-layer perception machine, and the output dimension is T x 4, where T is the prediction period time span, and each timestamp contains 2-dimensional predicted position information and 2-dimensional position uncertainty information.

[0131] In this embodiment, the trajectory encoder is composed of a multi-layer perception machine plus a max-pooling layer, which inputs the predicted trajectory of the agent and outputs the predicted trajectory encoding information with the same dimension as the historical trajectory encoding result.

[0132] After completing joint trajectory prediction for all modal combinations, the final multi-modal trajectory prediction result can be output in units of the current driving scene.

[0133] Exemplary computer programs and readable storage media;

[0134] In addition to the above method and system, the embodiments of the present disclosure can also be a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to perform the trajectory prediction model steps described in the “Exemplary Method” section of the present specification according to various embodiments of the present disclosure.

[0135] ​The above detailed description merely illustrates preferred and possible embodiments of the application, and is not intended to limit the scope of the application. Any equivalent variations or modifications of the above described embodiments are included within the scope of the present application.

Claims

1. A scene-level joint trajectory prediction method based on a multi-level interaction relationship graph, characterized in that, Comprise the following: S1, based on the intelligent car belongs to the driving scene to build multi-agent space-time interaction graph; S2, based on the space-time interaction relationship graph, the space interaction subset is divided, and the multi-level interaction relationship graph is divided based on the interaction level; S3, based on a single subset, realize the information interaction between groups, guide the element state coding and information fusion in the trajectory prediction model; S4, based on the embedded features of the target agent, the trajectory is predicted, and the decoding process is decoded from high level to low level, and the low level agent will decode the trajectory based on the fusion of high level agent features; S5, generate scene-level multi-modal prediction results; The implementation of S1 includes: S1.1, build an interaction relationship judgment model based on the interaction mask strategy, which is used to generate the interaction true value label of the agent on the training data set; S1.2, based on the interaction true value label, train the interaction relationship judgment model, build the multi-agent space-time interaction relationship graph, and predict the relationship between target agents; The implementation of S1.1 includes: S1.1.1 build a neural network model that realizes trajectory prediction function, as follows: The model will comprehensively process the road map elements and agent elements in the driving scene, and output the trajectory prediction result of the target agent in the fixed period; The process includes agent, road map element data preprocessing process, agent-road map element interaction coding process, agent-agent interaction coding process, and prediction trajectory decoding process; In order to guarantee the integrity of agent-agent interaction, self-attention mechanism will be used to fully integrate the coding information of all effective agents in the driving scene for each target agent; Complete the training of the above neural network trajectory prediction model on the training data set, and obtain the training weight; S1.1.2 build an interaction relationship judgment model based on the interaction mask strategy, analyze the interaction influence of adjacent agents on the target agent, judge the interaction relationship between them, as follows: The target agent will obtain the coding information from the adjacent agents through attention mechanism, which will be used for subsequent trajectory decoding; Using the interaction mask strategy will shield the adjacent agent elements of the target agent in turn, that is, an attention mask is built, so that the target agent cannot obtain coding information from the agents whose interaction relationship needs to be judged, but at the same time, it can interact with other agents normally; Compare the trajectory results of the target agent decoded after adding the mask with the decoding results without adding the mask, so as to judge the interaction relationship between the target agent and the currently selected other agents, and generate the interaction true value label; The label will be divided into two categories, the first category is that the target agent is affected by the currently selected agent, and the second category is that the target agent is not affected by the currently selected agent; S1.1.3 based on the above interaction mask strategy, generate the interaction relationship true value label for all agents in the training data set.

2. The method of claim 1, wherein, The S1.2 includes: Get the state information of various elements in the driving scene of the autonomous vehicle, including the position, speed and semantic attribute of the agent element, the position and semantic attribute of the road map element; The road map element coding module and the agent element coding module are constructed to code and interactively fuse the information of the road map elements and the historical trajectories of the agents; A graph attention neural network is constructed to realize the fusion of the agent features and the road map element features through a cross-attention mechanism to obtain agent embedding coding of the fused driving scene environment information; A full connection graph of agent elements is constructed, each agent acts as a node of the interaction graph, the node edges are bidirectional edges, and the node coding is initialized by using the agent embedding coding; The edge information is initialized by using the source node embedding coding, the target node embedding coding, and the state quantity coding of the source node relative to the target node through a multi-layer perception machine; For the target agent, the target node and the edge node are spliced, and the interaction relationship classification result is output through a multi-layer perception machine, and the classification result has three kinds, the first kind is that the source node influences the target node, the second kind is that the target node influences the source node, and the third kind is that there is no interaction relationship.

3. The method of claim 2, wherein, The S2 includes the following: S2.1, obtaining the relative relationship between the agents in the driving scene through S1, that is, including the source node influencing the target node, the target node influencing the source node, and the non-interaction relationship; S2.2, dividing a spatial interaction subset based on the relative relationship between the agents, including: Clustering the agents with interaction relationship in the current driving scene, and the agent with interaction relationship is the relationship between the source node and the target node which is not the non-interaction relationship; The above clustering operation will aggregate all the mutually influencing agents in the driving scene, and divide the driving scene into several subsets, each subset will independently process the internal interaction behavior, effectively reducing the scene size; After the preliminary initialization of the subset, for the larger subset in the current scene, secondary segmentation will be performed, that is, the single interaction part of the current subset agent is segmented through the interaction relationship probability result, so as to further reduce the scene size of the subset without affecting the conditional prediction, and improve the operation efficiency; S2.3, constructing a multi-level interaction relationship graph based on the relative relationship between the nodes in the subset, including: In this link, each subset in the current driving scene will be constructed as an independent scene to construct a multi-level interaction relationship graph; For each subset, the internal hierarchical division is that the lower level is influenced by the higher level, that is, the lower level will be influenced by the higher level agent; First, filter the agent with the highest interaction relationship in itself, that is, the target agent not influenced by other agents, as the highest level; Secondly, based on the highest level agent, the next level agent influenced by the current level is found in turn and used as the next level; Filtering in turn until there is no next level agent influenced by the current level agent, the search is ended; Based on the above operation, a multi-level interaction relationship graph is constructed in each subset.

4. The method of claim 3, wherein, The implementation of S3 includes: The multi-level interaction relationship graph in each subset obtained through S2 is used to construct the edge of the graph attention neural network. The information coding and interaction in the subset coverage scenario is guided based on a graph attention neural network, i.e., a low-level agent improves the weight of embedding coding of a high-level agent connected thereto, and coding information is preferentially obtained from the agent, thereby improving the efficiency of information fusion and reducing the calculation requirement; In addition, in order to better fuse global information, each subset will generate global features for information fusion of each target agent in each subset, and by obtaining the global features in the single subset coverage scenario, the embedding feature values of the target agent are updated based on the subset features and the features of adjacent scene elements, thereby realizing information interaction and fusion among multiple subsets in the overall driving scenario.

5. The method of claim 4, wherein, The implementation of the S4 includes: According to the multi-level interaction relationship graph in S2 and the fusion of the adjacent level agent target agent coding result in S3; the current level is initialized as the highest level, and the trajectory prediction of the agent belonging to the current level is performed through the trajectory decoding module, and the prediction result is output; After completing the trajectory decoding of the agent belonging to the current level, the trajectory prediction result of each agent under the current level is coded, and the coded features are saved, and the coded features are stored by using the multi-level interaction relationship graph; The trajectory prediction result of the next level is decoded, and the trajectory prediction result coding features of the agents in the current level are considered in the decoding process, so as to guarantee the scene consistency between the prediction result of the next level and the prediction result of the current level; The above steps are repeated until decoding to the lowest level.

6. The method of claim 1, wherein, The implementation of the S5 includes: generating a specific number of groups of interaction relationship graphs by using S2, i.e., multiple groups of interaction relationship graphs are generated for each subset, and each group of interaction relationship graphs represents a driving behavior mode, and each group of interaction relationship graphs serves a single scene level mode.

7. A scene-level joint trajectory prediction system based on multi-level interaction graph, characterized in that, It includes: The space-time relationship graph acquisition module is used to realize the content of step S1 in claim 1; The relationship reasoning module is used to realize the content of step S2 in claim 1; The information fusion module is used to realize the content of step S3 in claim 1; The trajectory prediction module is used to realize the content of steps S4 and S5 in claim 1. 8.A storage medium storing a scene-level joint trajectory prediction based on a multi-level interaction graph, characterized in that, The storage medium stores a computer program for executing the prediction method of any one of claims 1-6. The storage medium stores a computer program for executing the prediction method of any one of claims 1-6.

Images