A trajectory prediction method and device for two-way dynamic map and intelligent agent interaction
Through the trajectory prediction method of the interaction between two-way dynamic maps and agents, combined with the dynamic feature update mechanism, the limitations of static maps and one-way interaction in the existing technology are solved, and more accurate and flexible trajectory prediction is achieved to adapt to complex traffic environments.
Patent Information
- Application Number
- CN202410236795.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-01
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-03-01
AI Technical Summary
The existing autonomous driving trajectory prediction technology relies on static maps and cannot effectively utilize environmental dynamic information and the feedback from agents to the environment, resulting in limited prediction accuracy in complex and dynamically changing urban traffic environments.
The trajectory prediction method of interaction between two-way dynamic maps and agents is adopted. Through the agent encoder, map encoder, multiple bidirectional interactive modules and decoders, combined with long-term memory networks and spatial graph convolution algorithms, a dynamic feature update mechanism is designed. Including traffic flow prediction, agent path prediction and road node feature reconstruction, enhancing the model's understanding of the environment.
It improves the accuracy and dynamic nature of trajectory prediction, can effectively capture the two-way interaction between the agent and the map, adapt to complex and changeable real-world scenarios, and improves the prediction performance in a dynamic environment.
Smart Images

Figure CN118135503B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a trajectory prediction method and device for interaction between a two-way dynamic map and an intelligent agent. Background Art
[0002] In the field of autonomous driving, trajectory prediction is one of the key technologies, and its goal is to predict the future trajectory of vehicles or pedestrians based on historical information. With the development of deep learning and neural networks, trajectory prediction technology has made significant progress. In particular, in terms of utilizing environmental maps and agent (vehicle, pedestrian) information, existing methods are committed to understanding and predicting behavior more accurately and dynamically.
[0003] The current classic framework for trajectory prediction is to encode the agent's historical trajectory and map information, then interact with the map, the agent considers self-interaction, and finally connects to the trajectory decoding head. Moreover, the map-agent interaction part is often one-way, that is, only the influence of the map on the agent is considered. These trajectory prediction technologies mainly focus on the effect of static maps on agents. These methods usually cannot fully utilize the long-term influence of potential information in the map on the behavior of the agent. Although there are some attempts to improve prediction by adding an environmental perception layer, there are still problems of insufficient adaptability to dynamic environments and low computational efficiency.
[0004] There are two key limitations in existing technologies for trajectory prediction in the field of autonomous driving that significantly affect their performance and scope of application:
[0005] 1. Ignoring the impact of agents on the environment: Existing technologies mainly focus on predicting the behavior of agents, but often ignore the impact of agents on their surroundings. In practical applications, the behavior of agents (such as vehicles and pedestrians) is not only affected by their own characteristics, but also significantly affected by the surrounding environment. For example, the behavior of a pedestrian may be changed by the actions of nearby vehicles, and vice versa. Current prediction models are usually unable to effectively capture this complex interaction.
[0006] 2. Insufficient dynamic updating of maps: Existing trajectory prediction frameworks usually do not consider the update of map status, which limits the effectiveness of these methods in dynamic environments. Urban environments and traffic conditions are constantly changing, including factors such as emergencies, road construction, and temporary traffic control. These dynamic changes are often not reflected in static maps. Therefore, trajectory prediction methods that rely on static maps cannot accurately capture these changes, resulting in a significant reduction in the reliability of the prediction. Summary of the invention
[0007] In order to solve the technical problems that the existing autonomous driving trajectory prediction technology mainly relies on static maps and simple intelligent agent interactions, fails to effectively utilize the dynamic information of the environment, ignores the impact of the intelligent agent on the surrounding environment and the feedback of the environment on the intelligent agent's behavior, and has limited prediction accuracy in complex and dynamically changing urban traffic environments, for example, intelligent agents with similar histories have very different decisions in the same map but in different traffic congestion environments, and the existing models cannot capture this. An embodiment of the present invention provides a trajectory prediction method and device for two-way dynamic map and intelligent agent interaction. The technical solution is as follows:
[0008] On the one hand, a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent is provided, the method being implemented by a trajectory prediction device, the method comprising:
[0009] S1. Obtain the trajectory of the intelligent agent to be predicted and the map data.
[0010] S2. Input the agent trajectory and map data into the constructed two-way dynamic map and the trajectory prediction model of the agent interaction.
[0011] Among them, the trajectory prediction model includes: an agent encoder, a map encoder, multiple two-way interactive modules, an agent decoder and a map decoder.
[0012] S3. According to the agent trajectory, map data and trajectory prediction model, the future trajectory prediction result of the agent is obtained.
[0013] Optionally, the process of constructing the trajectory prediction model of the two-way dynamic map and the intelligent agent interaction in S2 includes:
[0014] S201, obtaining the vectorized agent historical trajectory and vectorized historical map data.
[0015] S202. Encode the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encode the vectorized historical map data according to the map encoder to obtain map features.
[0016] S203, inputting the agent features and the map features into the first-stage two-way interaction module to obtain the first-stage agent features and the first-stage map features.
[0017] S204, performing self-interaction on the first-stage agent features and the first-stage map features respectively to obtain the first-stage self-interaction agent features and the first-stage self-interaction map features.
[0018] S205, inputting the first-stage self-interactive intelligent agent features and the first-stage self-interactive map features into the second-stage two-way interactive module to obtain the second-stage intelligent agent features and the second-stage map features.
[0019] S206: Obtain a hidden layer map state prediction result according to the second-stage map features and the hidden layer map decoder of the map encoder.
[0020] S207, using multiple independent multi-layer perceptrons MLP to perform agent feature differentiation on the second-stage agent features, and obtain the second-stage agent features after feature differentiation.
[0021] S208, splicing the second-stage intelligent agent features after feature differentiation, and inputting the spliced intelligent agent features and the second-stage map features into the third-stage two-way interactive module to obtain the third-stage intelligent agent features and the third-stage map features.
[0022] S209: Obtain the trajectory prediction result of the intelligent agent based on the characteristics of the intelligent agent in the third stage and the intelligent agent decoder.
[0023] S210, obtaining a map state prediction result of an output layer according to the map features of the third stage and the output layer map decoder of the map encoder.
[0024] Optionally, in S202, encoding the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encoding the vectorized historical map data according to the map encoder to obtain map features include:
[0025] The long short-term memory network LSTM is used to encode the vectorized agent history trajectory to obtain the agent features.
[0026] The lane graph convolution operator LaneConv is used to encode and vectorize historical map data to obtain map features.
[0027] Optionally, the process of constructing the trajectory prediction model of the two-way dynamic map and the agent interaction in S2 further includes:
[0028] Design a dynamic feature update mechanism to train the trajectory prediction model.
[0029] Designing a dynamic feature update mechanism includes: designing auxiliary tasks related to the map node status.
[0030] Auxiliary tasks related to the map node status include: traffic flow prediction, agent path prediction, and feature reconstruction of road nodes.
[0031] Traffic flow prediction, including predicting the flow of agents through each map node at a specific time interval in the future.
[0032] Agent path prediction includes: each map node predicts whether a specific agent will pass through the map node within a specific time interval in the future.
[0033] The feature reconstruction of the road node includes: erasing some features of the map node features, and reconstructing the erased features through the trajectory prediction model.
[0034] Optionally, multiple two-way interaction modules include: a two-way interaction mechanism.
[0035] The implementation method of the two-way interaction mechanism includes:
[0036] The bidirectional interaction mechanism is realized by adopting the extended spatial graph convolution method shown in the following formula (1):
[0037]
[0038] In the formula, a′ i represents the i-th agent feature after convolution, Represents the graph convolution calculation method, a i represents the i-th agent feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, Δd ij represents the characteristics of the center and its neighbors, represents the map node neighbor set of the ith agent, m′ j represents the j-th map node feature after convolution, Represents the set of all agent neighbors of the j-th map node.
[0039] Optionally, the step of inputting the agent features and the map features into the first-stage two-way interaction module in S203 to obtain the first-stage agent features and the first-stage map features includes:
[0040] S2031. Obtain the neighbor pairing relationship between the intelligent agent and the map node, calculate the Euclidean distance of each neighbor pairing according to the neighbor pairing relationship, and perform dimension upgrading on the Euclidean distance through a multi-layer perceptron to obtain a distance feature.
[0041] S2032. Concatenate the agent features, distance features, and map features to obtain concatenated features.
[0042] S2033. According to the concatenated features and the multi-layer perceptron, updated agent features and weights of neighbor pairings of each map node are obtained.
[0043] S2034. According to the concatenated features and the multi-layer perceptron, updated map node features and weights of each agent neighbor pairing are obtained.
[0044] S2035. Perform point multiplication and addition on the updated agent features and the weights of the corresponding agent neighbor pairings to obtain the agent features after point multiplication and addition, and perform residual connection on the agent features and the agent features after point multiplication and addition to obtain the first-stage agent features.
[0045] S2036. Perform dot product addition on the updated map node features and the weights of the corresponding map node neighbor pairs to obtain the dot product added map node features, and perform residual connection on the map features and the dot product added map node features to obtain the first stage map features.
[0046] Optionally, the loss function of the trajectory prediction model of the two-way dynamic map and the agent interaction is as shown in the following formula (2):
[0047]
[0048] In the formula, Loss represents the loss function of the trajectory prediction model, Loss traj Represents the trajectory prediction loss function, Loss map represents the map state prediction loss function, α, β, ε are hyperparameters for adjusting the weights of the learning objectives of different auxiliary tasks, and Loss flow Represents the traffic flow prediction loss function, Loss path Represents the agent path prediction loss function, Loss fcon represents the feature reconstruction loss function of the road node, M represents the number of observations or samples in the dataset, and α c represents the weighting factor of category c, m i represents the classification result of the i-th map node, γ represents the focusing hyperparameter that reduces the relative loss of well-classified examples, and y i represents the true label of the i-th map node, σ represents the sigmoid function, represents the classification result output of the i-th map node, M′ represents the total number of erased map nodes, and f mask represents the set of node features to be erased, Indicates the real feature labels that are erased.
[0049] On the other hand, a trajectory prediction device for interaction between a two-way dynamic map and an intelligent agent is provided, and the device is applied to a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent, and the device comprises:
[0050] The acquisition module is used to obtain the trajectory of the intelligent agent to be predicted and the map data.
[0051] The input module is used to input the agent trajectory and map data into the trajectory prediction model of the constructed two-way dynamic map and agent interaction.
[0052] Among them, the trajectory prediction model includes: an agent encoder, a map encoder, multiple two-way interactive modules, an agent decoder and a map decoder.
[0053] The output module is used to obtain the future trajectory prediction results of the intelligent agent based on the intelligent agent trajectory, map data and trajectory prediction model.
[0054] Optionally, the input module is further configured to:
[0055] S201, obtaining the vectorized agent historical trajectory and vectorized historical map data.
[0056] S202. Encode the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encode the vectorized historical map data according to the map encoder to obtain map features.
[0057] S203, inputting the agent features and the map features into the first-stage two-way interaction module to obtain the first-stage agent features and the first-stage map features.
[0058] S204, performing self-interaction on the first-stage agent features and the first-stage map features respectively to obtain the first-stage self-interaction agent features and the first-stage self-interaction map features.
[0059] S205, inputting the first-stage self-interactive intelligent agent features and the first-stage self-interactive map features into the second-stage two-way interactive module to obtain the second-stage intelligent agent features and the second-stage map features.
[0060] S206: Obtain a hidden layer map state prediction result according to the second-stage map features and the hidden layer map decoder of the map encoder.
[0061] S207, using multiple independent multi-layer perceptrons MLP to perform agent feature differentiation on the second-stage agent features, and obtain the second-stage agent features after feature differentiation.
[0062] S208, splicing the second-stage intelligent agent features after feature differentiation, and inputting the spliced intelligent agent features and the second-stage map features into the third-stage two-way interactive module to obtain the third-stage intelligent agent features and the third-stage map features.
[0063] S209: Obtain the trajectory prediction result of the intelligent agent based on the characteristics of the intelligent agent in the third stage and the intelligent agent decoder.
[0064] S210, obtaining a map state prediction result of an output layer according to the map features of the third stage and the output layer map decoder of the map encoder.
[0065] Optionally, the input module is further configured to:
[0066] The long short-term memory network LSTM is used to encode the vectorized agent history trajectory to obtain the agent features.
[0067] The lane graph convolution operator LaneConv is used to encode and vectorize historical map data to obtain map features.
[0068] Optionally, the input module is further configured to:
[0069] Design a dynamic feature update mechanism to train the trajectory prediction model.
[0070] Designing a dynamic feature update mechanism includes: designing auxiliary tasks related to the map node status.
[0071] Auxiliary tasks related to the map node status include: traffic flow prediction, agent path prediction, and feature reconstruction of road nodes.
[0072] Traffic flow prediction, including predicting the flow of agents through each map node at a specific time interval in the future.
[0073] Agent path prediction includes: each map node predicts whether a specific agent will pass through the map node within a specific time interval in the future.
[0074] The feature reconstruction of the road node includes: erasing some features of the map node features, and reconstructing the erased features through the trajectory prediction model.
[0075] Optionally, multiple two-way interaction modules include: a two-way interaction mechanism.
[0076] The implementation method of the two-way interaction mechanism includes:
[0077] The bidirectional interaction mechanism is realized by adopting the extended spatial graph convolution method shown in the following formula (1):
[0078]
[0079] In the formula, a′ i represents the i-th agent feature after convolution, Represents the graph convolution calculation method, a i represents the i-th agent feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, Δd ij represents the characteristics of the center and its neighbors, represents the map node neighbor set of the ith agent, m′ j represents the j-th map node feature after convolution, Represents the set of all agent neighbors of the j-th map node.
[0080] Optionally, the input module is further configured to:
[0081] S2031. Obtain the neighbor pairing relationship between the intelligent agent and the map node, calculate the Euclidean distance of each neighbor pairing according to the neighbor pairing relationship, and perform dimension upgrading on the Euclidean distance through a multi-layer perceptron to obtain a distance feature.
[0082] S2032. Concatenate the agent features, distance features, and map features to obtain concatenated features.
[0083] S2033. According to the concatenated features and the multi-layer perceptron, updated agent features and weights of neighbor pairings of each map node are obtained.
[0084] S2034. According to the concatenated features and the multi-layer perceptron, updated map node features and weights of each agent neighbor pairing are obtained.
[0085] S2035. Perform point multiplication and addition on the updated agent features and the weights of the corresponding agent neighbor pairings to obtain the agent features after point multiplication and addition, and perform residual connection on the agent features and the agent features after point multiplication and addition to obtain the first-stage agent features.
[0086] S2036. Perform dot product addition on the updated map node features and the weights of the corresponding map node neighbor pairs to obtain the dot product added map node features, and perform residual connection on the map features and the dot product added map node features to obtain the first stage map features.
[0087] Optionally, the loss function of the trajectory prediction model of the two-way dynamic map and the agent interaction is as shown in the following formula (2):
[0088]
[0089] In the formula, Loss represents the loss function of the trajectory prediction model, Loss traj Represents the trajectory prediction loss function, Loss map represents the map state prediction loss function, α, β, ε are hyperparameters for adjusting the weights of the learning objectives of different auxiliary tasks, and Loss flow Represents the traffic flow prediction loss function, Loss path Represents the agent path prediction loss function, Loss fcon represents the feature reconstruction loss function of the road node, M represents the number of observations or samples in the dataset, and α c represents the weighting factor of category c, m irepresents the classification result of the i-th map node, γ represents the focusing hyperparameter that reduces the relative loss of well-classified examples, and y i represents the true label of the i-th map node, σ represents the sigmoid function, represents the classification result output of the i-th map node, M′ represents the total number of erased map nodes, and f mask represents the set of node features to be erased, Indicates the real feature labels that are erased.
[0090] On the other hand, a trajectory prediction device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the trajectory prediction methods for interaction between a two-way dynamic map and an intelligent agent is implemented.
[0091] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned trajectory prediction methods for interaction between a bidirectional dynamic map and an intelligent agent.
[0092] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0093] In the embodiment of the present invention, the proposed two-way dynamic map and agent interaction trajectory prediction method DyMAI adopts a two-way interaction mechanism in the map and agent interaction part, effectively combines map information and agent behavior, and improves the accuracy and dynamics of trajectory prediction. A dynamic feature update mechanism is added to the model to cope with complex and changeable real-world scenarios. The present invention introduces additional supervisory signals such as traffic flow, agent path, and road node feature reconstruction to assist in the update and evolution of map features, thereby enhancing the model's in-depth understanding of the environment. This method brings new challenges and possibilities to the concept of static maps in the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0094] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0095] Figure 1 It is a flow chart of a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent provided by an embodiment of the present invention;
[0096] Figure 2It is a network framework diagram of a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent provided by an embodiment of the present invention;
[0097] Figure 3 It is a trajectory prediction flow chart of a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent provided by an embodiment of the present invention;
[0098] Figure 4 is a framework diagram of a two-way interaction mechanism provided by an embodiment of the present invention;
[0099] Figure 5 is a flow chart of a two-way interaction mechanism provided by an embodiment of the present invention;
[0100] Figure 6 It is a block diagram of a trajectory prediction device for interaction between a two-way dynamic map and an intelligent agent provided by an embodiment of the present invention;
[0101] Figure 7 It is a structural schematic diagram of a trajectory prediction device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0102] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0103] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0104] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0105] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.
[0106] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0107] The embodiment of the present invention provides a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent, which can be implemented by a trajectory prediction device, which can be a terminal or a server. Figure 1 The process flow chart of the trajectory prediction method for the interaction between the two-way dynamic map and the intelligent agent shown in the figure may include the following steps:
[0108] S1. Obtain the trajectory of the intelligent agent to be predicted and the map data.
[0109] S2. Input the agent trajectory and map data into the constructed two-way dynamic map and the trajectory prediction model of the agent interaction.
[0110] Among them, the trajectory prediction model includes: an agent encoder, a map encoder, multiple two-way interactive modules, an agent decoder and a map decoder.
[0111] Alternatively, if Figure 2 , 3 As shown, the process of constructing the trajectory prediction model of the two-way dynamic map and the intelligent agent interaction in S2 may include the following steps S201-S210:
[0112] S201, obtaining the vectorized agent historical trajectory and vectorized historical map data.
[0113] S202. Use LSTM (Long Short-term Memory Networks) to encode the vectorized agent history trajectory to obtain agent features. Use lane graph convolution operator LaneConv to encode the vectorized historical map data to obtain map features. Both the trajectory prediction head and the map state prediction head are decoded using a multi-layer perceptron.
[0114] S203, inputting the agent features and the map features into the first-stage two-way interactive module, extracting the dynamic map features using the two-way interactive network, and obtaining the first-stage agent features and the first-stage map features.
[0115] In a feasible implementation, the first stage embeds basic information between the agent and the map through two-way interaction.
[0116] Alternatively, if Figure 4 , 5 As shown, the above step S23 may include the following steps S231-S237:
[0117] S2031, agent features and map node features pass through the fully connected layer respectively.
[0118] S2032. Obtain the neighbor pairing relationship between the intelligent agent and the map node, calculate the Euclidean distance of each neighbor pairing according to the neighbor pairing relationship, and perform dimension upgrading on the Euclidean distance through a multi-layer perceptron to obtain a distance feature.
[0119] In a feasible implementation, the neighbor pairing relationship between the agent and the map node is first found by distance, the Euclidean distance of each pairing is calculated, and the obtained distance is dimensionally upgraded through a multi-layer perceptron.
[0120] S2033, performing feature splicing on the intelligent agent features, the distance features and the map features to obtain spliced features.
[0121] S2034. According to the concatenated features and the multi-layer perceptron, updated agent features and weights of neighbor pairings of each map node are obtained.
[0122] S2035. Based on the concatenated features and the multi-layer perceptron, the updated map node features and the weights of each agent neighbor pairing are obtained.
[0123] S2036. Perform point multiplication and addition on the updated agent features and the weights of the corresponding agent neighbor pairings to obtain the agent features after point multiplication and addition, and perform residual connection on the agent features and the agent features after point multiplication and addition to obtain the first-stage agent features.
[0124] S2037. Perform dot product addition on the updated map node features and the weights of the corresponding map node neighbor pairs to obtain the dot product added map node features, and perform residual connection on the map features and the dot product added map node features to obtain the first stage map features.
[0125] Optionally, multiple two-way interactive modules include: a two-way interactive mechanism.
[0126] The implementation method of the two-way interaction mechanism includes:
[0127] Graph convolution is used to implement a two-way interaction mechanism between the map and the agent, but the present invention extends the convolution process to allow the agent and the map nodes to convolve with each other as neighbors, thereby simultaneously updating the features of the agent and the map nodes.
[0128] The commonly used spatial graph convolution method can be expressed as:
[0129]
[0130] Among them, a i represents the central feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, represents the set of map node neighbors of the ith agent, Δd ijRepresents the features of the center and its neighbors. This convolution algorithm only considers the impact of the neighbor feature m on the center feature a.
[0131] On this basis, the present invention extends the convolution process, taking into account the bidirectional influence of feature m and feature a, and allowing both to be neighbors of each other, such as Figure 3 That is, the bidirectional interaction mechanism is realized by adopting the extended spatial graph convolution method shown in the following formula (2):
[0132]
[0133] In the formula, a′ i represents the i-th agent feature after convolution, Represents the graph convolution calculation method, a i represents the i-th agent feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, Δd ij represents the characteristics of the center and its neighbors, represents the map node neighbor set of the ith agent, m′ j represents the j-th map node feature after convolution, Represents the set of all agent neighbors of the j-th map node.
[0134] In the present invention, the influence of the map on the agent and the influence of the agent on the map are considered to be highly symmetrical, so the set of {neighbor-center} between them is also symmetrical, so they can share the distance Δd ij In addition, due to this symmetry, except for the residual connection, the present invention allows the two convolution processes to share all calculation parameters:
[0135]
[0136] Where W is the fully connected layer, is a multilayer perceptron, ⊙ represents a dot product, and the two convolution processes share all computational parameters except for W3 and W5 used in the residual connection. In addition, the k-state features of the agent in the third stage are connected and interact with the map in parallel. This method reduces the computational load and avoids the impact caused by the different order of interaction between different modalities and the map.
[0137] S204, performing self-interaction on the first-stage agent features and the first-stage map features respectively to obtain the first-stage self-interaction agent features and the first-stage self-interaction map features.
[0138] In one feasible implementation, the self-interaction between agents uses a one-way graph convolution, which is the "common spatial graph convolution method" mentioned in the two-way interaction module. That is, there is no two-way process, and the expression is as follows:
[0139]
[0140] The self-interaction of map nodes is implemented using LaneConv, which uses the same graph convolution operator as the map encoder.
[0141] S205, inputting the first-stage self-interactive intelligent agent features and the first-stage self-interactive map features into the second-stage two-way interactive module to obtain the second-stage intelligent agent features and the second-stage map features.
[0142] In one possible implementation, agents consider social interactions with each other, while lane nodes consider information transfer from nearby lane nodes. At this point, both agents and map nodes are aware of a wider range of potential future states, which prompts the second stage of two-way interaction. At this point, both the agent's features and the map node's features have enhanced predictive power.
[0143] S206: Obtain a hidden layer map state prediction result according to the second-stage map features and the hidden layer map decoder of the map encoder.
[0144] S207, using multiple independent multi-layer perceptrons MLP to perform agent feature differentiation on the second-stage agent features, and obtain the second-stage agent features after feature differentiation.
[0145] In one feasible implementation, k MLPs (Multilayer Perceptrons) are used to branch out multiple possibilities for the agent.
[0146] S208, splicing the second-stage intelligent agent features after feature differentiation, and inputting the spliced intelligent agent features and the second-stage map features into the third-stage two-way interactive module to obtain the third-stage intelligent agent features and the third-stage map features.
[0147] In one feasible implementation, the third stage performs a two-way interaction on the map for each modality's agent features.
[0148] S209: Obtain the trajectory prediction result of the intelligent agent based on the characteristics of the intelligent agent in the third stage and the intelligent agent decoder.
[0149] S210, obtaining a map state prediction result of an output layer according to the map features of the third stage and the output layer map decoder of the map encoder.
[0150] In one feasible implementation, a three-stage bidirectional interaction mechanism dynamically evolves the features of the agent and the map, ensuring that each time the agent uses environmental features, both the agent and the map are dynamic entities, so as to extract dynamic features that are more suitable for complex driving scenarios for trajectory prediction.
[0151] Optionally, the process of constructing the trajectory prediction model of the two-way dynamic map and the agent interaction in S2 further includes:
[0152] Design a dynamic feature update mechanism to train the trajectory prediction model.
[0153] Designing a dynamic feature update mechanism includes: designing auxiliary tasks related to the map node status.
[0154] In one possible implementation, in order to enhance the dynamics of map updates, the present invention incorporates auxiliary tasks that are intended to supervise and guide the evolution of map features. These tasks are strategically aligned with trajectory prediction relevance. These auxiliary tasks include: traffic flow prediction, agent path prediction, and feature reconstruction of road nodes.
[0155] Traffic flow prediction, including: predicting the vehicle flow through each map node at a specific time interval in the future.
[0156] In a feasible implementation, the present invention predicts the traffic flow of each map node within a future time interval, enriches node features with dynamic, time-related information, and directly affects trajectory prediction. This includes using the vehicle flow through each map node as a label for supervised learning, corresponding to the same future time step as the trajectory prediction. Taking the Argoverse1 dataset as an example, the goal of the present invention is to predict the trajectory of the agent in the next three seconds. Therefore, the present invention counts the future paths (ground truth) of all agents passing through each node in the scene. If the center of an agent appears within two meters of the center of a node at any time in the future, the node is considered to have an agent "passing through".
[0157] Furthermore, considering the short prediction time interval and low traffic flow, different flow values may have completely different effects on trajectory prediction, so the present invention designs traffic flow prediction as a classification task. Here, zero flow is classified as class 0, and each additional agent constitutes another class with class number +1. This design ensures that numerically close but different node flows are spaced apart in the feature space, allowing the model to better exploit this dynamic property.
[0158] Agent path prediction includes: each map node predicts whether a specific agent will pass through the map node within a specific time interval in the future.
[0159] In one possible implementation, similar to the traffic flow prediction task of identifying the number of agents that will flow through each map node, the agent path prediction task seeks to discern which specific agents are present. Each node predicts whether a specific agent will pass through it within a specific time interval in the future, thereby encouraging the model to learn more sophisticated, agent-specific interaction patterns. This is achieved by assigning a binary code to each agent in the scene, and each map node predicts this binary code.
[0160] For example, if there are three agents in the scene, they can correspond to three binary codes, 001, 010, and 100. Then each map node outputs a three-bit binary code. If agent 001 and agent 100 pass through a node at the same time, the output of the node is 101. If no agent passes, the node outputs 000.
[0161] The feature reconstruction of the road node includes: erasing some features of the map node features, and reconstructing the erased features through the trajectory prediction model.
[0162] In a feasible implementation, in this work, the present invention also incorporates the inherent static features of the map into the scope of additional supervision. Specifically, the present invention erases a part of the map node features input to the network, and then lets the network reconstruct this part. Here, the present invention selects the directional features of the map nodes for erasing and prediction, aiming to enable the model to better mine the potential features of the map nodes, while revealing the impact of these potential features on the movement of the agent through a two-way interactive process. Specifically, in this auxiliary task, the present invention hopes that the model learns the strong connection between the moving direction of the agent and the inherent direction of the map node.
[0163] The present invention specifically designs auxiliary tasks that are strongly related to trajectory prediction and introduces relevant additional supervision labels to ensure that the model meaningfully updates the map through a two-way interactive process. This enables the model to explore the potential impact of information embedded in the map on trajectory prediction.
[0164] Optionally, the learning objectives of the present invention include trajectory regression and dynamic evolution of the map, so the learning objective of the model is the following formula (5):
[0165] Loss=Loss traj +Loss map (5)
[0166] Loss function of trajectory prediction part traj Including trajectory regression and score prediction, this part can use the conventional trajectory regression loss function (such as smooth L1, etc.).
[0167] Furthermore, the loss function Loss of the map state prediction part of the present invention is map Contains the losses of the three auxiliary tasks mentioned above: Loss map =αLoss flow +βLoss path +εLoss fcon In the present invention, α=10, β=0.3, and ε=1 are set. The specific settings will vary depending on the application.
[0168] 1) Traffic flow loss: In the time period that needs to be predicted, there are many nodes in the scene that have no intelligent agents passing through, that is, class 0, which is considered as the background class. The present invention needs to focus on the non-zero class, so it is suitable to use focal loss, that is, to pay more attention to the difficult-to-classify parts.
[0169]
[0170] Among them, m i represents the classification result of the i-th map node, α c It is the weighting factor of category c. The present invention sets the background category (labal 0) to 0.2 and the other categories to 1. In the case of 10 categories, γ is a focusing hyperparameter that reduces the relative loss of well-classified examples, and pays more attention to erroneous examples that are difficult to classify. M is the number of observations or samples in the dataset.
[0171] 2) Agent Path Loss: For this task, the present invention uses binary cross entropy loss BCEWith Logits Loss with logarithmic probability to learn binary encoding, which is very suitable for multi-label classification problems and helps to distinguish between individual agents at each node.
[0172]
[0173] in The classification result output of the i-th map node, y i is the true label, a binary code representing the number of all agents in the scene, a specific binary position corresponds to a specific agent number, and σ is the sigmoid function.
[0174] 3) Feature reconstruction Loss: The feature reconstruction uses the MSE loss function as follows:
[0175]
[0176] Where M′ is the total number of map nodes erased, f mask represents the set of node features to be erased, is the true feature label.
[0177] S3. According to the intelligent agent trajectory, map data and trajectory prediction model, the trajectory prediction result of the two-way dynamic map and the intelligent agent interaction is obtained.
[0178] Experiments show that the DyMAI of the present invention has achieved state-of-the-art performance on both Argoverse1 and Argoverse2 (hereinafter referred to as AV1 and AV2) data benchmarks. Through a two-way interactive mechanism, the agent and map node features can update each other, which not only captures the current state of the environment, but also reveals how the potential information in the map affects future agent behavior. The present invention obtains brier-minFDE=1.677 and minFDE=1.09 meters on the AV1 dataset test set. On the AV1 dataset test set, brier-minFDE=1.79 and minFDE=1.22 meters are obtained, and in terms of the official ranking indicator brier-minFDE, the AV1 dataset has achieved the best results of the current trajectory prediction algorithm, and the AV2 dataset is highly competitive with the latest results.
[0179] In view of the problems of existing technologies, such as ignoring the impact of intelligent agents on the environment and insufficient dynamic updating of maps, DyMAI effectively solves these problems by adopting a two-way interaction mechanism and a combination of auxiliary tasks. It can not only capture and utilize the dynamic information of the map, but also consider the impact of the intelligent agent on the environment and its feedback. This method not only improves the accuracy of trajectory prediction, but also enables the model to adapt to complex and dynamically changing real-world environments. The innovation of DyMAI lies in its ability to dynamically integrate the interaction between the environment and the intelligent agent, providing a new perspective to understand and predict the complex dynamics in the autonomous driving environment.
[0180] In the embodiment of the present invention, the proposed two-way dynamic map and agent interaction trajectory prediction method DyMAI adopts a two-way interaction mechanism in the map and agent interaction part, effectively combines map information and agent behavior, and improves the accuracy and dynamics of trajectory prediction. A dynamic feature update mechanism is added to the model to cope with complex and changeable real-world scenarios. The present invention introduces additional supervisory signals such as traffic flow, agent path, and road node feature reconstruction to assist in the update and evolution of map features, thereby enhancing the model's in-depth understanding of the environment. This method brings new challenges and possibilities to the concept of static maps in the field of autonomous driving.
[0181] Figure 6 The block diagram of a trajectory prediction device for interaction between a two-way dynamic map and an intelligent agent is shown according to an exemplary embodiment. The device is used in a trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent. Figure 6 The device includes an acquisition module 610, an input module 620 and an output module 630. Among them:
[0182] The acquisition module 610 is used to obtain the trajectory of the intelligent agent to be predicted and the map data.
[0183] The input module 620 is used to input the intelligent agent trajectory and map data into the constructed two-way dynamic map and the trajectory prediction model of the intelligent agent interaction.
[0184] Among them, the trajectory prediction model includes: an agent encoder, a map encoder, multiple two-way interactive modules, an agent decoder and a map decoder.
[0185] The output module 630 is used to obtain the future trajectory prediction result of the intelligent agent based on the intelligent agent trajectory, map data and trajectory prediction model.
[0186] Optionally, the input module 620 is further configured to:
[0187] S201, obtaining the vectorized agent historical trajectory and vectorized historical map data.
[0188] S202. Encode the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encode the vectorized historical map data according to the map encoder to obtain map features.
[0189] S203, inputting the agent features and the map features into the first-stage two-way interaction module to obtain the first-stage agent features and the first-stage map features.
[0190] S204, performing self-interaction on the first-stage agent features and the first-stage map features respectively to obtain the first-stage self-interaction agent features and the first-stage self-interaction map features.
[0191] S205, inputting the first-stage self-interactive intelligent agent features and the first-stage self-interactive map features into the second-stage two-way interactive module to obtain the second-stage intelligent agent features and the second-stage map features.
[0192] S206: Obtain a hidden layer map state prediction result according to the second-stage map features and the hidden layer map decoder of the map encoder.
[0193] S207, using multiple independent multi-layer perceptrons MLP to perform agent feature differentiation on the second-stage agent features, and obtain the second-stage agent features after feature differentiation.
[0194] S208, splicing the second-stage intelligent agent features after feature differentiation, and inputting the spliced intelligent agent features and the second-stage map features into the third-stage two-way interactive module to obtain the third-stage intelligent agent features and the third-stage map features.
[0195] S209: Obtain the trajectory prediction result of the intelligent agent based on the characteristics of the intelligent agent in the third stage and the intelligent agent decoder.
[0196] S210, obtaining a map state prediction result of an output layer according to the map features of the third stage and the output layer map decoder of the map encoder.
[0197] Optionally, the input module 620 is further configured to:
[0198] The long short-term memory network LSTM is used to encode the vectorized agent history trajectory to obtain the agent features.
[0199] The lane graph convolution operator LaneConv is used to encode and vectorize historical map data to obtain map features.
[0200] Optionally, the input module 620 is further configured to:
[0201] Design a dynamic feature update mechanism to train the trajectory prediction model.
[0202] Designing a dynamic feature update mechanism includes: designing auxiliary tasks related to the map node status.
[0203] Auxiliary tasks related to the map node status include: traffic flow prediction, agent path prediction, and feature reconstruction of road nodes.
[0204] Traffic flow prediction, including: predicting the vehicle flow through each map node at a specific time interval in the future.
[0205] Agent path prediction includes: each map node predicts whether a specific agent will pass through the map node within a specific time interval in the future.
[0206] The feature reconstruction of the road node includes: erasing some features of the map node features, and reconstructing the erased features through the trajectory prediction model.
[0207] Optionally, multiple two-way interaction modules include: a two-way interaction mechanism.
[0208] The implementation method of the two-way interaction mechanism includes:
[0209] The bidirectional interaction mechanism is realized by adopting the extended spatial graph convolution method shown in the following formula (1):
[0210]
[0211] In the formula, a′ i represents the i-th agent feature after convolution, Represents the graph convolution calculation method, a irepresents the i-th agent feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, Δd ij represents the characteristics of the center and its neighbors, represents the map node neighbor set of the ith agent, m′ j represents the j-th map node feature after convolution, Represents the set of all agent neighbors of the j-th map node.
[0212] Optionally, the input module 620 is further configured to:
[0213] S2031. Obtain the neighbor pairing relationship between the intelligent agent and the map node, calculate the Euclidean distance of each neighbor pairing according to the neighbor pairing relationship, and perform dimension upgrading on the Euclidean distance through a multi-layer perceptron to obtain a distance feature.
[0214] S2032. Concatenate the agent features, distance features, and map features to obtain concatenated features.
[0215] S2033. According to the concatenated features and the multi-layer perceptron, updated agent features and weights of neighbor pairings of each map node are obtained.
[0216] S2034. According to the concatenated features and the multi-layer perceptron, updated map node features and weights of each agent neighbor pairing are obtained.
[0217] S2035. Perform point multiplication and addition on the updated agent features and the weights of the corresponding agent neighbor pairings to obtain the agent features after point multiplication and addition, and perform residual connection on the agent features and the agent features after point multiplication and addition to obtain the first-stage agent features.
[0218] S2036. Perform dot product addition on the updated map node features and the weights of the corresponding map node neighbor pairs to obtain the dot product added map node features, and perform residual connection on the map features and the dot product added map node features to obtain the first stage map features.
[0219] Optionally, the loss function of the trajectory prediction model of the two-way dynamic map and the agent interaction is shown in the following formula (2):
[0220]
[0221] In the formula, Loss represents the loss function of the trajectory prediction model, Loss traj Represents the trajectory prediction loss function, Loss map represents the map state prediction loss function, α, β, ε are hyperparameters for adjusting the weights of the learning objectives of different auxiliary tasks, and Lossf lowRepresents the traffic flow prediction loss function, Loss path Represents the agent path prediction loss function, Loss fcon represents the feature reconstruction loss function of the road node, M represents the number of observations or samples in the dataset, and α c represents the weighting factor of category c, m i represents the classification result of the i-th map node, γ represents the focusing hyperparameter that reduces the relative loss of well-classified examples, and y i represents the true label of the i-th map node, σ represents the sigmoid function, represents the classification result output of the i-th map node, M′ represents the total number of erased map nodes, and f mask represents the set of node features to be erased, Indicates the real feature labels that are erased.
[0222] In the embodiment of the present invention, the proposed two-way dynamic map and agent interaction trajectory prediction method DyMAI adopts a two-way interaction mechanism in the map and agent interaction part, effectively combines map information and agent behavior, and improves the accuracy and dynamics of trajectory prediction. A dynamic feature update mechanism is added to the model to cope with complex and changeable real-world scenarios. The present invention introduces additional supervisory signals such as traffic flow, agent path, and road node feature reconstruction to assist in the update and evolution of map features, thereby enhancing the model's in-depth understanding of the environment. This method brings new challenges and possibilities to the concept of static maps in the field of autonomous driving.
[0223] Figure 7 is a schematic diagram of the structure of a trajectory prediction device provided by an embodiment of the present invention, such as Figure 7 As shown, the trajectory prediction device may include the above Figure 6 The trajectory prediction device 710 for the interaction between the two-way dynamic map and the intelligent agent is shown. Optionally, the trajectory prediction device 710 may include a first processor 2001.
[0224] Optionally, the trajectory prediction device 710 may further include a memory 2002 and a transceiver 2003 .
[0225] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0226] Combine the following Figure 7 The components of the trajectory prediction device 710 are described in detail:
[0227] The first processor 2001 is the control center of the trajectory prediction device 710, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).
[0228] Optionally, the first processor 2001 may perform various functions of the trajectory prediction device 710 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .
[0229] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.
[0230] In a specific implementation, as an embodiment, the trajectory prediction device 710 may also include multiple processors, such as Figure 7 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0231] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0232] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0233] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0234] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0235] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0236] It should be noted that Figure 7 The structure of the trajectory prediction device 710 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0237] In addition, the technical effects of the trajectory prediction device 710 can refer to the technical effects of the trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent described in the above method embodiment, and will not be repeated here.
[0238] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0239] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).
[0240] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0241] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0242] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0243] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0244] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0245] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0246] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0247] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0248] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0249] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0250] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A trajectory prediction method for two-way dynamic map and agent interaction, characterized in that: The method comprises: S1, obtain the trajectory of the intelligent body to be predicted and the map data; S2, inputting the agent trajectory and map data into the constructed two-way dynamic map and the trajectory prediction model of the agent interaction; The trajectory prediction model includes: an agent encoder, a map encoder, a plurality of two-way interactive modules, an agent decoder and a map decoder; S3, obtaining a future trajectory prediction result of the intelligent agent according to the intelligent agent trajectory, map data and trajectory prediction model; The process of constructing the trajectory prediction model of the two-way dynamic map and the intelligent agent interaction in S2 includes: S201, obtaining vectorized agent historical trajectory and vectorized historical map data; S202, encoding the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encoding the vectorized history map data according to the map encoder to obtain map features; S203, inputting the agent features and the map features into the first-stage two-way interaction module to obtain the first-stage agent features and the first-stage map features; S204, performing self-interaction on the first-stage agent features and the first-stage map features respectively to obtain the first-stage self-interaction agent features and the first-stage self-interaction map features; S205, inputting the first-stage self-interactive agent features and the first-stage self-interactive map features into the second-stage two-way interactive module to obtain the second-stage agent features and the second-stage map features; S206, obtaining a hidden layer map state prediction result according to the second-stage map features and a hidden layer map decoder of the map encoder; S207, using multiple independent multi-layer perceptrons (MLPs) to perform agent feature differentiation on the second-stage agent features to obtain the second-stage agent features after feature differentiation; S208, splicing the second-stage intelligent agent features after the feature differentiation, and inputting the spliced intelligent agent features and the second-stage map features into the third-stage two-way interaction module to obtain the third-stage intelligent agent features and the third-stage map features; S209, obtaining a trajectory prediction result of the agent according to the third-stage agent features and the agent decoder; S210, obtaining a map state prediction result of an output layer according to the map features of the third stage and an output layer map decoder of the map encoder; The multiple two-way interactive modules include: a two-way interactive mechanism; The method for implementing the two-way interaction mechanism includes: The bidirectional interaction mechanism is realized by adopting the extended spatial graph convolution method shown in the following formula (1): In the formula, a′ i represents the i-th agent feature after convolution, Indicates the graph convolution calculation method, a i represents the i-th agent feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, Δd ij represents the characteristics of the center and its neighbors, represents the map node neighbor set of the ith agent, m′ j represents the j-th map node feature after convolution, Represents the set of all agent neighbors of the j-th map node.
2. The trajectory prediction method for two-way dynamic map and agent interaction according to claim 1 is characterized in that: The step of encoding the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encoding the vectorized history map data according to the map encoder to obtain map features in S202 includes: Using a long short-term memory network (LSTM) to encode the vectorized agent history trajectory to obtain agent features; The lane graph convolution operator LaneConv is used to encode the vectorized historical map data to obtain map features.
3. The trajectory prediction method for two-way dynamic map and agent interaction according to claim 1 is characterized in that: The process of constructing the trajectory prediction model of the two-way dynamic map and the intelligent agent interaction in S2 also includes: Design a dynamic feature update mechanism to train the trajectory prediction model; The design of the dynamic feature update mechanism includes: designing auxiliary tasks related to the map node status; The auxiliary tasks related to the map node status include: traffic flow prediction, agent path prediction and feature reconstruction of road nodes; The traffic flow prediction includes: predicting the flow of intelligent agents passing through each map node within a specific time interval in the future; The agent path prediction includes: each map node predicts whether a specific agent will pass through the map node within a specific time interval in the future; The feature reconstruction of the road node includes: erasing part of the map node features, and reconstructing the erased part of the features through a trajectory prediction model.
4. The trajectory prediction method for two-way dynamic map and agent interaction according to claim 1 is characterized in that: The step of inputting the agent features and the map features into the first-stage two-way interaction module in S203 to obtain the first-stage agent features and the first-stage map features includes: S2031, obtaining a neighbor pairing relationship between an intelligent agent and a map node, calculating the Euclidean distance of each neighbor pairing according to the neighbor pairing relationship, and performing dimension-upgrading on the Euclidean distance through a multi-layer perceptron to obtain a distance feature; S2032, performing feature splicing on the agent feature, the distance feature, and the map feature to obtain a spliced feature; S2033, obtaining updated agent features and weights of neighbor pairings of each map node according to the concatenated features and the multi-layer perceptron; S2034, obtaining updated map node features and weights of neighbor pairings of each agent based on the concatenated features and the multi-layer perceptron; S2035, performing dot product addition on the updated agent feature and the weight of the corresponding agent neighbor pairing to obtain the agent feature after dot product addition, and performing residual connection on the agent feature and the agent feature after dot product addition to obtain the first stage agent feature; S2036. Perform dot product addition on the updated map node feature and the weight of the corresponding map node neighbor pairing to obtain the dot product added map node feature, and perform residual connection on the map feature and the dot product added map node feature to obtain the first stage map feature.
5. The trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent according to claim 1, characterized in that: The loss function of the trajectory prediction model of the two-way dynamic map and the agent interaction is shown in the following formula (2): In the formula, Loss represents the loss function of the trajectory prediction model, Loss traj Represents the trajectory prediction loss function, Loss map represents the map state prediction loss function, α, β, ε are hyperparameters for adjusting the weights of the learning objectives of different auxiliary tasks, and Loss flow Represents the traffic flow prediction loss function, Loss path Represents the agent path prediction loss function, Loss fcon represents the feature reconstruction loss function of the road node, M represents the number of observations or samples in the dataset, and α c represents the weighting factor of category c, m i represents the classification result of the i-th map node, γ represents the focusing hyperparameter that reduces the relative loss of well-classified examples, and y i represents the true label of the i-th map node, σ represents the sigmoid function, represents the classification result output of the i-th map node, M′ represents the total number of erased map nodes, and f mask represents the set of node features to be erased, Indicates the real feature labels that are erased.
6. A trajectory prediction device for interaction between a two-way dynamic map and an intelligent agent, the trajectory prediction device for interaction between a two-way dynamic map and an intelligent agent is used to implement the trajectory prediction method for interaction between a two-way dynamic map and an intelligent agent as claimed in any one of claims 1 to 5, characterized in that: The device comprises: The acquisition module is used to obtain the trajectory of the intelligent agent to be predicted and the map data; An input module, used to input the agent trajectory and map data into the constructed two-way dynamic map and the trajectory prediction model of the agent interaction; The trajectory prediction model includes: an agent encoder, a map encoder, a plurality of two-way interactive modules, an agent decoder and a map decoder; An output module, used to obtain a future trajectory prediction result of the intelligent agent based on the intelligent agent trajectory, map data and trajectory prediction model; The process of constructing the trajectory prediction model of the two-way dynamic map and the intelligent agent interaction includes: S201, obtaining vectorized agent historical trajectory and vectorized historical map data; S202, encoding the vectorized agent history trajectory according to the agent encoder to obtain agent features, and encoding the vectorized history map data according to the map encoder to obtain map features; S203, inputting the agent features and the map features into the first-stage two-way interaction module to obtain the first-stage agent features and the first-stage map features; S204, performing self-interaction on the first-stage agent features and the first-stage map features respectively to obtain the first-stage self-interaction agent features and the first-stage self-interaction map features; S205, inputting the first-stage self-interactive agent features and the first-stage self-interactive map features into the second-stage two-way interactive module to obtain the second-stage agent features and the second-stage map features; S206, obtaining a hidden layer map state prediction result according to the second-stage map features and a hidden layer map decoder of the map encoder; S207, using multiple independent multi-layer perceptrons (MLPs) to perform agent feature differentiation on the second-stage agent features to obtain the second-stage agent features after feature differentiation; S208, splicing the second-stage intelligent agent features after the feature differentiation, and inputting the spliced intelligent agent features and the second-stage map features into the third-stage two-way interaction module to obtain the third-stage intelligent agent features and the third-stage map features; S209, obtaining a trajectory prediction result of the agent according to the third-stage agent features and the agent decoder; S210, obtaining a map state prediction result of an output layer according to the map features of the third stage and an output layer map decoder of the map encoder; The multiple two-way interactive modules include: a two-way interactive mechanism; The method for implementing the two-way interaction mechanism includes: The bidirectional interaction mechanism is realized by adopting the extended spatial graph convolution method shown in the following formula (1): In the formula, a′ i represents the i-th agent feature after convolution, Indicates the graph convolution calculation method, a i represents the i-th agent feature that needs to be aggregated, m j Indicates the neighbor features that need to be aggregated, Δd ij represents the characteristics of the center and its neighbors, represents the map node neighbor set of the ith agent, m′ j represents the j-th map node feature after convolution, Represents the set of all agent neighbors of the j-th map node.
7. A trajectory prediction device, characterized in that: The trajectory prediction device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.
