Trajectory prediction method and system and storage medium
Through the dynamic flow self-attention transformation model, combining multi-stream and single-stream encoder, the real-time and accuracy problems of edge computing devices are solved, and fast and accurate prediction of agent trajectory is achieved, and the application of intelligent robot systems is supported.
Patent Information
- Application Number
- CN202510307196.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-16
- Publication Date
- 2025-07-01
AI Technical Summary
The prior art lacks an accurate agent trajectory prediction method that is required for real-time requirements of edge computing devices. The generative random probability model requires multiple random sampling, the computational burden of complex models is large, and the prediction accuracy of simple models is insufficient.
A dynamic flow self-attention transformation model combining a multi-stream encoder and a single-stream encoder is used to generate a multi-stream input vector through a multi-stream encoder. The single-stream encoder is converted into a single-stream input vector, and the position vector at the next moment is predicted based on environmental information to generate a predicted track of the preset time.
It realizes fast and accurate trajectory prediction in complex environments, meets real-time requirements, and provides support for intelligent robot systems.
Smart Images
Figure CN120235294A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a prediction method, in particular to a trajectory prediction method, system and storage medium, belonging to the technical field of trajectory prediction. Background Art
[0002] With the development of the concept of "solving the last mile" and the increasing entry of autonomous robots into people's living areas, fast and accurate pedestrian trajectory prediction is becoming increasingly important for autonomous robot systems.
[0003] In order to obtain accurate trajectory prediction, various methods have conducted in-depth explorations in aspects such as agent intention modeling, environmental information fusion, and agent target prior prediction. With continuous improvement of the modeling method and training with more data, relatively accurate trajectory prediction can be achieved even in complex scenarios.
[0004] However, there is currently no accurate agent trajectory prediction method that is oriented to actual deployment and meets the real-time requirements of edge computing devices. The most advanced methods currently have the following disadvantages:
[0005] Disadvantage 1: The output of the generative stochastic probability model needs to go through random sampling. It is necessary to select the maximum probability value of the random variable during the sampling process to obtain a deterministic output result. Usually, when evaluating such trajectory prediction models, different trajectories are generated through multiple random samplings for error evaluation, which is separated from the actual application needs.
[0006] Disadvantage 2: In order to understand the agent intention and improve the prediction accuracy, existing accurate prediction models mostly adopt computationally intensive designs such as self-attention mechanisms and deep diffusion models that are applied to the data of all agents throughout the process. They usually have the disadvantages of large parameter quantities and long inference times.
[0007] Disadvantage 3: Some depth prediction models with faster speeds but simple structures have a large gap in prediction accuracy compared with advanced models. Summary of the Invention
[0008] In view of this, the present invention provides a trajectory prediction method, system and storage medium to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.
[0009] The technical solution of the embodiment of the present invention is implemented as follows: A trajectory prediction method is provided, including the following steps: Obtain the historical positions of multiple agents, generate a target input vector based on the historical positions of the multiple agents, and encode the target input vector through a multi-stream encoder to generate a multi-stream input vector; Convert the multi-stream input vector into a single-stream input vector through a single-stream encoder, and generate a multi-stream output vector based on the single-stream input vector; And use the current historical positions of the multiple agents as the initial positions, and combine the multi-stream output vector to predict the position vector at the next moment, and repeat the above process to generate a predicted trajectory with a preset duration.
[0010] Further preferably: The agents at least include pedestrians, vehicles, and / or autonomous robots.
[0011] Further preferably: The generating of the target input vector includes: generating a position vector based on the historical positions of the multiple agents, and generating a target input vector according to the position vector.
[0012] Further preferably: It may further include the environmental information around the agents; The generating of the target input vector includes: encoding the environmental information, generating a position vector based on the historical positions of the multiple agents, and generating a target input vector according to the environmental information and the position vector.
[0013] Further preferably: The multi-stream encoder at least includes a multi-stream time encoder and a multi-stream social interaction encoder that are alternately cascaded; The generating of the multi-stream input vector includes: encoding the target input vector sequentially through the multi-stream time encoder and the multi-stream social interaction encoder, and then generating a multi-stream input vector.
[0014] Further preferably: The single-stream encoder at least includes a multi-stream to single-stream morphological transformation module, multiple full-dimensional sub-encoders, and a single-stream to multi-stream morphological transformation module that are cascaded; The generating of the multi-stream output vector includes: converting the multi-stream input vector into a single-stream input vector through the multi-stream to single-stream morphological transformation module; encoding the single-stream input vector through the multiple full-dimensional sub-encoders, and converting the encoded single-stream input vector into a multi-stream output vector through the single-stream to multi-stream morphological transformation module.
[0015] Further preferably: The predicting of the position vector at the next moment includes: initializing the current historical positions of the multiple agents to obtain the initial positions, and generating a predicted position vector at the current moment according to the initial positions.
[0016] Further preferably, the position vector for predicting the next moment further includes: decoding the multi-stream output vector through a multi-stream time decoder to obtain the position vector of the next moment by combining the decoded multi-stream output vector and the predicted position vector of the current moment.
[0017] Based on the same concept, the present invention further provides a trajectory prediction system, which adopts the above-mentioned trajectory prediction method. The system includes:
[0018] An input module, configured to obtain the historical positions of multiple agents, and generate a target input vector based on the historical positions of the multiple agents.
[0019] A content encoder, configured to encode the target input vector through a multi-stream encoder to generate a multi-stream input vector; convert the multi-stream input vector into a single-stream input vector through a single-stream encoder, and generate a multi-stream output vector based on the single-stream input vector.
[0020] A trajectory encoder, configured to use the current historical positions of multiple agents as the initial positions, and combine the multi-stream output vector to predict the position vector of the next moment, and repeatedly execute the input module and the content encoder to generate a predicted trajectory with a preset duration.
[0021] Based on the same concept, the present invention further provides a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the above-mentioned trajectory prediction method is implemented.
[0022] Due to the above technical solutions adopted in the embodiments of the present invention, it has the following advantages:
[0023] The present invention provides a fast and accurate multi-agent trajectory prediction method, which can be applied to complex environments, provide accurate trajectory prediction while meeting the real-time requirements, and support the application of intelligent robot systems in living environments.
[0024] The above summary is only for the purpose of the specification and is not intended to be limiting in any way. In addition to the above-described illustrative aspects, embodiments and features, further aspects, embodiments and features of the present invention will be readily apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0026] Figure 1 Flowchart of the trajectory prediction method according to the present invention.
[0027] Figure 2 Framework diagram of the trajectory prediction system according to the present invention. Detailed implementation manners
[0028] In the following, only some exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present invention. Therefore, the drawings and the description are considered to be exemplary in nature rather than restrictive.
[0029] The embodiments of the present invention will be described in detail below with reference to the drawings.
[0030] As Figure 1 shown, an embodiment of the present invention provides a trajectory prediction method, including the following steps: obtaining the historical positions of multiple agents, generating a target input vector based on the historical positions of the multiple agents, and encoding the target input vector through a multi-stream encoder to generate a multi-stream input vector; converting the multi-stream input vector into a single-stream input vector through a single-stream encoder, and generating a multi-stream output vector based on the single-stream input vector; and using the current historical positions of the multiple agents as the initial positions, and combining the multi-stream output vector to predict the position vector at the next moment, repeating the above process to generate a predicted trajectory with a preset duration.
[0031] Among them, the present invention develops a dynamic flow self-attention transformation model from a deep self-attention model, regards the input of each agent as a stream, classifies the network layer that focuses on a single dimension of the agent as a multi-stream layer to allow them to perform parallel calculations on multiple streams, and classifies the layer that simultaneously focuses on the time and interaction dimensions as a single-stream layer. The single-stream layer and the multi-stream layer are finally combined into a multi-stream - single-stream - multi-stream dynamic flow self-attention transformation model.
[0032] In this embodiment, specifically: the agents at least include pedestrians, vehicles, and / or autonomous robots.
[0033] Among them, an agent refers to an individual with autonomous will.
[0034] In this embodiment, specifically: the generating of the target input vector includes: generating a position vector based on the historical positions of the multiple agents, and generating a target input vector according to the position vector.
[0035] Among them, the input of the agent must include two-dimensional position information (i.e., the historical position) of past consecutive time nodes.
[0036] In this embodiment, specifically: it may further include the environmental information around the agent; the generating of the target input vector includes: encoding the environmental information, and generating a position vector based on the historical positions of the multi-agent, so as to generate a target input vector according to the environmental information and the position vector.
[0037] Among them, the input of the agent may further include the environmental information around the agent represented by a semantic map from an aerial view; the position information, environmental information, etc. of the agent can be obtained by sensors such as lidar, depth cameras, and color cameras. Those skilled in the art can also adopt other acquisition methods, which are not limited herein.
[0038] It should be noted that whether to obtain environmental information is optional. If environmental information is obtained, multiple usage scenarios need to be included.
[0039] Furthermore, the two-dimensional position information of the agent at each time node will be transformed into a position vector of a higher dimension through a linear layer. This dimension is usually 16, 32, 64 dimensions, etc., and needs to be set according to the computing power before training. When no environmental information input is provided, the position vector encoded by the sine-cosine time series is the input vector (i.e., the target input vector) of a single agent at a single historical time node.
[0040] Furthermore, when providing a semantic map, the interaction between the agent and the environment can be taken into account through an environmental information encoder. The environmental information encoder uses a widely used convolutional neural network to extract semantic map information and generates an environmental information vector through a linear layer, which is concatenated with the position vector to obtain the input vector (i.e., the target input vector) of a single agent at a single historical time node.
[0041] In this embodiment, specifically: the multi-stream encoder at least includes a multi-stream time encoder and a multi-stream social interaction encoder that are alternately cascaded; the generating of the multi-stream input vector includes: encoding the target input vector sequentially through the multi-stream time encoder and the multi-stream social interaction encoder, and then generating a multi-stream input vector.
[0042] Among them, the multi-stream encoder focuses on a single dimension and consists of two sub-encoders: a multi-stream time encoder that focuses on the historical information of the agent and a multi-stream social interaction encoder that focuses on the interaction information of the agent. Both of these sub-encoders adopt a self-attention structure and can perform multi-stream parallel computing in the dimensions they do not focus on. The multi-stream time encoder and the multi-stream social interaction encoder are alternately cascaded to form the final multi-stream encoder. The total number of cascaded sub-encoders needs to be greater than 2, which is determined according to the device computing power and the dataset size.
[0043] In this embodiment, specifically, the single-stream encoder at least includes a cascaded multi-stream to single-stream transformation module, a plurality of full-dimensional sub-encoders, and a single-stream to multi-stream transformation module; generating the multi-stream output vector includes: converting the multi-stream input vector into a single-stream input vector through the multi-stream to single-stream transformation module; encoding the single-stream input vector through the plurality of full-dimensional sub-encoders, and converting the encoded single-stream input vector into a multi-stream output vector through the single-stream to multi-stream transformation module.
[0044] Among them, the single-stream encoder focuses on all dimensions and converges multi-stream information into a single stream. The full-dimensional sub-encoder applies a self-attention structure to the already concatenated single-stream sequence, encoding both the temporal and social interaction dimensions simultaneously. Its computational complexity is relatively high, and usually no more than 2 in the cascade.
[0045] In this embodiment, specifically, predicting the position vector at the next moment includes: initializing the current historical positions of multiple agents to obtain an initial position, and generating a predicted position vector at the current moment based on the initial position.
[0046] In this embodiment, specifically, predicting the position vector at the next moment further includes: decoding the multi-stream output vector through a multi-stream temporal decoder to obtain a position vector at the next moment by combining the decoded multi-stream output vector and the predicted position vector at the current moment.
[0047] Among them, the position vector at the next moment is obtained through a trajectory encoder. The trajectory decoder is composed of cascaded multi-stream temporal decoders and generates the final predicted trajectory in an autoregressive manner. The multi-stream temporal decoder applies self-attention and cross-attention mechanisms, takes the predicted position vector at the current moment as input, and combines the output of the content encoder to predict the position vector at the next moment.
[0048] As can be seen from the above solution, according to the present invention, this dynamic stream self-attention model can effectively model a single dimension in the multi-stream stage, while maintaining information exchange between dimensions in the single-stream stage, fully considering the interactions between agents and the environment, and between agents. At the same time, it can be seen that compared with directly using a single-stream layer for all encoding and decoding, the introduction of the multi-stream layer significantly reduces the computational complexity of the model (the complexity of the self-attention model is quadratic with respect to the length of the input sequence). Through this organic combination of multi-stream and single-stream modules, the present invention achieves a good balance between accuracy and speed and realizes excellent prediction performance.
[0049] Based on the same concept, the present invention also provides a trajectory prediction system that adopts the above-mentioned trajectory prediction method. The system includes:
[0050] An input module for obtaining the historical positions of multiple agents to generate a target input vector based on the historical positions of the multiple agents.
[0051] A content encoder for encoding the target input vector through a multi-stream encoder to generate a multi-stream input vector; and converting the multi-stream input vector into a single-stream input vector through a single-stream encoder and generating a multi-stream output vector based on the single-stream input vector.
[0052] Among them, the content encoder includes the above-mentioned multi-stream encoder and single-stream encoder, with the multi-stream encoder in the front and the single-stream encoder in the back, and executes each step in the trajectory prediction method, and the specific implementation manner thereof will not be elaborated here.
[0053] A trajectory encoder for using the current historical positions of multiple agents as the initial positions and predicting the position vectors at the next moment in combination with the multi-stream output vector, and repeatedly executing the input module and the content encoder to generate a predicted trajectory with a preset duration.
[0054] Based on the same concept, the present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned trajectory prediction method is implemented.
[0055] As mentioned above, the above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various changes or substitutions thereof, and these should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims
1. A trajectory prediction method, characterized in that: The following steps are involved: Acquire multi-agent historical positions to generate a target input vector based on the multi-agent historical positions, and encode the target input vector through a multi-stream encoder to generate a multi-stream input vector; Converting the multi-stream input vector into a single-stream input vector by a single-stream encoder, and generating a multi-stream output vector based on the single-stream input vector; The current multi-agent historical position is used as the initial position, and the position vector at the next moment is predicted in combination with the multi-stream output vector. The above process is repeated to generate a predicted trajectory of a preset duration.
2. The trajectory prediction method according to claim 1, characterized in that: The intelligent agents include at least pedestrians, vehicles and / or autonomous robots.
3. The trajectory prediction method according to claim 2, characterized in that: The generating of the target input vector comprises: A position vector is generated based on the historical positions of the multi-agents to generate a target input vector according to the position vector.
4. The trajectory prediction method according to claim 2, characterized in that: It may also include environmental information around the agent; generating a target input vector includes: The environmental information is encoded, and a position vector is generated based on the historical positions of the multi-agents, so as to generate a target input vector according to the environmental information and the position vector.
5. The trajectory prediction method according to any one of claims 3 or 4, characterized in that: The multi-stream encoder comprises at least a multi-stream time encoder and a multi-stream social interaction encoder which are alternately cascaded; the generating of the multi-stream input vector comprises: The target input vector is encoded in sequence by the multi-stream temporal encoder and the multi-stream social interaction encoder to generate a multi-stream input vector.
6. The trajectory prediction method according to claim 5, characterized in that: The single-stream encoder at least includes a cascaded multi-stream to single-stream morphology transformation module, a plurality of full-dimensional sub-encoders and a single-stream to multi-stream morphology transformation module; the generating of the multi-stream output vector includes: Converting the multi-stream input vector into a single-stream input vector by a multi-stream to single-stream morphology transformation module; The single-stream input vector is encoded by the multiple full-dimensional sub-encoders, and the encoded single-stream input vector is converted into a multi-stream output vector by the single-stream to multi-stream morphological transformation module.
7. The trajectory prediction method according to claim 6, characterized in that: The predicting of the position vector at the next moment includes: The current multi-agent historical position is initialized to obtain an initial position, and a predicted position vector at the current moment is generated based on the initial position.
8. The trajectory prediction method according to claim 7, characterized in that: The predicting of the position vector at the next moment also includes: The multi-stream output vector is decoded by a multi-stream time decoder to combine the decoded multi-stream output vector and the predicted position vector at the current moment to obtain the position vector at the next moment.
9. A trajectory prediction system, characterized in that: Using the trajectory prediction method according to any one of claims 1 to 8, the system comprises: An input module, used for obtaining the historical positions of the multi-agents to generate a target input vector based on the historical positions of the multi-agents; A content encoder, configured to encode the target input vector by a multi-stream encoder to generate a multi-stream input vector; and convert the multi-stream input vector into a single-stream input vector by a single-stream encoder, and generate a multi-stream output vector based on the single-stream input vector; The trajectory encoder is used to take the current multi-agent historical position as the initial position, and combine the multi-stream output vector to predict the position vector at the next moment, repeatedly execute the input module and content encoder, and generate a predicted trajectory of a preset duration.
10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the trajectory prediction method according to any one of claims 1 to 8 is implemented.