A Fast Adaptive Trajectory Prediction Method Based on Online Multi-Source Transfer Learning

Abstract

This invention relates to a fast adaptive trajectory prediction method based on online multi-source transfer learning, comprising: using n base learners {f i The online learner f is pre-trained in the corresponding source domain for |i=1,...,n} and randomly initialized. t The parameters of the online learner f t It has the same structure as the base learner; data {X t ,Y t |t=1,2,...,T} arrive in chronological order, X t Y represents the observed trajectory arriving at time t. t This represents the future true trajectory reached at time t. When time t arrives, the online learner f... t Perform trajectory prediction and train the online learner f t Perform an update operation to obtain the online learner f. t+1 This invention can adapt to real-time changing scenarios and trajectory data, thereby accurately predicting future trajectories.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction, and more specifically to a fast adaptive trajectory prediction method based on online multi-source transfer learning. Background Technology

[0002] Trajectory prediction is a research hotspot in fields such as intelligent surveillance, autonomous driving, and social robotics. Currently, trajectory prediction is mainly achieved by training deep models using batch processing on fixed offline data. However, in the real world, prediction models need to deal with real-time updated data and changing scenarios. Traditional batch processing methods struggle to process newly acquired data in real time, limiting the application of trajectory prediction models in real-world scenarios.

[0003] While online learning has seen extensive research in image classification, its application in trajectory prediction is relatively limited. Trajectory prediction models primarily rely on offline learning on known public datasets. However, trajectory data is generated constantly in the real world, and scenarios are continuously changing, posing a challenge to the model's adaptability. Furthermore, offline learning methods jointly train data from different source domains, failing to consider the differences between these domains. Additionally, offline learning methods can prevent models from improving their inference capabilities in real time, requiring significant time and resources for retraining and validation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a fast adaptive trajectory prediction method based on online multi-source transfer learning. This method can adapt to real-time changing scenarios and trajectory data, and thus accurately predict future trajectories.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is: a fast adaptive trajectory prediction method based on online multi-source transfer learning, comprising:

[0006] Given n base learners {f i The online learner f is pre-trained in the corresponding source domain for |i=1,...,n} and randomly initialized. t The parameters of the online learner f t It has the same structure as the base learner;

[0007] Data {X t ,Y t |t=1,2,...,T} arrive in chronological order, X t Y represents the observed trajectory arriving at time t. t This represents the future true trajectory reached at time t. When time t arrives, the online learner f... t Perform trajectory prediction and train the online learner f tPerform an update operation to obtain the online learner f. t+1 ;

[0008] The update operation includes:

[0009] Computational online learner f t The predicted loss L at time t t Using the predicted loss L t Update online learning tool f t Get an online learning device

[0010] Calculate the prediction loss L for each base learner i and online predictors Predicted loss L n+1 ;

[0011] According to the predicted loss L i Adjusting the weights of the base learner Obtain weights Meanwhile, based on the predicted loss L n+1 Adjusting the online learning device weight Obtain weights

[0012] For n+1 weights Perform relaxation operations;

[0013] Using relaxed weights on n base learners and an online learner We perform a weighted summation to obtain the online learner f at time t+1. t+1 .

[0014] Furthermore, using the predicted loss L t Update online learning tool f t Get an online learning device The formula is:

[0015]

[0016] Where η represents the learning rate.

[0017] Furthermore, the formula for adjusting the weights based on the predicted loss is as follows:

[0018]

[0019] Where i = 1, ..., n+1, β ∈ (0, 1) is the weight decay factor.

[0020] Furthermore, the steps for calculating the learner's prediction loss at time t include:

[0021] The learner learns from the observed trajectory Xt Predicting the future trajectory of pedestrians, obtaining the predicted future trajectory.

[0022] Based on the future real trajectory Y t and future predicted trajectory Calculate the prediction loss at time t; where,

[0023] The learner includes an online learner f t Online learning devices and basis learner {f i |i=1,...,n}.

[0024] Furthermore, for the n+1 weights The formula for performing relaxation is:

[0025]

[0026] Where s is the smoothing coefficient.

[0027] Furthermore, the base learner incorporates a memory retrieval module and a trajectory prediction module, and the base learner f is pre-trained in the corresponding source domain. i The steps include:

[0028] In the source domain Pre-training memory retrieval module;

[0029] The key-value memory is extracted using a pre-trained memory retrieval module to obtain a key-value external memory M′, which stores each key-value memory [H]. j G j ] and the corresponding trajectory data C j , where j = 1, ..., |M′|

[0030] Based on k-means clustering, pedestrian trajectories are clustered into multiple groups, and the key-value memory corresponding to each group is selected to obtain the external memory M.

[0031] The trajectory prediction module in the source domain The optimal model f is obtained by performing offline training. i f, as the i-th base learner i .

[0032] Furthermore, the working process of the memory retrieval module includes:

[0033] The working process of the memory retrieval module includes:

[0034] The relative displacement of pedestrian p's historical trajectory Relative displacement of future trajectory The data are fed into encoders h and g respectively to generate features H of the observed trajectory and the future trajectory.p and G p And it is stored as the key value of external memory M';

[0035] The complete trajectory C of pedestrian p p ={X p ||Y p The corresponding key-value pairs are stored in external memory M', where || represents a cascading operation, used later to cluster the number of memories in external memory M' into a fixed number;

[0036] Finally, H p G p and the coordinates of the last observation The data is sent to decoder f to generate the future trajectory.

[0037] Furthermore, the trajectory prediction module includes an encoder h, a decoder d, and a multi-hop attention mechanism responsible for iteratively querying relevant key value storage from the external memory M. The encoder h is pre-trained in the memory retrieval module, and the decoder d is used to output the predicted trajectory. The external memory M' is clustered to become the external memory M.

[0038] By adopting the above technical solution, this invention combines online transfer learning and ensemble learning to train an online learner. The online learner can adapt to real-time changing scenarios and trajectory data, achieving rapid adaptive trajectory prediction. Attached Figure Description

[0039] Figure 1 The flowchart shows the fast adaptive trajectory prediction method based on online multi-source transfer learning of the present invention.

[0040] Figure 2 This is a structural diagram of the memory retrieval module of the base learner of the present invention;

[0041] Figure 3 This is a structural diagram of the trajectory prediction module of the base learner of the present invention. Detailed Implementation

[0042] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0043] Suppose there are m pedestrians in a given scene, and the coordinates of the p-th pedestrian at time t are given by... Let the observed trajectory and the future actual trajectory of pedestrian p be represented as follows: and Model P θ Based on the observed trajectory X p Generate future trajectory The above problem can be expressed as finding the optimal parameter θ, as shown in the following formula:

[0044]

[0045] Assume there are n source domains and target domain D T Each source domain pre-trains a base learner, and D T The data arrives sequentially. Using full feedback online learning, the observed trajectory data arriving at time t is denoted as X. t The online learner generates the prediction result at time t. and actual trajectory Y t The loss is calculated and used for fine-tuning the online learner. The fine-tuned online learner is then updated by integrating with the base learner using an ensemble learning method. Ultimately, the online learner continuously improves through new scenarios and data to enhance the model's inference capabilities.

[0046] The technical solutions involved in the above embodiments will be described in detail below with reference to specific examples.

[0047] like Figure 1 As shown, a fast adaptive trajectory prediction method based on online multi-source transfer learning includes:

[0048] Given n base learners {f i The online learner f is pre-trained in the corresponding source domain for |i=1,...,n} and randomly initialized. t The parameters of the online learner f t Having the same structure as the base learner, the base learner f i initial weights Set to

[0049] Data {X t ,Y t |t=1,2,...,T} arrive in chronological order, X t Y represents the observed trajectory arriving at time t. t This represents the future true trajectory reached at time t. When time t arrives, the online learner f... t Perform trajectory prediction and train the online learner f t Perform an update operation to obtain the online learner f. t+1 ;

[0050] The update operation includes:

[0051] Computational online learner f t The predicted loss L at time t t Using the predicted loss L tUpdate online learning tool f t Get an online learning device

[0052] Calculate the prediction loss L for each base learner i and online predictors Predicted loss L n+1 ;

[0053] According to the predicted loss L i Adjusting the weights of the base learner Obtain weights Meanwhile, based on the predicted loss L n+1 Adjusting the online learning device weight Obtain weights

[0054] Since the learner is combined by multiplying the weights by the learner parameters, to avoid excessively large weights causing large changes in the updated online learner parameters and resulting in poor prediction performance, n+1 weights are... Perform a relaxation operation.

[0055] Using relaxed weights on n base learners and an online learner We perform a weighted summation to obtain the online learner f at time t+1. t+1 Specifically

[0056] A single base learner based on a neural network may lead to underfitting or overfitting. To improve generalization performance, this embodiment trains multiple base learners to learn different tasks and forms a strong learner through a reasonable combination strategy. This embodiment combines online transfer learning and ensemble learning to train an online learner, which can adapt to real-time changing scenarios and trajectory data, achieving rapid adaptive trajectory prediction.

[0057] In one embodiment, the prediction loss L is used t Update online learning tool f t Get an online learning device The formula is:

[0058]

[0059] Where η represents the learning rate.

[0060] In one embodiment, the formula for adjusting the weights based on the predicted loss is:

[0061]

[0062] Where i = 1, ..., n+1, β ∈ (0, 1) is the weight decay factor.

[0063] In one embodiment, the step of calculating the learner's prediction loss at time t includes:

[0064] The learner predicts the future trajectory of a pedestrian based on the observed trajectory, thus obtaining the predicted future trajectory.

[0065] The prediction loss at time t is calculated based on the future actual trajectory and the future predicted trajectory.

[0066] The specific formula is as follows:

[0067]

[0068] Where ||·||2 represents the L2 norm, Let Y represent the j-th predicted future trajectory of pedestrian p. p Let f represent the future true trajectory of pedestrian p, and L represent the prediction loss. The learner includes an online learner f. t Online learning devices and basis learner {f i |i=1,...,n}.

[0069] In one embodiment, for n+1 weights The formula for performing relaxation is:

[0070]

[0071] Where s is the smoothing coefficient.

[0072] In one embodiment, the base learner incorporates a memory retrieval module and a trajectory prediction module, and the base learner f is pre-trained in the corresponding source domain. i The steps include:

[0073] In the source domain Pre-training memory retrieval module;

[0074] The key-value memory is extracted using a pre-trained memory retrieval module to obtain a key-value external memory M′, which stores each key-value memory [H]. j G j ] and the corresponding trajectory data C j , where j = 1, ..., |M′|

[0075] Based on k-means clustering, pedestrian trajectories are clustered into multiple groups, and the key-value memory corresponding to each group is selected to obtain the external memory M.

[0076] The trajectory prediction module in the source domain The optimal model f is obtained by performing offline training. i f, as the i-th base learner i .

[0077] It is important to note that after obtaining the external memory M, during prediction, the obtained historical trajectory is used to calculate the similarity with each key in M. The value corresponding to the key with the highest similarity is taken and sent to the online learner f. t The predicted trajectory is obtained from this.

[0078] Specifically, the memory retrieval module is responsible for reading and writing key-value memories, maintaining low redundancy and sample diversity in the external memory. The working process of the memory retrieval module includes:

[0079] The working process of the memory retrieval module includes:

[0080] The relative displacement of pedestrian p's historical trajectory Relative displacement of future trajectory The data are fed into encoders h and g respectively to generate features H of the observed trajectory and the future trajectory. p and G p And store it as the key value of external memory M'; at the same time, store the complete trajectory C of pedestrian p. p ={X p ||Y p The corresponding key-value pairs are stored in external memory M', where || represents a cascading operation, used later to cluster the memory counts in external memory M' into a fixed number; finally, H is stored in external memory M'. p G p and the coordinates of the last observation The data is sent to decoder f to generate the future trajectory.

[0081] To reduce redundancy and maintain diversity in the external memory M', read and write operations are performed through the Training Controller (CTR). The CTR locates the memory by calculating the cosine similarity score between the observed trajectory feature encoding and the memory key value, and then extracts typical pedestrian motion patterns from the current task and stores them in the external memory M'. At the start of model training, because the external memory M' lacks available observed trajectory encodings, future trajectory encodings, and the last moment position of the pedestrian observed trajectory, the memory M' is initially randomly initialized and updated via the CTR. The read and write operations are as follows:

[0082] Reading operation: In the entire process, the observation feature code H of pedestrian p is first calculated. p And all keys H in external memory M' d The corresponding cosine similarity s d The specific formula is as follows:

[0083]

[0084] Then, the top k keys with the highest cosine similarity are selected, and their corresponding values ​​G are retrieved.j Where j = 1,...,k, these k values ​​are encoded with the currently observed feature H. j Concatenate to form a new feature code N j =[H j G j Encode these k features into N j Coordinates of the last observation The cascaded trajectories are then fed into decoder f to generate k future trajectories, as shown in the following formula:

[0085]

[0086] Where ω f These are the learnable parameters of the decoder f, FC f and GRU f This indicates the fully connected (FC) and recurrent neural network (GRU) layers used in the decoder f.

[0087] Write operation: In this stage, each future predicted trajectory is first calculated. Pedestrian's actual trajectory Y p To address the error, the proposed trajectory error algorithm is as follows:

[0088]

[0089] Among them l t The period (.) indicates whether the error between the observed trajectory and the future trajectory is less than a certain threshold th at each time step. If the error of the predicted trajectory at each time step is within the circle, the output is 1; otherwise, the output is 0. The threshold th decreases linearly from the end point to the beginning point (from 8.0 to 0.8), after which {e j The minimum error e in the sequence is fed into the CTR to generate memory and modify the probability P. ω The calculation formula is as follows:

[0090] P ω =Sigmoid(FC(e;ω) FC ))

[0091] Where ω FC These are the learnable parameters of the FC (Functionally Connected) controller. Subsequently, the controller (CTR) adaptively increases or decreases the probability P through the loss function L. ω And update the controller (CTR) parameters. The loss function is calculated as follows:

[0092] L=e·(1-P ω )+(1-e)·P ω

[0093] The magnitude of e determines the model's predictive performance, and P ωThe decision to write the current sample to external memory is made based on the fact that the network's prediction performance is good when e→0, and the write probability P should be reduced. ω To avoid writing samples, otherwise, when e→1, the network model's prediction performance deteriorates, indicating that there are no similar pedestrian trajectories in the external memory, and P needs to be increased. ω To write down more new memories.

[0094] Ensemble learning requires the base learner to have a known structure. However, updates and modifications to the external memory during external memory retrieval cannot be controlled. Therefore, to ensure the consistency of the number of external memory units, k-means clustering is used to memorize the complete trajectory C of the corresponding pedestrian p based on the key value. i The number of memories is clustered and calculated using the following formula:

[0095] M = k-means(C p ,n c ),p=1,...,|M'| (7)

[0096] Where n c M represents the clustering item, and M is the external memory obtained after clustering M'.

[0097] In one embodiment, the trajectory prediction module includes an encoder h, a decoder d, and a multi-hop attention mechanism responsible for iteratively querying relevant key value storage from external memory M. The encoder h is pre-trained in the memory retrieval module, and the decoder d is used to output the predicted trajectory. The external memory M' is clustered to become external memory M.

[0098] The observed trajectory and the future trajectory are a pair of related information. The future trajectory is predicted through a multi-hop attention mechanism (MHA). MHA is used to ensure the diversity of the output trajectory, that is, to predict multiple possible trajectories. This mechanism performs multiple iterative inferences from external memory and uses the results of the multi-hops as the generated multimodal trajectory.

[0099] Given a pedestrian p in a given scene, observe the trajectory X. p It is fed into encoder h to generate feature vector G p eigenvector G p Sent to linear transformation φ q Generate query vector q j Trajectory prediction is then achieved through key addressing, value reading, trajectory prediction, and query vector. Specific measures are as follows:

[0100] Key addressing: External memory M for each key H d Through the linear function φ k The mapping is performed as a key feature embedding. Then, the softmax function is used to compute the relationship between each key and the current query vector q.j The similarity probability between them is calculated as follows:

[0101] p d =Softmax(q j ·φ k (H d ),d=1,...,|M|

[0102] Where p d Indicates the addressing weight.

[0103] Value read: Each value G in external memory M d The linear function φ is mapped to the value feature embedding. Then, the attention result o... j In addressing weight p d The definitions between sum-value feature embeddings are as follows:

[0104]

[0105] Trajectory prediction: Considering the influence of a pedestrian's historical trajectory on their future trajectory, the observed trajectory H... p and o j Connect them. Then compare the fusion result with the last observed location. The data is sent to decoder d to predict the trajectory, thereby further improving the model's ability to extract observational information. The calculation is as follows:

[0106]

[0107] Where ω d The learnable parameters of encoder d are represented by ||, which indicates a cascading operation, and FC is used. d and GRU d This indicates the FC layer and GRU used in encoder d.

[0108] Query vector update: After generating the trajectory, the current query vector q j And attention results o j Add them together and feed them into the linear function φ. q To generate the query vector q for the next round. j+1 The calculation is as follows:

[0109] q j+1 =φ q (q j +o j )

[0110] After k iterations, k predicted trajectories are generated. Where j = 1,...,k, to encourage the generation of multiple possible predicted trajectories, a variety of loss functions are used, calculated as follows:

[0111]

[0112] Where ||·||2 represents the L2 norm, Let j represent the predicted trajectory of pedestrian p.

[0113] It's important to note that the trajectory generated by the memory retrieval module is only used to train the network in the training set to extract features from observed and future trajectories; in other words, it trains the network to extract better memories M. The trajectory prediction module, on the other hand, outputs the actual predicted trajectory, essentially predicting the trajectory in the real world.

[0114] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A fast adaptive trajectory prediction method based on online multi-source transfer learning, characterized in that, include: Given n base learners {f i The online learner f is pre-trained in the corresponding source domain for |i=1,...,n} and randomly initialized. t The parameters of the online learner f t It has the same structure as the base learner; Data {X t ,Y t |t=1,2,...,T} arrive in chronological order, X t Y represents the observed trajectory arriving at time t. t This represents the future true trajectory reached at time t. When time t arrives, the online learner f... t Perform trajectory prediction and train the online learner f t Perform an update operation to obtain the online learner f. t+1 ; The update operation includes: Computational online learner f t The predicted loss L at time t t Using the predicted loss L t Update online learning tool f t Get an online learning device Calculate the prediction loss L for each base learner i and online predictors Predicted loss L n+1 ; According to the predicted loss L i Adjusting the weights of the base learner Obtain weights Meanwhile, based on the predicted loss L n+1 Adjusting the online learning device weight Obtain weights For n+1 weights Perform relaxation operations; Using relaxed weights on n base learners and an online learner We perform a weighted summation to obtain the online learner f at time t+1. t+1 .

2. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 1, characterized in that, Using prediction loss L t Update online learning tool f t Get an online learning device The formula is: Where η represents the learning rate.

3. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 1, characterized in that, The formula for adjusting the weights based on the predicted loss is: Where i = 1, ..., n+1, β+(0,1) is the weight decay factor.

4. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 1, characterized in that, The steps involved in calculating the learner's prediction loss at time t include: The learner learns from the observed trajectory X t Predicting the future trajectory of pedestrians, obtaining the predicted future trajectory. Based on the future real trajectory Y t and future predicted trajectory Calculate the prediction loss at time t; where, The learner includes an online learner f t Online learning devices and basis learner {f i |i=1,...,n}.

5. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 1, characterized in that, For n+1 weights The formula for performing relaxation is: Where s is the smoothing coefficient.

6. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 1, characterized in that, The base learner incorporates a memory retrieval module and a trajectory prediction module, and the base learner f is pre-trained in the corresponding source domain. i The steps include: In the source domain Pre-training memory retrieval module; The key-value memory is extracted using a pre-trained memory retrieval module to obtain a key-value external memory M', which stores each key-value memory [H]. j G j ] and the corresponding trajectory data C j , where j = 1, ..., |M′| Based on k-means clustering, pedestrian trajectories are clustered into multiple groups, and the key-value memory corresponding to each group is selected to obtain the external memory M. The trajectory prediction module in the source domain Offline training is performed to obtain the optimal model f. i f, as the i-th base learner i .

7. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 6, characterized in that, The working process of the memory retrieval module includes: The relative displacement of pedestrian p's historical trajectory Relative displacement of future trajectory The data are fed into encoders h and g respectively to generate features H of the observed trajectory and the future trajectory. p and G p And it is stored as the key value of external memory M'; The complete trajectory C of pedestrian p p ={X p ||Y p The corresponding key-value pairs are stored in external memory M', where || represents a cascading operation, used later to cluster the number of memories in external memory M' into a fixed number; Finally, H p G p and the coordinates of the last observation The data is sent to decoder f to generate the future trajectory.

8. The fast adaptive trajectory prediction method based on online multi-source transfer learning according to claim 6, characterized in that, The trajectory prediction module includes an encoder h, a decoder d, and a multi-hop attention mechanism responsible for iteratively querying relevant key value storage from external memory M. The encoder h is pre-trained in the memory retrieval module, and the decoder d is used to output the predicted trajectory. The external memory M' is clustered to become external memory M.

Images