Trajectory representation pre-training system based on deep learning

Abstract

The invention belongs to the technical field of intelligent transportation, and particularly relates to a track representation pre-training system based on deep learning. The invention discloses a track representation pre-training system based on deep learning. The method comprises the following steps: constructing a deep learning model oriented to a multi-scale track structure; the model fully combines static and dynamic features of a road network, and learns feature representation of a moving track from fine granularity to coarse granularity by mixing attention mechanisms of different scales and a self-supervised pre-training normal form, so that richer track space-time semantic information is obtained. The method specifically comprises a road network static feature construction module, a road network dynamic feature construction module, a multi-scale trajectory attention encoder and a travel trajectory self-supervised learning module. The whole pre-training system can enable track features to perceive more spatio-temporal information on different scales, and track vector representation obtained through self-supervised learning in massive tracks can be widely applied to various intelligent traffic system tasks such as travel time estimation and traffic condition prediction.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular relates to a trajectory representation pre-training system based on deep learning. Background Art

[0002] In recent years, with the development of economy and the improvement of living standards, the demand for intelligent transportation related applications has been increasing. Many applications in daily life are closely related to trajectories, including travel time estimation, destination prediction and traffic status prediction. Among them, trajectory representation learning is a very popular research task. It learns the general vector representation of trajectory from massive trajectory data through pre-training, and uses it as a basic model to provide unified features for different downstream trajectory-related tasks, so that these tasks can be efficiently deployed and applied. However, the current trajectory pre-training method lacks multi-scale spatiotemporal information and road network environment information, which limits the effective capture of trajectory semantics. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a multi-scale trajectory representation pre-training system based on deep learning to solve the problem that the current trajectory pre-training methods lack effective learning of multi-scale trajectory information and road network environment.

[0004] The trajectory representation pre-training system based on deep learning provided by the present invention comprises constructing a deep learning model for a multi-scale trajectory structure, wherein the deep learning model comprises a static and dynamic feature combination module of a road network, a multi-scale attention perception module and a self-supervised pre-training module; the model fully combines the static and dynamic features of the road network, and learns the feature representation of mobile trajectories from fine-grained to coarse-grained by mixing attention mechanisms of different scales with a self-supervised pre-training paradigm, thereby obtaining richer trajectory spatiotemporal semantic information; the pre-training system specifically comprises a static and dynamic feature construction module of a road network, a multi-scale trajectory attention encoder module, and a self-supervised pre-training module; the static and dynamic feature construction module of the road network, the multi-scale trajectory attention encoder module, and the self-supervised pre-training module are sequentially connected in series to form a complete system; wherein:

[0005] The static and dynamic feature construction modules of the road network include a module for constructing static features of the road network and a module for constructing dynamic features of the road network; they are used to capture static and dynamic features on the road network and input them into the subsequent attention encoder as the initial features of the trajectory; the static feature construction module is used to capture features such as unchanging topological structures and road types; the dynamic feature construction module is used to capture the features of each section on the road network that change dynamically over time, such as traffic status information such as speed and flow on the section. Here, the trajectory mainly refers to the path trajectory, which is composed of continuous sections on the road network, and a discrete point sequence composed of GPS sampling points is obtained through map matching; the speed and flow information mainly refers to the average speed of vehicles passing through each section per unit time and the number of vehicles passing through.

[0006] The multi-scale trajectory attention encoder module is used to capture the multi-scale feature representation of the trajectory on the road network. Specifically, it uses trajectory sequence representations at different granularity levels to perform intra-layer and inter-layer feature fusion to enrich the spatiotemporal semantic information of the trajectory.

[0007] The self-supervised learning module is used to obtain feature representations that can perceive rich trajectory contexts in massive trajectory data through self-supervision. It uses the masked trajectory restoration task as a supervisory signal to train the model to predict the missing parts of the trajectory in the case of missing data.

[0008] Further:

[0009] (i) In the static and dynamic feature construction modules of the road network:

[0010] (1) The road network static feature construction module may specifically include two parts: constructing the static features of the off-line road network and constructing the static features of the on-line road network. The former can obtain multi-scale static features at the road network level by offline preprocessing; the latter queries the corresponding static features on the road network according to the travel trajectory R, ​​and obtains the multi-scale static feature representation on the trajectory sequence as the model input.

[0011] (1.1) For the construction of static features of the off-line road network, the spectral clustering method is used to cluster the road segments of the road network according to the distance, so as to obtain a coarse-grained road network composed of road segment clusters. Specifically, spectral clustering divides the road network according to the adjacency matrix between nodes, thereby achieving coarse-graining of the network to obtain a specific number of clusters. The goal is to make the distance between nodes within the cluster close, while the average distance between different clusters is as far as possible. If there are road segments connecting two clusters, then the two clusters will also be connected by edges in the coarse-grained network. Then, by continuing to iterate and perform spectral clustering, a higher-scale road network G is obtained. j Then, the embedding vectors of nodes in cluster networks of different scales are obtained through graph embedding learning methods.

[0012] (1.2) For the static feature construction of the online network, for the trajectory R, ​​the sequence in the clustering network at the jth scale can be expressed as Then, for each R j , the static features of each trajectory point can be indexed from the multi-scale static features of the off-line network

[0013] (2) The road network dynamic feature construction module can be specifically divided into two parts, namely, constructing the off-line road network dynamic features and constructing the on-line road network dynamic features. The former can perform self-supervisory pre-training on the historical trajectory data through pre-training, and obtain an encoder model that can perceive the trajectory context. The latter, after obtaining the short-term historical sampling point trajectory set and the trajectory R to be queried, dynamically updates the multi-scale dynamic features of the trajectory from the dynamic feature memory of the road network.

[0014] (2.1) For offline road network multi-scale dynamic feature construction, self-supervised pre-training of historical trajectories is performed by restoring missing trajectories. The purpose is to use this self-supervised pre-training method to allow dynamic features to capture the model's ability to perceive contextual trajectory semantics and better learn the spatiotemporal characteristics of trajectory points. The specific training loss function is as follows:

[0015]

[0016]

[0017] in, Represents a pair of discrete GPS point trajectories τ O The neural network model for encoding can be a recurrent neural network structure or a multi-scale attention structure. The output is the vector representation of the j-th trajectory point feature, W τ ∈R d×d ,b τ ∈R d It is a neural network model The prediction layer parameters, d represents the dimension of the parameter vector; V represents the set of road segments in the road network, r p ∈V represents a specific road section; τ M is the set of obscured points in the historical trajectory τ, τ O It means replacing the trajectory τ with a mask mark in τ M The present invention is to obtain the trajectory after the point in Calculate the cross entropy loss function and pass the logits vector U j ′∈R |V| To predict which road segment the jth point on τ will belong to; After pre-training, Encode the historical trajectory and write the features into the dynamic feature memory.

[0018] (2.2) For online road network multi-scale dynamic feature construction, when the model needs to process the query time t q When the travel trajectory is obtained, the dynamic features stored in the memory can be read by constructing the following ordinary differential equation containing the neural network function [1] (Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud: Neural Ordinary Differential Equations. NeurIPS 2018: 6572-6583):

[0019]

[0020] Neural ODE is a deep learning model that combines neural networks with ODEs. It replaces discrete hierarchical structures with continuous ODE solutions to improve memory efficiency and computational efficiency while providing a continuous modeling method. represents the initial state of the ordinary differential equation, that is, f ψ (·) is a two-layer perceptron network; is a neural network function composed of spatiotemporal convolution modules [2] (Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep Residual Learning for Image Recognition. CoRRabs / 1512.03385 (2015)), with the static storage feature C corresponding to the cluster network of the jth layer j As input, it contains two temporal convolution structures, one spatial convolution structure and a linear output layer; Represents when the query time is t q The dynamic characteristics of the j-th layer cluster network at time , for each road segment r i and its corresponding nodes in the multi-scale network Acquiring multi-scale dynamic features N j Indicates the number of clusters in the j-th layer cluster network;

[0021] (ii) The multi-scale trajectory attention encoder module is used to capture trajectory features on cluster networks of different scales in the road network; and fuse the cross-scale trajectory features in the cluster network, and use the features of different scales to enhance the self-attention mechanism module, so as to achieve adaptive feature extraction and fusion in the neural network; here replaces the classic self-attention mechanism; specifically:

[0022] (1) Fusion of features at the same scale: The purpose of this module is to effectively interact trajectory features within a road segment or cluster network. For the coarse-grained trajectory R at the j∈[1,c]th scale j , whose output at the l-1th layer of the multi-scale attention encoder is Then the intermediate features of the trajectory can be calculated

[0023]

[0024] Among them, LN is the layer normalization operation, H represents the number of computing heads in the multi-head attention, and h represents the calculation of the h-th head; W q ,W k ,W v ∈R d×d Represents the parameters of the linear calculation; then, the features of each attention head are concatenated to form:

[0025]

[0026] Finally, the calculation is:

[0027]

[0028] Among them, FFNLN(x) means FFN(LN(x))+x, and FFN is a two-layer feedforward neural network.

[0029] (2) Cross-scale feature fusion: The purpose of this module is to effectively fuse the fine-grained segment-level trajectory features with the coarse-grained cluster-level trajectory features. The spatiotemporal features of In the features of , the result of cross-scale attention fusion of the u-th point of the trajectory of each scale is calculated:

[0030]

[0031] Concatenate all the features to get:

[0032] R' 0 =R′ 0,0 ‖…‖R′ 0,H-1 ; (14)

[0033] and through Obtaining fine-grained trajectory features

[0034] (3) Different scale enhancement: The purpose of this module is to use coarse-grained trajectory features to enhance the effective extraction of fine-grained trajectory features on road sections. By fusing features across scales, the fused fine-grained trajectory features can be obtained. After that, the internal nodes of the fine-grained trajectory are fused. The coarse-grained information of is used to enhance the extraction capability of trajectory features. For the features of the coarse-grained trajectory, its dot product is calculated in the attention mechanism to obtain the fine-grained trajectory features of the lth layer, namely:

[0035]

[0036] Among them, W α,q ,W α,k ∈R d×d represents the learnable parameters of the linear projection, represents the similarity of trajectory features at the granularity of the jth cluster network, β j is a parameter that controls the overall influence of the j-th cluster network.

[0037] (III) The self-supervised pre-training module aims to use the trajectory restoration task as a self-supervised pre-training task for trajectory representation to effectively learn the travel trajectory representation. The loss function of the training is defined as follows:

[0038]

[0039] Among them, R M Represents the set of masked sections in trajectory R. If there is a section in R M is randomly masked, then R O The corresponding position in will replace the corresponding road section in R with the mask symbol. Calculating cross entropy in represents the jth element of the fine-grained trajectory representation output by the Lth layer of the model.

[0040] The pre-training system of the present invention uses the topological structure of the road network and a large amount of vehicle trajectory data to pre-train the parameters of the model. The purpose is to enable the model to obtain well-trained model parameters and effectively output feature vectors representing the semantics of trajectory driving, so as to facilitate direct use of various downstream tasks, or to apply them after simple fine-tuning and adaptation.

[0041] Specifically, during pre-training, first, input the relevant information of the static road network, including road attributes and topological structure, and use the static feature construction module of the road network to obtain the static features of the road network. Then, input the data of historical trajectories and the road network structure, and use the dynamic feature construction module of the road network to obtain the dynamic features of the road network. Next, pre-train each trajectory in turn, and use the above self-supervised pre-training module to construct a self-supervised pre-training task for each trajectory. The main thing is to mask part of the trajectory and use the remaining trajectory information to predict the missing trajectory information.

[0042] During the prediction process, each masked trajectory is indexed from the saved road network static and dynamic feature memory according to the road section passed and the query time, and the static and dynamic features of each road section are obtained, which are input into the multi-scale attention encoder to obtain a vector representation representing the trajectory semantics, and then connected to a simple linear layer for classification prediction of the target road section.

[0043] The present invention effectively solves the defect that the vector representation of the vehicle's driving trajectory lacks multi-scale semantic information in space, and makes more full use of the characteristics of the road network topological structure. The present invention can effectively model the trajectory and road network, and obtain feature vectors representing the multi-scale semantic information of the trajectory journey from massive historical trajectory data, thereby facilitating the use of various traffic-related downstream tasks including travel time estimation, destination prediction, etc., and effectively improving the application scope of artificial intelligence in urban transportation. Compared with existing solutions, the present invention can capture more static and dynamic information of the road network, and use a multi-scale attention encoder to more effectively integrate the travel spatiotemporal information of different scales, and more effectively refine the traffic semantics represented by the trajectory, thereby obtaining a more general vector representation to adapt to a variety of traffic-related tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of the structure of the trajectory representation pre-training system based on deep learning proposed in the present invention.

[0045] Figure 2 A detailed schematic diagram of a multi-scale road network used by the trajectory representation pre-training system proposed in the present invention and a query trajectory involved. DETAILED DESCRIPTION

[0046] The present invention is further described below with reference to specific examples and accompanying drawings.

[0047] Figure 1This is a schematic diagram of the structure of the trajectory representation pre-training system based on deep learning proposed by the present invention. The system includes a static and dynamic feature construction module of the road network, a multi-scale trajectory attention encoder module, and a self-supervised learning module. The static and dynamic feature construction module of the road network, the multi-scale trajectory attention encoder module, and the self-supervised learning module are connected in series to form a complete system.

[0048] Figure 2 A detailed schematic diagram of a multi-scale road network and a multi-scale trajectory involved in the trajectory representation pre-training system proposed in the present invention. In this embodiment, the road network is subjected to two iterative clustering operations to form a three-layer progressive structure. In the first layer, 13 road sections are clustered with road sections as nodes to obtain a cluster structure composed of road section sets, forming a cluster network of 6 clusters in the second layer. In the next layer, the cluster structure of the second layer is clustered to obtain a third-layer cluster network of further 3 clusters. In this embodiment, there is a queried trajectory, and there is a corresponding representation in each layer of the road network.

[0049] The off-line network static feature construction module uses the spectral clustering method to cluster the 13 road sections of the road network at two scales, and obtains a coarse-grained road network with the road section cluster as the node, which contains 6 nodes and 3 nodes respectively. 1 ,r 2 ,…,r 22 , can be captured by graph embedding learning methods. For the online network static feature construction module, for the trajectory R, ​​its movement in the third scale clustering network can be expressed as:

[0050] R 0 = {r 1 ,r 2 ,r 3 ,r 4 ,r 5},R 1 = {r 14 ,r 15 ,r 16},R 2 = {r 21}.

[0051] Then, for each R j ,The static features of each trajectory point are indexed from the multi-scale static features of the off-line network.

[0052] The road network dynamic feature construction module can be specifically divided into two parts, namely, constructing offline road network dynamic features and constructing online road network dynamic features. The former can perform self-supervised pre-training on historical trajectory data through pre-training, and obtain an encoder model that can perceive trajectory context. After obtaining the short-term historical sampling point trajectory set and the trajectory R to be queried, the latter dynamically updates the multi-scale dynamic features of the trajectory from the dynamic feature memory of the road network. For the offline road network multi-scale dynamic feature construction, the purpose is to allow the dynamic features to capture the model's ability to perceive contextual trajectory semantics through this self-supervised pre-training method, and better learn the spatiotemporal characteristics of trajectory points. The specific training loss function is as follows:

[0053]

[0054] in, The vector representation of the j-th trajectory point feature output by the model, W τ ∈R 64×64 ,b τ ∈R 64 is the parameter of the model, τ M is the set of obscured points in the historical trajectory τ, τ O It means replacing the trajectory τ with a mask mark in τ M The trajectory obtained after the points in Calculate the cross entropy loss function and pass the logits vector U′ j ∈R 13 To predict which road segment the jth point on τ will belong to. After pre-training, Encode the historical trajectory and write the features into the dynamic feature memory.

[0055] For online road network multi-scale dynamic feature construction, when the model needs to process a travel trajectory with a query time of 30 minutes, the dynamic features stored in the memory are read in the following way:

[0056]

[0057] in, represents the initial state of the ordinary differential solver, that is, It is a two-layer perceptron network. It is a spatiotemporal convolution module, which contains two temporal convolution structures, one spatial convolution structure and a linear output layer. It represents the dynamic characteristics of the j-th layer cluster network when the query time is 30. For each road segment r i and its corresponding nodes in the multi-scale network Can obtain multi-scale dynamic features For example, the road segment r 1 , its corresponding node in the cluster is r 14 ,r 21 , then its corresponding dynamic feature is

[0058] The multi-scale trajectory attention encoder module uses feature fusion of different scales and multi-scale enhanced self-attention module to replace the original self-attention mechanism, and the H of the attention mechanism is set to 4. For the coarse-grained trajectory R of the j∈[1,2]th scale j , output from the previous layer of the model The trajectory characteristics are calculated in

[0059]

[0060] Then, we concatenate the trajectory features of each head aggregation to form Finally, we calculate Where FFNLN(x) represents FFN(LN(x))+x. In order to convert the coarse-grained trajectory The spatiotemporal features of Among the features of , the result of cross-scale attention fusion of the u-th point of the trajectory of each scale is calculated.

[0061]

[0062] Concatenate all features to get R' 0 =R′ 0,0 ‖R′ 0,1 ‖R′ 0,2 ‖R′ 0,3 and through Obtaining fine-grained trajectory features By fusing features across scales, we can obtain the fused fine-grained trajectory features. Perform feature fusion on the internal nodes of fine-grained trajectories. The coarse-grained information of is used to enhance the extraction capability of trajectory features. For the features of the coarse-grained trajectory, its dot product is calculated in the attention mechanism to obtain the fine-grained trajectory features of the lth layer, that is,

[0063]

[0064] The self-supervised pre-training module uses the trajectory restoration task as the pre-training task of the travel trajectory to effectively learn the travel trajectory representation. In this example, define R M = {R 2 ,R3},R O = {R 1 ,[mask],[mask],R 4 ,R 5 The loss function of the training is defined as follows:

[0065]

[0066] The present invention is achieved by Calculating cross entropy in represents the jth element of the fine-grained trajectory representation output by the Lth layer of the model.

[0067] The real trajectory datasets and road network data of Shanghai and Chengdu were used as experimental objects for testing. Among them, the Shanghai dataset has a total of 1.13 million trajectories, involving 7954 road sections, and the maximum trajectory length is 3.2km. The Chengdu dataset has a total of 1.26 million trajectories, involving 7055 road sections, and the maximum trajectory length is 3.4km. This paper selects several advanced methods for comparison, namely

[0068] HRNR(Ning Wu,Wayne Xin Zhao,Jingyuan Wang,Dayan Pan:LearningEffective Road Network Representation with Hierarchical Graph NeuralNetworks.KDD 2020:6-14),

[0069] Tremblr(FU TY,LEE W C.Trembr:Exploring road networks for trajectoryrepresentation learning[J].ACM Transactions on Intelligent Systems andTechnology,2020,11(1):1-25)

[0070] and START (JIANG J, PAN D, REN H, et al. Self-supervised trajectory representation learning with temporal regularities and travel semantics[C] / / Proceedings of the 39 thIEEE International Conference on DataEngineering.2023:843-855.).

[0071] In the downstream tasks, we select travel time estimation, destination prediction, next k-hop prediction, and moving distance prediction tasks to test the capabilities of the pre-trained trajectory encoding model. Among them, travel time estimation predicts the total time cost of the vehicle to complete the journey, with mean absolute percentage error (MAPE) and mean absolute error (MAE) as the measurement. Destination prediction predicts the destination of the journey given a partial trajectory, with top-20 and top-50 accuracy (A@20 and A@50) as the measurement. The next k-hop prediction task predicts the kth road segment in the future given a partial trajectory, with Ak#10 and Ak#15 representing the prediction accuracy of the next 10th and 15th road segments. Moving distance prediction predicts the actual mileage of the vehicle given the journey consisting of the passed road segments, with mean absolute percentage error (MAPE) and mean absolute error (MAE) as the measurement.

[0072] The following is a table of statistical results of the experiment. It can be seen from the experimental results that the method of the present invention has achieved the best performance in multiple downstream tasks and has a significant advantage in trajectory pre-training.

[0073]

Claims

1. A trajectory representation pre-training system based on deep learning, characterized in that: It includes building a deep learning model for multi-scale trajectory structure; the model fully combines the static and dynamic characteristics of the road network, and learns trajectory feature representations of different granularities by mixing attention mechanisms of different scales with self-supervised pre-training paradigms, thereby obtaining richer spatiotemporal semantic information; specifically, it includes a static and dynamic feature construction module for the road network, a multi-scale attention encoder module, and a self-supervised pre-training module; among which: The static and dynamic feature construction module of the road network includes a road network static feature construction module and a road network dynamic feature construction module; it is used to capture the static and dynamic features on the road network and input them into the subsequent trajectory encoding model as the initial features of the trajectory; the static feature construction module is used to capture the topological structure and road type of the road network; the dynamic feature construction module is used to capture the characteristics of each road section on the road network that change dynamically over time, including the speed and flow traffic state information on the road section; here, the trajectory refers to the path sequence of vehicle travel, which is composed of continuous road sections on the road network; the speed and flow information refers to the average speed of vehicles passing through each road section per unit time and the number of vehicles passing through; The multi-scale attention encoder module is used to capture the multi-scale feature representation of the trajectory on the road network; specifically, it uses trajectory sequence representations at different granularity levels to perform intra-layer and inter-layer feature fusion to enrich the spatiotemporal semantic information of the trajectory; The self-supervised pre-training module is used to obtain feature representations that can perceive rich trajectory contexts in massive trajectory data through self-supervision; specifically, the mask trajectory restoration task is used as a supervisory signal to construct a task of predicting missing segments in the trajectory.

2. The deep learning-based trajectory representation pre-training system according to claim 1, characterized in that: In the static and dynamic features building blocks of the road network: (1) The road network static feature construction module specifically includes two parts: constructing the static features of the off-line network and constructing the static features of the on-line network. The former obtains the multi-scale static features of the road network level by means of offline preprocessing. The latter queries the corresponding static features on the road network according to the travel trajectory, and obtains the multi-scale static feature representation on the trajectory sequence as the model input. The construction of the static features of the off-line road network uses spectral clustering technology to cluster the road segments in the road network, thereby forming a coarse-grained road network with road segment clusters as nodes. Specifically, spectral clustering divides the road network by analyzing the adjacency matrix between nodes to generate a specific number of clusters, and its goal is to minimize the distance between nodes in the same cluster and maximize the distance between different clusters. If there are interconnected road segments in two clusters, then there will be connecting edges between the two clusters in the generated coarse-grained network. By iteratively performing spectral clustering on road networks of different scales, a higher-scale road network G is obtained. j ; Then, the graph embedding learning method is used to learn the embedding vectors of nodes in cluster networks of different scales The construction of the static features of the online road network, for a modeled trajectory R, ​​the number of road sections it passes through is m, and its movement in the j-th scale clustering network is expressed as Then, for each R j , the static features of each trajectory point are indexed from the multi-scale static features of the off-line network (2) The road network dynamic feature construction module is specifically divided into two parts: constructing the off-line network dynamic features and constructing the on-line network dynamic features. The former performs self-supervisory pre-training on the historical trajectory data through pre-training, and obtains an encoder model that can perceive the trajectory context. The latter dynamically updates the multi-scale dynamic features of the trajectory from the dynamic feature memory of the road network after obtaining the short-term historical sampling point trajectory set and the trajectory R to be queried. The offline multi-scale dynamic features of the road network are constructed by self-supervising pre-training the historical trajectories by restoring the missing trajectory tasks. Through this self-supervised restoration method, the encoder-to-model ability to perceive the context trajectory semantics is improved to better learn the spatiotemporal features of the trajectory points. The specific training loss function is as follows: in, Represents a pair of discrete GPS point trajectories τ O The neural network model for encoding is a recurrent neural network structure or a multi-scale attention structure, in which The vector representation of the output j-th trajectory point feature, W τ ∈R d×d ,b τ ∈R d It is a neural network model The prediction layer parameters, d represents the dimension of the parameter vector; V represents the set of road segments in the road network, r p ∈V represents a specific road section; τ M is the set of obscured points in the historical trajectory τ, τ O It means replacing the trajectory τ with a mask marker in τ M The trajectory obtained after the point in ; use Calculate the cross entropy loss function and pass the logits vector U j ′ ∈R |V| To predict which road segment the jth point on τ will belong to; After pre-training, Encode the historical trajectory and write the features into the dynamic feature memory; The online construction of multi-scale dynamic features of the road network is performed when the model needs to process the query time t q When the travel trajectory is calculated, the dynamic features stored in the memory are read by constructing the following ordinary differential equation containing the neural network function: in, represents the initial state of the ordinary differential equation, that is, f ψ (·) is a two-layer perceptron network; It is a neural network function composed of spatiotemporal convolution modules, with static storage features C corresponding to the cluster network of the jth layer j As input, it contains two temporal convolution structures, one spatial convolution structure and a linear output layer; Represents when the query time is t q Dynamic features on the j-th layer cluster network at the time; for each road segment r i and its corresponding nodes in the multi-scale network Acquiring multi-scale dynamic features N j Represents the number of clusters in the j-th layer cluster network.

3. The deep learning-based trajectory representation pre-training system according to claim 2, characterized in that: The multi-scale trajectory attention encoder module is used to capture trajectory features on cluster networks of different scales in the road network; and fuse the cross-scale trajectory features in the cluster network, and use the features of different scales to enhance the self-attention mechanism module, so as to achieve adaptive feature extraction and fusion in the neural network; specifically: (1) Feature fusion at the same scale: For the coarse-grained trajectory R at the j∈[1,c]th scale j , whose output at the l-1th layer of the multi-scale attention encoder is Then the intermediate features of the trajectory can be calculated Among them, LN is the layer normalization operation, H represents the number of computing heads in the multi-head attention, and h represents the calculation of the h-th head; W q ,W k ,W v ∈R d×d Represents the parameters of the linear calculation; then, the features of each attention head are concatenated to form: Finally, the calculation is: Among them, FFNLN(x) means FFN(LN(x))+x, and FFN is a two-layer feedforward neural network; (2) Cross-scale feature fusion: In order to integrate coarse-grained trajectories The spatiotemporal features of In the features of , the result of cross-scale attention fusion of the u-th point of the trajectory of each scale is calculated: Concatenate all the features to get: R ‘ 0=R ′ 0,0 ‖…‖R ′ 0,H -1 ; (14) and through Obtaining fine-grained trajectory features (3) Enhancement at different scales: By fusing features across scales, we can obtain fused fine-grained trajectory features. After that, the internal nodes of the fine-grained trajectory are fused with features; the trajectory features from other cluster networks are used The coarse-grained information of is used to enhance the extraction ability of trajectory features; for the features of the coarse-grained trajectory, its dot product on the attention mechanism is calculated to obtain the fine-grained trajectory features of the lth layer, that is: Among them, W α,q ,W α,k ∈R d×d represents the learnable parameters of the linear projection, represents the similarity of trajectory features at the granularity of the jth cluster network, β j is a parameter that controls the overall influence of the j-th cluster network.

4. The deep learning-based trajectory representation pre-training system according to claim 3, characterized in that: The self-supervised pre-training module uses the trajectory restoration task as the pre-training task of the travel trajectory to learn the travel trajectory representation; its loss function is defined as follows: Among them, R M Represents the set of masked sections in trajectory R. If there is a section in R M is randomly masked, then R O The corresponding position in will replace the corresponding road segment in R with the mask symbol; Calculate cross entropy logits in represents the jth element of the fine-grained trajectory representation output by the Lth layer of the model.

Images