A deep learning based trajectory representation pre-training system

Abstract

The application belongs to the technical field of intelligent transportation, and particularly relates to a trajectory representation pre-training system based on deep learning. The trajectory representation pre-training system based on deep learning comprises a deep learning model for constructing a multi-scale trajectory structure; the model fully combines static and dynamic features of a road network, and learns feature representation of a moving trajectory from fine granularity to coarse granularity through a mixed attention mechanism of different scales and a self-supervised pre-training paradigm, so that richer trajectory spatiotemporal semantic information is obtained. The application specifically comprises a road network static feature construction module, a road network dynamic feature construction module, a multi-scale trajectory attention encoder, and a self-supervised learning module of a travel trajectory. The whole pre-training system can make the trajectory feature perceive more spatiotemporal information in different scales, and the trajectory vector representation obtained through self-supervised learning in a large amount of trajectories can be widely applied to various intelligent transportation system tasks such as travel time estimation and traffic condition prediction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent transportation, and particularly relates to a trajectory representation pre-training system based on deep learning. BACKGROUND

[0002] In recent years, with the development of economy and the improvement of living standards, the demand for intelligent transportation related applications has been rising. Many daily life applications are closely related to trajectories, including travel time estimation, endpoint prediction and traffic state prediction. Among them, trajectory representation learning is a very popular research task. It learns the general vector representation of trajectories from massive trajectory data through pre-training, and uses it as a basic model to provide unified features for different trajectory related tasks downstream, so as to efficiently deploy and apply these tasks. However, the current trajectory pre-training method lacks multi-scale spatio-temporal information and road network environment information, which limits the effective capture of trajectory semantics. SUMMARY

[0003] In view of the above problems, the purpose of the present application is to provide a multi-scale trajectory representation pre-training system based on deep learning to solve the problem of lack of effective learning of multi-scale trajectory information and road network environment in the current trajectory pre-training method.

[0004] The trajectory representation pre-training system based on deep learning provided by the present application includes a deep learning model for multi-scale trajectory structure construction, which includes a road network static and dynamic feature combination module, a multi-scale attention perception module and a self-supervised pre-training module. The model fully combines the static and dynamic features of the road network, and learns the feature representation of moving trajectories from fine granularity to coarse granularity by combining different scale attention mechanisms and self-supervised pre-training paradigm, thereby obtaining more rich trajectory spatio-temporal semantic information. The pre-training system specifically includes a road network static and dynamic feature construction module, a multi-scale trajectory attention encoder module and a self-supervised pre-training module. The road network static and dynamic feature construction module, the multi-scale trajectory attention encoder module and the self-supervised pre-training module are connected in series to form a complete system. Wherein:

[0005] The static and dynamic feature construction module of the road network includes a static feature construction module and a dynamic feature construction module. These modules capture static and dynamic features of the road network and input them as initial features for the trajectory into the subsequent attention encoder. The static feature construction module captures invariant topological structures and road types, while the dynamic feature construction module captures features that dynamically change over time for each road segment, such as traffic state information like speed and flow rate. Here, the trajectory mainly refers to a path trajectory, which is composed of continuous road segments on the road network and obtained through map matching from a discrete point sequence formed by GPS sampling points. The speed and flow rate information mainly refers to the average speed and number of vehicles passing each road segment per unit time.

[0006] The multi-scale trajectory attention encoder module is used to capture multi-scale feature representations of trajectories on the road network. Specifically, it uses trajectory sequence representations at different granularity levels to perform intra-layer and inter-layer feature fusion to enrich the spatiotemporal semantic information of the trajectory.

[0007] The self-supervised learning module is used to obtain feature representations that are aware of rich trajectory context from massive trajectory data through self-supervision. It uses the masked trajectory reconstruction task as a supervision signal to train the model to predict missing parts of the trajectory in the case of missing data.

[0008] Furthermore:

[0009] (a) In the static and dynamic feature construction module of the road network:

[0010] (1) The road network static feature construction module can specifically have two parts: constructing offline road network static features and constructing on-road road network static features. The former can obtain multi-scale static features at the road network level through offline preprocessing; the latter queries the corresponding static features on the road network based on the travel trajectory R and obtains the multi-scale static feature representation on the trajectory sequence as the model input.

[0011] (1.1) For the construction of static features of the off-road network, spectral clustering is used to cluster road segments according to their distance, resulting in a coarse-grained road network composed of segment clusters. Specifically, spectral clustering divides the road network based on the adjacency matrix between nodes, thereby achieving coarse-grained network formation to obtain a specific number of clusters. The goal is to make the distance between nodes within a cluster as close as possible, while maximizing the average distance between different clusters. If road segments in two clusters are connected, then these two clusters will also be connected by an edge in the coarse-grained network. Then, by continuing to iterate and perform spectral clustering, a higher-scale road network G is obtained. j Then, the embedding vectors of nodes in cluster networks of different scales are learned through graph embedding learning methods.

[0012] (1.2) For the static feature construction of the railway network, the sequence of trajectory R in the j-th scale cluster network can be represented as follows: Then, for each R j The static features of each trajectory point can be indexed from the multi-scale static features of the off-line network.

[0013] (2) The road network dynamic feature construction module can be divided into two parts: constructing off-line road network dynamic features and constructing on-line road network dynamic features. The former can be pre-trained using historical trajectory data through self-supervised pre-training to obtain an encoder model that can perceive the trajectory context. The latter, after obtaining the short-term historical sampling point trajectory set and the trajectory R to be queried, dynamically updates the multi-scale dynamic features of the trajectory from the road network's dynamic feature memory.

[0014] (2.1) For offline multi-scale dynamic feature construction of road networks, self-supervised pre-training of historical trajectories is performed by reconstructing missing trajectories. The aim is to enable dynamic features to capture the model's ability to perceive the contextual semantics of trajectories through this self-supervised pre-training method, thereby better learning the spatiotemporal features of trajectory points. The specific training loss function is as follows:

[0015]

[0016]

[0017] in, Represents a discrete GPS point trajectory τ O The neural network model used for encoding can be a recurrent neural network structure or a multi-scale attention structure. The output represents the vector representation of the features of the j-th trajectory point, W. τ ∈R d×d ,b τ ∈R d It is a neural network model The prediction layer parameters, where d represents the dimension of the parameter vector; V represents the set of road segments in the road network, r p ∈V represents a specific road segment; τ M It is the set of points that are obscured in the historical trajectory τ, τ O This means replacing the trajectory τ with a mask marker in τ. M The trajectory obtained after the points in the middle. This invention is for... Calculate the cross-entropy loss function and pass it through the logits vector U. j ′∈R |V| To predict which road segment the j-th point on τ will belong to; After pre-training, the history trajectory is encoded and the features are written into the dynamic feature memory.

[0018] (2.2) For online multi-scale dynamic feature construction of road network, when the model needs to process the query trajectory at time t q , the dynamic features stored in the memory can be read by constructing the following ordinary differential equation [1] (Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud: Neural Ordinary Differential Equations. NeurIPS 2018: 6572-6583) containing neural network functions:

[0019]

[0020] Neural ordinary differential equation is a deep learning model that combines neural network and ordinary differential equation. By replacing the discrete hierarchical structure with continuous ODE solution, it improves the memory and computing efficiency, and provides continuous modeling method. Among them, represents the initial state of the ordinary differential equation, i.e. f ψ (·) is a two-layer perceptron network; is a neural network function composed of spatio-temporal convolution module [2] (Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep Residual Learning for Image Recognition. CoRR abs / 1512.03385 (2015)), which takes the static storage features C j corresponding to the jth layer cluster network as input, containing two time convolution structures, one spatial convolution structure and one linear output layer; represents the dynamic feature of the jth layer cluster network when the query time is t q . For each road segment r i and its corresponding node in the multi-scale network, the multi-scale dynamic feature N is obtained. j represents the number of clusters of the jth layer cluster network;

[0021] ​(ii) the multi-scale trajectory attention encoder module is used to capture trajectory features on different scales of cluster networks in a road network, and to fuse cross-scale trajectory features in a clustering cluster network, and to enhance a self-attention mechanism module using different scale features, so as to achieve adaptive feature extraction and fusion in a neural network; here, the classical self-attention mechanism is replaced; specifically:

[0022] (1) Same scale feature fusion; the purpose of this module is to effectively interact the trajectory features within a road segment or a cluster network. For the j-th scale trajectory R j , the output of the l-1 layer of the multi-scale attention encoder is Then the intermediate feature of the trajectory can be calculated as

[0023]

[0024] Where LN is the layer normalization operation, H represents the number of calculation heads in the multi-head attention, h represents the calculation of the h-th head, and W q ,W k ,W v ∈R d×d represents the parameters of linear calculation; then, the features of each attention head are spliced to form:

[0025]

[0026] Finally, the following is calculated:

[0027]

[0028] Where FFNLN(x) represents FFN(LN(x))+x, and FFN is a two-layer feedforward neural network.

[0029] (2) Cross-scale feature fusion; the purpose of this module is to effectively fuse the fine-grained road segment-level trajectory features and the coarse-grained clustering cluster-level trajectory features. In order to fuse the spatio-temporal features of the coarse-grained trajectory into the features of the fine-grained trajectory , the cross-scale attention fusion result of the u-th point of each scale trajectory is calculated as:

[0030]

[0031] All features are spliced to obtain:

[0032] R‘0=R′ 0,0 ‖…‖R′ 0,H-1 ; (14)

[0033] and through Obtaining fine-grained trajectory features

[0034] (3) Different scale enhancement; the purpose of this module is to enhance the effective extraction of trajectory features on fine-grained road segments using coarse-grained trajectory features. By cross-scale fusion of features, the fused fine-grained trajectory features can be obtained After that, the internal nodes of the fine-grained trajectory are fused. The extracting ability of the trajectory features is enhanced by using the coarse-grained information of the trajectory features from other cluster networks. For the features of the coarse-grained trajectory, the dot product in the attention mechanism is calculated to obtain the fine-grained trajectory features of the lth layer, i.e.

[0035]

[0036] where W α,q ,W α,k ∈R d×d represents the learnable parameters of linear projection, represents the trajectory feature similarity at the granularity of the jth cluster network, β j is a parameter that controls the overall influence of the jth cluster network.

[0037] (Three) The self-supervised pre-training module aims to use the trajectory restoration task as a self-supervised pre-training task for trajectory representation to effectively learn the trajectory representation. The loss function of the training is defined as follows:

[0038]

[0039] where R M represents the set of road segments in the trajectory R that are masked, if there are road segments in R M that are randomly masked, the corresponding position in R O will be replaced by the mask symbol corresponding to the corresponding road segment in R. The present application calculates the cross-entropy represents the jth element of the fine-grained trajectory representation output by the Lth layer of the model.

[0040] The pre-training system of the present application will use the topological structure of the road network and the massive trajectory data of vehicle travel to pre-train the parameters of the model. The purpose is to make the model have well-trained model parameters, and can effectively output a feature vector representing the semantic of trajectory travel, so as to facilitate the direct use of various downstream tasks, or after simple fine-tuning and adaptation, it can be applied.

[0041] ​​Specifically, during pre-training, first, the relevant information of the static road network is input, including road attributes and topological structure, etc., and the static features of the road network are obtained by using the static feature construction module of the road network. Then, the data of the historical trajectory and the road network structure are input, and the dynamic features of the road network are obtained by using the dynamic feature construction module of the road network. Then, each trajectory is pre-trained in turn, and the self-supervised pre-training module is used to construct a self-supervised pre-training task for each trajectory. Mainly, part of the trajectory is masked, and the remaining trajectory information is used to predict the missing trajectory information.

[0042] In the prediction process, each masked trajectory is indexed from the saved road network static and dynamic feature storage according to the passed road section and query time to obtain the static and dynamic features of each road section, which are input into the multi-scale attention encoder to obtain a vector representation representing the semantic of the trajectory, and then access a simple linear layer for classification prediction of the target road section.

[0043] The present application effectively solves the defect that the vector representation of the driving trajectory of the vehicle lacks multi-scale semantic information in space, and more fully utilizes the features of the road network topological structure. The present application can effectively model the trajectory and the road network, obtain a feature vector representing the multi-scale semantic information of the trajectory trip from the massive historical trajectory data, so as to facilitate the use of various traffic-related downstream tasks including trip time estimation, endpoint prediction, etc., and effectively improve the application range of artificial intelligence in urban traffic. Compared with the existing scheme, the present application can capture more static and dynamic information of the road network, and more effectively integrate the different scale of trip space-time information using the multi-scale attention encoder, more effectively extract the traffic semantic represented by the trajectory, so as to obtain a more general vector representation to adapt to various traffic-related tasks. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The structure diagram of the trajectory representation pre-training system based on deep learning proposed by the present application.

[0045] Figure 2 The detail diagram of the multi-scale road network and the trajectory of a query involved in the trajectory representation pre-training system proposed by the present application. DETAILED DESCRIPTION

[0046] The present application will be further described below in combination with specific examples and drawings.

[0047] Figure 1The pre-training system structure diagram of trajectory representation based on deep learning is provided in the present application. The system comprises a static and dynamic feature construction module of road network, a multi-scale trajectory attention encoder module and a self-supervised learning module. The static and dynamic feature construction module of road network, the multi-scale trajectory attention encoder module and the self-supervised learning module are connected in series to form a complete system.

[0048] Figure 2 The details of a multi-scale road network and a multi-scale trajectory involved in the trajectory representation pre-training system provided in the present application are shown in the schematic diagram. In the embodiment, the clustering operation is performed twice on the road network, thereby forming a 3-layer progressive structure. In the first layer, 13 road segments are clustered to obtain a cluster structure composed of road segment sets, thereby forming a cluster network of 6 clusters in the second layer. In the next layer, the cluster structure in the second layer is further clustered to obtain a third layer cluster network of 3 clusters. In the embodiment, there is a query trajectory, and the trajectory has a corresponding representation in each layer of the road network.

[0049] The offline road network static feature construction module adopts a spectral clustering method to perform clustering on the 13 road segments of the road network in two scales, thereby obtaining a coarse-grained road network comprising 6 nodes and 3 nodes respectively in the form of road segment clusters. For each node r1, r2, …, r 22 , an embedding vector value For the online road network static feature construction module, the movement of the trajectory R in the third scale cluster network can be represented as:

[0050] R0={r1,r2,r3,r4,r5},R1={r 14 ,r 15 ,r 16},R2={r 21}。

[0051] Then, for each R j , the static feature of each trajectory point is indexed from the multi-scale static feature of the offline road network.

[0052] The road network dynamic feature construction module can be specifically divided into two parts, namely constructing offline road network dynamic features and constructing online road network dynamic features. The former can be pre-trained to perform self-supervised pre-training on historical trajectory data, and an encoder model capable of perceiving trajectory context is obtained. The latter dynamically updates the multi-scale dynamic features of the trajectory from the dynamic feature storage of the road network after obtaining the short-time historical sampling point trajectory set and the trajectory R to be queried. For offline road network multi-scale dynamic feature construction, the purpose is to enable the dynamic feature to capture the ability of the model to perceive the context trajectory semantics through such self-supervised pre-training, and better learn the spatiotemporal features of the trajectory points. The specific training loss function is as follows:

[0053]

[0054] wherein, represents the vector representation of the jth trajectory point feature output by the model, W τ ∈R 64×64 ,b τ ∈R 64 is the parameter of the model, τ M is the set of points in the historical trajectory τ that are masked, τ O represents the trajectory obtained by replacing the points in the trajectory τ M with a mask marker. The cross-entropy loss function is calculated by , and the logits vector U′ j ∈R 13 is used to predict which road segment the jth point on τ belongs to. After pre-training, , the historical trajectory is encoded by , and the features are written into the dynamic feature storage.

[0055] For online road network multi-scale dynamic feature construction, when the model needs to process a journey trajectory with a query time of 30 minutes, the dynamic features stored in the storage are read in the following manner,

[0056]

[0057] wherein, represents the initial state of the ordinary differential solver, i.e. is a double-layer perceptron network. is a spatiotemporal convolution module, which includes two convolution structures in time, one convolution structure in space, and a linear output layer. represents the dynamic features of the jth layer cluster network at the query time of 30. For each road segment r i and its corresponding node in the multi-scale network, multi-scale dynamic features can be obtained For example, road segment r1 corresponds to node r in the cluster. 14 ,r 21 Then its corresponding dynamic characteristics are

[0058] The multi-scale trajectory attention encoder module replaces the original self-attention mechanism with feature fusion at different scales and a multi-scale enhanced self-attention module, with the attention mechanism's H set to 4. For the coarse-grained trajectory R at the j-th scale [1,2]... j Output from the previous layer of the model Trajectory features are calculated

[0059]

[0060] Then, this paper concatenates the trajectory features aggregated from each head to form... Finally, the calculation yielded Where FFNLN(x) represents FFN(LN(x))+x, in order to convert the coarse-grained trajectory The spatiotemporal features are fused into fine-grained trajectories. In the features, the cross-scale attention fusion result of the trajectory at the u-th point of each scale is calculated.

[0061]

[0062] By concatenating all features, we obtain R'0 = R' 0,0 ‖R′ 0,1 ‖R′ 0,2 ‖R′ 0,3 and through Obtaining fine-grained trajectory features By fusing features across scales, fine-grained trajectory features can be obtained. Feature fusion is performed on the internal nodes of fine-grained trajectories. Trajectory features from other cluster networks are utilized. The coarse-grained information is used to enhance the trajectory feature extraction capability. For the features of the coarse-grained trajectory, its dot product in the attention mechanism is calculated to obtain the fine-grained trajectory features of the l-th layer, i.e.

[0063]

[0064] The self-supervised pre-training module utilizes the trajectory reconstruction task as the pre-training task for travel trajectories, effectively learning the travel trajectory representation. In this example, R is defined as... M ={R2,R3},R OThe loss function of the training = {R1, [mask], [mask], R4, R5} is defined as follows:

[0065]

[0066] The present application is through Calculate the cross-entropy wherein represents the jth element of the fine-grained trajectory representation output by the Lth layer of the model.

[0067] The real trajectory data set and road network data of Shanghai and Chengdu were tested as experimental objects. Among them, the Shanghai data set has a total of 1.13 million trajectories, involving 7954 road segments, and the maximum trajectory length is 3.2 km. The Chengdu data set has a total of 1.26 million trajectories, involving 7055 road segments, and the maximum trajectory length is 3.4 km. The present application selects several advanced methods for comparison, which are

[0068] HRNR (Ning Wu, Wayne Xin Zhao, Jingyuan Wang, Dayan Pan: Learning Effective Road Network Representation with Hierarchical Graph Neural Networks. KDD 2020: 6-14),

[0069] Tremblr (FU T Y, LEE W C. Trembr: Exploring road networks for trajectory representation learning [J]. ACM Transactions on Intelligent Systems and Technology, 2020, 11 (1): 1-25)

[0070] and START (JIANG J, PAN D, REN H, et al. Self-supervised trajectory representation learning with temporal regularities and travel semantics [C] / / Proceedings of the 39 th IEEE International Conference on Data Engineering. 2023: 843-855.).

[0071] On downstream tasks, the travel time estimation, end point prediction, next k-hop prediction, and moving distance prediction tasks are selected to test the ability of the pre-trained trajectory encoding model. Among them, the travel time estimation predicts the total time cost of the vehicle passing through the journey, and the mean absolute percentage error (MAPE) and the mean absolute error (MAE) are used as the measurement. The end point prediction gives a part of the form of the trajectory, and predicts the end point of the journey, and the top-20 and top-50 accuracy (A@20 and A@50) are used as the measurement. The next k-hop prediction task gives a part of the trajectory, and predicts what the kth road segment of the future is, and Ak#10 and Ak#15 represent the prediction accuracy of the next 10th road segment and the 15th road segment. The moving distance prediction gives a journey composed of road segments, and predicts the actual distance traveled by the vehicle, and the mean absolute percentage error (MAPE) and the mean absolute error (MAE) are used as the measurement.

[0072] The following is the table statistical result of the experiment. From the experimental results, it can be seen that the method of the present application has the best performance on multiple downstream tasks, and has a significant advantage in trajectory pre-training.

[0073]

Claims

1. A deep learning based trajectory representation pre-training system, characterized in that, The method comprises constructing a deep learning model oriented to a multi-scale trajectory structure; the model fully combines static and dynamic features of a road network, and learns trajectory feature representations of different granularities by mixing attention mechanisms of different scales and a self-supervised pre-training paradigm, so that richer spatiotemporal semantic information is obtained; the method comprises a static and dynamic feature construction module of the road network, a multi-scale attention encoder module, and a self-supervised pre-training module; wherein: The static and dynamic feature construction module of the road network comprises a static feature construction module of the road network and a dynamic feature construction module of the road network; the static and dynamic feature construction module is used for capturing static and dynamic features on the road network and serving as initial feature input of the trajectory to a subsequent trajectory encoding model; the static feature construction module is used for capturing a topological structure and road types of the road network; the dynamic feature construction module is used for capturing dynamic features of each road segment on the road network over time, including speed and flow traffic state information on the road segment; here, the trajectory refers to a path sequence of a vehicle, which is composed of continuous road segments on the road network; the speed and flow information refers to average speed and number of passing vehicles per unit time on each road segment; The multi-scale attention encoder module is used for capturing multi-scale feature representations of the trajectory on the road network; specifically, different granularity levels of trajectory sequence representations are adopted to perform feature fusion within and between layers, so as to enrich spatiotemporal semantic information of the trajectory; The self-supervised pre-training module is used for obtaining feature representations capable of perceiving rich trajectory context in a large amount of trajectory data in a self-supervised manner; specifically, a mask trajectory restoration task is used as a supervision signal to construct a task of predicting missing fragments in the trajectory. 2.The deep learning based trajectory representation pre-training system of claim 1, wherein, In the static and dynamic feature construction module of the road network: (1) The static feature construction module of the road network specifically comprises two parts of constructing offline road network static features and constructing online road network static features; the former obtains multi-scale static features of the road network level through offline preprocessing; the latter queries corresponding static features on the road network according to a travel trajectory, and obtains multi-scale static feature representations on the trajectory sequence as model input; The construction of the offline road network static feature adopts a spectral clustering technique to perform clustering processing on road segments in the road network, thereby forming a coarse-grained road network with road segment clusters as nodes. Specifically, the spectral clustering divides the road network by analyzing the adjacency matrix between nodes to generate a specific number of clusters, with the goal of minimizing the distance between nodes within the same cluster and maximizing the distance between different clusters. If there are mutually connected road segments in two clusters, there will also be a connecting edge between the two clusters in the generated coarse-grained network. By iteratively performing spectral clustering on road networks of different scales, a higher-scale road network G j is obtained. Then, the embedding vectors of the nodes under the cluster networks of different scales are learned through a graph embedding learning method The construction of the online road network static feature, for a modeled trajectory R, the number of road segments it passes through is m, and its movement in the jth scale clustering cluster network is represented as Then, for each R j From the multi-scale static features of the offline road network, the static features of each trajectory point are indexed (2) The dynamic feature construction module of the road network specifically comprises two parts of constructing offline road network dynamic features and constructing online road network dynamic features; the former performs self-supervised pre-training on historical trajectory data by pre-training, and obtains an encoder model capable of perceiving trajectory context; the latter dynamically updates multi-scale dynamic features of the trajectory from a dynamic feature storage of the road network after obtaining a short-time historical sampling point trajectory set and a trajectory R to be queried; The construction of offline road network multi-scale dynamic features pre-trains the historical trajectory in a self-supervised manner through a missing trajectory restoration task; in this way, the encoder model can better learn spatiotemporal features of the trajectory point by perceiving trajectory semantic context; a loss function for specific training is as follows: in, Represents a discrete GPS point trajectory τ O The neural network model used for encoding is either a recurrent neural network structure or a multi-scale attention structure, in which... W represents the vector representation of the features of the j-th trajectory point in the output. τ ∈R d×d ,b τ ∈R d It is a neural network model The prediction layer parameters, where d represents the dimension of the parameter vector; V represents the set of road segments in the road network, r p ∈V represents a specific road segment; τ M It is the set of points that are obscured in the historical trajectory τ, τ O This means replacing the trajectory τ with a mask label in τ. M The trajectory obtained after the point in the middle; using Calculate the cross-entropy loss function and pass it through the logits vector U. j ′ ∈R |V| To predict which road segment the j-th point on τ will belong to; After pre-training, through The historical trajectory is encoded, and the features are written into a dynamic feature memory; The online road network multi-scale dynamic feature construction, when the model needs to process the query time t q of the journey trajectory, the dynamic features stored in the memory are read by constructing the following ordinary differential equation containing the neural network function: wherein, represents the initial state of the ordinary differential equation, i.e. f ψ is a two-layer perceptron network; is a neural network function composed of spatio-temporal convolution modules, taking the static stored features C j as input, containing two temporal convolution structures, one spatial convolution structure and a linear output layer; represents the dynamic feature on the j-th cluster network at query time t q ; for each road segment r i and its corresponding node in the multi-scale network to obtain the multi-scale dynamic feature N j denotes the number of clusters of the j-th cluster network. 3.The deep learning based trajectory representation pre-training system of claim 2, wherein, The multi-scale trajectory attention encoder module is configured to capture trajectory features on cluster networks of different scales in a road network, and to fuse cross-scale trajectory features in the cluster network, and to enhance a self-attention mechanism module using different scale features, so as to achieve adaptive feature extraction and fusion in the neural network; in particular: (1) The same scale feature fusion; for the j-th ∈ [1, c] scale of the coarse-grained trajectory R j , which is the output of the l-1 layer of the multi-scale attention encoder Then the intermediate feature of the trajectory can be calculated where LN is a layer normalization operation, H denotes the number of heads in multi-head attention, h denotes the computation of the h-th head; W q ,W k ,W v ∈R d×d denotes the parameters of linear computation; then, the features of each attention head are spliced to form: Finally, the following is calculated: Where FFNLN(x) represents FFN(LN(x))+x, and FFN is a two-layer feedforward neural network. (2) Cross-scale feature fusion; to fuse the spatio-temporal features of the coarse-grained trajectories into the features of the fine-grained trajectories , the result of the cross-scale attention fusion of the u-th point of the trajectory of each scale is calculated: All features are spliced to obtain: R ‘ 0 = R ′ 0,0 ‖…‖R ′ 0,H -1 ; (14) And by Obtaining fine-grained trajectory features (3) Enhancement at different scales; by fusing features across scales, fine-grained trajectory features can be obtained after fusion. Then, feature fusion is performed on the internal nodes of the fine-grained trajectory; trajectory features from other cluster networks are utilized. The coarse-grained information is used to enhance the ability to extract trajectory features; for the features of the coarse-grained trajectory, its dot product on the attention mechanism is calculated to obtain the fine-grained trajectory features of the l-th layer, that is: where W α,q , α,k ∈R d×d denotes the learnable parameters of the linear projection, denotes the trajectory feature similarity at the granularity of the j-th cluster network, β j is a parameter that controls the overall influence of the j-th cluster network. 4.The deep learning based trajectory representation pre-training system of claim 3, wherein, The self-supervised pre-training module uses a trajectory restoration task as a pre-training task of the travel trajectory, and is configured to learn a travel trajectory representation; a loss function of the self-supervised pre-training module is defined as follows: where R M represents the set of road segments in trajectory R that are masked, if there are road segments in R M that are randomly masked, the corresponding locations in R O will be replaced with a mask symbol by the corresponding road segments in R; by computing the cross-entropy logits where represents the jth element of the fine-grained trajectory representation that is the output of the Lth layer of the model.

Images