Human movement trajectory prediction method and device based on multi-modal spatio-temporal data

Abstract

A human movement trajectory prediction method and device based on multi-modal spatio-temporal data, the method comprising: S1, obtaining a remote sensing image of a place, and obtaining an image representation through a pre-trained image encoder; S2, obtaining intra-modal spatio-temporal movement representation using a spatio-temporal relationship graph; S3, obtaining multi-modal spatio-temporal movement representation using a cross-modal spatio-temporal relationship graph; S4, aligning the image modal and the spatio-temporal graph modal through cross-modal contrast learning; S5, obtaining trajectory multi-modal representation through multi-modal fusion, and optimizing the target using a multi-task learning method. The present application effectively realizes the semantic fusion and alignment of different modes, largely retains and utilizes the unique information of each mode, and can introduce the information of other modes with less additional computational overhead, and has an advantage over existing methods in terms of accuracy in human movement trajectory prediction.

Description

Technical Field

[0001] This invention belongs to the field of spatiotemporal trajectory data mining, specifically referring to a method and device for predicting human movement trajectories based on multimodal spatiotemporal data. Background Technology

[0002] Human movement trajectory prediction plays a crucial role in socio-economic development and urban governance, with wide-ranging applications encompassing location recommendation, evacuation planning, and intelligent transportation systems. In location recommendation, service providers can infer user preferences based on historical movement data, thereby generating personalized travel plans and providing targeted location and transportation mode recommendations. In evacuation planning and intelligent transportation systems, by predicting future destinations, government departments can optimize traffic scheduling in advance to alleviate traffic congestion in metropolitan areas. Furthermore, in emergencies such as public safety incidents, it can dynamically replan routes for crowds, ensuring both safety and smooth traffic flow.

[0003] However, human mobility data inherently suffers from sparsity, posing a significant challenge to accurate prediction. Due to limited observation density, trajectory patterns are often incomplete when users travel to new locations or exhibit irregular movement behavior. This data sparsity makes accurately predicting human mobility dynamics from observed sparse data a critical research problem that urgently needs to be solved, and it has important implications for urban development.

[0004] Significant advances in deep learning have driven a shift in mobility prediction towards representation learning paradigms, primarily categorized into methods based on recurrent neural networks (RNNs) and graph-based methods. Early RNN-based methods effectively captured spatiotemporal effects in a personalized manner by modeling the spatial and temporal relationships within a user's historical location access sequences. However, data scarcity has prompted researchers to construct graphs relating locations to improve representation quality and enrich the diversity of mobility patterns. Nevertheless, representation learning based solely on mobility records neglects location-related multimodal knowledge, significantly limiting the perception of human mobility dynamics.

[0005] In recent years, the application of multimodal knowledge in various fields has gradually attracted attention. In recommender systems, existing research has reduced cross-modal semantic differences by constructing multimodal graphs and fusing representations of nodes from different modalities, or by employing contrastive learning. Inspired by these advances, some studies have begun to apply multimodal knowledge to human movement trajectory prediction. However, existing multimodal methods fail to fully consider the inherent dynamic characteristics of human movement in different modalities (such as time transitions and access preferences), and lack explicit spatiotemporal constraints between multiple modalities when fusing cross-modal data.

[0006] In summary, current methods for predicting human movement trajectories face numerous challenges and have certain limitations. In multimodal human movement trajectory prediction, two key issues remain: first, how to fully model the dynamics of human movement across multiple modalities; and second, how to clearly construct cross-modal relationships and align multimodal semantics. Solving these problems is crucial for improving the performance of multimodal models and achieving more accurate human movement trajectory prediction. Summary of the Invention

[0007] To address the above problems, this invention provides a method and apparatus for predicting human movement trajectories based on multimodal spatiotemporal data.

[0008] This invention takes into account the sparsity of human movement data, acquires remote sensing images of the corresponding areas of a location, uses ChatGPT to aggregate locations to obtain human activities, constructs multi-level transfer relationships using historical trajectories, integrates and aligns information from different modalities through cross-modal spatiotemporal graph neural networks and contrastive learning, and finally enhances the prediction of the next location through multi-task learning.

[0009] This invention is achieved through the following technical solution:

[0010] The method for predicting human movement trajectories based on multimodal spatiotemporal data includes the following steps:

[0011] S1. Obtain remote sensing images of the location and acquire image representations through a pre-trained image encoder;

[0012] Based on the latitude and longitude of each location, a Python script is used to crawl remote sensing images of the corresponding location from ArcGIS, and a pre-trained image encoder is used to obtain the remote sensing image representation:

[0013] Z img =Proj img (Image-Encoder(img)) (1)

[0014] Among them, Proj img It is a nonlinear projection layer composed of multiple MLPs.

[0015] S2. Use the spatiotemporal relationship diagram to obtain the spatiotemporal movement representation within the mode;

[0016] Based on the category text of each location, ChatGPT clustering is used to generate corresponding human activities, obtaining information at three levels: location, category, and activity.

[0017] The transfer edge weight w(p) between locations i p j ), the transition edge weight w(c) between categories i c j), the transition edge weight w(a) between activities i a j ) respectively from p in the same trajectory in the trajectory training set j It is p i Number of times to visit the next location, c j It is C i The number of times the next access category is, a j It is a i The number of visits to the next activity is defined. A time transition matrix M for the location is constructed using the transition edge weights of three levels of information. p ∈N p ×N p The time transition matrix M of the category c ∈N c ×N c The time transition matrix M of the activity a ∈N a ×N a Among them, N p N c N a These are the location, category, and number of activities, respectively.

[0018] Furthermore, to reduce space requirements, a corresponding intramodal spatiotemporal sparse graph is constructed using the k nearest neighbors:

[0019]

[0020] For node e i The set of k nearest neighbors. Further, construct a diagonal matrix D using the maximum element of each row. e Representations of different entities are obtained through GCN:

[0021]

[0022] Among them, Z e σ is the original one-hot encoding of the entity, W is the weight matrix, and σ(·) is the activation function.

[0023] S3. Use cross-modal spatiotemporal relationship graphs to obtain multimodal spatiotemporal mobility representations;

[0024] To better integrate representations from different modalities, in addition to intra-modal relationships, three different inter-modal relationships were constructed in the image modality and ID modality. The most direct relationship is the location and its corresponding image at the same location, and this relationship is assigned a weight of 1.

[0025] sim(img i p i ):=1 (4)

[0026] Furthermore, by using the ID mode p i The set of neighboring nodes Lend to the corresponding image (img) i The relationship between images and ID modalities is constructed; similarly, the relationship between ID modalities and image modalities is constructed. Finally, the cross-modal spatiotemporal graph is obtained.

[0027] The cross-modal spatiotemporal graph fusion representation of the ID modality and the image modality is obtained as follows:

[0028]

[0029] in, Construct a diagonal matrix for the largest element in each row.

[0030] The final multimodal fusion representation is obtained by adding residual connections between the original and fused representations of the image modality and the ID modality, respectively.

[0031]

[0032] Where α is a hyperparameter that controls the proportion of fused information.

[0033] S4. Use cross-modal contrastive learning to align image modalities and spatiotemporal graph modalities;

[0034] To facilitate spatiotemporal dynamic learning of different modalities, a contrastive learning loss is designed for image representation and location ID modality representation, injecting knowledge of the ID modality into the image. Specifically, the location ID modality representation is first input into the projection layer, aligned spatially with the image modality:

[0035] Z ID =Proj ID (Z p (8)

[0036] Among them, Proj ID The projection layer is composed of a single-layer MLP;

[0037] The contrast loss between the image modality and the location ID modality is calculated globally, matching the images corresponding to the same location. i and p i As positive samples, img at different positions i and p j As a negative sample:

[0038]

[0039] Here, sim(·) is the inner product function, which, through contrastive learning, can preserve information from the image to the location ID modality to the maximum extent.

[0040] S5. Obtain multimodal representations of the trajectory through multimodal fusion, and optimize the target using a multi-task learning approach;

[0041] Trajectory sequence representation can consist of multiple record representations. The multimodal representation of a single record can be represented by a multimodal fusion of location, category, activity, multi-level user preferences, and image information.

[0042]

[0043] Where fuison(·) = MLP(concat(·)). Z u It is a one-hot encoded representation of the user ID. A day is divided into 48 timestamps, each half hour. The time representation Z with dimension k+1 is obtained using time2vector. t :

[0044]

[0045] Among them, w and As learnable parameters, the sin(·) activation function is used to capture temporal periodicity.

[0046] Furthermore, the input is fed into the Transformer decoder to obtain the user u. i movement trajectory sequence Characterization:

[0047]

[0048] Among them, Transformer through Attention scores for historical locations are iteratively calculated. Since the next category and next time prediction tasks can enhance the prediction of the next location, three prediction heads composed of MLPs are constructed to perform three prediction tasks simultaneously:

[0049]

[0050] Finally, a multi-task learning approach is used to collaboratively optimize the three prediction losses and the cross-modal contrastive learning loss to enhance the next location prediction:

[0051]

[0052] in, The cross-entropy function is used to calculate the prediction losses for the next location and the next class, respectively. The next time prediction loss is calculated using the mean squared error function. λ1 and λ2 are respectively... The weight.

[0053] Specifically, the calculation process of the cross-entropy function is as follows:

[0054]

[0055] Where N is the number of samples, C is the number of classes, and the true class label of the i-th sample is the one-hot encoded vector y. i =[y i1 y i2 , ..., y iC ], where only the position corresponding to the real category is 1, and the rest are 0. Let be the predicted score of the model for the j-th class of the i-th sample.

[0056] A second aspect of the present invention relates to a human movement trajectory prediction device based on multimodal spatiotemporal data, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the human movement trajectory prediction method based on multimodal spatiotemporal data of the present invention.

[0057] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the human movement trajectory prediction method based on multimodal spatiotemporal data of the present invention.

[0058] The working principle of this invention is:

[0059] (1) Intramodal spatiotemporal graphs and cross-modal spatiotemporal graphs were designed, which effectively realized the fusion of information.

[0060] (2) Cross-modal contrastive learning was designed to achieve semantic alignment between image modality and ID modality.

[0061] (3) In this invention, ChatGPT is only used for data preprocessing. During the training process, the image encoder freezes the parameters. Compared with existing multimodal human trajectory prediction methods, the additional video memory introduced is small and the training time cost is low.

[0062] The advantages of this invention are:

[0063] (1) The present invention effectively realizes semantic fusion and alignment of different modalities, and retains and utilizes the unique information of each modality to a large extent.

[0064] (2) The present invention can introduce information from other modalities with less additional computational overhead.

[0065] (3) The present invention has an advantage over existing methods in terms of accuracy in predicting human movement trajectories. Attached Figure Description

[0066] Figure 1This is a flowchart of the human movement trajectory prediction method based on multimodal spatiotemporal data according to the present invention.

[0067] Figure 2 This is a schematic diagram of the edges between different modalities in the cross-modal graph of the present invention. The white circle represents the ID modality, and the gray circle represents the image modality.

[0068] Figure 3 This is a schematic diagram of cross-modal contrastive learning in this invention.

[0069] Figure 4 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0070] The technical solution of the present invention will be clearly and completely explained and described below with reference to the accompanying drawings.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment relates to a method for predicting human movement trajectories based on multimodal spatiotemporal data, including the following steps:

[0073] S1. Obtain remote sensing images of the location and acquire image representations through a pre-trained image encoder;

[0074] Based on the latitude and longitude of each location, a Python script is used to crawl remote sensing images of the corresponding location from ArcGIS, and a pre-trained image encoder is used to obtain the remote sensing image representation:

[0075] Z img =Proj img (Image-Encoder(img)) (1)

[0076] Among them, Proj img It is a nonlinear projection layer composed of multiple MLPs.

[0077] S2. Use the spatiotemporal relationship diagram to obtain the spatiotemporal movement representation within the mode;

[0078] Based on the category text of each location, ChatGPT clustering is used to generate corresponding human activities, obtaining information at three levels: location, category, and activity.

[0079] The transfer edge weight w(p) between locations i p j ), the transition edge weight w(c) between categories i c j ), the transition edge weight w(a) between activities i a j ) respectively from p in the same trajectory in the trajectory training set j It is pi Number of times to visit the next location, c j It is C i The number of times the next access category is, a j It is a i The number of visits to the next activity is defined. A time transition matrix M for the location is constructed using the transition edge weights of three levels of information. p ∈N p ×N p The time transition matrix M of the category c ∈N c ×N c The time transition matrix M of the activity a ∈N a ×N a Among them, N p N c N a These are the location, category, and number of activities, respectively.

[0080] Furthermore, to reduce space requirements, a corresponding intramodal spatiotemporal sparse graph is constructed using the k nearest neighbors:

[0081]

[0082] For node e i The set of k nearest neighbors. Further, construct a diagonal matrix D using the maximum element of each row. e Representations of different entities are obtained through GCN:

[0083]

[0084] Among them, Z e σ is the original one-hot encoding of the entity, W is the weight matrix, and σ(·) is the activation function.

[0085] S3. Use cross-modal spatiotemporal relationship graphs to obtain multimodal spatiotemporal mobility representations;

[0086] like Figure 2 As shown, to better integrate different modal representations, in addition to intra-modal relationships, three different inter-modal relationships were constructed in the image modality and ID modality. The most direct relationship is the location at the same position and the corresponding image, and the relationship weight is assigned a value of 1.

[0087] sim(imgi, p) i :=1 (4)

[0088] Furthermore, by using the ID mode p i The set of neighboring nodes Lend to the corresponding image (img) iThe relationship between images and ID modalities is constructed; similarly, the relationship between ID modalities and image modalities is constructed. Finally, the cross-modal spatiotemporal graph is obtained.

[0089] The cross-modal spatiotemporal graph fusion representation of the ID modality and the image modality is obtained as follows:

[0090]

[0091] in, Construct a diagonal matrix for the largest element in each row.

[0092] The final multimodal fusion representation is obtained by adding residual connections between the original and fused representations of the image modality and the ID modality, respectively.

[0093]

[0094] Where α is a hyperparameter that controls the proportion of fused information.

[0095] S4. Use cross-modal contrastive learning to align image modalities and spatiotemporal graph modalities;

[0096] like Figure 3 As shown, to facilitate spatiotemporal dynamic learning of different modalities, a contrastive learning loss is designed for image representation and location ID modality representation, injecting ID modality knowledge into the image. Specifically, the location ID modality representation is first input into the projection layer, aligned spatially with the image modality:

[0097] Z ID =Proj ID (Z p (8)

[0098] Among them, Proj ID The projection layer is composed of a single-layer MLP;

[0099] The contrast loss between the image modality and the location ID modality is calculated globally, matching the images corresponding to the same location. i and p i As positive samples, img at different positions i and p j As a negative sample:

[0100]

[0101] Here, sim(·) is the inner product function, which, through contrastive learning, can preserve information from the image to the location ID modality to the maximum extent.

[0102] S5. Obtain multimodal representations of the trajectory through multimodal fusion, and optimize the target using a multi-task learning approach;

[0103] Trajectory sequence representation can consist of multiple record representations. The multimodal representation of a single record can be represented by a multimodal fusion of location, category, activity, multi-level user preferences, and image information.

[0104]

[0105] Where fuison(·) = MLP(concat(·)). Z u It is a one-hot encoded representation of the user ID. A day is divided into 48 timestamps, each half hour. The time representation Z with dimension k+1 is obtained using time2vector. t :

[0106]

[0107] Among them, w and As learnable parameters, the sin(·) activation function is used to capture temporal periodicity.

[0108] Furthermore, the input is fed into the Transformer decoder to obtain the user u. i movement trajectory sequence Characterization:

[0109]

[0110] Among them, Transformer through Attention scores for historical locations are iteratively calculated. Since the next category and next time prediction tasks can enhance the prediction of the next location, three prediction heads composed of MLPs are constructed to perform three prediction tasks simultaneously:

[0111]

[0112] Finally, a multi-task learning approach is used to collaboratively optimize the three prediction losses and the cross-modal contrastive learning loss to enhance the next location prediction:

[0113]

[0114] in, The cross-entropy function is used to calculate the prediction losses for the next location and the next class, respectively. The next time prediction loss is calculated using the mean squared error function. λ1 and λ2 are respectively... The weight.

[0115] Specifically, the calculation process of the cross-entropy function is as follows:

[0116]

[0117] Where N is the number of samples, C is the number of classes, and the true class label of the i-th sample is the one-hot encoded vector y. i =[y i1 y i2 , ..., y iC ], where only the position corresponding to the real category is 1, and the rest are 0. Let be the predicted score of the model for the j-th class of the i-th sample.

[0118] The positive effects of this invention are as follows:

[0119] (1) The present invention effectively realizes semantic fusion and alignment of different modalities, and retains and utilizes the unique information of each modality to a large extent.

[0120] (2) The present invention can introduce information from other modalities with less additional computational overhead.

[0121] (3) The present invention has an advantage over existing methods in terms of accuracy in predicting human movement trajectories.

[0122] To verify the effectiveness of this invention and compare it with existing methods, this invention uses the NYC and SIN city datasets from the open-source Foursquare dataset for validation. The following data cleaning strategy was employed: records from the same day were grouped into the same trajectory; trajectories shorter than 3 lines, locations with fewer than 10 visits, and inactive users with fewer than 5 trajectories were removed. LSTPM and Graph-Flashback were used as baseline methods for comparison. The results show that the method of this invention is optimal in all three accuracy metrics. Table 1 compares the accuracy of the method of this invention with other methods on the NYC and SIN city datasets.

[0123] Table 1

[0124]

[0125] Example 2

[0126] Reference Figure 4 This embodiment relates to a human movement trajectory prediction device based on multimodal spatiotemporal data, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the human movement trajectory prediction method based on multimodal spatiotemporal data of Embodiment 1.

[0127] Example 3

[0128] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the human movement trajectory prediction method based on multimodal spatiotemporal data of Embodiment 1.

[0129] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for predicting human movement trajectories based on multimodal spatiotemporal data, characterized in that, Includes the following steps: S1. Obtain remote sensing images of the location and acquire image representations through a pre-trained image encoder; S2. Use the spatiotemporal relationship diagram to obtain the spatiotemporal movement representation within the mode; S3. Obtain multimodal spatiotemporal mobility representations using cross-modal spatiotemporal relationship graphs; specifically including: To better integrate representations from different modalities, in addition to intra-modal relationships, three different inter-modal relationships were constructed in the image modality and ID modality; the most direct relationship is the location and its corresponding image, with a weight of 1 assigned to this relationship. (4) Furthermore, by using the ID modality Neighbors Lend the corresponding image First, construct the relationship from image to ID modality; then, in the same way, construct the relationship from ID modality to image modality; finally, obtain the cross-modal spatiotemporal graph. ; The cross-modal spatiotemporal graph fusion representation of the ID modality and the image modality is obtained as follows: (5) in, Construct a diagonal matrix for the maximum element in each row; The final multimodal fusion representation is obtained by adding residual connections between the original and fused representations of the image modality and the ID modality, respectively. (6) (7) in, Hyperparameters for controlling the proportion of fused information; S4. Aligning image modalities and spatiotemporal graph modalities through cross-modal contrastive learning; specifically including: To facilitate spatiotemporal dynamic learning of different modalities, a contrastive learning loss is designed for image representation and location ID modality representation, injecting ID modality knowledge into the image; specifically, the location ID modality representation is first input into the projection layer, aligned in the same space with the image modality: (8) in, The projection layer is composed of a single-layer MLP; The contrast loss between the image modality and the location ID modality is calculated globally, matching the corresponding locations... and As positive samples, different positions and As a negative sample: (9) (10) in, It is an inner product function that, through contrastive learning, preserves the information from the image to the location ID modality to the maximum extent. S5. Obtain multimodal representations of the trajectory through multimodal fusion, and optimize the target using a multi-task learning approach.

2. The human movement trajectory prediction method based on multimodal spatiotemporal data as described in claim 1, characterized in that, Step 1 specifically includes: Based on the latitude and longitude of each location, a Python script is used to crawl remote sensing images of the corresponding location from ArcGIS, and a pre-trained image encoder is used to obtain the remote sensing image representation: (1) in, It is a nonlinear projection layer composed of multiple MLPs.

3. The human movement trajectory prediction method based on multimodal spatiotemporal data as described in claim 1, characterized in that, Step 2 specifically includes: Based on the category text of each location, ChatGPT clustering is used to generate corresponding human activities, obtaining information at three levels: location, category, and activity. Transfer of border rights between locations inter-category transfer edge weights Transfer of boundary weights between activities Each from the same trajectory in the trajectory training set yes Number of times to visit the next location yes The number of times the next category is visited. yes Define the number of visits; construct the time transition matrix of a location using the transition edge weights of three levels of information. Time transition matrix of the category Activity time transition matrix ;in, , , These are the location, category, and number of activities, respectively. Furthermore, in order to reduce space requirements, by... Construct the corresponding intramodal spatiotemporal sparse graph using the nearest neighbors: (2) For nodes of A set of nearest neighbor nodes; further, construct a diagonal matrix using the maximum element of each row. Representations of different entities are obtained through GCN: (3) in, For the original one-hot encoding of the entity, This is the weight matrix. This is the activation function.

4. The human movement trajectory prediction method based on multimodal spatiotemporal data as described in claim 1, characterized in that, Step 5 specifically includes: Trajectory sequence representation consists of multiple record representations. The multimodal representation of a single record is represented by a multimodal fusion of location, category, activity, multi-level user preferences, and image information. (11) in, ; It is a one-hot encoded representation of the user ID. A day is divided into 48 timestamps, each half hour. The time representation with dimension k+1 is obtained using time2vector. : (12) in, and For learnable parameters, Activation functions are used to capture temporal periodicity; Furthermore, the input is fed into the Transformer decoder to obtain the user's... movement trajectory sequence Characterization: (13) (14) in, pass Attention scores for historical locations are iteratively calculated. Since the next category and next time prediction tasks enhance the prediction of the next location, three prediction heads composed of MLPs are constructed, simultaneously performing three prediction tasks: (15) Finally, a multi-task learning approach is used to collaboratively optimize the three prediction losses and the cross-modal contrastive learning loss to enhance the next location prediction: (16) in, , The cross-entropy function is used to calculate the prediction losses for the next location and the next class, respectively. The next time prediction loss is calculated using the mean squared error function; , They are respectively , The weights; Specifically, the calculation process of the cross-entropy function is as follows: (17) Where N is the number of samples, C is the number of categories, and the nth category is the number of categories. The true class label of each sample is a one-hot encoded vector. Only the position corresponding to the true category is 1, and the rest are 0; For the model to the first The first sample Class prediction score.

5. A human movement trajectory prediction device based on multimodal spatiotemporal data, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the human movement trajectory prediction method based on multimodal spatiotemporal data as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the human movement trajectory prediction method based on multimodal spatiotemporal data as described in any one of claims 1-4.

Images