A POI Location Prediction Method, Device, and Medium Based on Self-Supervised Learning

Abstract

The present invention discloses a method, device and medium for predicting POI locations based on self-supervised learning. The method includes: obtaining user historical POI location trajectory data, extracting trajectory features at each time step, and using an embedding layer to represent to obtain user embeddings and a trajectory feature embedding matrix, which are input into a transformer network to extract long-term dependence feature vectors and aggregate them with the user embeddings; inputting the aggregated vectors into a prediction network, constructing a prediction loss function based on the prediction output and the true value; calculating virtual companions for each user under each weather condition based on the POI-User vectors, regarding the aggregated vectors of each user with their virtual companions and non-virtual companions as positive pairs and negative pairs respectively, and constructing a contrastive learning loss function; training the overall model parameters by combining the two loss functions; and using the trained overall model to predict and extract the aggregated vectors and predict the POI location of the user at the next time step. The present invention improves the accuracy of user POI location prediction.

Description

Technical Field

[0001] The present invention belongs to the fields of intelligent transportation and location prediction, and relates to a method for predicting POI locations based on self-supervised learning. Background Art

[0002] With the continuous increase in the number of location-based service applications, location prediction has also attracted extensive attention in the academic community. Location prediction is an important topic in the study of human mobility, aiming to extract user movement behavior characteristics from existing historical trajectory data, so as to predict the locations that users are most likely to visit next. By predicting the next location point, it can help location-based service application providers provide location-based advertisements, coupons, etc. more accurately and improve the user experience.

[0003] Regarding the problem of predicting the next location point, many excellent representation learning methods have been proposed in existing research, but they still have certain limitations. For example, the supervision signals are sparse. Most models perform location prediction tasks in the supervised learning paradigm, but their supervision signals all come from observed user-POI interactions. Compared with the entire interaction space, the observed interactions are very sparse. For example, most studies predict the next location that users will visit based on their historical and recent trajectory data, while ignoring the benefits from the movement information of peers with similar movement patterns.

[0004] In recent years, self-supervised learning has been widely used in computer vision (CV) and natural language processing (NLP). The idea of self-supervised learning is to construct auxiliary supervision tasks to extract additional supervision signals from the input data itself, thereby achieving significant improvements in downstream tasks. However, since trajectory data is different from image data or language text and it is difficult to construct augmentation operators, it has not been popularized in location prediction yet. The idea of self-supervised learning is to construct auxiliary supervision tasks to extract additional supervision signals from the input data itself, thereby achieving significant improvements in downstream tasks. How to use the virtual peer information of users to assist in prediction and introduce the advantages of self-supervised learning into the location prediction algorithm requires further research. Summary of the Invention

[0005] The present invention provides a method for predicting POI locations based on self-supervised learning. It calculates virtual friends for users under different weather conditions using user historical trajectory data, uses a network based on transformer to capture the movement patterns and periodicity in the trajectory data, regards the user and their weather virtual friends as positive pairs, and other users as negative pairs for contrastive learning, and finally improves the prediction accuracy through a multi-task training strategy.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solutions:

[0007] A method for predicting POI locations based on self-supervised learning, comprising:

[0008] S1. Obtain the historical POI location trajectory data of multiple users, and extract the trajectory features at each time step within the statistical time window;

[0009] S2. Use the embedding layer to represent the user features and the trajectory features at each time step, and obtain the user embedding and the trajectory feature embedding matrix;

[0010] S3. Input the trajectory feature embedding matrix of the user into the network based on transformer, extract the long-term dependence feature vector of the user trajectory, and aggregate the user embedding and the long-term dependence feature vector. Denote the aggregated vector as the aggregated vector of the user;

[0011] S4. Input the aggregated vector of the user into the POI location prediction network to obtain the POI location prediction output, and construct a prediction loss function based on the prediction output and the true POI location;

[0012] S5. Calculate the virtual companions of each user under each weather condition based on the probability vector of the user at each POI location, i.e., the POI-User vector; and regard the aggregated vectors of each user and their virtual companions as positive pairs, and the aggregated vectors of non-virtual companions as negative pairs to construct a contrastive learning loss function;

[0013] S6. Combine the contrastive learning loss function and the prediction loss function to train the parameters in the embedding layer, the network based on transformer, and the POI location prediction network;

[0014] S7. Use the trained embedding layer and the network based on transformer to extract the corresponding aggregated vector from the currently known POI location trajectory data of the user to be predicted, and then input the extracted aggregated vector into the trained POI location prediction network. The output is the POI location prediction result of the user to be predicted at the next time step.

[0015] Further, the trajectory features at any time step k extracted in step S1 include: the POI location l k , the weather condition w k , a certain time period h of the natural day when the trajectory starts k and a certain day d of the natural week where it is located k .

[0016] Further, in the embedding layer, a parameter matrix is used for mapping between the original vector and the real-valued vector, that is, the embedding layer uses different parameter matrices W l , W w , W h , W d , W u to map lk 、w k 、h k 、d k and user feature u (i) are embedded and represented. The specific representation method is as follows:

[0017]

[0018] Where: is the embedded representation vector corresponding to the trajectory feature, is the embedded representation vector of user i, denoted as user embedding; l k d k h k w k u (i) are respectively the one-hot encoding vectors corresponding to the trajectory features;

[0019] Then, by adding the embedded representation vector to the positional encoding PE, the trajectory feature embedding vector at each time step k is obtained

[0020]

[0021] where pos represents the position of the trajectory in the dataset, i represents the position of the POI location point in this trajectory, and d model represents the preset embedding dimension of the positional encoding;

[0022] Finally, for each user i, their user embedding is obtained. At the same time, the trajectory feature embedding matrix emb is constructed from the trajectory feature embedding vectors at all time steps of each user i within the statistical time window all .

[0023] Furthermore, step S3 uses a Transformer-based decoder module to extract the long-term dependency feature vectors of the user trajectory, specifically:

[0024] The Transformer-based decoder module consists of N identical blocks, and each block has two layers; the first layer is a masked multi-head attention mechanism, which contains H self-attention layers. The masking operation is used to only focus on the trajectory feature embedding vectors before time step k when calculating at time step k; the second layer is a fully connected feed-forward network with two linear layers and a ReLU activation function; a residual connection and a layer normalization component are added to each layer;

[0025] First, the trajectory feature embedding matrix emb allIt is passed into H different self-attention layers to obtain H output matrices head1, head2, head3…head H , and then these output matrices are concatenated:

[0026]

[0027] Among them, Multi-Head is the output of the multi-head attention mechanism with a mask, and W o is the weight matrix learned during the training process. The output matrix head i of any j-th self-attention layer is expressed as:

[0028]

[0029] Among them, represents the concatenation operation, are the learnable parameter matrices corresponding to Q, K, and V in the j-th self-attention layer respectively. In the self-attention layer of the first block, Q, K, and V are set to the trajectory feature embedding matrix emb all , and in the self-attention layers of the remaining blocks, Q, K, and V are the outputs of the previous block;

[0030] Then, the user embedding is concatenated with the long-term dependence feature vector out n of the final output of the decoder module, and then input into the fully connected residual block to obtain the aggregated vector

[0031]

[0032] of user i at the current time step n. Among them, FC(.) is the operation of the fully connected residual block, which consists of a single linear feedforward layer and a ReLU activation function, followed by a dropout, a residual connection, and a batch normalization layer.

[0033] Furthermore, a multi-class cross-entropy is used to construct the prediction loss function as:

[0034]

[0035]

[0036] Among them, represents the value of the prediction loss function; k is the index of the POI location type, and L is the number of POI location types; represents the aggregated vector of user i at the current time step n; P(l n+1 ) (k)is the true value for the user to access the k-th location at time step n+1. If the actually accessed POI location is the k-th location, then P(l n+1 ) (k) = 1, otherwise P(l n+1 ) (k) = 0; represents the probability of predicting that the k-th location will be accessed in the next time step n+1. Linear(·) and softmax(·) represent the linear projection and softmax function in the POI location prediction network respectively.

[0037] Furthermore, step S5 calculates the virtual companions of each user under each weather condition. Specifically, the spatial proximity score SC score between the user i to be calculated and each other user j is calculated:

[0038]

[0039] where pu i , pu j are the POI-User vectors of two different users i and j respectively, and one of them is the user to be calculated. The larger the spatial proximity score SC score, the greater the probability of frequently accessing the same POI location under the current calculated weather condition. Then, all corresponding users with a spatial proximity score SC score greater than the preset threshold are selected as the virtual companions of the user to be calculated under the current calculated weather condition.

[0040] Furthermore, the contrastive learning loss function is constructed as:

[0041]

[0042] where represents the value of the contrastive learning loss function; s(.) is the cosine similarity function used to measure the similarity between two vectors; τ is a hyperparameter; N u(i) is the set of virtual companions of the user i to be calculated, and j is the user among them; represents the user set, and u and k are the users among them.

[0043] Furthermore, the combination method of the contrastive learning loss function and the prediction loss function is:

[0044]

[0045] where and represent the value of the prediction loss function and the value of the contrastive learning loss function respectively, represents the value of the total loss function; λ is the hyperparameter for balancing and .

[0046] An electronic device includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor implements the POI location prediction method described in any one of the above.

[0047] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the POI location prediction method described in any one of the above is implemented.

[0048] The next location point prediction method based on self-supervised learning provided by the present invention takes into account that users have different movement patterns under different weather conditions, and can more accurately select virtual companions for users according to weather conditions; uses a transformer decoder structure to fully capture the regularity and periodicity in user movement trajectory data; and uses virtual friend trajectory data for contrast learning to optimize the prediction task. Brief Description of the Drawings

[0049] Figure 1 It is an overall framework diagram of the next location point prediction method based on self-supervised learning of the present invention;

[0050] Figure 2 It is a network structure diagram of a transformer decoder. Detailed Embodiments

[0051] The embodiments of the present invention will be described in detail below. Based on the technical solutions of the present invention, detailed implementation manners and specific operation processes are given, and the technical solutions of the present invention are further explained and illustrated.

[0052] The embodiments of the present invention provide a POI location prediction method based on self-supervised learning, as shown in Figure 1 and includes the following steps:

[0053] S1. Obtain historical POI location trajectory data of multiple users, and extract trajectory features at each time step within a statistical time window.

[0054] The obtained historical POI location trajectory data of users is to remove user data with sparse data, such as user trajectory data less than 5 days, and clean the abnormal trajectory data.

[0055] To solve the modeling problem caused by a long time period, the user trajectory data is divided by the hour. For example, if two hours are used as a statistical time window for division, each user can obtain the trajectory data of multiple statistical time windows. Then, the trajectory data of each statistical time window is classified according to the weather conditions, and the trajectory features of each time step within each statistical time window are extracted, including the trajectory features of any time step k, including: the POI location l k , the weather condition w k , the start time t k , a certain time period h of the natural day where it is located k (For example, if 15 minutes is used as a time period to divide a day, h k represents the number of the time period) and the day d of the natural week where it is located k (For example, d k = 3 represents Wednesday).

[0056] S2. Use the embedding layer to represent the user features and the trajectory features of each time step to obtain the user embedding and the trajectory feature embedding matrix.

[0057] In the embedding layer, a parameter matrix is used for the mapping between the original vector and the real-valued vector, that is, different parameter matrices W l , W w , W h , W d , W u are used to perform embedding representations on l k , w k , h k , d k and the user feature u (i) respectively. The specific representation method is as follows:

[0058]

[0059] Among them: is the embedding representation vector corresponding to the trajectory feature, is the embedding representation vector of user i, denoted as the user embedding; l k , d k , h k , w k , u (i) are the one-hot encoding vectors corresponding to the respective trajectory features;

[0060] Then, by adding the embedding representation vector to the positional encoding PE, the trajectory feature embedding vector of each time step k is obtained

[0061]

[0062] where pos represents the position of the trajectory (i.e., the divided trajectory) in the dataset, i represents the position of the position point in the trajectory, and d model represents the preset embedding dimension of the position encoding;

[0063] Finally, for each user i, their user embedding is obtained At the same time, the trajectory feature embedding matrix emb is constructed from the trajectory feature embedding vectors of each user i at all time steps within the statistical time window all .

[0064] S3. Input the trajectory feature embedding matrix of the user into the network based on Transformer, extract the long-term dependence feature vector of the user's trajectory, and aggregate the user embedding and the long-term dependence feature vector. Denote the aggregated vector as the aggregated vector of the user

[0065] In this step, the network based on Transformer is used to extract patterns and capture multi-level periodicity from the complex spatio-temporal historical trajectory sequence (i.e., the trajectory feature embedding matrix emb all ).

[0066] In this embodiment, a decoder module based on Transformer is used to extract the long-term dependence feature vector of the user's trajectory. Specifically

[0067] The decoder module based on Transformer is as Figure 2 shown, composed of N identical blocks, each block having two layers; the first layer is the masked Multi-Head Attention mechanism (with a masked multi-head attention mechanism), and this multi-head attention mechanism contains H self-attention layers. The input and output dimensions of each block are designed to be the same as the embedding vector, that is, d model =d base , and in addition, the masking operation is used to only focus on the trajectory feature embedding vectors before (including k) the time step k when calculating the time step k; the second layer is a fully connected feed-forward network with two linear layers and a ReLU activation function; residual connections and layer normalization components are added to each layer to facilitate learning

[0068] First, the trajectory feature embedding matrix emb all is passed into H different self-attention layers to obtain H output matrices head1, head2, head3... head H , and then these output matrices are concatenated

[0069]

[0070] Among them, Multi-Head is the output of the multi-head attention mechanism with a mask, and W o is the weight matrix learned during the training process. The output matrix head of any j-th self-attention layer i is expressed as:

[0071]

[0072] Among them, represents the concatenation operation, are the learnable parameter matrices corresponding to Q, K, and V in the j-th self-attention layer respectively. In the self-attention layer of the first block, Q, K, and V are set to the trajectory feature embedding matrix emb all , and in the self-attention layers of the remaining blocks, Q, K, and V are the outputs of the previous block.

[0073] In this embodiment, due to the adoption of the multi-head attention mechanism, the model retains multiple groups of parameter matrices and focuses on different positions in the historical sequence, thereby effectively capturing the multi-level periodicity of human mobility.

[0074] Then, the user embedding emb u(i) is concatenated with the long-term dependence feature vector out of the final output of the decoder module n , and then input into the fully connected residual block to obtain the aggregated vector of user i at the current time step n

[0075]

[0076] Among them, FC(.) is the operation of the fully connected residual block, which consists of a single linear feed-forward layer and a ReLU activation function, followed by a dropout, a residual connection, and a batch normalization layer.

[0077] Finally, the obtained aggregated vector representation f n encapsulates the location, time, weather conditions, and user information of the entire historical sequence and is used to predict the POI location of the user at the next time step.

[0078] S4. Input the aggregated vector of the user into the POI location prediction network to obtain the POI location prediction output at the next time step, and construct a prediction loss function based on the prediction output and the true POI location.

[0079] Specifically, the prediction loss function is constructed using multi-class cross-entropy as:

[0080]

[0081] Among them, denotes the predicted loss function value; k is the index of the POI location type, and L is the number of POI location types; denotes the aggregated vector of user i at the current time step n; P(l n+1 ) (k) is the true value that the user visits the k-th location at time step n + 1. If the actually visited POI location is the k-th location, then P(l n+1 ) (k) = 1; otherwise, P(l n+1 ) (k) = 0; denotes the probability of predicting that the k-th location will be visited at the next time step n + 1; Linear(·) and softmax(·) respectively represent the linear projection and the softmax function in the POI location prediction network.

[0082] S5. Based on the probability vectors of users at each POI location, that is, the POI-User vectors, calculate the virtual companions of each user under each weather condition; and regard the aggregated vectors of each user and their virtual companions as positive pairs, and the aggregated vectors of non-virtual companions as negative pairs to construct a contrastive learning loss function.

[0083] First, construct POI-User vectors with a length equal to the number of POI locations for each user under different weather conditions. Each element in the vector represents the probability that the user is seen at the corresponding POI location;

[0084] Then, according to different weather conditions, calculate the spatial proximity scores between the selected user and other users under the same weather condition. The calculation formula is as follows:

[0085]

[0086] where pu i , pu j (j ≠ i) are the POI-User vectors of the selected user i and any other user j respectively.

[0087] Those with a lower SC score are the people who often visit the same location as the selected user under the same weather condition. Conversely, the SC score will be higher. Therefore, all corresponding users with an SC score greater than the preset threshold can be selected based on the spatial proximity score SC score as the virtual companions of user i under the current calculated weather condition, and all virtual companions form a virtual companion set.

[0088] Then, the aggregated vectors of each user and their virtual companions can be regarded as positive pairs, and the aggregated vectors of non-virtual companions as negative pairs to construct a contrastive learning loss function:

[0089]

[0090] Among them, represents the value of the contrastive learning loss function; s(.) is the cosine similarity function, which is used to measure the similarity between two vectors; τ is a hyperparameter; is the set of virtual peers of user i to be calculated, and j is the user among them; represents the user set, and u and k are the users among them.

[0091] S6. Combine the contrastive learning loss function and the prediction loss function to train the parameters in the embedding layer, the transformer-based network, and the POI location prediction network.

[0092] The combination method of the contrastive learning loss function and the prediction loss function is as follows:

[0093]

[0094] Among them, and respectively represent the value of the prediction loss function and the value of the contrastive learning loss function, represents the value of the total loss function; λ is a hyperparameter for balancing and .

[0095] S7. Use the trained embedding layer and the transformer-based network to extract the corresponding aggregated vector from the currently known POI location trajectory data of the user to be predicted, and then input the extracted aggregated vector into the trained POI location prediction network. The output is the predicted result of the next POI location of the user to be predicted.

[0096] This embodiment plans to use the private car travel trajectories in Guangdong Province for model training and prediction evaluation. Specifically, 80% of the data in the experimental dataset is selected as the training set, and the remaining 20% is used as the test set. For the test data, the trajectory data is divided into trajectory segments (for example, every 2h of trajectory is taken as a segment), 80% of the trajectory segment data is input into the trained model for prediction, and the predicted result output by the model is compared with the actual visited location points in the trajectory data to evaluate the performance of the model.

[0097] The above embodiments are the preferred embodiments of the present application. Those of ordinary skill in the art can also make various transformations or improvements based on this. Without departing from the overall concept of the present application, these transformations or improvements should all fall within the scope of protection required by the present application.

Claims

1. A method for predicting POI locations based on self-supervised learning, characterized in that, Including: S1. Obtain the historical POI location trajectory data of multiple users, and extract the trajectory features at each time step within the statistical time window. S2. Use the embedding layer to represent the user features and the trajectory features at each time step, and obtain the user embedding and the trajectory feature embedding matrix. The parameter matrix is used in the embedding layer to map the original vector and the real-valued vector, that is, different parameter matrices W l 、W w 、W h 、W d 、W u are used to perform embedding representations on l k 、w k 、h k 、d k and the user feature u (i) respectively, and the specific representation method is as follows: Wherein: is the embedding representation vector corresponding to the trajectory feature, is the embedding representation vector of user i, denoted as user embedding; l k , d k , h k , w k , u (i) are the one-hot encoding vectors corresponding to the trajectory features respectively; Then, by adding the embedded representation vector to the positional encoding PE, the trajectory feature embedding vector at each time step k is obtained Among them, pos represents the position of the trajectory in the dataset, i represents the position of the POI location point in this trajectory, and d model represents the preset embedding dimension of the position encoding; Finally, for each user i, their user embedding is obtained At the same time, a trajectory feature embedding matrix is constructed from the trajectory feature embedding vectors of each user i at all time steps within the statistical time window S3. Input the trajectory feature embedding matrix of the user into the network based on Transformer to extract the long-term dependence feature vector of the user's trajectory, and aggregate the user embedding and the long-term dependence feature vector. Denote the aggregated vector as the aggregated vector of the user. S4. Input the aggregated vector of the user into the POI location prediction network to obtain the POI location prediction output, and construct a prediction loss function based on the prediction output and the true POI location. S5. Calculate the virtual companion of each user under each weather condition based on the probability vector of the user at each POI location, that is, the POI-User vector; and regard the aggregated vectors of each user and their virtual companions as positive pairs, and the aggregated vectors of non-virtual companions as negative pairs to construct a contrastive learning loss function. S6. Combine the contrastive learning loss function and the prediction loss function to train the parameters in the embedding layer, the network based on Transformer, and the POI location prediction network. S7. Use the trained embedding layer and the network based on Transformer to extract the corresponding aggregated vectors from the currently known POI location trajectory data of the user to be predicted, and then input the extracted aggregated vectors into the trained POI location prediction network. The output is the POI location prediction result of the user to be predicted at the next time step.

2. The POI location prediction method according to claim 1, wherein The trajectory features at any time step k extracted in step S1 include: the POI location l k , the weather condition w k , the start time t k , a certain time period h of the natural day where it is located k , and a certain day d of the natural week where it is located k .

3. The POI location prediction method according to claim 1, wherein Step S3 uses the decoder module based on Transformer to extract the long-term dependence feature vector of the user's trajectory. Specifically: The decoder module based on Transformer consists of N identical blocks, and each block has two layers; the first layer is a masked multi-head attention mechanism, and this multi-head attention mechanism contains H self-attention layers. The masking operation is used to only focus on the trajectory feature embedding vectors before time step k when calculating at time step k. The second layer is a fully connected feed-forward network with two linear layers and a ReLU activation function; a residual connection and a layer normalization component are added to each layer. First, embed the trajectory feature into the matrix emb all Pass it into H different self-attention layers to obtain H output matrices head1, head2, head3... head H , and then concatenate these output matrices: Among them, Multi-Head is the output of the multi-head attention mechanism with masking, and W o is the weight matrix learned during the training process. The output matrix head of any j-th self-attention layer i is expressed as: Among them, represents a connection operation, are the learnable parameter matrices corresponding to Q, K, and V in the j-th self-attention layer respectively. The Q, K, and V in the self-attention layer of the first block are set to the trajectory feature embedding matrix emb all , and the Q, K, and V in the self-attention layers of the remaining blocks are the outputs of the previous block; Then the user is embedded to the long-term dependence feature vector out of the final output of the decoder module n for connection, and then input into the fully connected residual block to obtain the aggregated vector of user i at the current time step n Among them, FC(.) is the operation of the fully connected residual block, which consists of a single linear feed-forward layer and a ReLU activation function, followed by a dropout, a residual connection, and a batch normalization layer.

4. The POI location prediction method according to claim 1, characterized in that The prediction loss function is constructed using multi-class cross-entropy as: Among them, represents the predicted loss function value; k is the index of the POI location type, and L is the number of POI location types; represents the aggregated vector of user i at the current time step n; P(l n+1 ) (k) is the true value that the user visits the k-th location at time step n + 1. If the actually visited POI location is the k-th location, then P(l n+1 ) (k) = 1, otherwise represents the probability of predicting that the k-th location will be visited at the next time step n + 1. Linear(·) and softmax(·) respectively represent the linear projection and the softmax function in the POI location prediction network.

5. The POI location prediction method according to claim 1, wherein Step S5 calculates the virtual companion of each user under each weather condition. Specifically, calculate the spatial proximity score SCscore between the user i to be calculated and each other user j: Among them, pu i , pu j are the POI-User vectors of two different users i and j respectively, where one is the user to be calculated; the larger the spatial proximity score SC score, the greater the probability of frequently visiting the same POI location under the current calculated weather conditions. Subsequently, all corresponding users with a spatial proximity score SC score greater than the preset threshold are selected as the virtual companions of the user to be calculated under the current calculated weather conditions.

6. The POI location prediction method according to claim 1, wherein The contrastive learning loss function is constructed as: Among them, represents the value of the contrastive learning loss function; s(.) is the cosine similarity function used to measure the similarity between two vectors; τ is a hyperparameter; is the set of virtual peers of user i to be calculated, and j is the user among them; represents the user set, and u and k are the users among them.

7. The POI location prediction method according to claim 1, wherein The combination method of the contrastive learning loss function and the prediction loss function is: Among them, and represent the predicted loss function value and the contrastive learning loss function value respectively, represents the total loss function value; λ is the hyperparameter for balancing and .

8. An electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that, When the computer program is executed by the processor, the processor is caused to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 7.

Images