Next point of interest recommendation method and device based on multi-modal spatio-temporal feature aggregation network

Abstract

The application belongs to the field of personalized recommendation, and discloses a next interest point recommendation method and device based on a multi-modal space-time feature aggregation network, which comprises the following steps: constructing a multi-modal knowledge graph based on a user's social network, a user check-in sequence and an interest point geographic location; mapping space-time features and multi-modal features into embedding representations respectively; fusing the embedding representations of the space-time features and the multi-modal features through a graph convolution network module to obtain a final embedding matrix; obtaining an aggregated hidden state of a user at a current time step based on the user check-in sequence, the embedding representations of the multi-modal features and the final embedding matrix; splicing the aggregated hidden state and user embedding in the embedding representations of the space-time features, and generating a next interest point interaction probability through a fully connected layer; and selecting the top pre-set number of interest points with the highest probability in the next interest point interaction probability as a recommendation result. The application can adapt to dynamic changes in user preferences and improve the accuracy of next interest point recommendation.

Description

Technical Field

[0001] This invention belongs to the field of personalized recommendation, specifically relating to a method and apparatus for recommending the next point of interest based on a multimodal spatiotemporal feature aggregation network. Background Technology

[0002] Next point of interest (POI) recommendation has become an important component of personalized recommendation, especially with the widespread adoption of mobile devices and the rapid development of location-based social networks. Unlike traditional recommendation systems based on content or item preferences, POI recommendation requires combining spatial and temporal context to predict the location a user is likely to visit next. This spatiotemporal dependency presents unique challenges to POI recommendation modeling, as both the user's geographic location and the specific time of interaction directly impact the recommendation results.

[0003] Existing research focuses on improving the accuracy and reliability of recommendations by optimizing spatiotemporal feature extraction, with knowledge graph-based spatiotemporal feature extraction methods showing particularly outstanding performance. However, users typically interact with only a limited number of points of interest, resulting in extremely limited available behavioral data. The long-standing challenge of data sparsity has constrained effective modeling of user preferences. To alleviate the data sparsity problem, some recent studies have attempted to introduce multimodal content to enhance the representation of points of interest. Although these methods have improved recommendation performance to some extent, key shortcomings remain in the mining and fusion of multimodal information.

[0004] First, while multimodal content is valuable for enriching interest point representations, it often suffers from insufficient information utilization or limited information fusion in practical recommendation tasks. Due to the semantic differences between modalities, extracting and integrating meaningful features from heterogeneous data presents a significant challenge. This not only diminishes the potential value of multimodal information but also makes it difficult for recommendation systems to improve prediction performance by comprehensively understanding the multimodal features of interest points.

[0005] Secondly, many existing methods struggle to effectively model the complex dependencies between spatiotemporal features and multimodal features. Most studies tend to isolate different types of features, failing to effectively characterize their potential interactions, thus limiting the model's ability to mine and model feature dependencies.

[0006] Third, many existing methods employ static or predefined aggregation strategies to integrate auxiliary information, including user social relationships, spatiotemporal information, and multimodal content. These strategies are difficult to adapt to dynamically changing user preferences. Summary of the Invention

[0007] The purpose of this invention is to provide a method and apparatus for recommending the next point of interest based on a multimodal spatiotemporal feature aggregation network, which can adapt to dynamically changing user preferences and improve the accuracy of the next point of interest recommendation.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] The first aspect: Provides a next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network, including:

[0010] A multimodal knowledge graph is constructed based on the user's social network, user check-in sequence, and geographic location of points of interest. The multimodal knowledge graph includes spatiotemporal features and multimodal features.

[0011] The multimodal spatiotemporal feature aggregation network includes an embedding module, a graph convolutional network module, an aggregation module, and a prediction module. The multimodal knowledge graph is input into the multimodal spatiotemporal feature aggregation network to predict the recommendation result for the next point of interest, including:

[0012] The embedding module is used to map spatiotemporal features and multimodal features into embedded representations, respectively;

[0013] The embedding representations of spatiotemporal features and multimodal features are fused through a graph convolutional network module to obtain the final embedding matrix.

[0014] The aggregation model obtains the user's aggregated hidden state at the current time step based on the user's check-in sequence, the embedding representation of multimodal features, and the final embedding matrix;

[0015] The prediction module concatenates the user embeddings in the embedded representations of the hidden state and spatiotemporal features, and passes the concatenated joint representation through a fully connected layer to generate the next interest point interaction probability. The top preset number of interest points with the highest probability among the next interest point interaction probabilities are selected as the recommendation results.

[0016] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0017] Preferably, the construction of a multimodal knowledge graph based on the user's social network, user check-in sequence, and geographic location of points of interest includes:

[0018] Extracting Spatiotemporal Features: Extracting Triples from Social Networks To build social relationships, representing two users and Friendship on social media platforms Extracting triples from user check-in sequences To establish access relationships, representing users With points of interest There is an access relationship Extracting triples based on the time sequence of user check-ins To establish temporal relationships and represent points of interest Points of interest Interviewed afterwards Extracting triples based on the geographic location of points of interest. To establish spatial relationships and represent two points of interest. and Geographic distance less than a predefined distance threshold ;

[0019] Extracting multimodal features: Extracting attribute triples based on multimodal attributes of interest points ,in, Represents entities of interest. Represents multimodal attributes, including metadata, images, and comments. Representative attribute The corresponding multimodal content.

[0020] Preferably, the step of mapping spatiotemporal features and multimodal features into embedded representations using an embedding module includes:

[0021] Initial embeddings of users and points of interest in spatiotemporal features;

[0022] The images in the multimodal features are converted into semantically aligned text descriptions, and the metadata, the text descriptions corresponding to the images, and the modality-specific embeddings of the comments are extracted respectively.

[0023] Preferably, the step of fusing the embedded representations of spatiotemporal features and multimodal features through a graph convolutional network module to obtain the final embedding matrix includes:

[0024] The spatiotemporal similarity matrix is ​​calculated based on the embedding representation of spatiotemporal features, and the normalized spatiotemporal adjacency matrix is ​​obtained.

[0025] The multimodal similarity matrix is ​​calculated based on the embedding representation of multimodal features, and the normalized multimodal adjacency matrix is ​​obtained.

[0026] By fusing the normalized spatiotemporal adjacency matrix and the normalized multimodal adjacency matrix, a multi-feature adjacency matrix is ​​obtained.

[0027] The multi-feature adjacency matrix is ​​passed through multiple layers of graph convolutional network to obtain the final embedding matrix.

[0028] Preferably, the embedding representation based on spatiotemporal features calculates the spatiotemporal similarity matrix and obtains the normalized spatiotemporal adjacency matrix, including:

[0029] Calculate the similarity between the initial embeddings of any two points of interest to obtain the spatiotemporal similarity matrix;

[0030] The k-nearest neighbor method is used to sparsify the spatiotemporal similarity matrix to obtain a sparse spatiotemporal adjacency matrix;

[0031] The sparse spatiotemporal adjacency matrix is ​​symmetrically normalized to obtain the normalized spatiotemporal adjacency matrix.

[0032] Preferably, the embedding representation based on multimodal features calculates the multimodal similarity matrix and obtains the normalized multimodal adjacency matrix, including:

[0033] For each modality, the similarity of the modality-specific embeddings of any two points of interest in that modality is calculated to obtain the modality similarity matrix for each modality;

[0034] The k-nearest neighbor method is used to sparsify the mode similarity matrix of each mode, so as to obtain the sparse mode adjacency matrix of each mode;

[0035] By concatenating the sparse modal adjacency matrices of all modes, a sparse multimodal adjacency matrix is ​​obtained.

[0036] The sparse multimodal adjacency matrix is ​​symmetrically normalized to obtain the normalized multimodal adjacency matrix.

[0037] Preferably, the aggregation model obtains the user's aggregated hidden state at the current time step based on the user check-in sequence, the embedding representation of multimodal features, and the final embedding matrix, including:

[0038] Extract the embeddings of the user's interest points visited at the current time step from the final embedding matrix, and use a recurrent neural network to output the user's hidden state at the current time step;

[0039] The embedding representation based on user check-in sequence and multimodal features calculates the user's preference weights between two time steps;

[0040] The aggregated hidden states of users at historical time steps and the current time step are aggregated and weighted using preference weights. Finally, after normalization, the aggregated hidden states of users at the current time step are obtained.

[0041] Preferably, the embedding representation based on user check-in sequence and multimodal features calculates the user's preference weights between two time steps, including:

[0042] Multimodal embeddings are obtained by concatenating the embedding representations of multimodal features;

[0043] A user-point of interest interaction matrix is ​​constructed based on the user's check-in sequence, and the user-point of interest interaction matrix is ​​weighted and corrected using the user's inverse access frequency;

[0044] The corrected matrix is ​​normalized using the L2 norm, and the multimodal embeddings of all visited interest points are weighted and aggregated using the L2 norm normalization structure to obtain the multimodal user preference embeddings.

[0045] Extract the current user's preference embedding from the multimodal user preference embedding. The user's preference weights between two time steps are calculated using the following formula:

[0046]

[0047] In the formula, Indicates the user at time step and time step The preference weights between them Time step and time step The time interval between Indicates time step Points of interest and time steps visited Spatial distance between visited points of interest It is a periodic function. As the input to a periodic function, and , and These represent the time decay factor and the spatial decay factor, respectively. Indicates time step Points of interest for visits Multimodal embedding.

[0048] The second aspect: provides a device for recommending the next point of interest based on a multimodal spatiotemporal feature aggregation network, including a processor and a memory storing a number of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the next point of interest recommendation method based on the multimodal spatiotemporal feature aggregation network.

[0049] This invention provides a method and apparatus for recommending next point of interest (POI) based on a Multi-Modal and Spatio-Temporal Feature Aggregation Network (MMSTFAN). It captures the complex dependencies between users, POIs, and the surrounding environment by constructing a Multi-Modal Knowledge Graph (MMKG), and utilizes advanced deep learning methods to achieve a unified representation and fusion of spatio-temporal and multi-modal features. Specifically, MMSTFAN consists of four key modules: an embedding module, which encodes the structured information and multi-modal content of the knowledge graph into a unified embedding representation; a Graph Convolutional Network (GCN) module, which dynamically updates the representation of POIs by aggregating the spatio-temporal and multi-modal features of neighboring nodes; an aggregation module, which fuses user historical behavior and current context information to capture user dynamic preferences; and a prediction module, which generates the interaction probability of the next POI based on the learned embeddings and optimizes the overall model training process by designing a loss function. Compared with existing technologies, the advantages of this application are:

[0050] (1) A multimodal knowledge graph integrating spatiotemporal information and multimodal content was constructed. Based on this, a pre-trained cross-modal Transformer and natural language processing model were used to map the multimodal content to a shared semantic space, thereby reducing the impact of semantic differences between different modalities on model prediction.

[0051] (2) A similarity function was designed to construct a similarity matrix based on multimodal features and spatiotemporal features, and integrate GCN to enhance the feature representation of interest points, thereby effectively characterizing the deep dependency relationship between spatiotemporal features and multimodal features.

[0052] (3) A novel user preference weighting strategy is proposed and combined with a standard recurrent neural network (RNN). This strategy aggregates user historical behavior and current context information by considering the spatiotemporal correlation of user behavior and multimodal user preference features, thereby achieving accurate modeling of user interests and preferences. Attached Figure Description

[0053] Figure 1 This is a flowchart of the next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network according to the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0056] Example 1:

[0057] Next point of interest (OPI) recommendation aims to predict a user's potential future locations by mining latent spatiotemporal patterns in their historical behavior. However, due to data sparsity, the prediction accuracy of this task has long been a challenge. Although existing research has made some progress in utilizing spatiotemporal information, it still has the following shortcomings: first, the feature processing methods are relatively isolated, making it difficult to achieve the organic integration of multimodal content; second, there is a lack of adaptive modeling for dynamic changes in user preferences. To address these issues, this paper proposes a multimodal spatiotemporal feature aggregation network to improve the performance of OPI recommendation. First, this invention constructs a multimodal knowledge graph that integrates spatiotemporal information and multimodal content, and introduces a pre-trained model to reduce the semantic differences between different modalities. Second, this invention designs a similarity function based on multimodal and spatiotemporal features to generate a similarity matrix, which provides feature neighborhood structure information for graph convolutional network computation, thereby further enriching the representation of OPIs. In addition, this invention proposes a novel user preference weighting strategy to explicitly consider the spatiotemporal relevance of user behavior and multimodal user preferences when integrating user historical behavior and current context information, thereby more effectively capturing user interests. Comprehensive experimental results based on the dataset show that MMSTFAN outperforms existing methods in both recommendation accuracy and stability.

[0058] like Figure 1 As shown in this embodiment, a next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network is presented. The next point of interest recommendation problem is defined as follows: given a multimodal knowledge graph... Target users and user check-in sequence Predict target users The next most likely visit One point of interest. The method specifically includes the following steps:

[0059] The user check-in sequence is defined as follows: Let the user set be... The set of points of interest is ,in and These represent the total number of users and points of interest, respectively. Each point of interest... It has corresponding latitude and longitude coordinates. For each user... Its historical behavior can be represented by a check-in sequence. This is represented, where each sign-in record is shown. Indicates user In timestamp Visited points of interest .

[0060] Multimodal knowledge graph representation is .in, This represents a collection of entities, encompassing all users and points of interest. It represents a set of relationships, including social, access, and temporal-spatial relationships; This represents a set of multimodal attributes, including "has comments", "has images", and "has metadata". This represents a collection of multimodal content, including comments, images, and metadata. and These represent the sets of relation triples and attribute triples, respectively.

[0061] Step 1: Construct a multimodal knowledge graph based on the user's social network, user check-in sequence, and geographic location of points of interest. The multimodal knowledge graph includes spatiotemporal features and multimodal features.

[0062] like Figure 1 As shown, the multimodal knowledge graph constructed in this embodiment consists of two layers: a spatiotemporal layer and a multimodal layer. The spatiotemporal layer extracts structured relationships from social networks, user check-in sequences, and geographic locations of points of interest, forming relation triples. The multimodal layer associates the multimodal content corresponding to each point of interest into attribute triples. The construction process of each layer will be described in detail below.

[0063] (1) The spatiotemporal layer contains four types of structured relationships: social relationships, access relationships, temporal relationships, and spatial relationships. These four types of structured relationships are denoted as spatiotemporal features. The specific construction method is as follows: extract triples from the social network. To build social relationships, representing two users and They are friends on social media platforms. Indicates a friend relationship. Extracting triples from user check-in sequences To establish access relationships, representing users Points of interest visited , Indicates access; extracts triples based on the time sequence of user check-ins. To construct a temporal relationship, representing points of interest within a user's check-in sequence. Immediately following the point of interest He was interviewed later. , Indicates subsequent visits; extracts triples based on the geographic location information of the points of interest. To establish spatial relationships and represent two points of interest. and The geographical distance is less than a predefined distance threshold (e.g., 0.2 km). This indicates a distance less than a predefined threshold.

[0064] (2) The multimodal layer uses multimodal content as attribute values ​​to associate with corresponding interest point entities, thus obtaining multimodal features. This layer covers three types of multimodal attributes related to interest points: metadata, images, and comments. These attributes are represented by attribute triples. It is represented in the form of . Among them, Represents entities of interest. Represents multimodal attributes, Representative attribute The corresponding multimodal content.

[0065] Step 2: Input the multimodal knowledge graph into the multimodal spatiotemporal feature aggregation network to predict the recommendation result of the next point of interest.

[0066] like Figure 1 As shown, the MMSTFAN proposed in this embodiment includes four modules: an embedding module, a GCN module, an aggregation module, and a prediction module. These four modules progressively complete the modeling and fusion of multimodal features and spatiotemporal features in the multimodal knowledge graph, ultimately achieving the prediction of the next point of interest.

[0067] Step 2.1: Use the embedding module to map spatiotemporal features and multimodal features into embedded representations. The goal of the embedding module is to uniformly map the structured spatiotemporal information and unstructured multimodal content in the multimodal knowledge graph into embedded representations that can be processed by downstream modules.

[0068] For the structured spatiotemporal information in the spatiotemporal layer, this embodiment uses the TransE (TranslatingEmbeddings for Modeling Multi-relational Data) model as the knowledge graph embedding model to learn the initial embeddings of users and points of interest. and Representing users respectively and points of interest The embedding vector, where This represents the embedding dimension. Each relation edge in the spatiotemporal layer is represented as a triple. The head entity Tail-end entity On behalf of users or points of interest , It encompasses social, access, spatial, and temporal relationships between entities. The TransE model measures the reasonableness of triples using the following scoring function:

[0069] (1)

[0070] in, Represents a triplet The score, and Let these represent the embedding vectors of the head and tail entities, respectively. The feature embeddings representing relations are lower scores indicating triples. The closer it is to the real relationship.

[0071] Subsequently, this embodiment employs a graph loss function. To optimize the knowledge graph embedding, thereby more effectively reflecting the real structure of the spatiotemporal layer, as shown in formula (2).

[0072] (2)

[0073] in, By using effective triples Tail entity in Replace with another entity Invalid triples obtained ; Indicates the activation function; Represents a triplet The score.

[0074] In multimodal knowledge graphs The multimodal layer integrates three modalities of content associated with points of interest: metadata, images, and comments. To fully extract semantic information from multimodal content, the key is to reduce the semantic differences between different modalities, especially the differences between visual and textual data. To this end, this embodiment uses BLIP (Bootstrapping Language-Image Pre-training model) as a pre-trained image-to-text (img2text) model to convert images associated with points of interest into semantically aligned text descriptions, as shown in formula (3).

[0075] (3)

[0076] in, Representation and points of interest A related set of images; This indicates that after BLIP processing and Corresponding text description; and These respectively represent points of interest. The number of associated images and the number of text descriptions corresponding to these images; .

[0077] Subsequently, this embodiment uses BERT (Bidirectional Encoder Representations from Transformers) as a unified pre-trained language model to extract modality-specific embeddings from three types of textual representations (including metadata, image-generated text representations, and user comments). Text representations of different modalities have different semantic focuses. Metadata describes the structured attributes of points of interest, such as category labels and opening hours; image-generated text representations focus on describing the goods and indoor environment offered by the points of interest; and comments reflect the user's subjective experience and emotional inclination. Therefore, to preserve the independent semantic information of each modality, this embodiment constructs independent embedding representations for each modality to avoid semantic confusion. The specific process is shown in formulas (4), (5), and (6).

[0078] (4)

[0079] in, Representation and points of interest A collection of associated metadata; Representative points of interest The amount of metadata; Indicate points of interest The metadata embedding is calculated by averaging the embedding results of each metadata item after BERT encoding. It represents a unified multimodal embedding dimension.

[0080] (5)

[0081] in, Indicate points of interest The image embedding, which is obtained by using BERT-encoded text descriptions. The embedding results are obtained by averaging. Indicate points of interest The number of images, Representation and points of interest A collection of related images.

[0082] (6)

[0083] in, Representation and points of interest Related collection of comments; Representatives and points of interest Number of related comments; Indicate points of interest Comment embedding, which is obtained by encoding the comments using BERT. The embedding results are obtained by averaging.

[0084] Step 2.2: The embedding representations of spatiotemporal features and multimodal features are fused through the GCN module to obtain the final embedding matrix.

[0085] The GCN module aims to update the interest point embedding by iteratively aggregating the spatiotemporal and multimodal features of neighboring nodes. Before GCN processing, the spatiotemporal similarity matrix and multimodal similarity matrix are calculated based on the output representation of the embedding module.

[0086] First, interest points are computed in the embedded representation of spatiotemporal features. The pairwise similarity between them is used to construct a spatiotemporal similarity matrix. Any two points of interest and The spatiotemporal similarity function between them is defined as shown in formula (7).

[0087] (7)

[0088] in, and Representing points of interest respectively and Spatiotemporal feature embedding, The modulo operator is used. This matrix captures a multimodal knowledge graph. Structural information of the spatiotemporal layer.

[0089] To reduce subsequent computational complexity and enable the model to focus on more meaningful similarity information, this paper employs the k-nearest neighbor (KNN) method to sparsify the spatiotemporal similarity matrix. For each point of interest Only retain its in The most similar to the first There are 10 adjacent points of interest, as shown in formula (8).

[0090] (8)

[0091] in, express Is with The most similar One of the points of interest. The resulting sparse spatiotemporal adjacency matrix. It can more accurately reflect the neighborhood structure of interest points based on spatiotemporal features.

[0092] To ensure numerical stability during the propagation of the graph neural network, this embodiment focuses on the sparse spatiotemporal adjacency matrix. Symmetric normalization is performed to obtain the normalized spatiotemporal adjacency matrix. As shown below:

[0093] (9)

[0094] (10)

[0095] in, This indicates adding a self-connect ( The sparse spatiotemporal adjacency matrix after ) ; express The corresponding degree matrix, the diagonal elements of the degree matrix are composed of Calculated.

[0096] For points of interest Corresponding multimodal content embedding , and Calculate the similarity between any two points of interest for each modality to construct a modality similarity matrix for each modality. , and As shown in formula (11). To reduce computational complexity and preserve key semantic connections, this embodiment uses the k-nearest neighbor method to sparsify the similarity matrix of each modality, thereby obtaining the sparse modal adjacency matrix corresponding to each modality. , and As shown in formula (12).

[0097] (11)

[0098] (12)

[0099] in," "Recommended" " "or" ", which correspond to metadata, images, and comment modalities, respectively; Indicate points of interest corresponding Modal embedding, Indicate points of interest corresponding Modal embedding.

[0100] In order to model the synergistic relationship of multimodal content, the sparse modal adjacency matrices of each modality are fused, as shown in formula (13).

[0101] (13)

[0102] The resulting sparse multimodal adjacency matrix It captures the comprehensive neighborhood structure between points of interest based on multimodal content embedding.

[0103] To further improve the stability of the graph structure and adapt it to the message passing mechanism of graph neural networks, this embodiment further improves... Perform symmetric normalization. First, [the process involves]... with self-connection Combined, we obtain As shown in formula (14). Then calculate... angle matrix Finally, the normalized multimodal adjacency matrix is ​​obtained. As shown in formula (15):

[0104] (14)

[0105] (15)

[0106] Finally, the normalized spatiotemporal adjacency matrix is... With normalized multimodal adjacency matrix Fusion is used to construct a multi-feature adjacency matrix that comprehensively considers spatiotemporal dynamics and multimodal semantics. As shown in formula (16).

[0107] (16)

[0108] in, It is a hyperparameter used to balance the relative importance of spatiotemporal features and multimodal features in graph relationship modeling.

[0109] This embodiment further incorporates a multi-feature adjacency matrix. After multiple GCN layers, the final embedding matrix is ​​obtained. A single GCN layer is based on the fused multi-feature adjacency matrix. Information is aggregated for interest point nodes with similar spatiotemporal and multimodal semantic features, thereby updating the feature embedding of interest points, as shown in formula (17).

[0110] (17)

[0111] in, , This represents the initial embedding matrix of all interest points computed in the embedding module; Indicates the first The updated interest point embedding output by the layer GCN; Indicates the first Layer weight matrix; Indicates the corresponding bias term; and Representing the first Layer and first Layer channel dimension; This represents a non-linear activation function. (Through...) The update of the layer GCN can obtain the final embedding matrix of the interest points. .

[0112] Step 2.3: The aggregation model obtains the user's aggregated hidden state at the current time step based on the user check-in sequence, the embedding representation of multimodal features, and the final embedding matrix.

[0113] In the aggregation module, for users Sign-in sequence The internal sequence patterns can be modeled using a standard RNN, and the time step can be used to model them. Generate the corresponding hidden state As shown in formula (18).

[0114] (18)

[0115] in, Indicates the previous time step The hidden state is used to store the characteristics of historical check-in behavior in the sequence; Indicates from the final embedding matrix The user who retrieved At time step Embedding of points of interest for access; and These represent the weight matrices of the previous hidden state and the current input, respectively. Indicates the bias term; This represents a non-linear activation function.

[0116] However, standard RNNs struggle to adequately model the spatiotemporal relevance of user behavior and multimodal user preferences, both of which significantly influence a user's decision-making process regarding their next point of interest. Specifically, recently accessed points of interest are typically more influential than those accessed earlier, and geographically proximate points of interest tend to be more correlated than those further apart. Furthermore, users often exhibit specific preferences for attributes of interest presented in multimodal content (such as service quality and atmosphere). To mitigate the limitations of standard RNNs in this regard, this embodiment further integrates multimodal user preferences into the spatiotemporal relevance model to achieve dynamic adaptive aggregation of historical behavior and current contextual information. This embodiment calculates the user's preference weights between two time steps based on the user's check-in sequence and the embedded representation of multimodal features, as shown below:

[0117] (19)

[0118] (20)

[0119] (twenty one)

[0120] (twenty two)

[0121] (twenty three)

[0122] First, by embedding multimodal content such as metadata, images, and text. , and Gain points of interest Multimodal embedding As shown in formula (19). Subsequently, a user-interest point interaction matrix is ​​constructed based on the user's check-in sequence. ,in Indicates user Points of interest The frequency of visits. To mitigate the impact of popular interests on user preferences, the inverse visit frequency of users is used to... After weighted correction, we obtain As shown in formula (20). Next, the corrected matrix... Perform L2 norm normalization on each row to obtain As shown in formula (21). Based on this, the multimodal user preference embedding is obtained by weighted aggregation of the multimodal embeddings of all visited points of interest. As shown in formula (22). Finally, the preference weights are defined. As shown in formula (23), this weight comprehensively considers the spatiotemporal correlation of user behavior and multimodal user preferences. Among them, the higher the similarity between the multimodal user preference embedding and the multimodal embedding of access interest points, the greater the weight assigned to the corresponding latent state, thereby realizing the dynamic modeling of user personalized preferences.

[0123] After introducing user preference weights, the aggregated hidden states of users at historical time steps and the current time step are aggregated and weighted using preference weights. Finally, after normalization, the aggregated hidden state of the user at the current time step is obtained. Aggregate hidden state It can be expressed by formula (24).

[0124] (twenty four)

[0125] in, This embodiment indicates that the user is at the time step. and time step Preference weights between By explicitly utilizing temporal and spatial information, we model the spatiotemporal correlations at different time steps, while also considering multimodal user preferences. Indicates time step The corresponding hidden historical state; Used for normalization to prevent any single hidden state from dominating the aggregation result. Time step and time step The time interval between; Indicates time step Points of interest and time steps visited Spatial distance between visited points of interest; It is a periodic function used to characterize users' periodic access preferences. As the input to a periodic function, and ; and These represent the time decay factor and the spatial decay factor, respectively, used to control the rate at which the correlation decreases with increasing time intervals and spatial distances; the exponential decay term... and The trend of natural decay of user behavior relevance with increasing time interval and spatial distance was modeled separately. Indicates time step Points of interest for visits Multimodal embedding.

[0126] Step 2.4: The prediction module concatenates the user embeddings in the embedded representations of the aggregated hidden states and spatiotemporal features, and passes the concatenated joint representation through a fully connected layer to generate the next interest point interaction probability. The top preset number of interest points with the highest probability among the next interest point interaction probabilities are taken as the recommendation results.

[0127] In the prediction module, the hidden states will be aggregated. With users initial embedding The data are concatenated to form a joint representation. This joint representation is then passed through a fully connected layer to generate the interaction probability for the next point of interest. , The probability distribution includes all points of interest as the next point of interest. Indicates user In the next time step Visit points of interest The probability of the next point of interest interaction is determined, and the highest probability is selected based on the probability of the next point of interest interaction. Each point of interest is used as the recommendation result. The specific calculation process is shown in formula (25).

[0128] (25)

[0129] in, This represents the trainable weight matrix of the fully connected layer.

[0130] To optimize the training process of the model, this embodiment adopts the cross-entropy loss function as shown in formula (26).

[0131] (26)

[0132] in, Indicates user The length of the historical access sequence; and These represent the user predicted by the model at time step [1]. Actual visit points of interest Other points of interest The probability of interaction; This represents a non-linear activation function. By minimizing this loss function, the trained model can improve the performance of actual access points of interest. Predicted probability And suppress other points of interest. probability This improves the ability to differentiate recommendations and enhances recommendation accuracy.

[0133] (a) Dataset:

[0134] To comprehensively evaluate the effectiveness and robustness of the MMSTFAN proposed in this invention, a comprehensive experiment was conducted on four real-world datasets: NYC, TKY, Philadelphia, and New Orleans. The sources and characteristics of each dataset are described below:

[0135] The NYC and TKY datasets are sourced from the Foursquare platform and contain user check-in data. The time span is from April 2012 to February 2013. The data includes check-in time, GPS coordinates, and information about the locations visited by users. In addition, multimodal content (such as images and comments) related to points of interest were collected.

[0136] The Philadelphia and New Orleans datasets are sourced from the Yelp platform and contain user check-ins and reviews of various businesses. The original Yelp dataset was filtered to retain only data from the Philadelphia and New Orleans areas. In addition to check-in data, the platform also provides rich multimodal content, such as user reviews, images, and business metadata (e.g., business descriptions, categories, and opening hours).

[0137] To reduce the interference of inactive users and less popular points of interest on the model training process, this experiment preprocessed all datasets as follows: users and points of interest with fewer than 10 check-ins were removed, and the check-in records of each user were sorted in ascending order of timestamp. Subsequently, the first 80% of each user's check-in data was divided into fixed-length sequences as the training set, and the remaining 20% ​​was used as the test set. Table 1 summarizes the key statistical information of the four datasets after preprocessing.

[0138] Table 1. Statistical information of the four datasets used in this paper.

[0139]

[0140] (ii) Baseline Model:

[0141] To verify the effectiveness of the proposed model, this experiment selected eight next interest point recommendation methods as baseline models, as follows:

[0142] LSTPM model: This model captures users’ long-term preferences by modeling contextual information of points of interest and uses a geographically extended RNN to model short-term preferences.

[0143] Flashback model: An RNN-based model that improves prediction performance by weighting historical hidden states in a spatiotemporal context.

[0144] PLSPL model: This model models long-term and short-term preferences through attention mechanism and LSTM, while taking into account contextual features such as category and check-in time.

[0145] STAN model: This model models the spatiotemporal dependencies in user trajectories through a dual attention architecture and provides personalized recommendations based on item frequency.

[0146] Graph-Flashback model: This model uses a graph-based approach to capture spatiotemporal dependencies and employs an RNN to model the sequence dynamics of user movement.

[0147] DisenPOI model: This model decouples the sequential and spatial factors in user behavior by constructing sequence graphs and geographic graphs, and combines contrastive learning to capture spatiotemporal dependencies.

[0148] GETNext model: A graph-enhanced Transformer model that integrates global patterns, user behavior patterns, spatial information, and temporal information.

[0149] MMPOI model: A multimodal model that enriches the representation of points of interest by integrating check-in data and additional modalities (e.g., images and comments).

[0150] To comprehensively evaluate the performance of MMSTFAN relative to the baseline model, this embodiment uses two widely used recommended performance metrics: Hit@K (Hit ratio at K) and NDCG@K (Normalized Discounted Cumulative Gain at K).

[0151] Hit@K is used to measure whether the true next point of interest appears before the model's prediction. In the recommended results, K={5,20} was set in the experiment to evaluate the model's recommendation performance, and the corresponding calculation method is as follows:

[0152] (27)

[0153] in, The model represents the user. The most likely prediction is before the next visit. A set of points of interest This indicates the user's next point of interest, i.e., the actual tag. Indicates the indicator function, when the actual label exist If it is in the middle, it is 1; otherwise, it is 0.

[0154] NDCG is a commonly used ranking quality evaluation metric in recommender systems. This metric measures actual ranking quality using DCG (Discounted Cumulative Gain) and normalizes it using IDCG (Ideal Discounted Cumulative Gain), thus enabling performance comparison across datasets. The specific definition is as follows:

[0155] (28)

[0156] (29)

[0157] (30)

[0158] In formula (28), Indicates the model's prediction of the previous Of the points of interest, the first one is... The relevance score of the interest point at the predicted location. In the next interest point recommendation task, if the... Interest points at predicted locations and the user's next actual visit interest points If matched, then ,otherwise . As a positional penalty factor, higher ranking results are given higher weight. In formula (29), This represents the relevance score under ideal ranking, which is the maximum DCG value that can be achieved when the next point of interest is in the first position. In formula (30), DCG is normalized to the interval [0,1], and the closer the value is to 1, the higher the ranking quality.

[0159] (III) Experimental Setup:

[0160] Computing environment: The MMSTFAN model is implemented based on the PyTorch framework and trained on a single machine. This machine is configured with 512GB of memory, an NVIDIA A800 GPU, and an x86_64 architecture processor, running Ubuntu 20.04.06 LTS.

[0161] Hyperparameter settings: When constructing the spatiotemporal adjacency matrix and the multimodal adjacency matrix, each interest point is considered The nearest neighbors. Weight parameters used to integrate spatiotemporal features and multimodal features. Set to 0.7. Based on empirical observation, the dimensions of the hidden state and the embedding dimensions of users and points of interest are both set to 64. Time decay factor. and spatial decay factor The values ​​are respectively and .

[0162] (iv) Performance comparison:

[0163] This experiment compared the proposed MMSTFAN with eight baseline models on four datasets to verify the effectiveness of the proposed MMSTFAN model. The experimental results are shown in Table 2.

[0164] Table 2 Performance comparison of different models on four datasets

[0165]

[0166] Based on the comparison of the values ​​in Table 2, the following conclusions can be drawn:

[0167] (1) Experimental results show that MMSTFAN generally achieves better recommendation performance than all baseline models on all datasets, which fully demonstrates the necessity of integrating spatiotemporal features, multimodal features and user preferences.

[0168] (2) Compared with sequence models using RNN or LSTM (including LSTPM, Flashback, PLSPL and STAN), graph-based models (including Graph-Flashback, DisenPOI, GETNext, MMPIOI and MMSTFAN) showed more stable recommendation performance on all four datasets. This result shows that graph-based models can capture the complex relationships between multi-user sequences more effectively than sequence models.

[0169] (3) Compared with existing models that do not incorporate multimodal content (including LSTPM, Flashback, PLSPL, STAN, Graph-Flashback, DisenPOI, and GETNext), MMSTFAN exhibits superior recommendation performance, indicating that incorporating multimodal content helps improve the accuracy of next point of interest recommendations. Further analysis of performance gains on different datasets reveals that MMSTFAN's performance on the Philadelphia and New_Orleans datasets is significantly higher than that on the NYC and TKY datasets. Since the check-in density on the Philadelphia and New_Orleans datasets is lower than that on NYC and TKY, this phenomenon indicates that MMSTFAN has a strong ability to alleviate data sparsity problems.

[0170] (4) Compared with the MMPIOI model, which also integrates multimodal content and spatiotemporal information, MMSTFAN still achieves better performance. This shows that by modeling spatiotemporal features through knowledge graph embedding and combining user preference weighting mechanism to aggregate user historical behavior, the complex relationship between spatiotemporal features and multimodal features can be modeled more effectively, thereby further improving the recommendation performance of the model.

[0171] (v) Ablation studies:

[0172] This experiment designed three simplified variants of MMSTFAN to systematically evaluate the impact of each core component on the overall model performance. The specific design of each variant is as follows:

[0173] w / o MM: Remove the multimodal adjacency matrix and retain only the spatiotemporal adjacency matrix. Specifically, the fusion weights in formula (16) are... Set to 1 to disable the integration mechanism of multimodal features.

[0174] w / o PW: Remove the user preference weights defined in formula (23). At this time, the aggregate hidden state in formula (24) is... Hidden state that relies solely on the output of a standard RNN We ignore the effects of time intervals, spatial distances, and multimodal user preferences.

[0175] w / o MMP: Directly use the formula The defined original weights do not incorporate the similarity between the multimodal embeddings of user preferences and the multimodal embeddings of access points of interest. .

[0176] To comprehensively evaluate the contribution of each component to the performance of MMSTFAN, ablation analysis was conducted on four real-world datasets. The experimental results are shown in Table 3.

[0177] Table 3 Ablation analysis of removed components

[0178]

[0179] Based on the results in Table 3, the following conclusions can be drawn:

[0180] (1) w / o MM: After removing the multimodal adjacency matrix, the model exhibits a significant performance degradation on all datasets. This indicates that multimodal features provide a useful supplement to interest point representation modeling in addition to spatiotemporal features. This performance degradation is particularly pronounced on sparse datasets (e.g., Philadelphia and New_Orleans), demonstrating that the fusion of multimodal features can enhance interest point representation and alleviate the data sparsity problem.

[0181] (2) w / o PW: The model performance degrades most significantly after removing the user preference weights, particularly in high-density datasets such as NYC and TKY. This component achieves dynamic weighted aggregation of user historical behavior by jointly modeling time intervals, spatial distances, and multimodal user preferences, thereby effectively coordinating the spatiotemporal context of users' recent travels with their long-term preferences. Removing this component weakens the model's adaptability to the evolution of user preferences. This indicates that user preference weights play a crucial role in balancing historical behavior and current context.

[0182] (3) w / o MMP: After removing multimodal user preferences, the model performance decreased to some extent, but the impact was relatively small compared to other modules. This indicates that although spatiotemporal dependence is the main driving factor of user behavior, combining multimodal personalized preferences can still further improve the model's ability to capture subtle user decision-making patterns. The above experimental results verify that MMSTFAN can effectively improve the accuracy of recommending the next point of interest.

[0183] Example 2:

[0184] This embodiment provides a next point of interest recommendation device based on a multimodal spatiotemporal feature aggregation network, including a processor and a memory storing a number of computer instructions. When the computer instructions are executed by the processor, they implement the steps of the next point of interest recommendation method based on the multimodal spatiotemporal feature aggregation network.

[0185] For specific limitations on the next point of interest recommendation device based on multimodal spatiotemporal feature aggregation network, please refer to the limitations on the next point of interest recommendation method based on multimodal spatiotemporal feature aggregation network mentioned above, which will not be repeated here.

[0186] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the method of the present invention by running the computer program stored in the memory.

[0187] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.

[0188] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0189] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for recommending next point of interest based on a multimodal spatiotemporal feature aggregation network, characterized in that, The next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network includes: A multimodal knowledge graph is constructed based on the user's social network, user check-in sequence, and geographic location of points of interest. The multimodal knowledge graph includes spatiotemporal features and multimodal features. The multimodal spatiotemporal feature aggregation network includes an embedding module, a graph convolutional network module, an aggregation module, and a prediction module. The multimodal knowledge graph is input into the multimodal spatiotemporal feature aggregation network to predict the recommendation result for the next point of interest, including: The embedding module is used to map spatiotemporal features and multimodal features into embedded representations, respectively; The embedding representations of spatiotemporal features and multimodal features are fused through a graph convolutional network module to obtain the final embedding matrix. The aggregation model obtains the user's aggregated hidden state at the current time step based on the user's check-in sequence, the embedding representation of multimodal features, and the final embedding matrix; The prediction module concatenates the user embeddings in the embedded representations of the hidden state and spatiotemporal features, and passes the concatenated joint representation through a fully connected layer to generate the next interest point interaction probability. The module then selects the top preset number of interest points with the highest probability among the next interest point interaction probabilities as the recommendation result. The aggregation model obtains the user's aggregated hidden state at the current time step based on the user check-in sequence, the embedded representation of multimodal features, and the final embedding matrix, including: Extract the embeddings of the user's interest points visited at the current time step from the final embedding matrix, and use a recurrent neural network to output the user's hidden state at the current time step; The embedding representation based on user check-in sequence and multimodal features calculates the user's preference weights between two time steps; The aggregated hidden states of users at historical time steps and the current time step are aggregated and weighted using preference weights. Finally, after normalization, the aggregated hidden states of users at the current time step are obtained. The embedding representation based on user check-in sequences and multimodal features calculates the user's preference weights between two time steps, including: Multimodal embeddings are obtained by concatenating the embedding representations of multimodal features; A user-point of interest interaction matrix is ​​constructed based on the user's check-in sequence, and the user-point of interest interaction matrix is ​​weighted and corrected using the user's inverse access frequency; The corrected matrix is ​​normalized using the L2 norm, and the multimodal embeddings of all visited interest points are weighted and aggregated using the L2 norm normalization structure to obtain the multimodal user preference embeddings. Extract the current user's preference embedding from the multimodal user preference embedding. The user's preference weights between two time steps are calculated using the following formula: In the formula, Indicates the user at time step and time step The preference weights between them Time step and time step The time interval between Indicates time step Points of interest and time steps visited Spatial distance between visited points of interest It is a periodic function. As the input to a periodic function, and , and These represent the time decay factor and the spatial decay factor, respectively. Indicates time step Points of interest for visits Multimodal embedding.

2. The next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network according to claim 1, characterized in that, The construction of a multimodal knowledge graph based on users' social networks, check-in sequences, and geographic locations of points of interest includes: Extracting Spatiotemporal Features: Extracting Triples from Social Networks To build social relationships, representing two users and Friendship on social media platforms Extracting triples from user check-in sequences To establish access relationships, representing users With points of interest There is an access relationship Extracting triples based on the time sequence of user check-ins To establish temporal relationships and represent points of interest Points of interest Interviewed afterwards Extracting triples based on the geographic location of points of interest. To establish spatial relationships and represent two points of interest. and Geographic distance less than a predefined distance threshold ; Extracting multimodal features: Extracting attribute triples based on multimodal attributes of interest points ,in, Represents entities of interest. Represents multimodal attributes, including metadata, images, and comments. Representative attribute The corresponding multimodal content.

3. The next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network according to claim 1, characterized in that, The method of mapping spatiotemporal features and multimodal features into embedded representations using an embedding module includes: Initial embeddings of users and points of interest in spatiotemporal features; The images in the multimodal features are converted into semantically aligned text descriptions, and the metadata, the text descriptions corresponding to the images, and the modality-specific embeddings of the comments are extracted respectively.

4. The next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network according to claim 1, characterized in that, The process of fusing the embedded representations of spatiotemporal features and multimodal features through a graph convolutional network module to obtain the final embedding matrix includes: The spatiotemporal similarity matrix is ​​calculated based on the embedding representation of spatiotemporal features, and the normalized spatiotemporal adjacency matrix is ​​obtained. The multimodal similarity matrix is ​​calculated based on the embedding representation of multimodal features, and the normalized multimodal adjacency matrix is ​​obtained. By fusing the normalized spatiotemporal adjacency matrix and the normalized multimodal adjacency matrix, a multi-feature adjacency matrix is ​​obtained. The multi-feature adjacency matrix is ​​passed through multiple layers of graph convolutional network to obtain the final embedding matrix.

5. The next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network according to claim 4, characterized in that, The embedding representation based on spatiotemporal features calculates the spatiotemporal similarity matrix and obtains the normalized spatiotemporal adjacency matrix, including: Calculate the similarity between the initial embeddings of any two points of interest to obtain the spatiotemporal similarity matrix; The k-nearest neighbor method is used to sparsify the spatiotemporal similarity matrix to obtain a sparse spatiotemporal adjacency matrix; The sparse spatiotemporal adjacency matrix is ​​symmetrically normalized to obtain the normalized spatiotemporal adjacency matrix.

6. The next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network according to claim 4, characterized in that, The embedding representation based on multimodal features calculates the multimodal similarity matrix and obtains the normalized multimodal adjacency matrix, including: For each modality, the similarity of the modality-specific embeddings of any two points of interest in that modality is calculated to obtain the modality similarity matrix for each modality; The k-nearest neighbor method is used to sparsify the mode similarity matrix of each mode, so as to obtain the sparse mode adjacency matrix of each mode; By concatenating the sparse modal adjacency matrices of all modes, a sparse multimodal adjacency matrix is ​​obtained. The sparse multimodal adjacency matrix is ​​symmetrically normalized to obtain the normalized multimodal adjacency matrix.

7. A device for recommending next point of interest based on a multimodal spatiotemporal feature aggregation network, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the next point of interest recommendation method based on a multimodal spatiotemporal feature aggregation network as described in any one of claims 1 to 6.

Images