A point of interest recommendation method based on multi-modal hypergraph high-order semantic modeling

Abstract

The application discloses a kind of interest point recommendation methods based on multi-modal hypergraph high-order semantic modeling, comprising: data preprocessing and multi-modal feature extraction, and construct user interest point check-in trajectory graph and geographic grid graph;With each modal feature respectively constructing hypergraph and carrying out hypergraph convolution, learn the high-order semantic embedding of each mode;Through adaptive gating mechanism, each road preference embedding is dynamically weighted and fused, to obtain multi-modal fusion embedding;Fusion user, interest point, time, geography and multi-modal embedding are sequentially modeled, and through multi-task uncertainty weighted joint training, output interest point recommendation result.The application can effectively depict the high-order semantic association behind interest point multi-modal feature by constructing hypergraph for each mode and carrying out hypergraph convolution, the application can improve the applicability and robustness of model in complex scene while ensuring the accuracy of recommendation by providing multi-modal hypergraph structure and dynamic fusion mechanism.

Description

Technical Field

[0001] This invention belongs to the field of point-of-interest (POI) recommendation technology, and mainly relates to a method for personalized location recommendation using multimodal content data of POIs and user check-in sequences. Specifically, it relates to a POI recommendation method that integrates multimodal hypergraphs and adaptive gating. Background Technology

[0002] With the development of mobile internet and location-aware technologies, location-based social networks (LBSNs) have accumulated a wealth of user check-in data and information related to points of interest (POIs), providing a rich data foundation for applications such as user travel services, urban computing, and precise recommendations. Point-of-interest (POI) recommendation aims to predict potential future locations based on users' historical browsing behavior, thereby helping users quickly discover target locations that match their preferences. It is an important research area in the field of location services.

[0003] In recent years, numerous research methods have been proposed for interest-based recommendation. Early methods primarily relied on techniques such as collaborative filtering or matrix factorization to mine user preferences by analyzing the interaction relationships between users and interest points. However, these methods are highly dependent on users' historical interaction data. When there are few check-in records or new interest points lack historical access data, they are easily affected by data sparsity and cold start problems, resulting in a significant decline in recommendation performance. Subsequently, researchers introduced sequence modeling methods such as recurrent neural networks and Transformers to learn the temporal dependencies in users' check-in behavior, which improved recommendation performance to some extent. However, these methods usually focus more on the user behavior sequence itself and do not make sufficient use of the content information contained in the interest points, making it difficult to fully reflect the potential high-order semantic connections between different interest points.

[0004] With the development of graph neural networks, more and more studies are beginning to use graph structures to describe the relationships between users and points of interest, improving node representation capabilities through neighborhood information propagation. Although graph neural networks can effectively integrate user interaction relationships and spatial proximity relationships, traditional graph structures can only describe pairwise connections between nodes. They lack the ability to effectively express complex relationships such as multiple points of interest sharing similar semantic features, common attributes, or the same content patterns, thus making it difficult to fully explore higher-order relationships between points of interest.

[0005] On the other hand, modern points of interest (POIs) are often accompanied by rich multimodal information such as images, comment text, and category attributes. This content can reflect the semantic features of POIs from different perspectives, which is of great value in improving recommendation accuracy. However, most existing multimodal POI recommendation methods directly concatenate or simply weight and fuse features from different modalities, without fully considering the differences in semantic expressive power and data distribution between modalities. They also lack a unified modeling mechanism for high-order relationships within multimodal content, making it difficult to fully leverage the complementary advantages between different modalities. Furthermore, the formation of user interests is influenced by a combination of factors, including historical behavior, POI content, time factors, and spatial factors. If feature fusion is performed using only a fixed method, it can easily lead to interference between information from different sources, limiting the recommendation model's ability to express complex user preferences.

[0006] Therefore, how to fully utilize the high-order semantic associations contained in the multimodal content of interest points, construct a representation model that can express complex relationships under a unified framework, and further combine user spatiotemporal behavior to achieve collaborative modeling of multi-source information has become a key technical problem that urgently needs to be solved in the current field of interest point recommendation. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes an interest point recommendation method based on high-order semantic modeling of multimodal hypergraphs. This method constructs a hypergraph for each modality and performs hypergraph convolution to characterize the high-order semantic relationships behind the multimodal features of interest points. An adaptive gating mechanism is used to dynamically weight and fuse the normalized multi-path preference embeddings. Finally, sequence modeling is performed by combining user, interest point, time, geographic, and multimodal fusion embeddings, and joint training is conducted using multi-task uncertainty weighting, thereby significantly improving recommendation accuracy in scenarios with sparse data and cold start.

[0008] The technical solution of this invention is as follows:

[0009] A method for recommending interest points that integrates multimodal hypergraphs and adaptive gating, the method comprising:

[0010] Step 1: Data preprocessing and multimodal feature extraction, and construction of user interest point check-in trajectory map and geographic grid map;

[0011] Step II: Construct hypergraphs based on the features of each modality and perform hypergraph convolution to learn the high-order semantic embeddings of each modality;

[0012] Step III: Dynamically weight and fuse the preference embeddings of each path through an adaptive gating mechanism to obtain a multimodal fused embedding;

[0013] Step IV: Integrate user, interest point, time, geography and multimodal embeddings to perform sequence modeling, and output interest point recommendation results through multi-task uncertainty weighted joint training.

[0014] The dataset refers to a collection of user check-in data and multimodal content of points of interest in a location-based social network. It includes user ID, point of interest ID, point of interest category ID, longitude and latitude coordinates of the location of the point of interest, timestamp of user check-in at the point of interest, and multimodal content such as images, user comment text and attribute metadata of each point of interest.

[0015] The term "check-in" refers to the act of a user visiting a point of interest at a certain time.

[0016] The user's check-in sequence refers to the sequence obtained by sorting the user's historical check-ins in chronological order. ={ , ,…, }, where u represents a user, n represents the total number of check-ins for that user, and q represents a triple. Where l represents the point of interest ID, c represents the point of interest category ID, and t represents the check-in timestamp.

[0017] The multimodal content refers to images, comment text, and attribute metadata associated with points of interest; this invention uses a pre-trained encoder to map them into image features, comment features, and metadata features, respectively.

[0018] The hypergraph refers to a graph structure that allows a single hyperedge to connect two or more nodes. Compared to ordinary graphs, which can only express pairwise relationships between nodes, hypergraphs can characterize higher-order semantic relationships between multiple points of interest through hyperedges.

[0019] Furthermore, according to the aforementioned interest point recommendation method that integrates multimodal hypergraphs and adaptive gating, step I includes the following steps:

[0020] Step I-1: Data Preprocessing. Perform 10-core filtering on the raw check-in data, removing users with fewer than 10 check-in records and points of interest with fewer than 10 check-in records. Collect multimodal content such as images, user comments, and attribute metadata for the points of interest in the dataset, ensuring that each point of interest has content information in all three modalities. Segment each user's check-in sequence at 24-hour intervals to obtain the user's check-in trajectory, and filter out trajectories containing only one check-in after segmentation.

[0021] Step I-2: Construct a user interest point check-in trajectory graph. Divide the preprocessed check-in data into a training set and a test set in an 8:2 ratio according to time sequence. Use interest points as nodes and the transition between two adjacent check-ins within the same trajectory as directed edges. The weight of the edge is the frequency of the transition in the training set, thereby constructing a user interest point check-in trajectory graph to express the transition relationships between interest points.

[0022] Step I-3: Construct a geographic grid map. Geohash encoding is performed on the latitude and longitude of the points of interest to obtain a geographic grid. Using the geographic grid as nodes and the grid transitions corresponding to adjacent check-ins as directed edges, a geographic grid map is constructed to represent the geographic proximity relationships of the points of interest.

[0023] Step I-4: Multimodal Feature Extraction. For each point of interest, pre-trained visual encoders and text encoders are used to extract image features, comment features, and metadata features from its image, comment text, and attribute metadata, respectively, as the original feature matrix for each modality.

[0024] Furthermore, according to the aforementioned interest point recommendation method that integrates multimodal hypergraphs and adaptive gating, step II includes the following steps:

[0025] Step II-1: Calculate intra-modal similarity. For each modality, calculate the cosine similarity between the feature vectors of any two interest points. :

[0026]

[0027] in, This represents the feature vector of the i-th interest point in the m-th modality; Represents the dot product of vectors; The L2 norm of a vector; This represents the cosine similarity between interest point i and interest point j in the m-th mode. The larger the value, the more similar the two interest points are in that mode.

[0028] Step II-2: Construct the modality association matrix. For each point of interest, retain only its k nearest neighbors with the highest similarity and binarize them to obtain the modality association matrix. The elements of the matrix... :

[0029]

[0030] in, Let represent the nearest neighbor set of the k interest points with the highest cosine similarity to interest point i in the m-th mode; This represents the correlation matrix of the m-th mode, with its elements taking values ​​of 0 or 1, used to characterize the nearest neighbor connections between points of interest.

[0031] Step II-3: Construct the hypergraph adjacency matrix. Construct the hypergraph adjacency matrix using the product of the incidence matrix and its transpose. Then, self-loops are removed and symmetric normalization is performed to obtain the symmetric normalized hypergraph adjacency matrix. :

[0032]

[0033] in, Indicates matrix transpose; Let represent the hypergraph adjacency matrix of the m-th modality. The element in the i-th row and j-th column is equal to the number of neighbors that interest point i and interest point j are connected to, i.e. the number of shared hyperedges. The larger the value, the stronger the higher-order semantic association between the two.

[0034]

[0035] in, Indicates by Subtracting this term from the diagonal matrix formed by the main diagonal elements removes the self-loop; Denotes the degree matrix, its diagonal elements ; Represents the degree matrix raised to the power of -1 / 2; Let represent the hypergraph adjacency matrix after symmetric normalization.

[0036] Step II-4: Hypergraph Convolution. Multiple hypergraph convolutions are performed on the modality features using the normalized hypergraph adjacency matrix to obtain the higher-order semantic embeddings of the modality. :

[0037]

[0038] in, This represents the node representation of the m-th mode in the l-th layer of the hypergraph convolution, initially... = This is the original feature matrix of this mode; This represents the learnable weight matrix of the l-th layer; This represents the LeakyReLU nonlinear activation function; after propagation through L layers, it outputs the higher-order semantic embedding of this mode. = .

[0039] Furthermore, according to the interest point recommendation method that integrates multimodal hypergraphs and adaptive gating, step III includes the following steps:

[0040] Step III-1: On the user interest point check-in trajectory map constructed in Step I-2, interest point graph embeddings are learned through multi-layer graph convolution; then, L2 normalization is performed on each embedding to obtain the normalized hypergraph embeddings for each modality. :

[0041]

[0042] in, Let m be the embedding, which includes hypergraph embeddings of three modalities: image, comment, and metadata, as well as graph embeddings of interest points learned through graph convolution on the user's interest point check-in trajectory graph. This represents its L2 normalization result.

[0043] Step III-2: The gating unit calculates adaptive weights for each embedding path. :

[0044]

[0045] in, , This represents the learnable weights and biases of the m-th gating unit; Represents the Sigmoid function; The model represents the gating weights adaptively learned by the m-th embedding, reflecting the contribution of the information from that path to the current fusion.

[0046] Step III-3: Weighted summation of each embedding according to the gating weights to obtain the multimodal fusion embedding. :

[0047]

[0048] in, This indicates element-wise multiplication; This represents summing over all embeddings; This represents the multimodal fusion embedding obtained after adaptive weighted fusion. Since the gating weights change dynamically with the input, this step achieves dynamic fusion of the contributions of each information stream, avoiding information distortion caused by fixed-weight fusion.

[0049] Furthermore, according to the aforementioned interest point recommendation method that integrates multimodal hypergraphs and adaptive gating, step IV includes the following steps:

[0050] Step IV-1: Embedding Representation. Learn user embeddings and interest point embeddings; considering that timestamps are continuous values ​​and difficult to embed directly, Time2Vec is used to map time scalars to time embeddings. ; and learn geographic grid embeddings on the geographic grid map through graph convolution.

[0051]

[0052] in, Indicates the input time scalar; , represents the learnable frequency and phase parameters in the i-th dimension; represents the time embedding dimension; the 0th dimension is a linear term to preserve the absolute trend of time, and the remaining dimensions are sinusoidal periodic terms to characterize the periodicity of time; This represents the time embedding vector obtained by mapping a time scalar using Time2Vec.

[0053] Step IV-2: Sequence Representation Concatenation. The user, point of interest, time, and geographic embeddings are concatenated with the multimodal fusion embedding obtained in Step III to obtain the input representation of a single check-in in the check-in sequence. :

[0054]

[0055] in, , , , These represent the embedding vectors for user, point of interest, time, and geographic grid, respectively. [ ; ] indicates multimodal fusion embedding; [ ; ] indicates vector concatenation operation; This represents the input representation of a single sign-in in the sign-in sequence.

[0056] Step IV-3: Sequence Encoding. The input representation of the check-in sequence is fed into the Transformer encoder, which captures the dependencies of user movement preferences through multi-head self-attention. :

[0057]

[0058] in, , , These represent the query, key, and value matrices obtained by linear mapping from sequence representations, respectively. Indicates the dimension of the key vector; is the scaling factor; softmax is the normalization function; express The transpose of .

[0059] Step IV-4: Multi-task prediction. Three decoders are set up above the encoder output to predict the next point of interest, the category of the point of interest, and the geographic grid, respectively. Cross-entropy loss is used for each task. :

[0060]

[0061] Where k represents the three prediction tasks: point of interest, category, and geography; M represents the number of samples; s iterates through each prediction location in the sequence; and c iterates through the candidate categories for the corresponding task. The indicator variable is the actual label (1 for a hit, 0 otherwise); The probability predicted by the model; This represents the cross-entropy loss of the k-th task.

[0062] Step IV-5: Joint Training. An adaptive balancing of the losses from each task is performed using a multi-task uncertainty weighting method to form the final joint training loss. The model is trained by minimizing the joint loss through backpropagation.

[0063]

[0064] in, This represents the loss of the k-th task; This represents the log-variance weighting parameter that can be learned by the task; To improve the task accuracy learned by the model's adaptive learning, the contributions of each task are dynamically balanced based on their own uncertainties; additional terms This is a regularization term to prevent the weight from increasing indefinitely. This represents the total weighted loss across multiple tasks, which is the final joint training loss of the model.

[0065] Step IV-6: Recommendation Output. The top-k points of interest with the highest predicted probabilities are output as recommendations to the user.

[0066] In summary, the technical solutions conceived in this invention offer the following advantages over existing technologies: By constructing a hypergraph for each modality and performing hypergraph convolution, this invention effectively characterizes the high-order semantic relationships behind the multimodal features of points of interest, overcoming the limitation of ordinary graph models that can only express pairwise relationships. This allows for more comprehensive modeling of point-of-interest relationships in sparse and cold-start scenarios. Furthermore, this invention first performs L2 normalization on each preference embedding to eliminate differences in the dimensions of features across different modalities. Then, it dynamically assigns weights to each embedding through a gating mechanism, adaptively fusing multimodal content preferences such as images, comments, and metadata with collaborative transfer preferences derived from the check-in trajectory graph. In the sequence modeling stage, it further combines time and geographic embeddings to jointly characterize multi-factor-driven user preferences, avoiding information loss, confusion, and distortion caused by simple splicing and fixed-weight fusion, thus achieving dynamic adaptive fusion of multi-source information. In addition, by providing a multimodal hypergraph structure and dynamic fusion mechanism, this invention can improve the applicability and robustness of the model in complex scenarios while ensuring recommendation accuracy. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0068] Figure 2 This is a schematic diagram of the multimodal hypergraph construction and hypergraph convolution of the present invention;

[0069] Figure 3 This is a schematic diagram of the adaptive gated weighted fusion of the present invention;

[0070] Figure 4 This is a schematic diagram of the overall model architecture of the present invention;

[0071] Figure 5 This is a bar chart comparing the recommended performance of the method of this invention with that of the comparative method. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The specific embodiments described herein are merely illustrative and not intended to limit the invention. Figure 1 and Figure 4 As shown, this embodiment uses the public dataset Foursquare NYC as an example to give a specific implementation process, but the scope of protection of this invention is not limited to this implementation process.

[0073] Step 1: Data preprocessing and feature and graph construction.

[0074] This implementation first performs a 10-core filter on the original check-in data to remove users and points of interest with fewer than 10 check-in records. For each point of interest in the dataset, multimodal content such as images, user comments, and attribute metadata is collected to ensure that each point of interest has content information in all three modalities. Then, the check-in sequence of each user is segmented at 24-hour intervals to obtain check-in trajectories, and trajectories containing only one check-in are filtered out. The preprocessed data is divided into training and testing sets in an 8:2 ratio according to time sequence. A user point of interest check-in trajectory map is constructed using points of interest as nodes and adjacent check-in transitions within the same trajectory as directed edges, with the edge weight being the frequency of the transition in the training set. Geohash encoding of latitude and longitude is performed with a precision parameter of 6, and a geographic grid map is constructed. For each point of interest, a pre-trained encoder is used to extract image features, comment features, and metadata features, where the text feature dimension of comments and metadata is 768.

[0075] Step 2: Multimodal hypergraph construction and hypergraph convolution.

[0076] like Figure 2As shown, this implementation constructs hypergraphs and performs hypergraph convolution on the three modalities of image, comment, and metadata respectively. First, for each modality, the cosine similarity between the feature vectors of any two interest points is calculated. :

[0077]

[0078] in, This represents the feature vector of the i-th interest point in the m-th modality; Represents the dot product of vectors; The L2 norm of a vector; The larger the value, the more similar interest point i and interest point j are in that mode.

[0079] Subsequently, for each point of interest, only the k nearest neighbors with the highest cosine similarity to that point are retained and binarized. In this implementation, k is set to 5, thus obtaining the elements of the modal correlation matrix. :

[0080]

[0081] in, Let represent the nearest neighbor set of the k interest points with the highest cosine similarity to interest point i in the m-th mode; Let m represent the correlation matrix of the m-th mode.

[0082] Next, the hypergraph adjacency matrix of this mode is constructed by multiplying the incidence matrix by its transpose. :

[0083]

[0084] in, Indicates matrix transpose; The element in the i-th row and j-th column is equal to the number of neighbors that interest point i and interest point j are connected to, that is, the number of hyperedges they share. The larger the value, the stronger the higher-order semantic association.

[0085] Next, remove self-loops from the hypergraph adjacency matrix and perform symmetric normalization to obtain the symmetric normalized hypergraph adjacency matrix. :

[0086]

[0087] in, Indicates by Subtracting this term from the diagonal matrix formed by the main diagonal elements removes the self-loop; Denotes the degree matrix, its diagonal elements ; This represents the degree matrix raised to the power of -1 / 2.

[0088] Finally, the modality feature is subjected to multi-layer hypergraph convolution using the normalized hypergraph adjacency matrix. In this embodiment, the dimensions of the hidden layers in the hypergraph convolution are 32 and 64 respectively, and the output modality embedding dimension is 128, with a total of 3 layers of hypergraph convolution and a dropout rate of 0.3. The node representation of the (l+1)th layer... :

[0089]

[0090] Wherein, initial input = This is the original feature matrix of this mode; This represents the learnable weight matrix of the l-th layer; This represents the LeakyReLU nonlinear activation function. After three layers of propagation, the higher-order semantic embedding of this modality is output. = .

[0091] Step 3: Adaptive gated weighted fusion.

[0092] like Figure 3 As shown, firstly, a 128-dimensional interest point graph embedding is learned using graph convolution on the user interest point check-in trajectory graph constructed in step I; then, L2 normalization is performed on the hypergraph embeddings of the three modalities of image, comment, and metadata, as well as the interest point graph embedding, to obtain the normalized m-th embedding. :

[0093]

[0094] in, This represents the m-th embedding before normalization; This represents the L2 norm of a vector.

[0095] Subsequently, the gating unit calculates adaptive gating weights for each embedding path. :

[0096]

[0097] in, , This represents the learnable weights and biases of the m-th gating unit; This represents the Sigmoid function.

[0098] Finally, the embeddings of each path are weighted and summed according to the gating weights to obtain a 128-dimensional multimodal fusion embedding. :

[0099]

[0100] in, This indicates element-wise multiplication; This represents the summation of the embeddings from each path. Since the gating weights change dynamically with the input, this step achieves dynamic fusion of the contributions from each path.

[0101] Step 4: Preference sequence modeling and multi-task joint training.

[0102] like Figure 4 As shown, this implementation learns user embedding and interest point embedding, both with a dimension of 128. Considering that timestamps are continuous values ​​and difficult to embed directly, Time2Vec is used to embed the time scalar. Mapped to 128-dimensional temporal embedding :

[0103]

[0104] in, , This represents the learnable frequency and phase parameters in the i-th dimension; This indicates the time embedding dimension.

[0105] Simultaneously, a 128-dimensional geographic grid embedding is learned on the geographic grid map using graph convolution.

[0106] The user, point of interest, time, and geographic embeddings are concatenated with the multimodal fusion embedding obtained in step III to obtain the input representation of a single check-in in the check-in sequence. :

[0107]

[0108] in, , , , These represent the embedding vectors for user, point of interest, time, and geographic grid, respectively. [ ; ] indicates multimodal fusion embedding; [ ; ] indicates vector concatenation operation.

[0109] The input representation of the check-in sequence is fed into the Transformer encoder. In this implementation, the encoder has 2 layers, 2 attention heads, a feedforward hidden dimension of 1024, and a dropout rate of 0.1. The encoder captures the dependencies of user movement preferences through multi-head self-attention. :

[0110]

[0111] in, , , These represent the query, key, and value matrices obtained by linear mapping from sequence representations, respectively. Indicates the dimension of the key vector; is the scaling factor; softmax is the normalization function; express The transpose of .

[0112] Three decoders are set above the encoder output to predict the next point of interest, the category of the point of interest, and the geographic grid, respectively. Cross-entropy loss is used for each task. :

[0113]

[0114] Where k represents the three prediction tasks: point of interest, category, and geography; M represents the number of samples; s iterates through each prediction location in the sequence; and c iterates through the candidate categories for the corresponding task. This is an indicator variable for the actual label; it takes the value 1 if the label is hit, and 0 otherwise. This represents the probability predicted by the model.

[0115] The loss of each task is adaptively balanced by weighting the multi-task uncertainty to obtain the final joint training loss. :

[0116]

[0117] in, This represents the log-variance weight parameter that can be learned for the k-th task, initialized to 0.5; To improve the task accuracy learned by the model's adaptive learning, the contributions of each task are dynamically balanced based on their own uncertainties; additional terms This is a regularization term to prevent the weight from increasing indefinitely.

[0118] Training uses the Adam optimizer with a learning rate of The batch size is 64, the default number of training rounds is 200, and an early stopping mechanism is adopted, which stops training if the number of recalls does not increase after five recalls. The model parameters are learned by minimizing the joint training loss through backpropagation.

[0119] Finally, the top few interest points with the highest predicted probabilities are selected as the recommendation results and output to the user.

[0120] Effect verification. For example... Figure 5As shown, on the Foursquare NYC dataset, the Recall@5, Recall@20, and NDCG@20 of the method of this invention are 0.5418, 0.7260, and 0.4468, respectively; compared with the above three indicators of Method 1, they are 0.4797, 0.6397, and 0.3843, respectively; and compared with the above three indicators of Method 2, they are 0.5141, 0.7100, and 0.4235, respectively. The three indicators of the method of this invention are all higher than those of the two comparative methods, indicating that the present invention can obtain better recommendation results in scenarios with sparse data and cold start.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. An interest point recommendation method based on high-order semantic modeling of multimodal hypergraphs, characterized in that, The method includes: Step 1: Data preprocessing and multimodal feature extraction, and construction of user interest point check-in trajectory map and geographic grid map; Step II: Construct hypergraphs based on the features of each modality and perform hypergraph convolution to learn the high-order semantic embeddings of each modality; Step III: Dynamically weight and fuse the preference embeddings of each path through an adaptive gating mechanism to obtain a multimodal fused embedding; Step IV: Integrate user, interest point, time, geography and multimodal embeddings to perform sequence modeling, and output interest point recommendation results through multi-task uncertainty weighted joint training; The dataset refers to a collection of user check-in data and interest point multimodal content from a location-based social network. It includes the user ID, interest point ID, interest point category ID, longitude and latitude coordinates of the location of the interest point, timestamp of the user's check-in at the interest point, and multimodal content of each interest point's image, user comment text, and attribute metadata. The check-in refers to the user's behavior of accessing a point of interest at a certain time; The user's check-in sequence refers to the sequence obtained by sorting the user's historical check-ins in chronological order. ={ , ,…, }, where u represents a user, n represents the total number of check-ins for that user, and q represents a triple. Where l represents the point of interest ID, c represents the point of interest category ID, and t represents the check-in timestamp; The multimodal content refers to images, comment text, and attribute metadata associated with points of interest; this invention uses a pre-trained encoder to map them into image features, comment features, and metadata features, respectively. A hypergraph is a graph structure that allows a single hyperedge to connect two or more nodes.

2. The interest point recommendation method based on multimodal hypergraph high-order semantic modeling according to claim 1, characterized in that, Step I includes the following steps: Step I-1: Data preprocessing; Perform 10-core filtering on the raw check-in data, i.e., remove users with fewer than 10 check-in records and interest points with fewer than 10 check-in records; collect multimodal content such as images, user comments, and attribute metadata for the interest points in the dataset to ensure that each interest point has content information in three modalities; segment the check-in sequence of each user at 24-hour intervals to obtain the user check-in trajectory, and filter the trajectory that contains only 1 check-in after segmentation; Step I-2: Construct a user interest point check-in trajectory map; divide the preprocessed check-in data into training and test sets in an 8:2 ratio according to time sequence; Using points of interest as nodes and the transitions between two adjacent check-ins within the same trajectory as directed edges, with the weight of the edge being the frequency of that transition in the training set, a user point of interest check-in trajectory graph is constructed to express the transition relationships between points of interest. Step I-3: Construct a geographic grid map; The latitude and longitude of the points of interest are geohash encoded to obtain a geographic grid. A geographic grid graph is constructed with the geographic grid as nodes and the grid transitions corresponding to adjacent check-ins as directed edges to express the geographic proximity associations of the points of interest. Step I-4: Multimodal feature extraction; For each point of interest, pre-trained visual encoders and text encoders are used to extract image features, comment features, and metadata features from its image, comment text, and attribute metadata, respectively, as the original feature matrix for each modality.

3. The interest point recommendation method based on multimodal hypergraph high-order semantic modeling according to claim 1, characterized in that, Step II includes the following steps: Step II-1: Calculate intra-modal similarity; for each modality, calculate the cosine similarity between the feature vectors of any two interest points. : ； in, This represents the feature vector of the i-th interest point in the m-th modality; Represents the dot product of vectors; The L2 norm of a vector; This represents the cosine similarity between interest point i and interest point j in the m-th mode. The larger the value, the more similar the two interest points are in that mode. Step II-2: Construct the modality association matrix; for each point of interest, retain only its k nearest neighbors with the highest similarity and binarize them to obtain the modality association matrix. The elements of the matrix... : ； in, Let represent the nearest neighbor set of the k interest points with the highest cosine similarity to interest point i in the m-th mode; The correlation matrix represents the m-th mode, with its elements taking values ​​of 0 or 1, used to characterize the nearest neighbor connections between points of interest; Step II-3: Construct the hypergraph adjacency matrix; construct the hypergraph adjacency matrix by multiplying the incidence matrix by its transpose. Then, self-loops are removed and symmetric normalization is performed to obtain the symmetric normalized hypergraph adjacency matrix. : ； in, Indicates matrix transpose; Let represent the hypergraph adjacency matrix of the m-th modality. The element in the i-th row and j-th column is equal to the number of neighbors that interest point i and interest point j are connected to, i.e. the number of shared hyperedges. The larger the value, the stronger the higher-order semantic association between the two. ； in, Indicates by Subtracting this term from the diagonal matrix formed by the main diagonal elements removes the self-loop; Denotes the degree matrix, its diagonal elements ; Represents the degree matrix raised to the power of -1 / 2; Represents the symmetric normalized hypergraph adjacency matrix; Step II-4: Hypergraph Convolution; Perform multi-layer hypergraph convolution on the modality features using the normalized hypergraph adjacency matrix to obtain the higher-order semantic embedding of the modality. : ； in, This represents the node representation of the m-th mode in the l-th layer of the hypergraph convolution, initially... = This is the original feature matrix of the mode; This represents the learnable weight matrix of the l-th layer; This represents the LeakyReLU nonlinear activation function; after propagation through L layers, it outputs the higher-order semantic embedding of this mode. = .

4. The interest point recommendation method based on multimodal hypergraph high-order semantic modeling according to claim 1, characterized in that, Step III includes the following steps: Step III-1: On the user interest point check-in trajectory map constructed in Step I-2, interest point graph embeddings are learned through multi-layer graph convolution; then, L2 normalization is performed on each embedding to obtain the normalized hypergraph embeddings for each modality. : ； in, Let m be the embedding, which includes hypergraph embeddings of three modalities: image, comment, and metadata, as well as graph embeddings of interest points learned through graph convolution on the user's interest point check-in trajectory graph. This indicates its L2 normalization result; Step III-2: The gating unit calculates adaptive weights for each embedding path. : ； in, , This represents the learnable weights and biases of the m-th gating unit; Represents the Sigmoid function; The model represents the gate weights adaptively learned by the m-th embedding, reflecting the degree of contribution of the information from that path to the current fusion. Step III-3: Weighted summation of each embedding according to the gating weights to obtain the multimodal fusion embedding. : ； in, This indicates element-wise multiplication; This represents summing over all embeddings; This represents the multimodal fusion embedding obtained after adaptive weighted fusion.

5. The interest point recommendation method based on multimodal hypergraph high-order semantic modeling according to claim 1, characterized in that, Step IV includes the following steps: Step IV-1: Embedding Representation; Learning User Embeddings and Interest Embeddings; Considering that timestamps are continuous values ​​and difficult to embed directly, Time2Vec is used to map time scalars to time embeddings. ; and learn geographic grid embeddings on the geographic grid map through graph convolution; ； in, Indicates the input time scalar; , represents the i-th learnable frequency and phase parameters; represents the time embedding dimension; the 0th dimension is a linear term to preserve the absolute trend of time, and the remaining dimensions are sinusoidal periodic terms to characterize the periodicity of time; This represents the time embedding vector obtained by mapping a time scalar using Time2Vec; Step IV-2: Sequence Representation Concatenation; Concatenate the user, point of interest, time, and geographic embeddings with the multimodal fusion embedding obtained in Step III to obtain the input representation of a single check-in in the check-in sequence. : ； in, , , , These represent the embedding vectors for user, point of interest, time, and geographic grid, respectively. [ ; ] indicates multimodal fusion embedding; [ ; ] indicates vector concatenation operation; The input representation of a single sign-in in the sign-in sequence; Step IV-3: Sequence Encoding; The input representation of the check-in sequence is fed into the Transformer encoder, which captures the dependencies of user movement preferences through multi-head self-attention. : ； in, , , These represent the query, key, and value matrices obtained by linear mapping from sequence representations, respectively. Indicates the dimension of the key vector; is the scaling factor; softmax is the normalization function; express Transpose of; Step IV-4: Multi-task prediction; Three decoders are set up above the encoder output to predict the next point of interest, the category of the point of interest, and the geographic grid, respectively. Cross-entropy loss is used for each task. : ； Where k represents the three prediction tasks: point of interest, category, and geography; M represents the number of samples; s iterates through each prediction location in the sequence; and c iterates through the candidate categories for the corresponding task. Indicator variable for the actual label; The probability predicted by the model; This represents the cross-entropy loss of the k-th task; Step IV-5: Joint Training; Adaptively balance the losses of each task using multi-task uncertainty weighting to form the final joint training loss. Train the model by minimizing the joint loss through backpropagation; ； in, This represents the loss of the k-th task; This represents the log-variance weighting parameter that can be learned by the task; To improve the task accuracy learned by the model's adaptive learning, the contributions of each task are dynamically balanced based on their own uncertainties; additional terms This is a regularization term to prevent the weight from increasing indefinitely. This represents the total weighted loss across multiple tasks, which is the final joint training loss of the model. Step IV-6: Recommendation Output; The top-k points of interest with the highest predicted probabilities are output to the user as recommendation results.

Images