A POI recommendation method based on multimodal temporal information fusion
By using a multimodal temporal information fusion method, a temporal information model TIHN and a long-short-term preference model LSTHN are constructed. Combined with a spatiotemporal gate embedding GRU model, the shortcomings of existing POI recommendation algorithms in terms of the impact of noisy data and the capture of time scale are solved, and more accurate and timely POI recommendations are achieved.
Patent Information
- Application Number
- CN202410989997.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Existing POI recommendation algorithms are easily affected by noisy data when acquiring users' current preferences, and cannot effectively capture users' behavioral characteristics at different time scales, resulting in insufficient recommendation accuracy and timeliness.
A multimodal temporal information fusion approach is adopted, which constructs a temporal information model TIHN and a long-short-term preference model LSTHN, and combines them with a spatiotemporal gate-embedded GRU model to fuse user behavioral features at different time scales. By using a hypergraph neural network to capture complex relationships, the accuracy and timeliness of recommendations are improved.
It effectively captures users' behavioral patterns at different times, improves the accuracy and timeliness of POI recommendations, enhances the robustness of the recommendation system, and can promptly capture changes in user interests to provide users with more accurate POI recommendations.
Smart Images

Figure CN118779531B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer intelligence technology, specifically a POI recommendation method based on multimodal temporal information fusion. Background Technology
[0002] In modern society, with the acceleration of urbanization and the widespread adoption of mobile internet, location-based services (LBS) applications have become increasingly important. Research on Point of Interest (POI) recommendation aims to predict a user's next destination by analyzing their historical behavioral data and geographic location information. This research not only has profound academic value but also broad practical significance. From a technical perspective, POI recommendation research has driven the development of fields such as data mining, machine learning, and artificial intelligence. How to extract useful information from massive amounts of data and how to improve the accuracy and real-time performance of recommendation algorithms are all issues worthy of in-depth exploration and urgent solutions. From a research significance perspective, firstly, accurate POI recommendations can significantly improve users' convenience, helping them discover interesting nearby locations such as restaurants, attractions, and shops. Users can obtain location information that matches their interests and needs without spending a lot of time searching, thus enhancing the user experience, which is especially important in today's fast-paced life. Secondly, for businesses, POI recommendation systems can provide precise marketing tools. By analyzing users' interests and behavioral patterns, businesses can push relevant information to potential customers at appropriate times, attracting more foot traffic and improving sales performance. Meanwhile, this precision marketing can also help businesses better manage and optimize resource allocation, reduce blind investment, and improve operational efficiency. In short, researching and developing the next POI recommendation technology can not only improve users' quality of life but also promote the intelligent development of society, possessing significant practical importance and broad application prospects.
[0003] Currently, in order to recommend POI locations of interest to users from massive amounts of data, many researchers are deeply mining the contextual information in users' check-in sequences to obtain users' location preferences.
[0004] However, current POI recommendation algorithms still have some limitations: On the one hand, when obtaining a user's current preferences, considering the entire historical check-in sequence, including future check-in information, is easily affected by noisy data, thus reducing the accuracy of recommendations. On the other hand, current methods generally rely on fixed time windows for long-term and short-term preference modeling, which may fail to capture users' behavioral characteristics at different time scales. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention aims to propose a POI recommendation method based on multimodal temporal information fusion, specifically including the following steps:
[0006] Step 1: Obtain the dense vector of Points of Interest (POIs) and the encoded vector of time intervals. and the encoding vector of spatial interval The points of interest are locations;
[0007] Step 2: Construct the temporal information model TIHN, specifically, construct the temporal correlation matrix H in the temporal hypergraph neural network for temporal information. t and hyperedge weight matrix W t The time information includes POIs accessed by the user within a preset time range, based on a time hypergraph neural network and the time association matrix H in the time hypergraph neural network. t and hyperedge weight matrix W t Obtain the time-related characteristics X of user visits to POIs. t X t Characterizes users' time preferences;
[0008] Step 3: Construct a Long Short-Term Preference (LSTHN) model. Specifically, based on the user's check-in sequence information, construct multiple LSTHN-based hypergraph neural networks. The check-in sequence information includes at least the order of check-in records in the check-in sequence. Use these multiple LSTHN-based hypergraph neural networks to obtain the user's POI visit features X based on long short-term preferences. a X a Characterizes users' short-term and long-term preferences;
[0009] Step 4: Construct the spacetime gate embedding GRU model ST-GRU
[0010] By fusing the features output by the TIHN and LSTHN models, the POI embedding features are obtained, specifically expressed by the following formula:
[0011]
[0012] Where, ω e X is a hyperparameter. a X represents the feature output of the long-short-term preference model LSTHN. t The features output by the aggregated temporal information model TIHN; when ω e When = 1, This represents the impact of short-term and long-term preferences on user access to POIs; when ω e When = 0, This indicates the impact of time information on user access to POIs;
[0013] By combining the time interval coding vector and the spatial interval coding vector, and the T-shift gating for the time interval coding vector and the S-shift gating for the spatial interval coding vector, the formulas for T-shift gating and S-shift gating are obtained, specifically expressed by the following formulas:
[0014]
[0015] Where, ω t It is the weight of the time interval encoding vector, ω s It is the weight of the spatially spaced encoding vector, ω tx ω represents the weights of the POI embedding features in the T-shift gate. sx b is the weight of the POI embedding features in the S-shift gate. t This indicates the bias introduced by T-shift gating, b s Ts represents the bias introduced by S-shift gating, σ represents the sigmoid activation function, and Ts represents the bias introduced by S-shift gating. i This is the output of the T-shift gate, representing the impact of the time interval encoding vector on the POI embedding features, Ss i This is the output of the T-shift gate, representing the influence of the spatially spaced encoding vector on the POI embedding features. express The l-th POI embedding feature in;
[0016] The output of S-shift gating is compared with the output of T-shift gating. Each POI embedding feature in the matrix is multiplied by a dot product, specifically represented by the following formula:
[0017]
[0018] Where, ω st This represents the weight of the embedded features in the ST-shift gate, which is a gate obtained by integrating the S-shift and T-shift gates. st This indicates the bias introduced by ST-shift gating, and ⊙ indicates element-wise multiplication. It is the output of ST-shift gated. This represents a feature that includes both time interval and spatial interval characteristics;
[0019] Will After the R-gated control is reset and the Z-gated control is updated, the updated features are obtained, resulting in the output of the ST-shift-GRU unit, which is expressed by the following formula:
[0020]
[0021] Where, ω R The previous hidden state h in the R-gated state i-1 The weight, ω Z The previous hidden state h in Z-gating i-1 The weight, b R The previous hidden state h in the R-gated state i-1 The bias, b Z The previous hidden state h in Z-gating i-1 The bias, ω Rx POI embedding features in R-gated The weight, ω Zx Embedded features in Z-gating The weight, b Rx POI embedding features in R-gated The bias, b Zx Embedded features in Z-gating The bias, h ′ For the candidate hidden state, ω hx POI embedding features The weight, ω h For the hidden state h i-1 The weight, b hx POI embedding features The bias, b h For the hidden state h i-1 The bias;
[0022] The check-in sequence contains multiple check-in records, each corresponding to an ST-shift-GRU unit. The hidden state h output by the ST-shift-GRU unit corresponding to the current check-in record is... i The result is passed to the ST-shift-GRU unit corresponding to the next check-in record for calculation, and the ST-shift-GRU unit corresponding to the current check-in record also outputs Y. i f Y i f For the node features output by the i-th ST-shift-GRU unit in layer f, Y i f Passed to the (i+1)th ST-shift-GRU unit in layer f, where Y i f This can be expressed by the following formula:
[0023]
[0024] Among them, ST-shift-GRU CellIt is a single ST-shift-GRU unit. This represents the hidden state of the (i-1)th ST-shift-GRU unit in layer f. Represents the spatially spaced encoding vector. Y represents the time interval encoding vector. i f-1 This represents the node characteristics output by the i-th ST-shift-GRU unit in the (f-1)-th layer;
[0025] The outputs of all ST-shift-GRU units in the last layer are concatenated, as shown by the following formula:
[0026]
[0027] Where Concat() represents a join operation, Y F This represents the output characteristics of all ST-shift-GRU units in layer F, which is the last layer. The output characteristics of the first ST-shift-GRU unit in layer F, The output characteristics of the second ST-shift-GRU unit in layer F, The output feature of the nth ST-shift-GRU unit in layer F;
[0028] For Y F After performing a nonlinear transformation, the following formula is obtained:
[0029]
[0030] Where, ω Y Let b represent the weight matrix. Y Represents the bias vector; Indicates the user's preference for accessing Points of Interest (POIs);
[0031] Step 5: Next POI Prediction and Model Optimization
[0032] Predict the POI that the user will visit next, and obtain the preference score of POIl at time t+1. Specifically, this is expressed by the following formula:
[0033]
[0034] In POIl, l is the POI number, POIl represents the POI with number l, and POIl is the POI to be predicted. This represents the user's preference for accessing POIs at time t. This represents the transpose of the matrix representing the position of POIl in the check-in sequence. This represents the user's preference score for accessing POIl at time t+1.
[0035] Optionally, step 1 specifically includes the following steps:
[0036] Step 1.1: Encode the POI ID number by converting the POI ID number into a P-dimensional dense vector;
[0037] Step 1.2: Obtain the encoding vector of the time interval and the encoding vector of the spatial interval between two adjacent check-in records in the user's check-in sequence. The check-in sequence contains multiple check-in records arranged in chronological order. Each check-in record includes at least the latitude and longitude of the POI where the user is located, and the check-in time when the user checked in at the POI.
[0038] Calculate the sign-in sequence S u Two adjacent sign-in records and time interval Specifically, it is calculated using the following formula:
[0039]
[0040] in, S represents the sign-in sequence. u The i-th sign-in record in the middle, S represents the sign-in sequence. u The (i-1)th sign-in record in the database. express The corresponding check-in time, express The corresponding check-in time;
[0041] Calculate the sign-in sequence S u Two adjacent sign-in records and Spatial interval Specifically, it is calculated using the following formula:
[0042]
[0043] Here, haversine(·) is a function used to calculate the distance between two locations. express The latitude and longitude of the corresponding check-in location, express The latitude and longitude of the corresponding check-in location;
[0044] Set the time interval threshold Δt max and spatial interval threshold Δd max The formula is as follows:
[0045]
[0046] in, This represents the time interval after the threshold update. This represents the spatial interval after threshold update;
[0047] calculate and relative time intervals between as well as and The relative spatial interval between Specifically, it is calculated using the following formula:
[0048]
[0049] in, This represents the time interval with the smallest value among multiple time intervals. This represents the spatial interval with the smallest value among multiple spatial intervals;
[0050] right and relative time intervals between as well as and The relative spatial interval between Normalization is performed using the following formula:
[0051]
[0052] in, This represents the time interval with the largest value among multiple time intervals. This represents the spatial interval with the largest value among multiple spatial intervals. This represents the normalized relative time interval. Indicates the normalized relative spatial interval;
[0053] Will and By converting it into a P-dimensional vector using equal-division encoding, we obtain... and The encoding vector is the time interval, representing the check-in sequence. With the sign-in sequence The time difference between the transfers The spatially spaced encoded vector represents the check-in sequence. With the sign-in sequence The difference in transfer space between them.
[0054] Optionally, step 2 specifically includes the following steps:
[0055] Step 2.1: Construct the temporal correlation matrix H in the hypergraph neural network for temporal information. t and hyperedge weight matrix W t The time information includes the POIs accessed by the user within a preset time range;
[0056] The time information is divided into 24 time periods by hour, and a time correlation matrix H is created. t Time correlation matrix H t The formula is:
[0057]
[0058] Where v represents a node of POI in the hypergraph based on time information, e t Represents a hyperedge in a time-based hypergraph; h t (v,e t ) is a function representing the hyperedge e t Does it have nodes v and h? t (1,1) indicates whether hyperedge 1 is connected to node 1; h t (1,2) indicates whether hyperedge 2 is connected to node 1, h t (1,24) indicates whether the hyperedge 24 is connected to node 1, h t (2,1) indicates whether hyperedge 1 connects to node 2, h t (2,2) indicates whether hyperedge 2 is connected to node 2, h t (2,24) indicates whether the hyperedge 24 connects to node 2, h t (|L|,1) indicates whether hyperedge 1 is connected to node |L|, h t (|L|,2) indicates whether hyperedge 2 is connected to node |L|, h t (|L|,24) indicates whether the superedge 24 is connected to node |L|;
[0059] When h t (v,e t When ) = 1, it indicates that the hyperedge e t Node v connected to POI; when h t (v,e t When ) = 0, it indicates that the hyperedge e t Node v has no connection to the POI;
[0060] Create a time-based hyperedge weight matrix W t The hyperedge weight matrix W for time information t This represents the number of POIs accessed by a user at each time step, and is the hyperedge weight matrix W for time information. t After normalization, the following formula is obtained:
[0061]
[0062] Where, diag represents a diagonal matrix. This indicates the number of nodes connected by the superedge e1. This indicates the number of nodes connected by the hyperedge e2. Indicates the superedge e 24 The number of connected nodes, where m represents the number of check-in records in the check-in sequence;
[0063] Step 2.2: Obtain user time preferences using a temporal hypergraph neural network
[0064] After obtaining the time correlation matrix H t and the weight of the superedge W t Then, through hypergraph convolution operations, an activation function is introduced to obtain the f-th layer hypergraph neural network, as shown in the formula:
[0065]
[0066] Where ReLU is a nonlinear activation function. Represents the time correlation matrix H t transpose, This is the feature matrix of the nodes in the f-th layer of the time hypergraph neural network. When f = 0, This represents the feature matrix of the nodes in layer 0 of the temporal hypergraph neural network, where X is a dense vector; D v,t D is a diagonal matrix representing the node degree of a time hypergraph neural network. e,t The diagonal matrix representing the hyperedge degree of the temporal hypergraph neural network. It is the weight matrix of the f-th layer of the time hypergraph neural network;
[0067] Set the diagonal matrix D for hyperedge degree. e,t The constraints are expressed by the following formula:
[0068] D e,t =max{D e,t ,ξ};
[0069] Where ξ is a constraint condition and ξ is a preset value;
[0070] Aggregation of multilayer hypergraph neural networks using aggregation functions is expressed by the following formula:
[0071]
[0072] Among them, X t X represents the features output by the aggregated temporal information model TIHN. t Characterizes users' time preferences.
[0073] Optionally, step 3 specifically includes the following steps:
[0074] Step 3.1: Construct multiple hypergraph neural networks based on the user's check-in sequence information, wherein the check-in sequence information includes at least the order of the check-in records in the check-in sequence;
[0075] Since multiple check-in records in the check-in sequence may contain the same POI, the sequence interval between POIs is calculated to obtain the sequence interval matrix A. The sequence interval is the distance between the positions of the check-in records corresponding to two different POIs in the check-in sequence. The sequence interval matrix A is specifically represented by the following formula:
[0076]
[0077] Where, l1 is the POI number, l1 corresponds to POIl1; l2 is the POI number, l2 corresponds to POIl2; |L| For the POI number, l |L| Corresponding to POIl |L| a(l1,l1) is used to calculate the minimum sequence interval between POIl1 and POIl2; a(l1,l2) is used to calculate the minimum sequence interval between POIl1 and POIl2; a(l1,l2) is used to calculate the minimum sequence interval between POIl1 and POIl2. |L| ) used to calculate POIl1 and POIl |L| The minimum sequence interval between POIl2 and POIl1; a(l2,l2) is used to calculate the minimum sequence interval between POIl2 and POIl1; a(l2,l2) is used to calculate the minimum sequence interval between POIl2 and POIl1; a(l2,l1) is used to calculate the minimum sequence interval between POIl2 and POIl1. |L| ) used to calculate POIl2 and POIl |L| The smallest sequence interval between them; a(l |L| ,l1) is used to calculate POIl |L| The minimum sequence interval between POIl1 and POIl1; a(l |L| ,l2) is used to calculate POIl |L| The minimum sequence interval between POIl2 and POIl2; a(l |L| ,l |L| ) used to calculate POIl |L| and POIl |L| The smallest sequence interval between them;
[0078] For each check-in sequence, K different sequence spans are set. These sequence spans are used to construct a sequence association matrix. For each check-in sequence with a given sequence span, a hypergraph neural network is constructed to obtain K hypergraph neural networks. The K sequence spans are as follows: The weights of each hypergraph neural network are represented as: {ω1, ω2, ..., ω K},and
[0079] For each check-in sequence with a given span, a hypergraph neural network is constructed based on the corresponding span. Specifically, for the k-th span, a k-level sequence association matrix is constructed. Specifically, this is expressed by the following formula:
[0080]
[0081] Where 1≤i≤|L|, Let i be the node i of the hypergraph corresponding to the k-th sequence span based on long-short-term preferences; 1≤j≤|L|, Let j represent the hyperedge j of the hypergraph corresponding to the k-th sequence span based on long-term and short-term preferences;
[0082] Indicates the superedge Does it have nodes? when hour, Indicates POIl i and POIl j The minimum sequence interval between them is less than the set k-th sequence span, and the user accesses POIl i I have already visited POIl. j , Indicates the superedge There are nodes on when hour, Indicates POIl i With another POIl j The minimum sequence interval between them is greater than the set k-th sequence span, or the user is accessing POIl i I haven't visited POIl before. j , Indicates the superedge There are no nodes on
[0083] Set the number of times a user accesses a POI to the hypergraph. Hyperedge weight matrix based on long-short-term preferences In this context, the hyperedge weights of the K different hypergraph neural networks are identical, and the hyperedge weight matrix represents the long-short-term preference. After normalization, the following formula is obtained:
[0084]
[0085] Where, diag represents a diagonal matrix. Indicates that the user accessed POIl i The frequency, m represents the number of check-in records in the check-in sequence;
[0086] Step 3.2: Obtain users' short-term and long-term preferences through multiple hypergraph neural networks.
[0087] After obtaining the k-level sequence correlation matrix and superedge weights based on long-term and short-term preferences Then, through hypergraph convolution operations, the ReLU activation function is introduced to obtain the l-th layer hypergraph neural network corresponding to the k-th sequence span. The f-th layer hypergraph neural network is represented by the following formula:
[0088]
[0089] in, Represents the k-level sequence incidence matrix transpose, Let be the node feature matrix of the k-th sequence span. is the node feature matrix of the f-th layer, where the node feature matrix of the 0th layer is a dense vector; The diagonal matrix representing the node degree of the hypergraph neural network with the k-th sequence span. The diagonal matrix representing the hyperedge degree of the hypergraph neural network with the span of the k-th sequence is... It is the weight matrix of the f-layer hypergraph neural network;
[0090] The multi-layer hypergraph neural network that aggregates the k-th check-in sequence using the sum-pooling aggregation function is represented by the following formula:
[0091]
[0092] in, The features output by the hypergraph neural network after aggregating the k-th sequence span;
[0093] Through hierarchical weight ω k The features output by the hypergraph neural network with K sequence spans are stacked, specifically represented by the following formula:
[0094]
[0095] Among them, X a These are the features output by the long-short-term preference model LSTHN.
[0096] Optionally, the spacetime gate-embedded GRU model ST-GRU is optimized using a cross-entropy loss function, the formula for which the cross-entropy loss function is optimized is:
[0097]
[0098] Where J represents the loss function value, u represents user u, U represents all users, S(u) is the check-in sequence, and i represents the POI number in the check-in sequence. The prediction parameters of the space-time gated embedding GRU model ST-GRU are: (i) the prediction output of the space-time gated embedding GRU model ST-GRU. denoted as user u’s preference score for POIi, where POIi represents the POI with number i, λ is the regularization coefficient, and ||θ||2 represents the L2 regularization of all weight parameters of the space-time gate embedded GRU model ST-GRU.
[0099] The beneficial effects of adopting the above technical solution are as follows:
[0100] A new POI recommendation method based on multimodal temporal information fusion offers several advantages over existing approaches: First, the MTFNR (Multimodal Temporal Information Fusion for Next POI Recommendation) model uses a hypergraph neural network to acquire POI features. This method, which aggregates node feature information through hyperedge connections, effectively captures complex relationships between data. Second, considering that user preferences may change over time, the MTFNR model integrates the long-short term preference model (LSTHN) and the time information model (TIHN) to learn changes in user interests and habits across different periods. Time information modeling captures user behavior patterns across different time periods, providing a better understanding of user behavior patterns; LSTHN dynamically captures changes in user interests, increasing the timeliness of POI recommendations. Finally, the MTFNR model incorporates user spatiotemporal information into the GRU model, effectively capturing the dependencies in check-in sequences, thus enhancing the robustness of the recommendation system and improving overall model performance. In addition, the MTFNR model can capture changes in user interests in a timely manner, providing users with more accurate POI recommendations. It can help users quickly find locations that match their current interests and needs, saving time and effort, and has certain application value. This method can be applied to recommending users' next destinations and is suitable for future prediction and recommendation based on known historical location trajectories, such as trajectory prediction of mobile devices. Attached Figure Description
[0101] Figure 1 A flowchart illustrating a POI recommendation method based on multimodal temporal information fusion in an embodiment of the present invention;
[0102] Figure 2 A bar chart of ablation experiment results in an embodiment of the present invention;
[0103] Figure 3 A bar chart of the results of a sequence spacing study in an embodiment of the present invention. Detailed Implementation
[0104] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0105] To address the shortcomings of existing technologies, this paper proposes a Multimodal Temporal Information Fusion for Next POI Recommendation (MTFNR) model. First, to capture user preferences and habits across different time periods, multiple hypergraph neural networks are constructed to analyze user behavior patterns at different times. Furthermore, to avoid introducing interference, only the check-in sequence visited in the current period is considered, reducing the impact of noise on the model's recommendation performance. Second, the next POI recommendation task is modeled by fusing temporal information and long-term / short-term preferences, thus comprehensively capturing user preferences and improving the timeliness and accuracy of recommendations. Extensive experiments on the real LBSN dataset demonstrate the superior performance of the MTFNR model. Experimental results show that the MTFNR model outperforms current methods in terms of recall. This invention provides a multimodal temporal information fusion POI recommendation method, combining... Figure 1 Specifically, this may include the following steps:
[0106] Step 1: Obtain the dense vector of Points of Interest (POIs) and the encoded vector of time intervals. and the encoding vector of spatial interval The points of interest are locations, such as shops, schools, etc.
[0107] Step 1.1: Encode the POI ID number by converting the POI ID number into a P-dimensional dense vector;
[0108] The encoding can be Embedding encoding; the POI ID number can be 15487, 24586, or 1254, and after encoding, the POI ID number can be 0, 1, or 2; the P-dimensional dense vector contains the features of the POI nodes, thus the node feature matrix is represented as X, where X∈R N*P Where N is the number of POI nodes, R represents the dimension, and X∈R N*P The dimension of the node feature matrix is N*P.
[0109] Step 1.2: Obtain the encoding vector of the time interval and the encoding vector of the spatial interval between two adjacent check-in records in the user's check-in sequence. The check-in sequence contains multiple check-in records arranged in chronological order. Each check-in record includes at least the latitude and longitude of the POI where the user is located, and the check-in time of the user at the POI. The check-in record may also include information such as the user ID and the ID of the POI.
[0110] Calculate the sign-in sequence S u Two adjacent sign-in records and time interval Specifically, it is calculated using the following formula:
[0111]
[0112] in, S represents the sign-in sequence. u The i-th sign-in record in the middle, S represents the sign-in sequence. u The (i-1)th sign-in record in the database. express The corresponding check-in time, express The corresponding check-in time;
[0113] Calculate the sign-in sequence S u Two adjacent sign-in records and Spatial interval Specifically, it is calculated using the following formula:
[0114]
[0115] Here, haversine(·) is a function used to calculate the distance between two locations. express The latitude and longitude of the corresponding check-in location, express The latitude and longitude of the corresponding check-in location;
[0116] Considering that excessively large spatiotemporal interval information may cause redundancy in the preference model, this invention sets a time interval threshold Δt. max and spatial interval threshold Δd max To control the time and space intervals within a certain range, the time interval threshold Δt max and spatial interval threshold Δd max The formula is as follows:
[0117]
[0118]
[0119] in, This represents the time interval after the threshold update. This represents the spatial interval after threshold update;
[0120] calculate and relative time intervals between as well as and The relative spatial interval between Specifically, it is calculated using the following formula:
[0121]
[0122] in, This represents the time interval with the smallest value among multiple time intervals. This represents the spatial interval with the smallest value among multiple spatial intervals;
[0123] right and relative time intervals between as well as and The relative spatial interval between Normalization is performed using the following formula:
[0124]
[0125] in, This represents the time interval with the largest value among multiple time intervals. This represents the spatial interval with the largest value among multiple spatial intervals. This represents the normalized relative time interval. Indicates the normalized relative spatial interval;
[0126] Will and By converting it into a P-dimensional vector using equal-division encoding, we obtain... and The encoding vector is the time interval, representing the check-in sequence. With the sign-in sequence The time difference between the transfers, and express The dimension is P. The spatially spaced encoded vector represents the check-in sequence. With the sign-in sequence The difference in transfer space between them express The dimension is P;
[0127] Step 2: Construct the temporal information model TIHN, specifically, construct the temporal correlation matrix H in the temporal hypergraph neural network for temporal information. t and hyperedge weight matrix W t The time information includes POIs accessed by the user within a preset time range, based on a time hypergraph neural network and the time association matrix H in the time hypergraph neural network. t and hyperedge weight matrix W tObtain the time-related characteristics X of user visits to POIs. t X t Characterizes users' time preferences;
[0128] Step 2.1: Construct the temporal correlation matrix H in the hypergraph neural network HGNN for time information. t and hyperedge weight matrix W t The time information includes the POIs accessed by the user within a preset time range;
[0129] This invention creates a time correlation matrix H t The time information was divided into 24 time periods by hour, and it was determined whether a user accessed a POI at each time point. Based on this, a time correlation matrix H was created. t Time correlation matrix H t The formula is:
[0130]
[0131] Among them, H t ∈R N*24 , representing the time correlation matrix H t The dimension is N*24, v∈V t e represents a node of POI in a time-based hypergraph. t ∈E t This represents a hyperedge in a time-based hypergraph, which can also be understood as a specific moment in time; h t (v,e t ) is a function representing the hyperedge e t Does it have a node v? This can also be understood as e. t At any given time, has the user accessed the POI corresponding to node v? t (1,1) indicates whether hyperedge 1 is connected to node 1, which can also be understood as whether the user visited node 1 at point 1; h t (1,2) indicates whether the superedge 2 is connected to node 1, which can also be understood as whether the user visited node 1 at point 2. t (1,24) indicates whether the superedge 24 is connected to node 1, which can also be understood as whether the user visited node 1 at point 24. t (2,1) indicates whether hyperedge 1 is connected to node 2, which can also be understood as whether the user visited node 2 at point 1. t (2,2) indicates whether the superedge 2 is connected to node 2, which can also be understood as whether the user visited node 2 from point 2. t (2,24) indicates whether the superedge 24 connects to node 2, which can also be understood as whether the user visited node 2 at point 24. t(|L|,1) indicates whether the superedge 1 is connected to node |L|, which can also be understood as whether the user visited node |L| at point 1. t (|L|,2) indicates whether hyperedge 2 is connected to node |L|, h t (|L|,24) indicates whether the superedge 24 is connected to node |L|, which can also be understood as whether the user visited node |L| at point 24.
[0132] When h t (v,e t When ) = 1, it indicates that the hyperedge e t A node v connected to a POI can also be understood as a user in e t The POI corresponding to node v was accessed at time h; t (v,e t When ) = 0, it indicates that the hyperedge e t A node v without a connection to a POI can also be understood as the user in e t The POI corresponding to node v was not visited at any time.
[0133] In hypergraph neural networks, the weight matrix of a hyperedge typically represents the correlation strength or weight ratio between hyperedges. When creating a temporal hypergraph neural network, the hyperedge weight matrix W, which is based on temporal information... t This represents the number of POIs visited by a user at each time step, thus allowing for a better understanding of the temporal periodicity of POI visits. Simultaneously, to help the model better capture the distribution characteristics of the data and improve its performance and generalization ability, the hyperedge weight matrix W is adjusted. t After normalization, the following formula is obtained:
[0134]
[0135] Among them, W t ∈R 24*24 , indicating W t The dimension is 24*24; diag represents a diagonal matrix. This represents the number of nodes connected by the superedge e1, which can also be understood as the number of POIs visited by user u at time e1. This represents the number of nodes connected by the superedge e2, which can also be understood as the number of POIs visited by user u at time e2. Indicates the superedge e 24 The number of connected nodes can also be understood as the number of nodes connected to user u in e. 24 The total number of POIs visited at any given time, where m represents the number of check-in records in the check-in sequence;
[0136] Step 2.2: Obtain user time preferences using a temporal hypergraph neural network
[0137] After obtaining the time correlation matrix H t and the weight of the superedge W t Then, through hypergraph convolution operations, an activation function is introduced to obtain the f-th layer hypergraph neural network, as shown in the formula:
[0138]
[0139] Where ReLU is a nonlinear activation function. Represents the time correlation matrix H t transpose, It is the feature matrix of the nodes in the f-th layer of the time hypergraph neural network. express The dimension is N*P, where N is the number of nodes and P represents the number of features for each node. When f = 0, ... This represents the feature matrix of the nodes in layer 0 of the temporal hypergraph neural network, where X is a dense vector; D v,t D is a diagonal matrix representing the node degree of a time hypergraph neural network. e,t These are diagonal matrices representing the hyperedge degrees of a temporal hypergraph neural network. They have all undergone normalization operations, which can help reduce the impact of degree differences between different nodes or hyperedges on model training. It is the weight matrix of the f-th layer of the time hypergraph neural network;
[0140] Since a user may not visit any POIs at a certain time, the hyperedge degree matrix D... e,t The presence of zeros in the matrix can render the inverse of the hyper-edge degree matrix undesirable, preventing hypergraph convolution operations. To address this, constraints are added to change the behavior of D. e,t The value of the diagonal matrix D e,t The constraints are expressed by the following formula:
[0141] D e,t =max{D e,t ,ξ};
[0142] Where ξ is a constraint condition and ξ is a preset value;
[0143] To facilitate the capture of information at different levels, multiple hypergraph aggregation layers are stacked using an aggregation function for propagation to update node embeddings, resulting in richer and more expressive feature representations. The formula for aggregating the multi-layer hypergraph neural network using the aggregation function is as follows:
[0144]
[0145] Among them, X t X represents the features output by the aggregated temporal information model TIHN. t Characterizes users' time preferences;
[0146] Step 3: Construct a Long Short-Term Preference (LSTHN) model. Specifically, based on the user's check-in sequence information, construct multiple LSTHN-based hypergraph neural networks. The check-in sequence information includes at least the order of check-in records in the check-in sequence. Use these multiple LSTHN-based hypergraph neural networks to obtain the user's POI visit features X based on long short-term preferences. a X a Characterizes users' short-term and long-term preferences;
[0147] Step 3.1: Construct multiple Hypergraph Neural Networks (HGNNs) based on the user's check-in sequence information, wherein the check-in sequence information includes at least the order of the check-in records in the check-in sequence;
[0148] Since multiple check-in records in the check-in sequence may contain the same POI, the sequence interval between POIs is calculated to obtain the sequence interval matrix A. The sequence interval is the distance between the positions of the check-in records corresponding to two different POIs in the check-in sequence. The sequence interval matrix A is specifically represented by the following formula:
[0149]
[0150] Where A∈R N*N Let A represent the dimension N*N, l1 be the POI number, l1 corresponds to POIl1; l2 be the POI number, l2 corresponds to POIl2; l |L| For the POI number, l |L| Corresponding to POIl |L| a(l1,l1) is used to calculate the minimum sequence interval between POIl1 and POIl2; a(l1,l2) is used to calculate the minimum sequence interval between POIl1 and POIl2; a(l1,l2) is used to calculate the minimum sequence interval between POIl1 and POIl2. |L| ) used to calculate POIl1 and POIl |L| The minimum sequence interval between POIl2 and POIl1; a(l2,l2) is used to calculate the minimum sequence interval between POIl2 and POIl1; a(l2,l2) is used to calculate the minimum sequence interval between POIl2 and POIl1; a(l2,l1) is used to calculate the minimum sequence interval between POIl2 and POIl1. |L| ) used to calculate POIl2 and POIl |L| The smallest sequence interval between them; a(l |L| ,l1) is used to calculate POIl |L| The minimum sequence interval between POIl1 and POIl1; a(l |L| ,l2) is used to calculate POIl |L| The minimum sequence interval between POIl2 and POIl2; a(l |L| ,l |L|) used to calculate POIl |L| and POIl |L| The smallest sequence interval between them;
[0151] In this invention, only the sequence interval of other POIs that the user has visited before visiting a certain POI is calculated. That is, if the user is visiting POI location l i I have visited POIl before. j Then it is considered POIl i and POIl j There is a sequence interval between them, which needs to be calculated. And if a user accesses POIl... i I haven't visited POIl before. j Then POIl is not considered i and POIl j The intervals between POIs. This is because only the sequence intervals between visited POIs can be correlated with user behavior and preferences. This approach also simplifies data processing and analysis, making the results clearer and more interpretable.
[0152] To more comprehensively learn users' short-term and long-term preferences, K different sequence spans are set for the check-in sequence. These sequence spans are used to construct a sequence association matrix. For each check-in sequence with a different sequence span, a hypergraph neural network is constructed to obtain K hypergraph neural networks. The K sequence spans are as follows: The weights of each hypergraph neural network are represented as: {ω1, ω2, ..., ω K},and
[0153] For each check-in sequence with a given span, a hypergraph neural network is constructed based on the corresponding span. Specifically, for the k-th span, a k-level sequence association matrix is constructed. Specifically, this is expressed by the following formula:
[0154]
[0155] in, express The dimension is N*N, 1≤i≤|L|, Let i be the node i of the hypergraph corresponding to the k-th sequence span based on long-short-term preferences; 1≤j≤|L|, Let j represent the hyperedge j of the hypergraph corresponding to the k-th sequence span based on long-term and short-term preferences;
[0156] Indicates the superedge Does it have nodes? when hour, Indicates POIl i and POIl j The minimum sequence interval between them is less than the set k-th sequence span, and the user accesses POIl i I have already visited POIl. j , Indicates the superedge There are nodes on when hour, Indicates POIl i With another POIl j The minimum sequence interval between them is greater than the set k-th sequence span, or it indicates that the user is accessing POIl. i I haven't visited POIl before. j , Indicates the superedge There are no nodes on
[0157] Set the number of times a user accesses a POI to the hypergraph. Hyperedge weight matrix based on long-short-term preferences In this context, the hyperedge weights of the K different hypergraph neural networks are identical, and the hyperedge weight matrix represents the long-short-term preference. After normalization, the following formula is obtained:
[0158]
[0159] in, express The dimension is N*N, and diag represents a diagonal matrix. Indicates that the user accessed POIl i The frequency, m represents the number of check-in records in the check-in sequence;
[0160] Step 3.2: Obtain users' short-term and long-term preferences through multiple hypergraph neural networks.
[0161] After obtaining the k-level sequence correlation matrix and superedge weights based on long-term and short-term preferences Then, through hypergraph convolution operations, the ReLU activation function is introduced to obtain the l-th layer hypergraph neural network corresponding to the k-th sequence span. The f-th layer hypergraph neural network is represented by the following formula:
[0162]
[0163] in, Represents the k-level sequence incidence matrix transpose, Let be the node feature matrix of the k-th sequence span, and express The dimension is N*P. is the node feature matrix of the f-th layer, where the node feature matrix of the 0th layer is a dense vector; The diagonal matrix representing the node degree of the hypergraph neural network with the k-th sequence span. The diagonal matrix representing the hyperedge degree of the hypergraph neural network with the span of the k-th sequence is... It is the weight matrix of the f-layer hypergraph neural network;
[0164] The multi-layer hypergraph neural network that aggregates the k-th check-in sequence using the sum-pooling aggregation function is represented by the following formula:
[0165]
[0166] in, The features output by the hypergraph neural network after aggregating the k-th sequence span;
[0167] To help capture information at different levels, hierarchical weights ω are used. k The features output by the hypergraph neural network with K sequence spans are stacked, specifically represented by the following formula:
[0168]
[0169] Among them, X a These are the features output by the long-short-term preference model LSTHN.
[0170] Step 4: Construct the spacetime gate embedding GRU model ST-GRU
[0171] To obtain enhanced time-aware user preferences, the aim is to fuse POI feature representations learned from the TIHN and LSTHN models. Therefore, after learning POI node representations from both models, the feature embeddings of the POIs are obtained by element-wise addition, and then determined according to a hyperparameter ω. e To control the influence of different aspects and thus more accurately represent the features of POIs, the features output by the TIHN model and the LSTHN model are fused to obtain the POI embedding features, which are specifically expressed by the following formula:
[0172]
[0173] Where, ω e X is a hyperparameter. a X represents the feature output of the long-short-term preference model LSTHN. t The features output by the aggregated temporal information model TIHN; ω eIt is an adjustable parameter, when ω e When = 1, This represents the impact of short-term and long-term preferences on user access to POIs; when ω e When = 0, This indicates the impact of time information on user access to POIs;
[0174] Based on experience, since users prefer to visit POIs that are close to their current location, the smaller the transition time and space, the greater the probability of recommending the next location. Therefore, a trainable weight parameter ω can be used to determine this. t and ω s The effects of time and space transitions were adjusted, and T-shift gating for time transition characteristics and S-shift gating for space transition characteristics were designed respectively. These two gating methods are used to improve the problem that nonlinear spatiotemporal information cannot be embedded in the GRU model.
[0175] Based on this, the present invention combines the time interval coding vector and the spatial interval coding vector, as well as the T-shift gating for the time interval coding vector and the S-shift gating for the spatial interval coding vector, to obtain the formulas for the T-shift gating and the S-shift gating, which are specifically expressed by the following formulas:
[0176]
[0177] Where, ω t It is the weight of the time interval encoding vector, ω t ∈R P , representing ω t The dimension is P; ω s It is the weight of the spatially spaced encoding vector, ω s ∈R P , representing ω s The dimension is P; ω tx ω represents the weights of the POI embedding features in the T-shift gate. tx ∈R P , representing ω tx The dimension is P; ω sx ω represents the weights of the POI embedding features in the S-shift gate. sx ∈R P , representing ω sx The dimension is P; b t This indicates the bias introduced by T-shift gating, b t ∈R P , indicating b t The dimension is P; b s This indicates the introduction of S-shift gating bias, b s ∈RP , indicating b s The dimension is P; σ represents the Sigmoid activation function, Ts i This is the output of the T-shift gate, representing the impact of the time interval encoding vector on the POI embedding features, Ts i ∈R P , indicating Ts i The dimension is P; Ss i This is the output of the T-shift gate, representing the influence of the spatially spaced encoding vector on the POI embedding features, Ss i ∈R P , indicating Ss i The dimension is P; express The l-th POI embedding feature in;
[0178] The output of S-shift gating is compared with the output of T-shift gating. Each POI embedding feature in the matrix is multiplied by a dot product, thereby loading the influence of the spatiotemporal transfer features onto each dimension, as expressed by the following formula:
[0179]
[0180] The ST-shift gating is a gating system obtained by integrating the S-shift and T-shift gating, ω st ω represents the weight of the embedded features in the ST-shift gate. st ∈R P , representing ω st The dimension is P; b st This indicates the bias introduced by ST-shift gating, b st ∈R P , indicating b st The dimension is P; ⊙ represents element-wise multiplication. It is the output of ST-shift gated. This represents a feature that includes both time interval and spatial interval characteristics;
[0181] Will After control reset (R-gated) and control update (Z-gated), specifically, h is calculated through the reset attribute of the R-gated control. ′ By controlling the update and forgetting through Z-gating, the updated features are obtained and passed to the hidden state of the next unit. The output of the ST-shift-GRU unit is specifically expressed by the following formula:
[0182]
[0183] Where, ω R The previous hidden state h in the R-gated state i-1 The weight, ω R ∈R P , representing ω R The dimension is P; ω Z The previous hidden state h in Z-gating i-1 The weight, ω Z ∈R P , representing ω Z The dimension is P; b R The previous hidden state h in the R-gated state i-1 The bias, ω Z ∈R P , representing ω Z The dimension is P; b Z The previous hidden state h in Z-gating i-1 The bias, b Z ∈R P , indicating b Z The dimension is P; ω Rx POI embedding features in R-gated The weight, ω Rx ∈R P , representing ω Rx The dimension is P; ω Zx Embedded features in Z-gating The weight, ω Zx ∈R P , representing ω Zx The dimension is P; b Rx POI embedding features in R-gated The bias, b Rx ∈R P , indicating b Rx The dimension is P; b Zx Embedded features in Z-gating The bias, b Zx ∈R P , indicating b Zx The dimension is P; h ′ For the candidate hidden state, ω hx POI embedding features The weight, ω hx ∈R P , representing ω hx The dimension is P; ω h For the hidden state h i-1 The weight, ω h ∈R P , representing ω h The dimension is P; b hx POI embedding features The bias, b hx ∈R P , indicating b hx The dimension is P; b h For the hidden state h i-1 The bias; b h ∈R P , indicating b h The dimension is P.
[0184] The check-in sequence contains multiple check-in records, each corresponding to an ST-shift-GRU unit. The hidden state h output by the ST-shift-GRU unit corresponding to the current check-in record is... i The result is passed to the ST-shift-GRU unit corresponding to the next check-in record for calculation, and the ST-shift-GRU unit corresponding to the current check-in record also outputs Y. i f Y i f For the node features output by the i-th ST-shift-GRU unit in layer f, Y i f Passed to the (i+1)th ST-shift-GRU unit in layer f, where Y i f This can be expressed by the following formula:
[0185]
[0186] Among them, ST-shift-GRU Cell It is a single ST-shift-GRU unit. This represents the hidden state of the (i-1)th ST-shift-GRU unit in layer f. Represents the spatially spaced encoding vector. Y represents the time interval encoding vector. i f-1 This represents the node characteristics output by the i-th ST-shift-GRU unit in the (f-1)-th layer;
[0187] In addition, to help the ST-GRU model better understand and learn complex feature representations and improve its performance and generalization ability, multiple layers of ST-shift-GRU units are connected and stacked, and the outputs of all ST-shift-GRU units in the last layer are concatenated, as shown in the following formula:
[0188]
[0189] Where Concat() represents a join operation, Y FThis represents the output characteristics of all ST-shift-GRU units in layer F, which is the last layer. The output characteristics of the first ST-shift-GRU unit in layer F, The output characteristics of the second ST-shift-GRU unit in layer F, The output feature of the nth ST-shift-GRU unit in layer F;
[0190] To improve the robustness of the model, for Y F After performing a nonlinear transformation, the following formula is obtained:
[0191]
[0192] Where, ω Y Let b represent the weight matrix. Y Represents the bias vector; Indicates the user's preference for accessing Points of Interest (POIs);
[0193] Finally, through the above steps, multiple ST-shift-GRU units are connected in series, the weight of the current node is updated and the hidden state of the upper level is passed, and the memory features of the historical nodes are recursively processed step by step, thus converging to the last node. By stacking multiple layers of ST-GRU, the effect of feature convergence and effective extraction is achieved.
[0194] Step 5: Next POI Prediction and Model Optimization
[0195] To estimate a user's preference for a Point of Interest (POI) for their next visit, a preference score for each candidate POI is calculated using the softmax function. Typically, this is done by... The last node in the algorithm is used as the estimation object because it aggregates the features of all nodes. Assuming... If the timestamp of the last node is t, then at timestamp t+1, the POI to be visited by the user at the next node is predicted, and the preference score of POIl at time t+1 is obtained. Specifically, this is expressed by the following formula:
[0196]
[0197] In POIl, l is the POI number, POIl represents the POI with number l, and POIl is the POI to be predicted. This represents the user's preference for accessing POIs at time t. This represents the transpose of the matrix representing the position of POIl in the check-in sequence. This represents the user's preference score for accessing POIl at time t+1;
[0198] After obtaining the user preference estimate, the objective function is optimized using the cross-entropy loss function. L2 regularization (also known as weight decay) is introduced to prevent overfitting and improve the model's generalization ability. The formula for optimizing the cross-entropy loss function is as follows:
[0199]
[0200] Where J represents the loss function value, u represents user u, U represents all users, S(u) is the check-in sequence, and i represents the POI number in the check-in sequence. The prediction parameters of the space-time gated embedding GRU model ST-GRU are: (i) the prediction output of the space-time gated embedding GRU model ST-GRU. denoted as user u’s preference score for POIi, where POIi represents the POI with number i, λ is the regularization coefficient, and ||θ||2 represents the L2 regularization of all weight parameters of the space-time gate embedded GRU model ST-GRU.
[0201] To verify the technical effectiveness of the method of the present invention, the following tests were conducted on this embodiment:
[0202] (1) Data acquisition and preprocessing
[0203] Experiments were conducted on three publicly available real-world datasets: Foursquare-NYC (NYC), Foursquare-TKY (TKY), and Gowalla (GOW), all collected from location-based social media platforms. The NYC and TKY datasets collected check-ins in New York and Tokyo, covering a time span of approximately 10 months (from April 12, 2012 to February 16, 2013); the GOW dataset was collected on the Gowalla platform in California and Nevada, with a longer time period and broader geographic coverage. The datasets include anonymized user IDs, POI location IDs with latitude and longitude information, POI location category IDs, and check-in timestamps.
[0204] During preprocessing, POIs and users with fewer than 5 historical access records in the NYC and TKY datasets, and POIs and users with fewer than 10 historical access records in the GOW dataset, were filtered out. Finally, all check-in records were sorted chronologically. For a user with n check-in records, the [1, n-2]th data point was selected as the input for the training set, and the (n-1)th data point was used as the label for the training set; the [2, n-1]th data point was selected as the input for the test set, and the nth data point was used as the label for the test set. The statistical results of the preprocessed datasets are summarized in Table 1.
[0205] Table 1 Description of the dataset
[0206]
[0207] (2) Model parameter settings
[0208] The model was developed using the PyTorch framework. The Adam optimizer used the default beta value, the learning rate was 2e-3, and the maximum sequence length was 100. Dropout was used in the long-short term preferences with hypergraph neural network (LSTHN), time information with hypergraph neural network (TIHN), and spatiotemporal gated GRU (ST-GRU) models. In the LSTHN and TIHN models, Dropout was set to 0.3, and in the ST-GRU model, it was set to 0.2. The sequence span in the LSTHN model was set to {25, 50, 75, 100}.
[0209] (3) Baseline Model
[0210] •MC: A common method for solving recommendation problems, which learns latent representations of users and points of interest (POIs) through matrix factorization.
[0211] •FPMC: A method that combines matrix factorization (MF) and Markov chains, and introduces the Bayesian Personalized Ranking (BPR) framework to learn model parameters for recommendation.
[0212] PRME: A personalized ranking metric embedding method that integrates ordinal information, personal preferences, and geographic influences.
[0213] STGN: An LSTM model that uses a gate mechanism to model the spatiotemporal intervals in a sign-in sequence to capture users’ short-term and long-term preferences.
[0214] • PLSPL: A model that combines users' short-term and long-term preferences through linear combination units of users.
[0215] STAN: An advanced model that learns patterns between non-adjacent locations and non-contiguous visits through a two-layer attention architecture.
[0216] GETNext: A model that enhances next POI predictions by using a user-unknown global trajectory flow graph and a novel graph augmentation converter.
[0217] (4) Experimental Results and Analysis
[0218] Experiments were conducted on three datasets to compare the performance of the proposed Multimodal Temporal Information Fusion for Next POI Recommendation (MTFNR) model with the baseline model, and the results on recall@k (k=5, 10) were obtained. The results are shown in Table 2.
[0219] Table 2 Performance comparison on the three datasets
[0220]
[0221]
[0222] Here, recall represents the recall rate, and recall@k represents the top K recall rate, which can also be understood as the recall rate of the top K items.
[0223] As can be seen from the table, the evaluation metrics of the MTFNR model are superior to the baseline model on all three datasets. In particular, the MTFNR model achieved improvements in Top-5 recall of 2.32%, 15.46%, and 63.78% on the NYC, TKY, and GOW datasets, respectively, and improvements in Top-10 recall of 16.91%, 33.06%, and 84.04%, respectively. This demonstrates the effectiveness of the MTFNR model and the accuracy of modeling through multimodal temporal information fusion. This improvement is mainly attributed to the following aspects: (1) The MTFNR model obtains the feature representation of POI through Hypergraph Neural Network (HGNN). Compared with the traditional Graph Neural Network (GNN), HGNN aggregates node information through the connection relationship of hyperedges to realize the updating and propagation of node features. This method can more effectively capture the multi-dimensional and deep relationships between data, enabling the hypergraph to more naturally represent the complex relationships in the real world. (2) The MTFNR model takes into account that users' behavioral preferences may change over time. Therefore, it creates a time information model and a long-short-term preference model. The time information model can obtain users' behavioral patterns in different time periods, which can better understand the user's behavioral patterns. The long-short-term preference model can dynamically capture changes in users' interests and increase the timeliness of POI recommendations. (3) The MTFNR model introduces users' spatiotemporal interval information on the basis of the GRU model, effectively capturing the dependency relationship of the check-in sequence, making the recommendation system more robust, thereby improving the overall performance of the model.
[0224] (5) Ablation Experimental Study
[0225] To verify the effectiveness of each component in the MTFNR model, ablation experiments were conducted on the NYC, TKY, and GOW datasets. Figure 2 As shown:
[0226] Among them, the no_TIHN and LSTHN variants remove the influence of time information modeling and user long-term and short-term preference modeling, and represent the influence of the model only through random embedding encoding; no_LSTHN: this variant removes the influence of long-term and short-term preference modeling, and only considers the influence of time information modeling; no_TIHN: this variant removes the influence of time information modeling, and only considers the influence of long-term and short-term preference modeling.
[0227] From the results, (1) the performance of the no_LSTHN variant and the no_TIHN variant is somewhat lower than that of the MTFNR model. This indicates that the modeling of time information and the modeling of users' long-term and short-term preferences have an impact on the model performance and need to be fully modeled. In addition, in the TKY dataset, the performance of the no_LSTHN variant is better than that of the no_TIHN variant; while in the NYC and GOW datasets, the performance of the no_TIHN variant is better than that of the no_LSTHN variant, indicating that the importance of each component is different in different datasets. (2) The recommendation performance of the no_TIHN and LSTHN variants is significantly lower than that of other variants. This indicates that the user's historical check-in sequence contains rich contextual information data. Modeling time information by check-in timestamps and modeling users' long-term and short-term preferences by sequence intervals can improve the recommendation effect of the model.
[0228] (6) Sequence spacing study
[0229] To demonstrate the effectiveness of local sequence interval modeling of users' long-term and short-term preferences, we modeled users' long-term and short-term preferences using both local and global sequence intervals, as shown in the results. Figure 3 As shown.
[0230] Local sequence interval refers to calculating the sequence interval of a specific point of interest (POI) by considering only check-in information preceding that POI, ignoring check-in information following it. Global sequence interval, on the other hand, considers all check-in information. The data in the figure shows that, across the three datasets, modeling user long-short-term preferences using local sequence interval performs better than using global sequence interval. This is because local sequence interval only considers a portion of the check-in information, resulting in a denser sequence interval containing more useful information about user behavior. In contrast, global sequence interval needs to consider more check-in information, which may lead to information sparsity and affect recommendation accuracy. Furthermore, global sequence interval, including all check-in information, may be more susceptible to noise and outliers, thus affecting the accuracy of recommendation results; local sequence interval, considering only the most recent check-in information, better reflects the user's current interests and preferences, reduces the impact of noise, and improves recommendation accuracy.
[0231] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A POI recommendation method based on multimodal temporal information fusion, characterized in that, include: Step 1: Obtain the dense vector of Points of Interest (POIs) and the encoded vector of time intervals. and the encoding vector of spatial interval The points of interest are locations; Step 2: Construct the temporal information model TIHN, specifically, construct the temporal correlation matrix H in the temporal hypergraph neural network for temporal information. t and hyperedge weight matrix W t The time information includes POIs accessed by the user within a preset time range, based on a time hypergraph neural network and the time association matrix H in the time hypergraph neural network. t and hyperedge weight matrix W t Obtain the time-related characteristics X of user visits to POIs. t X t Characterizes users' time preferences; Step 3: Construct a Long Short-Term Preference (LSTHN) model. Specifically, based on the user's check-in sequence information, construct multiple LSTHN-based hypergraph neural networks. The check-in sequence information includes at least the order of check-in records in the check-in sequence. Use these multiple LSTHN-based hypergraph neural networks to obtain the user's POI visit features X based on long short-term preferences. a X a Characterizes users' short-term and long-term preferences; Step 4: Construct the spacetime gate embedding GRU model ST-GRU By fusing the features output by the TIHN and LSTHN models, the POI embedding features are obtained, specifically expressed by the following formula: Where, ω e X is a hyperparameter. a X represents the feature output of the long-short-term preference model LSTHN. t The features output by the aggregated temporal information model TIHN; when ω e When = 1, This represents the impact of short-term and long-term preferences on user access to POIs; when ω e When = 0, This indicates the impact of time information on user access to POIs; By combining the time interval coding vector and the spatial interval coding vector, and the T-shift gating for the time interval coding vector and the S-shift gating for the spatial interval coding vector, the formulas for T-shift gating and S-shift gating are obtained, specifically expressed by the following formulas: Where, ω t It is the weight of the time interval encoding vector, ω s It is the weight of the spatially spaced encoding vector, ω tx ω is the weight of the POI embedding features in the T-shift gate. sx b is the weight of the POI embedding features in the S-shift gate. t This indicates the bias introduced by T-shift gating, b s Ts represents the bias introduced by S-shift gating, σ represents the sigmoid activation function, and Ts represents the bias introduced by S-shift gating. i This is the output of the T-shift gate, representing the impact of the time interval encoding vector on the POI embedding features, Ss i This is the output of the T-shift gate, representing the influence of the spatially spaced encoding vector on the POI embedding features. express The l-th POI embedding feature in; The output of S-shift gating is compared with the output of T-shift gating. Each POI embedding feature in the matrix is multiplied by a dot product, specifically represented by the following formula: Where, ω st This represents the weight of the embedded features in the ST-shift gate, which is a gate obtained by integrating the S-shift and T-shift gates. st This indicates the bias introduced by ST-shift gating, and ⊙ indicates element-wise multiplication. It is the output of ST-shift gated. This represents a feature that includes both time interval and spatial interval characteristics; Will After the R-gated control is reset and the Z-gated control is updated, the updated features are obtained, resulting in the output of the ST-shift-GRU unit, which is expressed by the following formula: Where, ω R The previous hidden state h in the R-gated state i-1 The weight, ω Z The previous hidden state h in Z-gating i-1 The weight, b R The previous hidden state h in the R-gated state i-1 The bias, b Z The previous hidden state h in Z-gating i-1 The bias, ω Rx POI embedding features in R-gated The weight, ω Zx Embedded features in Z-gating The weight, b Rx POI embedding features in R-gated The bias, b Zx Embedded features in Z-gating The bias, h ′ For candidate hidden states, ω hx POI embedding features The weight, ω h For the hidden state h i-1 The weight, b hx POI embedding features The bias, b h For the hidden state h i-1 The bias; The check-in sequence contains multiple check-in records, each corresponding to an ST-shift-GRU unit. The hidden state h output by the ST-shift-GRU unit corresponding to the current check-in record is... i The result is passed to the ST-shift-GRU unit corresponding to the next check-in record for calculation, and the ST-shift-GRU unit corresponding to the current check-in record also outputs Y. i f Y i f For the node features output by the i-th ST-shift-GRU unit in layer f, Y i f Passed to the (i+1)th ST-shift-GRU unit in layer f, where Y i f This can be expressed by the following formula: Among them, ST-shift-GRU Cell It is a single ST-shift-GRU unit. This represents the hidden state of the (i-1)th ST-shift-GRU unit in layer f. Represents the spatially spaced encoding vector. Y represents the time interval encoding vector. i f-1 This represents the node characteristics output by the i-th ST-shift-GRU unit in the (f-1)-th layer; The outputs of all ST-shift-GRU units in the last layer are concatenated, as shown by the following formula: Where Concat() represents a join operation, Y F This represents the output characteristics of all ST-shift-GRU units in layer F, which is the last layer. The output characteristics of the first ST-shift-GRU unit in layer F, The output characteristics of the second ST-shift-GRU unit in layer F, The output feature of the nth ST-shift-GRU unit in layer F; For Y F After performing a nonlinear transformation, the following formula is obtained: Where, ω Y Let b represent the weight matrix. Y Represents the bias vector; Indicates the user's preference for accessing Points of Interest (POIs); Step 5: Next POI Prediction and Model Optimization Predict the POI that the user will visit next, and obtain the preference score of POIl at time t+1. Specifically, this is expressed by the following formula: In POIl, l is the POI number, POIl represents the POI with number l, and POIl is the POI to be predicted. This represents the user's preference for accessing POIs at time t. This represents the transpose of the matrix representing the position of POIl in the check-in sequence. This represents the user's preference score for accessing POIl at time t+1.
2. The POI recommendation method based on multimodal temporal information fusion according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1.1: Encode the POI ID number by converting the POI ID number encoding into a P-dimensional dense vector; Step 1.2: Obtain the encoding vector of the time interval and the encoding vector of the spatial interval between two adjacent check-in records in the user's check-in sequence. The check-in sequence contains multiple check-in records arranged in chronological order. Each check-in record includes at least the latitude and longitude of the POI where the user is located, and the check-in time when the user checked in at the POI. Calculate the sign-in sequence S u Two adjacent sign-in records and time interval Specifically, it is calculated using the following formula: in, S represents the sign-in sequence. u The i-th sign-in record in the middle, S represents the sign-in sequence. u The (i-1)th sign-in record in the database. express The corresponding check-in time, express The corresponding check-in time; Calculate the sign-in sequence S u Two adjacent sign-in records and Spatial interval Specifically, it is calculated using the following formula: Here, haversine(·) is a function used to calculate the distance between two locations. express The latitude and longitude of the corresponding check-in location, express The latitude and longitude of the corresponding check-in location; Set the time interval threshold Δt max and spatial interval threshold Δd max The formula is as follows: in, This represents the time interval after the threshold update. This represents the spatial interval after the threshold update; calculate and relative time intervals between as well as and The relative spatial interval between Specifically, it is calculated using the following formula: in, This represents the time interval with the smallest value among multiple time intervals. This represents the spatial interval with the smallest value among multiple spatial intervals; right and relative time intervals between as well as and The relative spatial interval between Normalization is performed using the following formula: in, d represents the time interval with the largest value among multiple time intervals. u max This represents the spatial interval with the largest value among multiple spatial intervals. This represents the normalized relative time interval. Indicates the normalized relative spatial interval; Will and By converting it into a P-dimensional vector using equal-division encoding, we obtain... and The encoding vector is the time interval, representing the check-in sequence. With the sign-in sequence The time difference between the transfers The spatially spaced encoded vector represents the check-in sequence. With the sign-in sequence The difference in transfer space between them.
3. The POI recommendation method based on multimodal temporal information fusion according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Construct the temporal correlation matrix H in the hypergraph neural network for temporal information. t and hyperedge weight matrix W t The time information includes the POIs accessed by the user within a preset time range; The time information is divided into 24 time periods by hour, and a time correlation matrix H is created. t Time correlation matrix H t The formula is: Where v represents a node of POI in the hypergraph based on time information, e t Represents a hyperedge in a time-based hypergraph; h t (v,e t ) is a function representing the hyperedge e t Does it have nodes v and h? t (1,1) indicates whether hyperedge 1 is connected to node 1; h t (1,2) indicates whether hyperedge 2 is connected to node 1, h t (1,24) indicates whether the hyperedge 24 is connected to node 1, h t (2,1) indicates whether hyperedge 1 connects to node 2, h t (2,2) indicates whether hyperedge 2 is connected to node 2, h t (2,24) indicates whether the hyperedge 24 connects to node 2, h t (|L|,1) indicates whether hyperedge 1 is connected to node |L|, h t (|L|,2) indicates whether hyperedge 2 is connected to node |L|, h t (|L|,24) indicates whether the superedge 24 is connected to node |L|; When h t (v,e t When ) = 1, it indicates that the hyperedge e t Node v connected to POI; when h t (v,e t When ) = 0, it indicates that the hyperedge e t Node v has no connection to the POI; Create a time-based hyperedge weight matrix W t The hyperedge weight matrix W for time information t This represents the number of POIs accessed by a user at each time step, and is the hyperedge weight matrix W for time information. t After normalization, the following formula is obtained: Where, diag represents a diagonal matrix. This indicates the number of nodes connected by the superedge e1. This indicates the number of nodes connected by the hyperedge e2. Indicates the superedge e 24 The number of connected nodes, where m represents the number of check-in records in the check-in sequence; Step 2.2: Obtain user time preferences using a temporal hypergraph neural network After obtaining the time correlation matrix H t and the weight of the superedge W t Then, through hypergraph convolution operations, an activation function is introduced to obtain the f-th layer hypergraph neural network, as shown in the formula: Where ReLU is a nonlinear activation function. Represents the time correlation matrix H t transpose, This is the feature matrix of the nodes in the f-th layer of the time hypergraph neural network. When f = 0, This represents the feature matrix of the nodes in layer 0 of the temporal hypergraph neural network, where X is a dense vector; D v,t D is a diagonal matrix representing the node degree of a time hypergraph neural network. e,t The diagonal matrix representing the hyperedge degree of the temporal hypergraph neural network. It is the weight matrix of the f-th layer of the temporal hypergraph neural network; Set the diagonal matrix D for hyperedge degree. e,t The constraints are expressed by the following formula: D e,t =max{D e,t ,ξ}; Where ξ is a constraint condition and ξ is a preset value; Aggregation of multilayer hypergraph neural networks using aggregation functions is expressed by the following formula: Among them, X t X represents the features output by the aggregated temporal information model TIHN. t Characterizes users' time preferences.
4. The POI recommendation method based on multimodal temporal information fusion according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Construct multiple hypergraph neural networks based on the user's check-in sequence information, wherein the check-in sequence information includes at least the order of the check-in records in the check-in sequence; Since multiple check-in records in the check-in sequence may contain the same POI, the sequence interval between POIs is calculated to obtain the sequence interval matrix A. The sequence interval is the distance between the positions of the check-in records corresponding to two different POIs in the check-in sequence. The sequence interval matrix A is specifically represented by the following formula: Where, l1 is the POI number, l1 corresponds to POIl1; l2 is the POI number, l2 corresponds to POIl2; |L| For the POI number, l |L| Corresponding to POIl |L| a(l1, l1) is used to calculate the minimum sequence interval between POIl1 and POIl2; a(l1, l2) is used to calculate the minimum sequence interval between POIl1 and POIl2; a(l1, l2) is used to calculate the minimum sequence interval between POIl1 and POIl2. |L| ) used to calculate POIl1 and POIl |L| The minimum sequence interval between POIl2 and POIl1; a(l2, l2) is used to calculate the minimum sequence interval between POIl2 and POIl1; a(l2, l2) is used to calculate the minimum sequence interval between POIl2 and POIl1; a(l2, l1 ... |L| ) used to calculate POIl2 and POIl |L| The smallest sequence interval between them; a(l |L| l1) is used to calculate POIl |t| The minimum sequence interval between POIl1 and POIl1; a(l |L| l2) is used to calculate POIl |L| The smallest sequence interval between POIl2 and POIl2; a(l |L| , l |L| ) used to calculate POIl |L| and POIl |L| The smallest sequence interval between them; For each check-in sequence, K different sequence spans are set. These sequence spans are used to construct a sequence association matrix. For each check-in sequence with a given sequence span, a hypergraph neural network is constructed to obtain K hypergraph neural networks. The K sequence spans are as follows: The weights of each hypergraph neural network are represented as: {ω1, ω2, ..., ω...} K },and For each check-in sequence with a given span, a hypergraph neural network is constructed based on the corresponding span. Specifically, for the k-th span, a k-level sequence association matrix is constructed. Specifically, this is expressed by the following formula: Where 1≤i≤|L|, Let i be the node i of the hypergraph corresponding to the k-th sequence span based on long-short-term preferences; 1≤j≤|L|, Let j represent the hyperedge j of the hypergraph corresponding to the k-th sequence span based on long-term and short-term preferences; Indicates the superedge Does it have nodes? when hour, Indicates POIl i and POIl j The minimum sequence interval between them is less than the set span of the k-th sequence, and the user visits Ask POIl i I have already visited POIl. j , Indicates the superedge There are nodes on when hour, Indicates POIl i With another POIl j The minimum sequence interval between them is greater than the set k-th sequence span, or the user is accessing POIl i I haven't visited POIl before. j , Indicates the superedge There are no nodes on Set the number of times a user accesses a POI to the hypergraph. Hyperedge weight matrix based on long-short-term preferences In this context, the hyperedge weights of the K different hypergraph neural networks are identical, and the hyperedge weight matrix represents the long-short-term preference. After normalization, the following formula is obtained: Where, diag represents a diagonal matrix. Indicates user access to POIl i The frequency, m represents the number of check-in records in the check-in sequence; Step 3.2: Obtain users' short-term and long-term preferences through multiple hypergraph neural networks. After obtaining the k-level sequence correlation matrix and superedge weights based on long-term and short-term preferences Then, through hypergraph convolution operations, the ReLU activation function is introduced to obtain the l-th layer hypergraph neural network corresponding to the k-th sequence span. The f-th layer hypergraph neural network is represented by the following formula: in, Represents the k-level sequence incidence matrix transpose, Let be the node feature matrix of the k-th sequence span. is the node feature matrix of the f-th layer, where the node feature matrix of the 0th layer is a dense vector; The diagonal matrix representing the node degree of the hypergraph neural network with the span of the k-th sequence. The diagonal matrix representing the hyperedge degree of the hypergraph neural network with the span of the k-th sequence is... It is the weight matrix of the f-layer hypergraph neural network; The multi-layer hypergraph neural network that aggregates the k-th check-in sequence using the sum-pooling aggregation function is represented by the following formula: in, The features output by the hypergraph neural network after aggregating the k-th sequence span; Through hierarchical weight ω k The features output by the hypergraph neural network with K sequence spans are stacked, specifically represented by the following formula: Among them, X a These are the features output by the long-short-term preference model LSTHN.
5. The POI recommendation method based on multimodal temporal information fusion according to claim 1, characterized in that, The spacetime gate-embedded GRU model, ST-GRU, is optimized using a cross-entropy loss function, the formula for which the cross-entropy loss function is optimized is as follows: Where J represents the loss function value, u represents user u, U represents all users, S(u) is the check-in sequence, and i represents the POI number in the check-in sequence. The prediction parameters of the space-time gated embedding GRU model ST-GRU are: (i) the prediction output of the space-time gated embedding GRU model ST-GRU. denoted as user u’s preference score for POIi, where POIi represents the POI numbered i, λ is the regularization coefficient, and ‖θ‖2 represents the L2 regularization of all weight parameters of the space-time gate embedded GRU model ST-GRU.
Citation Information
Patent Citations
Interest point recommendation method and device based on multi-modal knowledge graph
CN117609605A
Points of interest (POI) ranking based on mobile user related data
US20130262479A1
