Multi-modal fusion space-time cross attention personalized trajectory recommendation method

Abstract

The invention discloses a multi-modal fusion space-time cross attention personalized trajectory recommendation method, which comprises the following steps: capturing a group interaction rule through collaborative embedding, generating user personalized embedding based on a mask language model, and constructing space-time data representation in combination with multi-granularity time embedding and quadtree space embedding; a personalized space-time cross attention mechanism is designed, and time stamp cross attention and time interval perception are fused to strengthen space-time relevance; a dynamic gating fusion mechanism is utilized to realize deep integration of three modes of collaboration, individuation and space-time, and finally, a multi-constraint individuation track is output through a low-rank adaptation fine tuning large language model. According to the method, the accuracy, the space-time rationality and the personalized adaptation capability of trajectory recommendation are remarkably improved, and the method is particularly suitable for scenes such as smart tourism and urban travel planning.

Description

Technical Field

[0001] This invention relates to the field of trajectory recommendation technology, and in particular to a multimodal fusion spatiotemporal cross-attention personalized trajectory recommendation method. Background Technology

[0002] With the rapid development of mobile internet and location-based services, trajectory recommendation has become a core technology connecting user needs and spatial services. Its core objective is to generate a sequence of Points of Interest (POIs) that combines personalized preferences, group collaboration patterns, and spatiotemporal rationality for users. In real-world scenarios such as smart tourism and urban travel, users' travel decisions face multiple constraints: they must align with their own interests (e.g., photography enthusiasts prefer natural landscapes, while families prefer amusement facilities), conform to the group selection patterns of similar users (e.g., young tourists tend to visit popular tourist attractions), and simultaneously satisfy the geographical accessibility (e.g., the distance between adjacent POIs is moderate) and the temporal rationality (e.g., the trip time meets expectations).

[0003] Existing trajectory recommendation methods can be divided into traditional recommendation methods and methods based on Large Language Models (LLM). Traditional recommendation methods, such as matrix factorization and Graph Neural Networks (GCN), can capture some group collaboration patterns or spatiotemporal correlations, but their performance drops significantly in cold-start scenarios (new users / new POIs lack historical interaction data), and they struggle to deeply integrate multimodal information. LLM-based recommendation methods, with their powerful semantic understanding capabilities, show certain advantages in cold-start scenarios, but still have significant technical limitations:

[0004] 1. Insufficient multimodal fusion: Existing methods mostly focus on "collaboration-text" bimodal fusion, without modeling spatiotemporal constraints as an independent modality, which leads to spatiotemporal misalignment in the generated trajectory (such as excessive distance between adjacent POIs or unreasonable time intervals).

[0005] 2. Limitations of personalized expression: Personalized representation relies on the direct splicing of text semantics from users' historical interactions, which cannot capture users' differentiated preferences for the same POI (such as the same scenic spot, photographers prefer to visit in the early morning, while families prefer to visit in the afternoon).

[0006] 3. Rigid fusion mechanism: The static weight allocation strategy is used to fuse multimodal information, which cannot adapt to the dynamic changes in user behavior (such as the difference in historical data between new users and old users, and the dynamic migration of user preferences).

[0007] The spatiotemporal modeling is crude: it lacks modeling of multi-scale time periods (such as hours, weeks, and months) and fine geographical features, making it difficult to accurately capture users' spatiotemporal behavior patterns.

[0008] Therefore, there is an urgent need for a personalized trajectory recommendation method that integrates multimodal fusion and spatiotemporal cross-attention to achieve deep integration of personalization, collaboration, and spatiotemporal constraints, thereby improving the overall performance of trajectory recommendation. Summary of the Invention

[0009] This invention aims to solve at least one of the technical problems existing in the prior art, and proposes a multimodal fusion spatiotemporal cross-attention personalized trajectory recommendation method.

[0010] In a first aspect, embodiments of the present invention provide a multimodal fusion-based spatiotemporal cross-attention personalized trajectory recommendation method, including:

[0011] S100. Obtain user set, POI set and user historical access trajectory data, perform multimodal feature encoding on the data, and generate collaborative embedding, personalized embedding, spatiotemporal data embedding and standardized trajectory sequence;

[0012] S200. Design a dynamic gating fusion mechanism that adaptively adjusts the weight ratio of collaborative embedding and personalized embedding through user dependency gating and POI dependency gating to generate collaborative and personalized fused embedding;

[0013] S300. Construct a personalized spatiotemporal cross-attention mechanism, which integrates personalized timestamp cross-attention and personalized time interval perception, dynamically couples user personalized preferences with spatiotemporal correlation, and generates spatiotemporally optimized embeddings;

[0014] S400. The collaborative and personalized fusion embedding and the spatiotemporal optimization embedding are semantically concatenated to construct the input sequence of the large language model; the attention layer parameters of the large language model are fine-tuned based on the low-rank adaptation method, the model is optimized through the joint loss function, and the recommendation trajectory is generated based on the optimized model.

[0015] Furthermore, in S100, the method for generating the collaborative embedding includes: aggregating user and POI embeddings using a three-layer graph convolution, fusing multi-scale interaction features through layer weights, and then performing linear transformation and nonlinear mapping through a two-layer MLP to generate user and POI collaborative embeddings adapted to the LLM semantic space.

[0016] Further, in S100, the method for generating the personalized embedding includes:

[0017] A masked language model is designed based on the frequency of user interaction with POIs. When the interaction frequency is high, the inherent semantics of the POI are preserved; when the interaction frequency is low, the contextual semantics are mined.

[0018] For each POI in the user's historical trajectory, a personalized embedding is generated according to a dynamic masking strategy; the average of the personalized embeddings of all POIs in the user's historical trajectory is taken to generate a personalized embedding for the user, accurately depicting differences in user preferences.

[0019] Furthermore, in S100, the spatiotemporal data embedding generation method includes:

[0020] Time embedding: Decompose the timestamp into five-dimensional granularity: month, weekday, date, hour, and quarter-hour, covering multi-scale time periods, and generate time embeddings adapted to large language models through linear transformation;

[0021] Spatial embedding: GPS coordinates are encoded using a pre-defined level quadtree, converting two-dimensional continuous geographic coordinates into discrete vectors of a pre-defined dimension. This quantifies geographic features while preserving spatial proximity, and generates spatial embeddings through linear transformation.

[0022] Furthermore, in S100, the standardized trajectory sequence generation method includes: adopting a fixed-length processing strategy, and unifying the historical trajectories of all users into a sequence of length by truncation or zero-padding methods, so as to provide the model with input in a unified format.

[0023] Furthermore, in S200, the method for implementing the dynamic gating fusion mechanism includes: adaptively adjusting weights based on the amount of user historical interaction data; when new users have scarce historical data, the gating mechanism reduces the weight of personalized embedding and increases the weight of collaborative embedding, ensuring recommendation reliability based on the group patterns of similar users; when old users have abundant historical data, the gating mechanism increases the weight of personalized embedding and reduces the weight of collaborative embedding, highlighting individual user preferences; and generating gating values ​​through the sigmoid activation function to achieve smooth adjustment of weights and avoid recommendation performance fluctuations caused by sudden changes.

[0024] Furthermore, in S300, the personalized timestamp cross-attention is used to focus on the periodicity of user behavior under similar timestamps, and the specific implementation method includes:

[0025] Generate query vectors, key vectors, and value vectors. The query vectors are used to carry the query intent for future time, the key vectors are used to carry the historical time matching benchmark, and the value vectors are used to carry POI features. User preference weights for POI categories are incorporated into the vector generation process.

[0026] We calculate attention scores to measure the degree of matching between future time queries and historical time keys, construct time period similarity and masking mechanisms, and accurately capture users' personalized time period preferences.

[0027] Furthermore, in S300, the personalized time interval perception is used to enhance the spatiotemporal coherence of adjacent POIs, and the specific implementation method includes:

[0028] Based on the interval relationship between historical and future timestamps, a personalized time interval weight matrix is ​​constructed to capture the correlation between time intervals and user preferences;

[0029] Cross-attention is calculated using query vectors, key vectors, and value vectors. The weight matrix is ​​incorporated into the calculation of cross-attention to enhance the impact of time intervals on spatial correlation. Residual connections and layer normalization are introduced to accelerate model training convergence and improve the stability of feature representation.

[0030] Furthermore, the methods for implementing S400 include:

[0031] The collaborative and personalized embedding method, along with spatiotemporal optimization embedding, is semantically concatenated to introduce [CLS] classification tokens and [SEP] separator tokens, constructing a sequence that conforms to the input format of a large language model. LoRA technology is used to fine-tune the LLM attention layer parameters, reducing the computational cost of full-scale fine-tuning while retaining the powerful semantic understanding capabilities of LLM. A joint loss function integrating recommendation loss, masked language model loss, and spatiotemporal constraint loss is designed to optimize recommendation accuracy, personalized representation capabilities, and spatiotemporal rationality, thereby optimizing the model. The optimized model generates recommendation trajectories through three steps: candidate selection, personalized ranking, and spatiotemporal optimization.

[0032] Secondly, the present invention also discloses an electronic device, comprising:

[0033] One or more processors;

[0034] Memory, used to store one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0036] This invention discloses a multimodal fusion-based spatiotemporal cross-attention personalized trajectory recommendation method. It captures group interaction patterns through collaborative embedding, generates personalized user embeddings based on a masked language model, and constructs spatiotemporal data representation by combining multi-granularity temporal embedding and quadtree spatial embedding. A personalized spatiotemporal cross-attention mechanism is designed, integrating timestamp cross-attention and time interval awareness to enhance spatiotemporal correlation. A dynamic gating fusion mechanism is used to achieve deep integration of collaborative, personalized, and spatiotemporal modalities. Finally, low-rank adaptation fine-tunes the large language model to output personalized trajectories that meet multiple constraints. This invention significantly improves the accuracy, spatiotemporal rationality, and personalized adaptation capability of trajectory recommendation, making it particularly suitable for scenarios such as smart tourism and urban travel planning.

[0037] Compared with the prior art, the present invention has the following technical effects:

[0038] 1. Deep multimodal fusion: A dynamic gating fusion mechanism is proposed to achieve seamless integration of collaborative, personalized, and spatiotemporal three modalities, solving the problem of multimodal fragmentation in existing methods;

[0039] 2. Precise Personalized Expression: By applying MLM (Mask Language Model) to trajectory point semantic processing, it breaks away from the limitations of traditional text semantic splicing, autonomously learns user-personalized embeddings, and accurately portrays differences in user preferences;

[0040] 3. Refined spatiotemporal modeling: Design a personalized spatiotemporal cross-attention mechanism that integrates timestamp cross-attention and time interval perception to effectively capture user spatiotemporal behavior patterns and improve the spatiotemporal rationality of trajectories;

[0041] 4. Strong scenario adaptability: It performs well in both cold and warm start scenarios with a smooth performance degradation, making it suitable for users with different activity levels and diverse POI category recommendations, especially for practical application scenarios such as smart tourism and urban travel planning. Attached Figure Description

[0042] Figure 1 A flowchart illustrating a multimodal fusion-based spatiotemporal cross-attention personalized trajectory recommendation method provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the implementation of personalized spatiotemporal attention in an embodiment of the present invention;

[0044] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0046] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0047] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0048] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0049] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0050] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0051] To address at least one of the technical problems existing in the aforementioned related technologies, this embodiment provides a personalized trajectory recommendation method based on multimodal fusion and spatiotemporal cross-attention, such as... Figure 1 ,include:

[0052] S100. Obtain user set, POI set and user historical access trajectory data, perform multimodal feature encoding on the data, and generate collaborative embedding, personalized embedding, spatiotemporal data embedding and standardized trajectory sequence;

[0053] In S100 of this embodiment, the method for generating collaborative embedding includes: aggregating user and POI embeddings using a three-layer graph convolution, fusing multi-scale interaction features through layer weights, and then performing linear transformation and nonlinear mapping through a two-layer MLP to capture the group collaborative patterns of user-POI interactions, providing a basis for group behavior in recommendations. Users' trajectory choices are not entirely independent; similar users often have similar travel preferences (e.g., young tourists tend to visit popular tourist attractions, while families prefer POIs with well-developed family-friendly facilities). By aggregating group interaction information, implicit group patterns can be mined, avoiding overly niche recommendation results.

[0054] This module uses a three-layer graph convolution [8] to aggregate user and POI embeddings, fuses multi-scale interaction features through layer weights, and then completes linear transformation and nonlinear mapping through a two-layer MLP to generate user and POI co-embeddings adapted to the LLM semantic space. The calculation method is as follows:

[0055] (1)

[0056] (2)

[0057] Among them, W mid1 b mid1 W mid2 and b mid2 These are the weight matrix and bias vector of the dual MLP, respectively. α l ∈[0,1] represents the layer weights, which are used to control the contribution of the embedding vector generated by the graph convolution of the l-th layer to the final co-embedding. and These are the user / POI embedding vectors output by the graph convolution of the l-th layer, respectively.

[0058] In S100 of this embodiment, the method for generating the personalized embedding includes:

[0059] A masked language model is designed based on the frequency of user interaction with POIs. When the interaction frequency is high, the inherent semantics of the POI are preserved; when the interaction frequency is low, the contextual semantics are mined.

[0060] For each POI in the user's historical trajectory, a personalized embedding is generated according to a dynamic masking strategy; the average of the personalized embeddings of all POIs in the user's historical trajectory is taken to generate a personalized embedding for the user, accurately depicting differences in user preferences.

[0061] Specifically, personalized embedding captures the differentiated preferences of individual users. Different users may have different preferences for the same POI (e.g., at the Summer Palace, photography enthusiasts prefer to go in the early morning to shoot the sunrise, while families tend to visit in the afternoon for rest). Relying solely on group collaboration or textual semantics cannot distinguish individual differences.

[0062] Existing methods mostly rely on direct concatenation of text semantics. This method is the first to apply MLM to trajectory point semantic processing. By dynamically optimizing the POI access sequence, it autonomously learns the semantic association of POI context and generates embeddings that can characterize differences in user preferences. The calculation method is as follows:

[0063] (3)

[0064] Among them, e p This is the original POI embedding, carrying the inherent semantic information of the POI. M is a dynamic masking flag used to control whether to mask the original semantics of the POI; M=1 indicates masking, and M=0 indicates no masking. E[MASK] is the masking embedding vector.

[0065] When a user has high interaction with a POI, the inherent semantics of the POI become more crucial for personalized user characterization. In this case, the masking probability of the inherent semantics of that POI should be reduced. Conversely, the masking probability can be increased to enable the model to capture the deep semantic relationships in the user-POI interaction.

[0066] Different users have different historical trajectories, and the semantic embeddings of POIs in each trajectory are dynamically processed by MLM. Therefore, the generated personalized user embeddings can effectively characterize the differences in preferences among users. The calculation method is as follows:

[0067] (4)

[0068] Where Traju=[(pu1,tu1),...,(puk,tuk)] is the historical trajectory of user u, k is the trajectory length, and Sum(Traju) is the total number of POIs in the historical trajectory of user u.

[0069] In S100 of this embodiment, the spatiotemporal data embedding generation method includes:

[0070] Time embedding: Decompose the timestamp into five-dimensional granularity: month, weekday, date, hour, and quarter-hour, covering multi-scale time periods, and generate time embeddings adapted to large language models through linear transformation;

[0071] Spatial embedding: GPS coordinates are encoded using a pre-defined level quadtree, converting two-dimensional continuous geographic coordinates into discrete vectors of a pre-defined dimension. This quantifies geographic features while preserving spatial proximity, and generates spatial embeddings through linear transformation.

[0072] Among them, the spatiotemporal data embedding captures the spatiotemporal physical constraints of the trajectory, ensuring that the recommended trajectory conforms to geographical accessibility and temporal rationality. This module is divided into temporal embedding and spatial embedding.

[0073] The time embedding transforms the timestamp t=[m,w,d,h,q] (representing month, weekday, date, hour, and quarter-hour respectively) into a 5-dimensional vector with multiple time granularities, breaking through the limitation of LLM-Traj's single-hour granularity and covering multiple time periods. A timestamp embedding adapted to the LLM semantic space is generated through linear transformation. The calculation method is as follows:

[0074] (5)

[0075] Among them, W t ∈Rd×5、b t ∈Rd are trainable parameters.

[0076] Spatial embedding uses a 17-level quadtree to encode GPS coordinates (latp, lonp), converting two-dimensional geographic coordinates into 17-dimensional discrete encoded vectors. Geographic features are quantified to generate spatial embeddings. The calculation method is as follows:

[0077] (6)

[0078] Where Wgeo∈Rd×17 and bgeo∈Rd are trainable parameters, Codep=QuadTree(latp,lonp)∈{0,1}, and 17 is the quadtree encoding result.

[0079] In S100 of this embodiment, the standardized trajectory sequence generation method includes: adopting a fixed-length processing strategy, and unifying the historical trajectories of all users into a sequence of length by truncation or zero-padding methods, so as to provide the model with a unified input format.

[0080] Specifically, the lengths of user historical trajectories vary significantly, and directly inputting them into the model can lead to training instability. This module employs BERT4Rec's fixed-length sequence processing method, standardizing user historical trajectories of varying lengths into fixed-length sequences through truncation or zero-padding. This provides a uniform format for subsequent model input, ensuring the model can stably handle inputs from different users. The calculation method is as follows:

[0081] (7)

[0082] (8)

[0083] Where i∈[1,k], k is the trajectory length, 0 3d It is a 3D zero vector to ensure that the length of all user trajectory sequences is uniform.

[0084] S200. Design a dynamic gating fusion mechanism that adaptively adjusts the weight ratio of collaborative embedding and personalized embedding through user dependency gating and POI dependency gating to generate collaborative and personalized fused embedding;

[0085] In S200 of this embodiment, the method for implementing the dynamic gating fusion mechanism includes: adaptively adjusting weights based on the amount of user historical interaction data; when new users have scarce historical data, the gating mechanism reduces the personalized embedding weight and increases the collaborative embedding weight, ensuring recommendation reliability based on the group patterns of similar users; when old users have abundant historical data, the gating mechanism increases the personalized embedding weight and reduces the collaborative embedding weight, highlighting individual user preferences; and generating a gating value through a sigmoid activation function to achieve smooth weight adjustment and avoid recommendation performance fluctuations caused by sudden changes.

[0086] Specifically, existing fusion methods mostly use static proportional weight allocation, which cannot adapt to dynamic changes in user behavior (such as differences in historical data volume between new and old users). This paper designs a user dependency gating (gu) and a POI dependency gating (gp) to dynamically balance the contribution weights of group collaboration patterns and individual user preferences.

[0087] This gating mechanism can adaptively adjust weights based on user behavior characteristics. For example, if a new user lacks rich historical trajectory data, the model will increase the weight of collaborative embedding to ensure the reliability of recommendations based on the travel patterns of similar users; if an experienced user has rich historical trajectory data, the model will increase the weight of personalized embedding to highlight individualized needs. The calculation method is as follows:

[0088] (9)

[0089] (10)

[0090] (11) (12)

[0091] Where Wg is the gating weight matrix, bg is the bias term, and σ is the sigmoid activation function.

[0092] This implementation employs user-dependent gating and POI-dependent gating, adaptively adjusting the fusion weights of collaborative embedding and personalized embedding based on user behavior characteristics. When new user interaction data is scarce, the weight of collaborative embedding is increased; when existing users have rich historical data, the weight of personalized embedding is increased, achieving a dynamic balance between group patterns and individual preferences.

[0093] S300. Construct a personalized spatiotemporal cross-attention mechanism, which integrates personalized timestamp cross-attention and personalized time interval perception, dynamically couples user personalized preferences with spatiotemporal correlation, and generates spatiotemporally optimized embeddings;

[0094] In S300 of this embodiment, the personalized timestamp cross-attention is used to focus on the periodicity of user behavior under similar timestamps, and the specific implementation method includes:

[0095] Generate query vectors, key vectors, and value vectors. The query vectors are used to carry the query intent for future time, the key vectors are used to carry the historical time matching benchmark, and the value vectors are used to carry POI features. User preference weights for POI categories are incorporated into the vector generation process.

[0096] Specifically, existing large-scale modeling methods lack fine-grained modeling of user spatiotemporal behavior, resulting in insufficient spatiotemporal coherence of trajectories. This paper's personalized spatiotemporal attention layer dynamically couples user personalized preferences with the spatiotemporal attention mechanism through personalized timestamp cross-attention and personalized time interval perception modules. This achieves fine-grained modeling of the correlation between user spatiotemporal behavior and personalized preferences, enhancing the model's ability to perceive the spatiotemporal characteristics of trajectories, such as... Figure 2 As shown.

[0097] First, generate a query vector (carrying the query intent with a future timestamp), a key vector (carrying the matching criteria with historical timestamps), and a value vector (carrying the feature representation of the POI). The calculation method is as follows:

[0098] (13)

[0099] Here, etf represents a future timestamp embedding. Wq and Wk are the user's preference weights for candidate POI category cpf, Wq and Wk are the query weight matrix / key weight matrix, and Wv is the value weight matrix. Let be the embedding vector of the i-th historical timestamp. Let Lmax be the preference weight of user u for the category ci to which the i-th historical POI belongs, and Lmax be the historical timestamp sequence.

[0100] We calculate attention scores to measure the degree of matching between future time queries and historical time keys, construct time period similarity and masking mechanisms, and accurately capture users' personalized time period preferences.

[0101] Attention scores are calculated to measure the degree of matching between future time queries and historical time keys. A time period similarity and masking mechanism is constructed to accurately capture users' personalized time period preferences. The calculation method is as follows:

[0102] (14)

[0103] in, This is the Query-Key standard attention score. λScycle is a weighted term for time period similarity to enhance the influence of similar time periods. Mask is an upper triangular mask used to mask future time information, preventing future information leakage and ensuring that the model predicts the future based solely on historical information.

[0104] In S300 of this embodiment, the personalized time interval perception is used to enhance the spatiotemporal coherence of adjacent POIs, and the specific implementation method includes:

[0105] Based on the interval relationship between historical and future timestamps, a personalized time interval weight matrix is ​​constructed to capture the correlation between time intervals and user preferences;

[0106] Cross-attention is calculated using query vectors, key vectors, and value vectors. The weight matrix is ​​incorporated into the calculation of cross-attention to enhance the impact of time intervals on spatial correlation. Residual connections and layer normalization are introduced to accelerate model training convergence and improve the stability of feature representation.

[0107] Specifically, the interval weight matrix enhances the correlation between time-adjacent POIs and optimizes spatiotemporal coherence by combining users' personalized dwell preferences.

[0108] First, a personalized time interval weight matrix is ​​constructed to capture time interval and preference information. The calculation method is as follows:

[0109] (15)

[0110] (16)

[0111] Where tf is the predicted future timestamp, tui is the historical timestamp of user u accessing POI, Ri is the original time interval between the i-th historical timestamp and the future timestamp, used to reflect the "far / near" relationship in time. max(R) is the maximum value in the original time interval, diag(·) is the diagonal matrix function, and Lmax is the maximum length of the historical timestamp sequence.

[0112] Secondly, cross-attention is calculated by combining the "Query-Key-Value" triple to enhance spatiotemporal correlation and achieve personalized spatiotemporal correlation modeling. The calculation method is as follows:

[0113] (17)

[0114] (18)

[0115] in, A learnable query, key, and value weight matrix. The Query-Key standard attention score is used to capture the underlying spatiotemporal relationships.

[0116] To accelerate model training convergence and enhance the stability of feature representations, residual connections and layer normalization are introduced. The calculation method is as follows:

[0117] (19)

[0118] LayerNorm(·) is the layer normalization function.

[0119] The method disclosed in this implementation constructs a "query-key-value" triple through personalized timestamp cross-attention, integrates user personalized preferences and time period similarity, avoids future information leakage through a masking mechanism, and accurately captures the user's access behavior patterns under similar timestamps; it constructs a personalized time interval weight matrix through personalized time interval perception, strengthens the correlation between time-adjacent POIs, and combines cross-attention to calculate spatiotemporal correlation features to improve the spatiotemporal coherence of the trajectory; and it accelerates model training convergence and enhances the stability of feature representation through residual connections and layer normalization.

[0120] S400. The collaborative and personalized fusion embedding and the spatiotemporal optimization embedding are semantically concatenated to construct the input sequence of the large language model; the attention layer parameters of the large language model are fine-tuned based on the low-rank adaptation method, the model is optimized through the joint loss function, and the recommendation trajectory is generated based on the optimized model.

[0121] The method implemented in S400 of this embodiment includes:

[0122] The collaborative and personalized embedding method, along with spatiotemporal optimization embedding, is semantically concatenated to introduce [CLS] classification tokens and [SEP] separator tokens, constructing a sequence that conforms to the input format of a large language model. LoRA technology is used to fine-tune the LLM attention layer parameters, reducing the computational cost of full-scale fine-tuning while retaining the powerful semantic understanding capabilities of LLM. A joint loss function integrating recommendation loss, masked language model loss, and spatiotemporal constraint loss is designed to optimize recommendation accuracy, personalized representation capabilities, and spatiotemporal rationality, thereby optimizing the model. The optimized model generates recommendation trajectories through three steps: candidate selection, personalized ranking, and spatiotemporal optimization.

[0123] This embodiment designs a deep fusion of LLM and multimodal features to balance recommendation accuracy, spatiotemporal rationality, and personalization needs. Low-Rank Adaptation (LoRA) technology is used to fine-tune the attention layer parameters, ultimately outputting a recommended trajectory that satisfies multiple constraints.

[0124] First, embed user collaboration and personalization. Instead of a single text input, semantically concatenate two types of features, Aspatio-temp, and spatiotemp, to construct an LLM input sequence. The calculation method is as follows:

[0125] (20)

[0126] Among them, [CLS] and [SEP] are BERT

[19] standard tokens.

[0127] Secondly, design a joint loss function to integrate the recommendation loss. MLM loss With spatiotemporal constraint loss This ensures that the generated trajectory meets the requirements of accuracy, personalization, and spatiotemporal rationality.

[0128] Specifically, the loss function is calculated as follows:

[0129] (twenty one)

[0130] (twenty two)

[0131] Where D is the training dataset, and the (u,p,y) triples are the sample set consisting of user, POI, and access label. Predict the probability of a user visiting a POI. The first target POI Nearest neighbor negative samples Predict the probability of a user accessing a negative sample. The negative sample weighting coefficient is T, which is the temperature coefficient that allows the model to assign higher weights to negative samples that are difficult to distinguish.

[0132] Let M[·] be the embedding corresponding to the masked POI, and M[·] be the expectation of all masked samples. Lpred is the predicted trajectory length. The time interval between the i-th and (i+1)-th POIs in the trajectory. Let vmax be the spatial distance between the i-th and (i+1)-th POIs, vmax be the average user movement speed, and max(0,·) be the hinge function, which only constrains discontinuous spatiotemporal patterns. These are the weighting coefficients.

[0133] The optimized model generates recommended trajectories through three steps: candidate selection, personalized ranking, and spatiotemporal optimization. First, the probability of visiting a Point of Interest (POI) is output by Vicuna-LoRA, combined with collaborative similarity. To measure the match between POIs and user collaborations, a Top-20 candidate POI was selected to narrow down the scope; then, the candidate POIs were categorized by "..." "Sorting is performed, and user category preferences are combined with model prediction confidence to improve personalized matching. Finally, POIs that do not meet the time and velocity constraints of spatiotemporal constraint loss are removed to generate a trajectory PredTraju=[ps,p1,...,pe] that conforms to real-world patterns."

[0134] This embodiment discloses a multimodal fusion-based spatiotemporal cross-attention personalized trajectory recommendation method. It captures group interaction patterns through collaborative embedding, generates personalized user embeddings based on a masked language model, and constructs spatiotemporal data representation by combining multi-granularity temporal embedding and quadtree spatial embedding. A personalized spatiotemporal cross-attention mechanism is designed, fusing timestamp cross-attention and time interval awareness to strengthen spatiotemporal correlation. A dynamic gating fusion mechanism is used to achieve deep integration of collaborative, personalized, and spatiotemporal modalities. Finally, a low-rank adaptation fine-tunes the large language model to output personalized trajectories that meet multiple constraints. This invention significantly improves the accuracy, spatiotemporal rationality, and personalized adaptation capability of trajectory recommendation, and is particularly suitable for scenarios such as smart tourism and urban travel planning.

[0135] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the recommended methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processors and the memory, configured to enable information interaction between the processors and the memory.

[0136] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0137] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0138] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0139] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the recommended methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0140] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the recommended method described above.

[0141] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0142] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0143] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0144] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0145] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0146] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0150] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multimodal fusion spatiotemporal cross-attention personalized trajectory recommendation method, characterized in that, include: S100. Obtain user set, POI set and user historical access trajectory data, perform multimodal feature encoding on the data, and generate collaborative embedding, personalized embedding, spatiotemporal data embedding and standardized trajectory sequence; S200. Design a dynamic gating fusion mechanism that adaptively adjusts the weight ratio of collaborative embedding and personalized embedding through user dependency gating and POI dependency gating to generate collaborative and personalized fused embedding; S300. Construct a personalized spatiotemporal cross-attention mechanism, which integrates personalized timestamp cross-attention and personalized time interval perception, dynamically couples user personalized preferences with spatiotemporal correlation, and generates spatiotemporally optimized embeddings; S400. The collaborative and personalized fusion embedding and the spatiotemporal optimization embedding are semantically concatenated to construct the input sequence of the large language model; the attention layer parameters of the large language model are fine-tuned based on the low-rank adaptation method, the model is optimized through the joint loss function, and the recommendation trajectory is generated based on the optimized model.

2. The trajectory recommendation method according to claim 1, characterized in that, In S100, the method for generating the collaborative embedding includes: using a three-layer graph convolution to aggregate user and POI embeddings, fusing multi-scale interaction features through layer weights, and then performing linear transformation and nonlinear mapping through a two-layer MLP to generate user and POI collaborative embeddings that are adapted to the semantic space of a large language model.

3. The trajectory recommendation method according to claim 1, characterized in that, In S100, the method for generating the personalized embedding includes: A masked language model is designed based on the frequency of user interaction with POIs. When the interaction frequency is high, the inherent semantics of the POI are preserved; when the interaction frequency is low, the contextual semantics are mined. For each POI in the user's historical trajectory, a personalized embedding is generated according to a dynamic masking strategy; the average of the personalized embeddings of all POIs in the user's historical trajectory is taken to generate a personalized embedding for the user, accurately depicting differences in user preferences.

4. The trajectory recommendation method according to claim 1, characterized in that, In S100, the spatiotemporal data embedding generation method includes: Time embedding: Decompose the timestamp into five-dimensional granularity: month, weekday, date, hour, and quarter-hour, covering multi-scale time periods, and generate time embeddings adapted to large language models through linear transformation; Spatial embedding: GPS coordinates are encoded using a pre-defined level quadtree, converting two-dimensional continuous geographic coordinates into discrete vectors of a pre-defined dimension. This quantifies geographic features while preserving spatial proximity, and generates spatial embeddings through linear transformation.

5. The trajectory recommendation method according to claim 1, characterized in that, In S100, the standardized trajectory sequence generation method includes: adopting a fixed-length processing strategy, and unifying the historical trajectories of all users into a sequence of length by truncation or zero-padding methods, so as to provide the model with input in a unified format.

6. The trajectory recommendation method according to claim 1, characterized in that, In S200, the method for implementing the dynamic gating fusion mechanism includes: adaptively adjusting weights based on the amount of historical user interaction data; when new users have scarce historical data, the gating mechanism reduces the weight of personalized embedding and increases the weight of collaborative embedding, ensuring recommendation reliability based on the group patterns of similar users; when old users have abundant historical data, the gating mechanism increases the weight of personalized embedding and reduces the weight of collaborative embedding, highlighting individual user preferences; and generating gating values ​​through the sigmoid activation function to achieve smooth adjustment of weights and avoid recommendation performance fluctuations caused by sudden changes.

7. The trajectory recommendation method according to claim 1, characterized in that, In S300, the personalized timestamp cross-attention is used to focus on the periodicity of user behavior under similar timestamps, and the specific implementation method includes: Generate query vectors, key vectors, and value vectors. The query vectors are used to carry the query intent for future time, the key vectors are used to carry the historical time matching benchmark, and the value vectors are used to carry POI features. User preference weights for POI categories are incorporated into the vector generation process. We calculate attention scores to measure the degree of matching between future time queries and historical time keys, construct time period similarity and masking mechanisms, and accurately capture users' personalized time period preferences.

8. The trajectory recommendation method according to claim 1, characterized in that, In S300, the personalized time interval awareness is used to enhance the spatiotemporal coherence of adjacent POIs, and the specific implementation method includes: Based on the interval relationship between historical and future timestamps, a personalized time interval weight matrix is ​​constructed to capture the correlation between time intervals and user preferences; Cross-attention is calculated using query vectors, key vectors, and value vectors. The weight matrix is ​​incorporated into the calculation of cross-attention to enhance the impact of time intervals on spatial correlation. Residual connections and layer normalization are introduced to accelerate model training convergence and improve the stability of feature representation.

9. The trajectory recommendation method according to claim 1, characterized in that, The methods for implementing S400 include: The collaborative and personalized embedding method, along with spatiotemporal optimization embedding, is semantically concatenated to introduce [CLS] classification tokens and [SEP] separator tokens, constructing a sequence that conforms to the input format of a large language model. LoRA technology is used to fine-tune the LLM attention layer parameters, reducing the computational cost of full-scale fine-tuning while retaining the powerful semantic understanding capabilities of LLM. A joint loss function integrating recommendation loss, masked language model loss, and spatiotemporal constraint loss is designed to optimize recommendation accuracy, personalized representation capabilities, and spatiotemporal rationality, thereby optimizing the model. The optimized model generates recommendation trajectories through three steps: candidate selection, personalized ranking, and spatiotemporal optimization.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the recommended method.

Images