A multi-modal and time-aware based multi-task point of interest recommendation method

Abstract

The present application belongs to the technical field of intelligent recommendation system and location service, and in particular to a multi-task point-of-interest recommendation method based on multi-modal and time perception. The method comprises the following steps: S1: data collection; S2: feature extraction; S3: construction of MAST-POI model. The present application introduces a large language model text encoder to improve semantic expression and cross-modal interaction modeling capability; introduces similar historical check-in trajectory embedding in the modeling process to effectively improve the recommendation accuracy in the data sparse scene; adopts a rotating position encoder to process the check-in time difference to enhance the model's ability to capture the time context; uses a gating weighting module and a hybrid expert feedforward network to realize dynamic adjustment of feature weights and expert selection, improve the model's personalized adaptability and generalization ability; can improve the accuracy, real-time performance and personalization level of the sequence POI recommendation system, and is suitable for large-scale user and multi-task real-time prediction scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent recommendation systems and location services, specifically a multi-modal and time-aware multi-task point of interest recommendation method. Background Technology

[0002] With the popularization of location-based social networks (LBSNs) and the development of mobile internet, sequential point of interest (POI) recommendation technology has become an important branch of intelligent recommendation. Its goal is to predict the points of interest for a user's next visit based on a given sequence of historical check-ins. Early methods included sequence modeling techniques based on matrix factorization, multilayer perceptrons (MLPs), recurrent neural networks (RNNs), and convolutional neural networks (CNNs). In recent years, the Transformer architecture, with its self-attention mechanism and advantages in long-distance dependency modeling, has become the mainstream framework for sequential POI recommendation.

[0003] However, user behavior patterns in urban environments are often influenced by multiple dimensions such as spatial location, interest categories, and time slices. Modeling methods that rely solely on time series or POI sequences are insufficient to fully capture users' complex spatiotemporal preferences, affecting prediction accuracy and personalized experience.

[0004] Existing methods for recommending sequence points of interest (POIs) have the following main shortcomings:

[0005] Insufficient utilization of time information: Existing methods generally do not incorporate the actual target time as an explicit condition into the decision-making process, resulting in a disconnect between the recommendation results and the user's current or specified access time; even when timestamps are introduced, they are mostly treated as auxiliary factors, failing to form a "recommendation mechanism under a given time scenario," making it difficult to match the temporal rhythm of human behavior.

[0006] Limited multimodal information fusion capabilities: In addition to POI IDs, user check-in data also includes various modal information such as category, geographic region, and time slice. Existing methods lack efficient representation and interaction modeling capabilities when fusing these heterogeneous features.

[0007] Insufficient utilization of historical trajectories: Most models rely only on the current trajectory or complete historical sequences, failing to fully explore the auxiliary role of similar historical trajectories in semantic information.

[0008] Decoupling of recommendation and ranking: Existing methods often design the recommendation module separately from the final ranking, lacking an end-to-end optimization mechanism, which makes it impossible to fully utilize the deep features learned by the model in the ranking stage.

[0009] To address these issues, we propose a multi-modal and time-aware multi-task interest point recommendation method. Summary of the Invention

[0010] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0011] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0012] A multi-modal and time-aware multi-task interest point recommendation method includes the following steps:

[0013] S1: Data Collection: Collect users' historical check-in data using mobile applications and GPS sensors;

[0014] S2: Feature Extraction: This includes data cleaning, feature extraction, and dataset construction;

[0015] S3: Constructing the MAST-POI model: A multimodal-multi-task joint modeling framework is adopted, setting corresponding prediction task headers for the three modalities of POI ID, POI category, and POI region. These three are embedded and encoded as different modal inputs in the model and participate in subsequent cross-modal interactive modeling and prediction tasks. The framework implements the following mechanisms within the model:

[0016] Shared representation + modality-specific task head: The multi-head hybrid attention module generates cross-modal interaction features. Each modality retrieves its own relevant feature tensor at the Query position as the input to the task.

[0017] Gated weighting and expert routing: The gated weighting module and the hybrid expert feedforward network module dynamically adjust the feature weights based on the time slice to which the user request belongs to the task.

[0018] Multi-task joint training: Each modality task calculates its loss function during the training phase, and the overall loss is formed by weighted summation. The loss weight coefficient used for weighted summation is an adjustable hyperparameter, which is used to adjust the loss weight of different tasks. During the inference phase, the prediction results of each modality are output separately.

[0019] The model includes an encoder module and a POI recommendation module. The encoder module is used to convert the input data into a format suitable for model processing and consists of a check-in encoder and a rotation position encoder. The POI recommendation module analyzes the user's historical behavior data to recommend places of interest to the user. After obtaining the joint embedding, it is input into the backbone structure of the basic POI recommendation model to learn the user's preference for POIs and capture preliminary user behavior timestamp features.

[0020] The check-in encoder includes a POI encoder, a category encoder, a region encoder, and a time slice encoder. The check-in encoder first combines all the check-in records in the user's historical check-in trajectory set and the similar historical check-in trajectory set into the user's historical check-in trajectory and the similar historical check-in trajectory, and forms the user's current check-in trajectory embedding, the user's historical check-in trajectory embedding, and the similar historical check-in trajectory embedding.

[0021] The rotary position encoder acts on the attention mechanism. and Regarding historical check-in data and Its relationship with the initial time The time differences are respectively and , corresponding Vector is , corresponding Vector is Then its opposite and The rotation encoding operations are as follows:

[0022] ;

[0023] ;

[0024] Rotational position encoding will replace the original attention mechanism Replace with ,Right now ;

[0025] The POI recommendation module consists of a multi-head hybrid attention module, a gated weighting module, a hybrid expert feedforward network module, and a similarity ranking module. The multi-head hybrid attention module models the features of the user's current trajectory, historical trajectory, and similar trajectories from multiple perspectives through multiple parallel attention heads. The gated weighting module uses the embedding of the target time slice currently requested by the user as a gating signal to perform weighted fusion of cross-modal features, so that the time scene directly affects the feature contribution. The hybrid expert feedforward network module receives the output of the gated weighting module and consists of multiple parallel feedforward expert networks and a gated network. The gated network dynamically allocates the weights of each expert according to the input features, so that different experts can focus on processing features under different modes or scenes. The similarity ranking module calculates the similarity between the feature representation output by the hybrid expert feedforward network module and the initial embedding of the candidate POIs, and ranks them according to the similarity score, selecting the Top-K results as the final recommendation list.

[0026] The gated weighting module is used to perform temporal context-aware weighted fusion of features from each modality task head within a multimodal-multi-task framework. The input to this gated weighting module comes from the multi-head hybrid attention module for each modality. The goal of the generated cross-modal feature vector set is to dynamically adjust the contribution of different cross-modal features in the corresponding modal task by combining the target time slice currently requested by the user.

[0027] As a preferred embodiment of the multimodal and time-aware multi-task point of interest recommendation method described in this invention, in step S2, the data cleaning method is as follows: user and POI interaction data containing missing or outlier values ​​are removed, and the historical check-in data of each user is sorted in chronological order, and then divided into trajectories with 24-hour boundaries; the feature extraction method is as follows: filtering technology is used to remove interactions of inactive users and unpopular POIs, in the following order: removing POIs with fewer than 10 total visits, removing users with fewer than 10 check-ins, and removing trajectories with only one check-in; the dataset construction method is as follows: extracting valid check-in sequences and time difference sequences from the processed user check-in sequences to construct the dataset.

[0028] As a preferred embodiment of the multimodal and time-aware multitask point of interest recommendation method described in this invention, the multi-head hybrid attention module is responsible for establishing interactive representations between different modalities, wherein the time slice participates in the interaction as a conditional modality, and generates dedicated feature inputs for three types of task heads: POI, the category to which the POI belongs, and the region where the POI is located; the multi-head hybrid attention module includes an encoder-decoder multi-head hybrid attention module and a decoder-only multi-head hybrid attention module;

[0029] The encoder-decoder multi-head hybrid attention module consists of an encoder and a decoder, wherein the encoder is composed of a multi-head hybrid attention block, and the input is the user's historical check-in trajectory embedding. Or embed similar historical check-in trajectories , recorded as The decoder consists of a masked multi-head hybrid attention block and a multi-head hybrid attention block. The input is the embedding of the user's current check-in trajectory. , recorded as Encoder input Output Decoder input After passing through the masked multi-head hybrid attention block, the result is... , and then As , As , and and The input multi-head hybrid attention block is mapped one-to-one to obtain the final output. or ;

[0030] The decoder multi-head hybrid attention module contains only one mask multi-head hybrid attention block, and the input... Output ,in Where b is the number of channels, Let n be the embedding dimension, and n be the length of the check-in sequence in the historical check-in data.

[0031] As a preferred embodiment of the multimodal and time-aware multi-task interest point recommendation method described in this invention, the hybrid expert feedforward network module is used to perform further nonlinear transformation and task-specific optimization on the fusion features of each modality task head within the multimodal-multi-task framework. This hybrid expert feedforward network module dynamically selects the most suitable expert network for the current input through a gating mechanism, so that different tasks can retain their specialized modeling capabilities for specific modes while sharing the basic structure.

[0032] As a preferred embodiment of the multimodal and time-aware multi-task interest point recommendation method described in this invention, the similarity ranking module is used to represent the high-level features of each modality task head. The mapping is used to arrive at the final prediction result, and the multi-task joint loss is calculated during the training phase. This similarity ranking module processes each modality task independently, ensuring that the task outputs are targeted and do not interfere with each other.

[0033] Compared with the prior art, the beneficial effects of this invention are: multimodal deep fusion: it integrates multimodal features such as POI ID, category, region, and time slice, and introduces a large language model text encoder to improve semantic expression and cross-modal interaction modeling capabilities;

[0034] Similar historical trajectory enhancement: Introducing similar historical check-in trajectory embedding during the modeling process effectively improves recommendation accuracy in data-sparse scenarios;

[0035] Refined temporal modeling: A rotary position encoder is used to handle check-in time differences, enhancing the model's ability to capture temporal context;

[0036] Gating and expert dynamic optimization: By using a gated weighting module and a hybrid expert feedforward network, the feature weights are dynamically adjusted and experts are selected, thereby improving the model's personalized adaptability and generalization ability.

[0037] Multi-task end-to-end training: Through joint optimization of the similarity ranking module and the backbone network, the accuracy of POI recommendation and ranking is improved, realizing collaborative modeling and complementary enhancement of tasks such as POI, category, and region;

[0038] In summary, this invention can improve the accuracy, real-time performance, and personalization of sequence POI recommendation systems, and is suitable for large-scale user and multi-task real-time prediction scenarios. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0040] Figure 1 This is a flowchart of a multi-modal and time-aware multi-task interest point recommendation method according to the present invention;

[0041] Figure 2 This is a flowchart of the MAST-POI model, which is a multi-modal and time-aware multi-task point of interest recommendation method according to the present invention.

[0042] Figure 3 This is a flowchart of the POI recommendation module of a multi-modal and time-aware multi-task point of interest recommendation method according to the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0046] This invention provides a multi-modal and time-aware multi-task point of interest recommendation method. Under a unified framework, it deeply encodes and fuses multimodal information such as POI ID, point of interest category, region, and time slice. Through multi-task learning, under a given access time, it simultaneously predicts the user's next check-in point of interest, its category, and region, forming a recommendation result consistent with the time scenario, thereby comprehensively modeling the user's spatiotemporal interest pattern.

[0047] The core innovations of this invention include:

[0048] Multimodal feature representation and semantic enhancement: By combining a large language model text encoder with structured embedding, semantic enrichment is performed on modalities such as category, region, and time slice, thereby improving the expressive power of cross-modal features.

[0049] Cross-modal multi-head hybrid attention mechanism: Establish bidirectional information interaction between different modal tasks, and use the "query by modality + multi-head fusion" approach to capture both intra-sequence dependencies and cross-modal complementary features.

[0050] Time-aware gated weighted and hybrid expert network: Using the target time slice embedding as the gating signal, the channel weights and expert selection are dynamically adjusted to enable the model to adaptively optimize feature representation under a given time scenario.

[0051] Similarity ranking-driven multi-task joint optimization: In end-to-end training, the recommendation and ranking processes are integrated. The similarity ranking module is used to accurately rank the prediction results of each task, and collaborative optimization is achieved through multi-task weighted loss.

[0052] This invention can be widely applied to scenarios such as location-based social networks, map navigation, travel, and commercial site selection, and is particularly suitable for real-time, multi-task parallel personalized recommendation systems. The task flowchart is as follows: Figure 1 As shown.

[0053] Specifically, a multi-modal and time-aware multi-task interest point recommendation method includes the following steps:

[0054] S1: Data Collection

[0055] Using mobile applications and GPS sensors, historical check-in data of users is collected regularly, including each POI and its corresponding timestamp, and the data is transmitted to the server in real time and stored.

[0056] S2: Feature Extraction

[0057] The feature extraction process includes three sub-steps: data cleaning, feature extraction, and dataset construction.

[0058] 1. First, perform data cleaning to remove user and POI interaction data containing missing or outliers, sort the historical check-in data of each user in chronological order, and then divide it into trajectories with 24-hour boundaries.

[0059] 2. Use filtering technology to remove interactions from inactive users and unpopular POIs in the following order: (1) Remove POIs with less than 10 total visits, (2) Remove users with less than 10 check-ins, and (3) Remove tracks with only one check-in (i.e., at least two check-ins within a day).

[0060] 3. Extract the valid check-in sequence and time difference sequence from the processed user check-in sequence to construct the dataset.

[0061] S3: Construct the MAST-POI model

[0062] This invention employs a multimodal-multitask joint modeling framework, simultaneously encoding modalities such as POI ID, category, region, and time slice; and sets prediction task heads for the three types of tasks: POI, category, and region. The target check-in time slice is only used as an input condition for feature modeling and gating routing, and is not used as a prediction target.

[0063] The framework implements the following mechanisms internally within the model:

[0064] Shared representation + modality-specific task head: The multi-head hybrid attention module generates cross-modal interactive features. Each modality retrieves its own feature tensor at the Query position as the input to the task. Features from other modalities will not participate in the subsequent prediction of the task.

[0065] Gated weighting and expert routing: The gated weighting module and the hybrid expert feedforward network dynamically adjust the feature weights according to the time slices corresponding to the tasks, so that the input features of each task head can be adaptively optimized for the task objectives.

[0066] Multi-task joint training: During the training phase, the loss function of each modality is calculated separately, and the overall loss is formed by weighted summation, where is an adjustable hyperparameter used to balance the importance of different tasks. During the inference phase, the prediction results of each modality are output separately.

[0067] The MAST-POI model consists of two modules: an encoder module and a POI recommendation module. The flowchart of MAST-POI is as follows: Figure 2 As shown.

[0068] (1) Encoder module

[0069] An encoder is used to convert input data into a format suitable for model processing and is a commonly used structure in machine learning. In the MAST-POI model, the encoder module consists of a sign-in encoder and a rotational position encoder. (Sign-in encoder) Includes POI encoder Category encoder Area encoder and time slice encoder .

[0070] ① Check-in encoder

[0071] For a check-in sequence The corresponding POI sequence is The category sequence is The region sequence is The time-slice sequence is The check-in encoder transforms it using embedding technology. As shown in formula (1):

[0072] (1);

[0073] The specific operation is as follows:

[0074] (2);

[0075] in, The operation represents stacking elements according to the new dimension. , , This refers to the number of channels, which is 4 in this model. The embedding dimension.

[0076] To fully utilize semantic information, the encoder , and Text encoder using a large language model For category sequences The corresponding text is directly input into the text encoder and embedded:

[0077] (3);

[0078] (4);

[0079] in, .

[0080] For regional sequences First, the area Coordinates of valid POIs The average coordinates are obtained by weighting the access frequency. Then use reverse geocoding Convert the coordinates into address text, then concatenate it with the text describing the number of POIs within the region, and input it into a text encoder to obtain the embedded data.

[0081] (5);

[0082] (6);

[0083] (7);

[0084] in, The operation is text concatenation. , For the region The total number of categories included.

[0085] For time-slice sequences First, divide the week into Each time slice is then converted into weekday text and hour text, concatenated, and input into a text encoder to obtain the embedded data.

[0086] (8);

[0087] (9);

[0088] in, The operation involves retrieving the weekday text corresponding to the time slice. The operation is to retrieve the hour text corresponding to the time slice.

[0089] POI encoder Standard embedding techniques are used to capture user interactions with Points of Interest (POIs) during training, as shown in the following formula:

[0090] (10);

[0091] (11);

[0092] The encoder module's input consists of three parts: the user's current trajectory. User historical check-in trajectory set Similar historical check-in tracks Given a user's current trajectory The corresponding user's historical check-in trajectory By the user in the trajectory The most recent It consists of several trajectories; if the trajectory is the user's first trajectory, then an empty trajectory is used. The formula is shown below:

[0093] (12);

[0094] (13);

[0095] For the training set In Its corresponding similar historical check-in trajectory From the training set and The text with the highest similarity The check-in trajectory consists of individual check-in records; for the test set In The corresponding similar historical check-in trajectory By Previously and with The text with the highest similarity The check-in trajectory is composed of several check-in records, and the formula is as follows:

[0096] (12);

[0097] (13);

[0098] in, Its operation represents cosine similarity.

[0099] The check-in encoder first sets the user's historical check-in trajectory. Similar historical check-in tracks All check-in records are combined to form the user's historical check-in trajectory. Similar historical check-in patterns Then , and Together with the encoders that input formula (1), they form the embedding of the user's current check-in trajectory. Embedding user's historical check-in trajectory Embedding similar historical check-in trajectories The formula is shown below:

[0100] (14);

[0101] (15);

[0102] ② Rotary position encoder

[0103] In this model, the rotary position encoder acts on the attention mechanism. and Regarding check-in and Its relationship with the initial time The time differences are respectively and , corresponding Vector is , corresponding Vector is Then its opposite and The rotation encoding operations are as follows:

[0104] (16);

[0105] (17);

[0106] Rotational position encoding will replace the original attention mechanism Replace with ,Right now .

[0107] (2) POI Recommendation Module

[0108] In the POI recommendation system, the model can recommend locations of interest to users by analyzing their historical behavior data. After obtaining the joint embedding through formula (1), it is input into the backbone structure of the basic POI recommendation model. This process enables the model to learn users' preferences for POIs and capture preliminary user behavior timestamp features. The flowchart of this module is as follows: Figure 3 As shown.

[0109] The POI recommendation module consists of a multi-head hybrid attention module, a gated weighting module, a hybrid expert feedforward network module, and a similarity ranking module. The multi-head hybrid attention module uses multiple parallel attention heads to model the features of the user's current trajectory, historical trajectories, and similar trajectories from multiple perspectives, simultaneously capturing feature correlations across different channels and time scales. The hybrid attention mechanism combines the advantages of self-attention and cross-attention, enabling the model to learn dependencies within sequences and integrate complementary information between different trajectories, thereby enhancing representational capabilities. The gated weighting module uses the target time slice embedding of the user's current request as a gating signal to perform weighted fusion of cross-modal features, allowing the time context to directly influence feature contributions. Through the gating mechanism, the model can dynamically adjust the weights of features from each channel based on time slice information, more accurately modeling the impact of temporal context on user behavior. The hybrid expert feedforward network module consists of multiple parallel feedforward expert networks and a gated network. The gated network dynamically assigns weights to each expert based on input features, allowing different experts to focus on processing features under different modes or scenarios. This approach enhances the model's expressive diversity and improves its adaptability to various POI recommendation tasks. The similarity ranking module calculates the similarity between the fused features and the initial embeddings of candidate POIs, ranks them according to the similarity scores, and selects the Top-K results as the final recommendation list. During the training phase, this module is also used to calculate the loss function, improving ranking accuracy and recommendation relevance by optimizing the similarity between features and target POI embeddings.

[0110] ① Multi-head hybrid attention module

[0111] The multi-head hybrid attention module is the core structure of the multimodal-multitask framework of this invention. It is responsible for establishing interactive representations between different modalities (POI ID, category, region, time slice). The time slice participates in the interaction as a conditional modality and generates dedicated feature inputs for the three types of task heads: POI, category, and region.

[0112] This module comprises two types of substructures: encoder-decoder multi-head hybrid attention modules and decoder-only multi-head hybrid attention modules. Its basic unit—the multi-head hybrid attention block—first targets four modalities (…). )and Each attention head is assigned an independent linear mapping matrix, and the input features are transformed into three types: Query, Key, and Value, thereby obtaining modality-specific and multi-head independent representation vectors.

[0113] Subsequently, in a certain modality As a query, with all modalities Attention is calculated on the Key and Value respectively to obtain the cross-modal attention result. At this time This is already the result of fusing multiple attention heads (first concatenated head by head, then processed through the output mapping matrix). (Fusion) can simultaneously capture feature interaction information from different subspaces.

[0114] The entire process will generate Cross-modal features (each group incorporates multi-head information). For modality The task header, whose input is obtained by stacking all cross-modal results when this modality is used as a query:

[0115] (18);

[0116] Should The vector group is only used for the prediction task head corresponding to the mode; for other modes... They are then sent to their respective task branches, thus ensuring the independence and specificity of features between tasks.

[0117] This "modal query retrieval + multi-head fusion" design allows the model to share cross-modal information while retaining dedicated feature representations for each task, facilitating subsequent optimization of the task by the gating weighting module and the hybrid expert module.

[0118] Multi-head hybrid attention block has Query weight matrix , Key weight matrix , Value weight matrix and Output weight matrix ,in For the number of attention heads, Can be Divisible, and , is the hidden vector dimension of each attention head. It is the channel number. This is the attention head number. First, embedding... and Each is mapped to a Query via a weight matrix. Key and Value :

[0119] (19);

[0120] Next, regarding and Execute formula (16) Operation and .

[0121] Subsequently, Perform hybrid attention operations:

[0122] (20);

[0123] Finally, Concatenate attention points and map them using a weight matrix:

[0124] (twenty one);

[0125] (twenty two);

[0126] in, The final output of this block is: .

[0127] A variant of the multi-head hybrid attention block is the masked multi-head hybrid attention block, which requires adding a mask matrix to formula (20). , becomes:

[0128] (twenty three);

[0129] Among them, let Element is Then when hour, For a standard causal mask matrix, we have

[0130] (twenty four);

[0131] when hour, Few codes, many masks

[0132] (25);

[0133] when hour, For a regular causal mask matrix, when hour, The condition boundary is shifted one unit to the right, which exposes the time of the next check-in to the model, allowing it to capture the semantic information of the constraint time slice.

[0134] The encoder-decoder multi-head hybrid attention module consists of an encoder and a decoder. The encoder is composed of a multi-head hybrid attention block, and the input is the user's historical check-in trajectory embedding. Or embed similar historical check-in trajectories , recorded as The decoder consists of a masked multi-head hybrid attention block and a multi-head hybrid attention block. The input is the embedding of the user's current check-in trajectory. , recorded as Encoder input Output Decoder input After passing through the masked multi-head hybrid attention block, the result is... , and then As , As , and and The input multi-head hybrid attention block is mapped one-to-one to obtain the final output. or The decoder multi-head hybrid attention module contains only one mask multi-head hybrid attention block, with the input... Output ,in .

[0135] ②Gated weighted module

[0136] The gated weighting module is used to perform temporal context-aware weighted fusion of features from various modal task heads within a multimodal-multi-task framework. The input to this module comes from the multi-head hybrid attention module tailored to the modality. The goal of generating cross-modal feature vector sets is to dynamically adjust the contribution of different channel features in the task by combining the target time slice currently requested by the user.

[0137] Specifically, for modal The task head retrieves the results from the multi-head hybrid attention module. splice it together And retrieve the time slice representation matrix embedded in the user's current trajectory. Reorganized into Subsequently, That is, the time slice embedding matrix for the next check-in, and the gating weight matrix. Multiplying yields the weighted average. Then use this weight to... The output is obtained by weighted summation. The specific formula is as follows:

[0138] (26);

[0139] (27);

[0140] in, Indicates the first The weight vector of each channel, Indicates the first Characteristics of each channel This indicates element-wise multiplication or element-wise multiplication after broadcasting. Operations at the cross-group channel dimension ( Normalization, that is, normalizing the weights of all channels from historical check-ins, similar check-ins and current check-ins.

[0141] This process will be performed on each modal task header ( Independent execution ensures that each task is targeted and does not interfere with each other when using time information for feature selection. By introducing time slices as gating signals, the model can capture the impact of temporal context on user behavior, thereby improving the accuracy of multi-task prediction.

[0142] ③ Hybrid expert feedforward network module

[0143] The Mixture-of-Experts Feed-Forward Network (MoE-FFN) module is used to perform further nonlinear transformations and task-specific optimizations on the fused features of task heads from various modalities within a multimodal-multi-task framework. This module dynamically selects the most suitable expert network for the current input through a gating mechanism, allowing different tasks to share the basic structure while retaining their specialized modeling capabilities for specific modes.

[0144] In this invention, each modal task head Each has an independent gating weight matrix With the corresponding A feedforward expert network .

[0145] The input to the module is the fused features obtained by the gated weighting module for this modality task head. First, embedding task-specific time slices. Generate expert weight matrix:

[0146] (28);

[0147] in, Operation by Expert Dimension Normalization.

[0148] Then, input features The data are fed into all expert networks and summed according to their weights to obtain the output. :

[0149] (29);

[0150] in:

[0151] (30);

[0152] It is a non-linear activation function, such as ReLU.

[0153] ④ Similarity ranking module

[0154] The similarity ranking module is used to represent the high-level features of each modality task head. The mapping is used to arrive at the final prediction result, and the joint loss of multiple tasks is calculated during the training phase. This module processes each modality task independently, ensuring the specificity and non-interference of task outputs.

[0155] During the training phase, for modal The task header, its high-level feature matrix vectors in Initial embedding vector of the true result By performing similarity calculations, the loss function for this trajectory in this modality task is obtained:

[0156] (31);

[0157] in, It is the first in the sequence The high-level feature vector of the project represents the feature vector of the project. The prediction of the nth project, therefore the loss function adopts the nth... Initial embedding of the project Calculate the similarity. The overall loss function for multi-task is:

[0158] (32);

[0159] in, , and This is a hyperparameter that controls the loss weights for different task predictions.

[0160] During the reasoning phase, for modalities Task header, calculation Initial embeddings of all candidate items in this modality The similarity scores are calculated and sorted in descending order. The Top-K items are selected as the final prediction results.

[0161] (33);

[0162] (34);

[0163] in, , For modality Size of the collection of items For sequence position In modality The prediction candidate set under the task.

[0164] S4: Model Application

[0165] When applied, the MAST-POI model of this invention first cleans and extracts features from user check-in data, constructing a trajectory sequence with multi-channel embeddings including POI ID, category, region, and time slice. Then, the user's current trajectory, historical check-in trajectories, and similar historical check-in trajectories are encoded separately and input into a multi-head hybrid attention module to fuse spatiotemporal features from different modalities and time scales. The fusion result is then passed through a gated weighting module and a hybrid expert feedforward network module, dynamically adjusting feature weights and selecting the optimal expert network for feature transformation based on the time slice information corresponding to the task. Finally, the high-level features of each modality task head are input into a similarity ranking module to generate Top-K prediction results for POI, category, and region, respectively. This process can be deployed in online recommendation systems to achieve real-time, multi-task parallel personalized recommendation services.

[0166] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1.A multi-modal and time-aware based multi-task point-of-interest recommendation method, characterized in that, The method comprises the following steps: S1: data collection: collecting historical check-in data of a user by using a mobile application and a GPS sensor; S2: feature extraction: including data cleaning, feature extraction and data set construction; S3: constructing a MAST-POI model: adopting a multi-modal and multi-task joint modeling framework, setting corresponding prediction task heads for tasks corresponding to POI ID, POI category and POI region, embedding and coding the three as different modal inputs in the model, and participating in subsequent cross-modal interaction modeling and prediction tasks, and the framework realizes the following mechanisms in the model: Shared representation + modal-specific task head: a multi-head mixed attention module generates cross-modal interaction features, and each modality takes out the feature tensor related to itself at the Query position as the input of the task; Gated weighting and expert routing: a gated weighting module and a mixed expert feedforward network module dynamically adjust feature weights according to the time slice embedding of the user request to which the task belongs; Multi-task joint training: each modality task calculates a loss function in the training stage, and forms an overall loss by weighted summation, wherein the loss weight coefficient for weighted summation is an adjustable hyperparameter, which is used to adjust the loss weight of different tasks, and the prediction results of each modality are output in the inference stage; The model comprises an encoder module and a POI recommendation module; The encoder module is used to convert input data into a format suitable for model processing and is composed of a check-in encoder and a rotating position encoder; the POI recommendation module recommends places of interest to the user by analyzing the user's historical behavior data, inputs the joint embedding into the backbone structure of the basic POI recommendation model after obtaining the joint embedding, learns the user's preference for POI, and captures the preliminary user behavior timestamp features; The check-in encoder comprises a POI encoder, a category encoder, a region encoder and a time slice encoder; the check-in encoder first combines all the check-in records in the user historical check-in trajectory set and the similar historical check-in trajectory set into user historical check-in trajectories and similar historical check-in trajectories, and forms user current check-in trajectory embedding, user historical check-in trajectory embedding and similar historical check-in trajectory embedding; The rotation position encoder acts on the attention mechanism and , for historical check-in data and , the time difference from the initial moment is and , The corresponding vector is , The corresponding vector is , and the rotation encoding operation on and is respectively: ； ； Rotary position encoding replaces the original attention mechanism with i.e. ; The POI recommendation module is composed of a multi-head mixed attention module, a gated weighting module, a mixed expert feedforward network module and a similarity ranking module; the multi-head mixed attention module models the features of the user's current trajectory, historical trajectory and similar trajectory through multiple sets of attention heads in parallel; the gated weighting module uses the target time slice embedding of the user's current request as a gating signal to weight and fuse the cross-modal features, so that the time scene directly affects the feature contribution; The mixed expert feedforward network module receives the output of the gating weighting module, is composed of a plurality of parallel feedforward expert networks and a gating network, the gating network dynamically allocates the weight of each expert according to the input feature, so that different experts can focus on processing features in different modes or scenes, the similarity ranking module calculates the similarity between the feature representation output by the mixed expert feedforward network module and the initial embedding of the candidate POI, and ranks according to the similarity score, and selects the top-K results as the final recommendation list. The gating weighting module is configured to perform time context-aware weighted fusion on the features of the respective modality task head under a multi-modal-multi-task framework, and the input of the gating weighting module comes from the multi-head mixed attention module for the modality The generated cross-modal feature vector group aims to dynamically adjust the contribution of different cross-modal features in the corresponding modality task in combination with the target time slice of the current request of the user. 2.The multi-modal and time-aware multi-task POI recommendation method of claim 1, wherein, In S2, the method of data cleaning is to eliminate user and POI interaction data containing missing values or abnormal values, and to sort the historical check-in data of each user in time sequence, and then divide it into trajectories with 24 hours as the boundary; the method of feature extraction is to remove the interaction of inactive users and unpopular POIs by using filtering technology, in the following order: remove POIs with less than 10 total visits, remove users with less than 10 check-ins, and remove trajectories with only one check-in; the method of data set construction is to extract effective check-in sequences and time difference sequences from the processed user check-in sequences, thereby constructing a data set. 3.The multi-modal and time-aware multi-task POI recommendation method of claim 1, wherein, The multi-head mixed attention module is responsible for establishing interaction representation between different modalities, wherein the time slice participates in the interaction as a conditional modality, and generates special feature input for the POI, the category to which the POI belongs, and the region where the POI is located. The multi-head mixed attention module includes an encoder-decoder multi-head mixed attention module and a decoder-only multi-head mixed attention module. The encoder-decoder multi-head hybrid attention module is composed of an encoder and a decoder, wherein the encoder is composed of one multi-head hybrid attention block, and the input is a user historical check-in trajectory embedding or a similar historical check-in trajectory embedding , denoted as The decoder is composed of one masked multi-head hybrid attention block and one multi-head hybrid attention block, and the input is a user current check-in trajectory embedding , denoted as The encoder input is , and the output is The decoder input is , and after passing through the masked multi-head hybrid attention block, the output is , and is taken as , as , and and are input into the multi-head hybrid attention block one by one to obtain the final output or ; The decoder-only multi-head mixed attention module only comprises one masked multi-head mixed attention block, input , output , wherein , wherein b is the number of channels, is the embedding dimension, and n is the length of the check-in sequence in the historical check-in data. 4.The multi-modal and time-aware multi-task POI recommendation method of claim 1, wherein, The mixed expert feedforward network module is used for further nonlinear transformation and task-specific optimization of the fusion features of each modality task head in a multi-modal multi-task framework, the mixed expert feedforward network module dynamically selects the expert network most suitable for the current input through the gating mechanism, so that different tasks share the basic structure while retaining the ability to model specific modes. 5.The multi-modal and time-aware multi-task POI recommendation method of claim 1, wherein, The similarity ranking module is configured to map the high-level feature representations of the task heads of each modality to final prediction results, and calculate a multi-task joint loss in a training phase. The similarity ranking module independently processes each modality task to ensure the pertinence and non-interference of the task outputs.

Images