Multimodal sequential recommendation method based on double-gated hybrid expert model and fourier denoising

Abstract

The application discloses a multimodal sequence recommendation method based on a double-gated hybrid expert model and Fourier denoising, and comprises the following steps: S1, multimodal feature coding: using a BERT / ViT pre-training model to extract text / visual features; S2, frequency domain feature denoising: adopting Fourier transform to filter frequency domain noise; S3, double-gated hybrid expert model: adopting parallel paths to realize adaptive multimodal fusion and time sequence interest modeling; and S4, multi-task joint optimization: integrating triple auxiliary contrast learning to jointly optimize the model, using the pre-training model BERT / ViT to extract text and image features, introducing Fourier transform to perform frequency domain denoising on the multimodal features, and significantly improving the robustness of the modal representation, so that input-dependent experts realize fine-grained modal interaction through dynamic routing, shared experts model the periodic evolution of user interest by using time coding with a gating mechanism, the quality of feature fusion is optimized, and the FT-MSR integrates triple auxiliary learning tasks, thereby relieving the data sparsity problem.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sequence recommendation, in particular to a multi-modal sequence recommendation method based on a double-gated hybrid expert model and Fourier denoising. BACKGROUND

[0002] The core role of a recommendation system is to filter out items of potential interest to users from a vast amount of data, thereby significantly improving user experience. As an important branch of recommendation systems, sequence recommendation aims to accurately predict the next interaction item of a user by modeling the temporal dependencies in the user interaction sequence. Early sequence recommendation models mainly rely on item ID embedding for preference modeling. Although such methods have high computational efficiency and can capture user preference patterns, they are essentially limited by data sparsity, resulting in limited performance.

[0003] A training method and device for a sequence recommendation model are disclosed in Chinese Patent No. 202211284644.9, which includes obtaining a historical behavior sequence of an associated object recommendation service, generating an original interest representation based on historical objects contained in the historical behavior sequence, calculating the object similarity between each historical object and a reference preference object based on the attention influence parameters of each historical object in at least one dimension, determining a target historical object in the historical behavior sequence based on the object similarity, and generating a counterfactual interest representation based on the target historical object. An optimized parameter pair is generated based on the original interest representation, the reference preference object corresponding to the reference preference representation, and the counterfactual interest representation, and the sequence recommendation model for the associated object recommendation service is trained based on the optimized parameter pair. The present application does not explicitly model the long-term and short-term preferences of users, cannot accurately generate personalized recommendations, does not fully utilize the application of item relationships in sequence recommendation, and has less than ideal recommendation performance when facing cold start problems or sparse data.

[0004] A sequence recommendation method and system based on contrast learning is disclosed in Chinese Patent Application No. 202311366797.2, which includes collecting historical interaction behavior sequence data of users and items, encoding according to embedding layers to obtain initial embedding representations of users and items that have interaction records with the users; inputting the initial embedding representations into a filter bank for filtering to obtain denoised embedding representations; obtaining user intent information through a clustering algorithm according to the behavior sequence data; using a sequence encoder based on a feedforward neural network as the main body in combination with the denoised embedding representations as the first training input; combining the user intent information and the output sequence representation of the sequence encoder through a contrast self-supervised algorithm as the second training input; using the first training input and the second training input simultaneously for training of the sequence recommendation model; obtaining a predicted score for final sequence prediction; the method has the disadvantage that the clustering algorithm cannot effectively distinguish fine-grained information of users, and when using the contrast learning technology, the construction of positive and negative sample pairs for items is easily affected by noise information of a single item, resulting in low recommendation accuracy.

[0005] A sequence recommendation data enhancement method based on graph contrast learning is disclosed in Chinese Patent Application No. 202211604474.8, which includes obtaining historical behavior sequences of users in real application scenarios and pre-processing the original data; according to the user behavior sequences obtained by pre-processing, the multi-order connection information between each item pair is counted, and then the weight of the edge between each item in the global graph is calculated and normalized according to the statistical information, and a threshold is set to filter noise information, and finally a global graph is constructed; for each item, neighborhood sampling is carried out to obtain two enhanced subgraphs with a certain probability, and LightGCN is used as an encoder, and shared parameters are used in different views; negative sampling is performed using the bucket clustering method to alleviate the negative effects of uneven distribution of item representations; interest extraction is performed using a capsule network; the prediction task and the contrast learning task are jointly optimized; the method has the disadvantage that when using the contrast learning technology, the construction of positive and negative sample pairs for items is easily affected by noise information of a single item, and when fusing embedding vectors output by multiple modules, a direct merging operation is performed, which cannot build connections between multiple semantic information, and the recommended item sequence provided for users is not accurate enough.

[0006] Multi-modal learning promotes the integration of semantic information and recommendation systems, providing a new way to solve the problem of sequence recommendation being limited by data sparsity, which limits performance. By integrating rich semantic information such as text and images, multi-modal recommendation significantly enhances user preference modeling capabilities. However, existing methods still face two major challenges: 1) severe modal noise interference; traditional methods struggle to effectively filter noise in multi-modal data, resulting in dilution of useful information; 2) neglecting temporal dynamics; existing feature fusion strategies often overlook fine-grained cross-modal relationships, leading to semantic loss and feature bias; at the same time, the model relies too much on behavior sequences themselves, ignoring temporal context information, making it difficult to capture dynamic changes in user interest. SUMMARY

[0007] The purpose of the present application is to solve the problems in the prior art and provide a multi-modal sequence recommendation method based on a dual-gated hybrid expert model and Fourier noise reduction, which can combat noise interference and data sparsity through a time-frequency dual-driven learning approach.

[0008] To achieve the above-mentioned purpose, the present application provides a multi-modal sequence recommendation method based on a dual-gated hybrid expert model and Fourier noise reduction, comprising the following steps:

[0009] S1, multi-modal feature encoding: using BERT / ViT pre-training model to extract text / visual features;

[0010] S2, frequency domain feature denoising: using Fourier transform for frequency domain noise filtering;

[0011] S3, dual-gated hybrid expert model: using parallel paths to realize adaptive multi-modal fusion and temporal interest modeling;

[0012] S4, multi-task joint optimization: integrating triple auxiliary contrast learning to jointly optimize the model.

[0013] As a preferred embodiment, the multi-modal representation of each item in step S1, multi-modal feature encoding, is constructed through the following three-stage process:

[0014] S1-1, ID modal embedding encoding: initialize an ID embedding matrix to map discrete item IDs to dense vectors;

[0015] S1-2, text modal feature extraction: given the text description associated with the item, add a special token "CLS" in front of it to input the BERT encoder to learn the text embedding;

[0016] S1-3, image modal feature extraction: divide the image associated with the item into blocks, serialize and splice the blocks, and add a special token "CLS" in front of them to input the ViT model to obtain the visual embedding.

[0017] As preferred, in the step S1-3 image modality feature extraction, a learnable linear projection layer is introduced to convert both text and image embeddings into an ID embedding space for cross-modality feature alignment.

[0018] As preferred, the step S2 frequency domain feature denoising comprises the following steps:

[0019] S2-1, giving the embedding representation of each modality;

[0020] S2-2, using fast Fourier transform to convert the feature embedding from time domain to frequency domain;

[0021] S2-3, calculating the frequency cutoff threshold according to the pre-defined reservation ratio, and constructing a binary mask matrix for spectral filtering;

[0022] S2-4, reconstructing the denoised frequency domain representation back to the time domain through inverse fast Fourier transform;

[0023] S2-5, obtaining the multi-modality feature embedding through layer normalization and residual connection.

[0024] As preferred, the step S3 double-gated hybrid expert model is constructed through the following three-stage process:

[0025] S3-1, input-dependent Top-k expert: adaptive multi-modality fusion is realized through dynamic expert routing;

[0026] S3-2, time-aware shared expert: a fixed number of shared experts are set, and user interest evolution is explicitly modeled through time encoding;

[0027] S3-3, user preference modeling and score prediction: a sequence modeling framework SASRec is used as the backbone network to model user preferences, where an independent SASRec encoder is used to process the feature sequence of each modality; the multi-modality prediction results are aggregated to obtain the predicted scores between users and candidate items.

[0028] As preferred, the step S3-1 input-dependent Top-k expert comprises the following steps:

[0029] S3-1-1, concatenating the feature vectors of the three modalities along the feature dimension;

[0030] S3-1-2, constructing a MoE architecture, where each expert is an independent feedforward network with the same structure but different parameters;

[0031] S3-1-3, calculating the expert weight using the gating mechanism;

[0032] S3-1-4, activate the Top-k experts with the highest weights to achieve fine-grained feature interaction modeling.

[0033] As preferred, the step S3-2 time-aware shared expert includes the following steps:

[0034] S3-2-1, generate two types of complementary time embedding encodings for a given interaction time series: absolute time encoding: capture user preferences at specific time points; interval encoding: model the periodicity of user behavior;

[0035] S3-2-2, combine the above two embedding representations to represent the temporal context;

[0036] S3-2-3, calculate the weight of each shared expert using a time gating mechanism;

[0037] S3-2-4, cooperatively model by all shared experts to obtain the final output;

[0038] S3-2-5, integrate the modeling results of the above two paths to obtain the final item embedding.

[0039] As preferred, in the step S4 multi-task joint optimization, the main task optimizes interaction prediction through cross-entropy loss, and the following three auxiliary contrast learning tasks are introduced to enhance representation learning and alleviate data sparsity:

[0040] S4-1, ID contrast, construct an ID contrast loss for each batch to maximize the similarity between user sequence embedding and its interaction item embedding;

[0041] S4-2, difficult negative sample generation, synthesize difficult negative samples by linearly interpolating the normalized representations of ID and text modalities to enhance the discriminability of the model, obtain an augmented training sample set, construct a contrast loss to maximize the similarity between the sequence representations of the same user in ID and text modalities;

[0042] S4-3, cross-modal sequence contrast, given text sequence embedding and image sequence embedding, randomly mask the features of the text sequence embedding and replace them with the features at the same position in the image sequence embedding to obtain an augmented text sequence embedding, construct a contrast loss to maximize the similarity between the augmented text sequence embedding and the original text sequence embedding of the same user to enhance the quality of cross-modal representation, similarly, calculate the contrast loss of the image modality, the total loss is the average of the two, integrate the auxiliary task with the main task to optimize the model parameters.

[0043] The application has the beneficial effects that: the application extracts text and image features by using a pre-trained model BERT / ViT, introduces Fourier transform to perform frequency domain noise reduction on multi-modal features, significantly improves the robustness of modal representation, integrates a double-gated MoE architecture of input-dependent non-shared experts and time-aware shared experts, the input-dependent experts realize fine-grained modal interaction through dynamic routing, the shared experts model the periodic evolution of user interest by using time coding with a gating mechanism, optimize the feature fusion quality, and the FT-MSR integrates three auxiliary learning tasks: ID contrast learning strengthens item representation; difficult negative sample generation optimizes the decision boundary; cross-modal sequence contrast learning promotes modal semantic alignment, and alleviates the data sparseness problem.

[0044] The features and advantages of the application will be described in detail with examples in conjunction with the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is the overall architecture diagram of the multi-modal sequence recommendation method based on the double-gated hybrid expert model and Fourier noise reduction of the application;

[0046] Figure 2 is a result graph of the influence of the frequency domain retention ratio on the recommendation performance in the parameter sensitivity analysis experiment of the multi-modal sequence recommendation method based on the double-gated hybrid expert model and Fourier noise reduction of the application;

[0047] Figure 3 is a result graph of the influence of the number of input-aware experts on the recommendation performance in the parameter sensitivity analysis experiment of the multi-modal sequence recommendation method based on the double-gated hybrid expert model and Fourier noise reduction of the application;

[0048] Figure 4 is a result graph of the influence of the number of time-aware experts on the recommendation performance in the parameter sensitivity analysis experiment of the multi-modal sequence recommendation method based on the double-gated hybrid expert model and Fourier noise reduction of the application;

[0049] Figure 5 is a result graph of the influence of the loss weight on the recommendation performance in the parameter sensitivity analysis experiment of the multi-modal sequence recommendation method based on the double-gated hybrid expert model and Fourier noise reduction of the application. DETAILED DESCRIPTION

[0050] Overall framework

[0051] The application provides a multimodal sequence recommendation method based on a double-gated hybrid expert model and Fourier denoising, the overall architecture of which is shown in Figure 1. The model comprises four core components: multimodal feature encoding: text / visual features are extracted using a BERT / ViT pre-training model; frequency domain feature denoising: Fourier transform is used for frequency domain noise filtering; double-gated hybrid expert model: adaptive multimodal fusion and time series interest modeling are realized through parallel paths; and multi-task joint optimization: the model is jointly optimized through triple auxiliary contrast learning.

[0052] Multimodal feature encoding

[0053] Each item is represented by a multimodal representation constructed through a three-stage process:

[0054] ID modal embedding encoding: an ID embedding matrix is initialized to map discrete item IDs into dense vectors ;

[0055] Text modal feature extraction: given the text description associated with an item , a special token "CLS" is added in front of it to input the BERT encoder to learn the text embedding:

[0056] (1),

[0057] where denotes the string concatenation operation;

[0058] Image modal feature extraction: the image associated with an item is divided into blocks , the blocks are serialized and concatenated, and a special token "CLS" is added in front of them to input the ViT model to obtain the visual embedding:

[0059] (2),

[0060] To realize cross-modal feature alignment, a learnable linear projection layer is introduced to convert both text and image embeddings into the ID embedding space:

[0061] (3),

[0062] where and are weight matrices, and are bias terms.

[0063] Frequency domain feature denoising

[0064] Multi-modal features are usually susceptible to noise interference: the text modality can distort semantic expression due to redundant advertisements or spelling errors, while the visual modality can cause image distortion due to cluttered backgrounds or low resolution. To enhance the robustness of feature embedding, the present application uses Fourier transform to realize frequency domain noise filtering, the specific process is as follows:

[0065] Given the embedding representation of each modality , , B denotes the batch size, L denotes the sequence length, D denotes the embedding dimension; first, the feature embedding is converted from the time domain to the frequency domain using the Fast Fourier Transform (FFT):

[0066] (4),

[0067] where denotes the spectrum matrix, denotes the frequency vector;

[0068] Then, according to the pre-defined reservation ratio r , the frequency truncation threshold is calculated to construct a binary mask matrix for spectrum filtering:

[0069] (5),

[0070] (6),

[0071] (7),

[0072] where is the unit indicator function (all elements equal to 1), denotes element-wise multiplication;

[0073] Next, the noise-reduced frequency domain representation is reconstructed back to the time domain through the Inverse Fast Fourier Transform (IFFT):

[0074] (8),

[0075] Finally, the multi-modal feature embedding is obtained through layer normalization and residual connection:

[0076] (9).

[0077] Dual-gated hybrid expert model

[0078] (1) Input-dependent Top-k experts

[0079] This module realizes adaptive multi-modal fusion through dynamic expert routing, and the process is as follows:

[0080] First, the feature vectors of the three modalities are concatenated along the feature dimension:

[0081] (10),

[0082] Then, a MoE architecture is constructed, and each expert is an independent feedforward network with the same structure but different parameters:

[0083] (11),

[0084] where and represent the trainable parameters of the i th expert network, is an activation function;

[0085] Next, the expert weights are calculated using a gating mechanism:

[0086] (12),

[0087] where and are the trainable parameters of the i th gating network;

[0088] Finally, the Top-k experts with the highest weights are activated to realize fine-grained feature interaction modeling:

[0089] (13),

[0090] where represents the number of input-dependent experts, i.e. k value in Top-k;

[0091] (2) Time-aware shared experts

[0092] This module contains a fixed number of shared experts, which explicitly model the evolution of user interest through time encoding; for a given interaction time sequence , two complementary time embedding encodings are generated:

[0093] Absolute time encoding: captures the user's preferences at a specific time point

[0094] ​ (14),

[0095] where denotes the dimension j of the wavelength, T is the maximum timestamp, d is the embedding dimension;

[0096] Time interval encoding: modeling periodicity of user behavior

[0097] (15),

[0098] where denotes the time interval between events, is the smoothing factor, is the scaling factor;

[0099] Temporal context combines the above two embedding representations:

[0100] (16),

[0101] The weight of each shared expert is computed using a temporal gating mechanism:

[0102] (17),

[0103] where and denote the trainable parameters of the i th gating network;

[0104] The final output is obtained by jointly modeling from all shared experts:

[0105] (18),

[0106] where denotes the i th shared expert network, denotes the number of shared experts;

[0107] The modeling results of the above two paths are integrated to obtain the final item embedding:

[0108] (19).

[0109] User preference modeling and rating prediction: the sequence modeling framework SASRec is used as the backbone network to model user preferences, where an independent SASRec encoder is used to process the feature sequence of each modality:

[0110] (20),

[0111] where denotes the feature sequence of a specific modality, denotes the encoder parameters, is the user preference vector in modality m ;

[0112] Aggregate the multi-modal prediction results to obtain the predicted score between the user and the candidate item:

[0113] (21),

[0114] where denotes the candidate item embedding, is the Sigmoid activation function.

[0115] Multi-task joint optimization

[0116] In the FT-MSR model, the main task optimizes the interaction prediction through cross-entropy loss:

[0117] (22),

[0118] where denotes the predicted interaction probability of item v , denotes the real interaction label, denotes the candidate item set.

[0119] In order to enhance representation learning and alleviate the problem of data sparsity, three auxiliary contrastive learning tasks are introduced:

[0120] (1) ID contrast

[0121] For each batch , an ID contrast loss is constructed to maximize the similarity between the user sequence embedding and the interaction item embedding:

[0122] (23),

[0123] where denotes the ID vector of user i , denotes the ID embedding of its interaction item, denotes the cosine similarity, is the temperature scaling factor;

[0124] (2) Difficult negative sample generation

[0125] To enhance the discriminability of the model, difficult negative samples are synthesized by linear interpolation ID and normalized representation of text modality:

[0126] (24),

[0127] where denotes L2 normalization, is a random mixing coefficient, denotes stop gradient update;

[0128] Now, an augmented training sample set is obtained:

[0129] (25),

[0130] where and denote the original samples in ID and text modalities, respectively;

[0131] Contrastive loss is constructed to maximize the similarity between the sequence representations of the same user in ID and text modalities:

[0132] (26);

[0133] (3) Cross-modal sequence contrast

[0134] Given the text sequence embedding and image sequence embedding , some features of the random mask are replaced using the features at the same position in to obtain the augmented text sequence embedding .

[0135] To enhance the quality of cross-modal representation, contrastive loss is constructed to maximize the similarity between the augmented text sequence embedding and the original text sequence embedding of the same user:

[0136] (27),

[0137] The contrastive loss of the image modality is calculated in a similar way, and the total loss is the average of the two:

[0138] (28),

[0139] Finally, the auxiliary task is integrated with the main task to optimize the model parameters:

[0140] (29),

[0141] where , and denote the normalized weights of auxiliary tasks.

[0142] Experiments

[0143] This study conducts experimental tests based on the RecBole framework to address the following research questions:

[0144] RQ1: How does the performance of FT-MSR compare to the current state-of-the-art models?

[0145] RQ2: How do the core components of FT-MSR contribute to performance improvement?

[0146] RQ3: How do the key hyperparameters of FT-MSR affect model performance?

[0147] (1) Datasets

[0148] The experiment uses three publicly available Amazon datasets: Beauty, Games, and Home, which cover different application scenarios; the dataset statistics are shown in Table 1:

[0149] Table 1. Dataset statistics table

[0150] ;

[0151] (2) Baseline models

[0152] Eight representative baseline models are selected, including traditional sequence recommendation models: SASRec, GRU4Rec, LRURec, and UniSRec, as well as multi-modal sequence recommendation models: MISSRec, IISAN, TedRec, and HM4SR;

[0153] (3) Evaluation metrics and experimental settings

[0154] The normalized discounted cumulative gain NDCG@K and the average reciprocal rank MRR@K are used as evaluation metrics, where K ∈ {5, 10, 20}; the leave-one-out method is followed during the experiment process, reserving the last interaction of each sequence for testing; all models are trained on an NVIDIA RTX 4090 GPU with 24GB of memory, and the experimental configuration is as follows: batch size of 1024, hidden dimension of 64, maximum sequence length of 50, Adam optimizer, learning rate of 1 × 10 -3, dropout rate is 0.5; hyperparameter tuning range is as follows: frequency domain reservation ratio r∈[0.1, 0.9] step 0.1, input dependent expert number k1∈{4, 6, 8, 10, 12}, time-aware expert number k2∈{2, 4, 6, 8, 10}, loss weight Step 0.1;

[0155] (4) Experimental results

[0156] a. Comparative experiment RQ1

[0157] As shown in Table 2, the traditional sequence recommendation model GRU4Rec, SASRec, LRURec has poor performance due to dependence on a single ID mode; the multi-modal model UniSRec, TedRec, although fusing text / visual information, its performance improvement is still limited by modal noise and coarse-grained feature fusion; the model FT-MSR of the application is significantly better than all baseline models on all data sets, with NDCG@10 improved by 5.94% on Games and MRR@10 improved by 3.80% on Home; the reason is that: the frequency domain denoising effectively filters the noise in the modal representation, the double-gated MoE realizes fine-grained modal fusion and interest evolution modeling, and the auxiliary contrastive learning task alleviates the data sparsity problem, further improving the recommendation performance;

[0158] Table 2 Performance comparison table of different methods

[0159] ;

[0160] b. Ablation experiment RQ2

[0161] In order to verify the necessity and contribution of the core components of FT-MSR, five variant models are constructed and evaluated:

[0162] w / o Denoise: remove the frequency domain denoising module;

[0163] w / o MoE: remove the double-gated MoE module;

[0164] w / o ID_CL: remove the ID contrastive learning task;

[0165] w / o Hard_CL: remove the difficult negative sample generation and contrastive task;

[0166] w / o Seq_CL: remove the cross-modal sequence contrastive learning task;

[0167] As shown in Table 3, each core component plays a key role in driving significant recommendation performance; removing the dual-gated MoE module results in the most severe performance drop, with a 50.5% drop in MRR@10 on Home, confirming its status as a core pillar for fine-grained feature interaction modeling; removing the frequency domain denoising results in a 7.6% and 4.9% loss in NDCG@10 on Games and Beauty, respectively, highlighting the value of noise filtering in multi-modal encoding, ID contrastive learning plays a key role in ultra-sparse scenarios, for example, on the Home dataset (with a sparsity of 99.97%), removing it results in a 16.5% drop in MRR@10, and only an 8.3% drop on Games, difficult negative sample generation and cross-modal sequence contrastive learning contribute stable gains, with an average increase of 3.8%;

[0168] Table 3 Ablation experiment results table

[0169] ;

[0170] c. Parameter sensitivity analysis RQ3

[0171] As shown in FIG. 2, the recommendation performance shows a unimodal pattern with respect to the frequency domain retention ratio r; too low r<0.3 and too high r>0.6 values will cause information loss and noise retention, respectively, to reduce performance; it should be noted that the optimal r depends on the dataset: as shown in Figure 2 (a) the results on the Games dataset, Games reaches a peak NDCG@10=0.0724 at r=0.5, as shown in Figure 2 (b) the results on the Beauty dataset, Beauty maintains high performance NDCG@10>0.0500 in the range of r=0.4-0.6, as shown in Figure 2 (c) the results on the Home dataset, the sparse Home dataset requires a smaller retention ratio r=0.1, NDCG@10=0.0202, which proves the fundamental difference in noise characteristics in different domains.

[0172] As shown in FIG. 3, the optimal value of the number of input-dependent experts k1 in the hybrid MoE module shows significant dataset dependence: as shown in Figure 3 (a) the results on the Games dataset and Figure 3(b) shows the results on the Beauty dataset. Both the Games and Beauty datasets peak performance at k1=8: Games: MRR@10=0.0549, NDCG@10=0.0724; Beauty: MRR@10=0.0405, NDCG@10=0.0506, indicating a preference for uniform capacity in dense datasets; in contrast, as... Figure 3 (c) shows the results on the Home dataset. The sparse Home dataset requires significantly fewer experts, peaking at k1=6. This difference highlights that the input complexity of different tasks directly determines the capacity requirement of experts.

[0173] Figure 4 reveals the general principle for the number k2 of time-aware shared experts in a hybrid MoE module: such as Figure 4 (a) shows the results on the Games dataset. Figure 4 (b) shows the results on the Beauty dataset and Figure 4 (c) shows the results on the Home dataset. All datasets peak at k2=2. Games: NDCG@10=0.0731; beyond this threshold, performance monotonically decreases, and k2>4 leads to a significant drop, with Home dropping by as much as 12.3%. Crucially, sparse datasets suffer disproportionately at excessively large k2 values ​​because redundant experts introduce noise that disrupts the generalization of behavioral patterns.

[0174] As shown in Figure 5, the normalized weights λ of the auxiliary learning task have a significant impact on the model effectiveness: refer to formula (29), where , and Represents the normalized weights of auxiliary tasks, such as Figure 5 As shown in (a) Performance results on the Games dataset Figure 5 (b) Performance results on the Games dataset and Figure 5 (c) shown Performance results on the Games dataset show that excessively large weights λ>1.0 dominate and inhibit the learning of the main task, while excessively small weights λ<0.3 fail to utilize the benefits of auxiliary tasks, both leading to performance degradation. Experiments show that the optimal range is λ∈[0.3,0.8]. It is worth noting that the three auxiliary tasks show different degrees of optimization effects, as follows: ID comparison is the most effective > difficult negative sample generation is the most effective > cross-modal sequence comparison has the least impact.

[0175] The application proposes FT-MSR, a multi-modal sequence recommendation framework integrating frequency domain noise reduction and hybrid MoE, which realizes the most advanced performance; the key innovations of the research include: ID, text, image multi-modal feature noise reduction based on Fourier transform to enhance the robustness of feature coding; double-gated MoE architecture integrating input-dependent non-shared experts for adaptive multi-modal fusion and time-aware shared experts for modeling interest evolution through time coding, for fine-grained feature interaction and multi-modal feature adaptive fusion; triple auxiliary learning tasks ID contrast, difficult negative sample generation, cross-modal sequence contrast, jointly solving the data sparsity problem; a large number of experiments on three benchmark datasets Games, Beauty, Home prove the superiority of FT-MSR, which improves 3.29%-6.24% compared with the current most advanced model HM4SR, TedRec in NDCG@10 and MRR@10 indicators, and ablation experiments further verify the necessity of each module contained in FT-MSR.

[0176] The above examples are descriptions of the application, not limitations of the application, and any simple transformation of the application also belongs to the protection scope of the application.

Claims

1. A multi-modal sequential recommendation method based on a dual-gated hybrid expert model and Fourier denoising, characterized in that: The method comprises the following steps: S1, multi-modal feature coding: using a BERT / ViT pre-training model to extract text / visual features; S2, frequency domain feature denoising: using Fourier transform to filter frequency domain noise; S3, double-gated hybrid expert model: using parallel paths to realize adaptive multi-modal fusion and time interest modeling; S4, multi-task joint optimization: integrating triple auxiliary contrast learning to optimize the model; The step S3 double-gated hybrid expert model is constructed through the following three-stage process: S3-1, input-dependent Top-k expert: realizing adaptive multi-modal fusion through dynamic expert routing; S3-2, time-aware shared expert: setting a fixed number of shared experts, and explicitly modeling user interest evolution through time coding; S3-3, user preference modeling and score prediction: using a sequence modeling framework SASRec as the backbone network to model user preferences, wherein an independent SASRec encoder is used to process the feature sequence of each modality; aggregating multi-modal prediction results to obtain the predicted score between the user and the candidate item; The step S3-1 input-dependent Top-k expert comprises the following steps: S3-1-1, concatenating the feature vectors of the three modalities along the feature dimension; S3-1-2, constructing a MoE architecture, wherein each expert is an independent feedforward network with the same structure but different parameters; S3-1-3, calculating expert weights using a gating mechanism; S3-1-4, activating the Top-k experts with the highest weights to realize fine-grained feature interaction modeling; The step S3-2 time-aware shared expert comprises the following steps: S3-2-1, generating two types of complementary time embedding codes for a given interaction time sequence: absolute time coding: capturing user preferences at a specific time point; time interval coding: modeling the periodic characteristics of user behavior; S3-2-2, combining the above two embedding representations to represent the time context; S3-2-3, calculating the weight of each shared expert using a time gating mechanism; S3-2-4, modeling by all shared experts to obtain the final output; S3-2-5, integrating the modeling results of the above two paths to obtain the final item embedding; In the step S4 multi-task joint optimization, the main task is optimized by cross-entropy loss for interaction prediction, and the following three auxiliary contrast learning tasks are introduced to enhance representation learning and alleviate data sparsity: S4-1, ID contrast, constructing an ID contrast loss for each batch to maximize the similarity between the user sequence embedding and the interaction item embedding; S4-2, difficult negative sample generation, synthesizing difficult negative samples by linearly interpolating the normalized representations of ID and text modalities to enhance the discriminability of the model, obtaining an augmented training sample set, and constructing a contrast loss to maximize the similarity between the sequence representations of the same user in ID and text modalities; S4-3, cross-modal sequence contrast, given the text sequence embedding and the image sequence embedding, randomly mask the features of the text sequence embedding and replace them with the features at the same position in the image sequence embedding, obtain the augmented text sequence embedding, construct a contrastive loss to maximize the similarity between the augmented text sequence embedding and the original text sequence embedding of the same user, which enhances the quality of cross-modal representation, similarly, calculate the contrastive loss of the image modality, take the average of the two to integrate the auxiliary task with the main task to optimize the model parameters.

2. The multi-modal sequential recommendation method based on dual-gated hybrid expert model and Fourier denoising of claim 1, wherein: The multi-modal representation of each item in the step S1 multi-modal feature encoding is constructed through the following three-stage process: S1-1, ID modality embedding encoding: initialize an ID embedding matrix to map discrete item IDs to dense vectors; S1-2, text modality feature extraction: given the text description associated with the item, add a special token "CLS" in front of it to input the BERT encoder to learn the text embedding; S1-3, image modality feature extraction: divide the image associated with the item into blocks, serialize and splice the blocks, add a special token "CLS" in front of it, and input it into the ViT model to obtain the visual embedding.

3. The multi-modal sequential recommendation method based on dual-gated hybrid expert model and Fourier denoising of claim 2, wherein: In the step S1-3 image modality feature extraction, a learnable linear projection layer is introduced to convert both text and image embeddings into the ID embedding space for cross-modal feature alignment.

4. The multi-modal sequential recommendation method based on dual-gated hybrid expert model and Fourier denoising of claim 1, wherein: The step S2 frequency domain feature denoising includes the following steps: S2-1, given the embedding representation of each modality; S2-2, use the fast Fourier transform to convert the feature embedding from the time domain to the frequency domain; S2-3, calculate the frequency cutoff threshold according to the pre-defined retention ratio, and construct a binary mask matrix for spectral filtering; S2-4, reconstruct the denoised frequency domain representation back to the time domain through the inverse fast Fourier transform; S2-5, obtain the multi-modal feature embedding through layer normalization and residual connection.

Images