Multi-modal sequence recommendation method based on double-gating hybrid expert model and Fourier noise reduction

Abstract

The invention discloses a multi-modal sequence recommendation method based on a double-gating hybrid expert model and Fourier denoising. The method comprises the following steps: S1, multi-modal feature coding: extracting text / visual features by using a BERT / ViT pre-training model; s2, frequency domain feature denoising: carrying out frequency domain noise filtering by adopting Fourier transform; s3, establishing a double-gating hybrid expert model: adopting a parallel path to realize self-adaptive multi-modal fusion and time sequence interest modeling; and S4, multi-task joint optimization: performing joint optimization on the model by integrating triple auxiliary contrast learning, extracting text and image features by utilizing a pre-training model BERT / ViT, performing frequency domain noise reduction on multi-modal features by introducing Fourier transform, remarkably improving the robustness of modal representation, realizing fine-grained modal interaction through dynamic routing by an input dependent expert, and performing multi-task joint optimization. A shared expert uses time coding with a gating mechanism to model periodic evolution of user interests, the feature fusion quality is optimized, the FT-MSR integrates triple auxiliary learning tasks, and the problem of data sparsity is relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sequence recommendation, in particular to a multi-modal sequence recommendation method based on a double-gated hybrid expert model and Fourier denoising. BACKGROUND

[0002] The core role of a recommendation system is to filter out items of potential interest to users from a vast amount of data, thereby significantly improving user experience. As an important branch of recommendation systems, sequence recommendation aims to accurately predict the next interaction item of a user by modeling the temporal dependencies in the user interaction sequence. Early sequence recommendation models mainly rely on item ID embedding for preference modeling. Although such methods have high computational efficiency and can capture user preference patterns, they are essentially limited by data sparsity, resulting in limited performance.

[0003] A training method and device for a sequence recommendation model are disclosed in Chinese Patent No. 202211284644.9, which includes obtaining a historical behavior sequence of an associated object recommendation service, generating an original interest representation based on historical objects contained in the historical behavior sequence, calculating the object similarity between each historical object and a reference preference object based on the attention influence parameters of each historical object in at least one dimension, determining a target historical object in the historical behavior sequence based on the object similarity, and generating a counterfactual interest representation based on the target historical object. An optimized parameter pair is generated based on the original interest representation, the reference preference object corresponding to the reference preference representation, and the counterfactual interest representation, and the sequence recommendation model for the associated object recommendation service is trained based on the optimized parameter pair. The present application does not explicitly model the long-term and short-term preferences of users, cannot accurately generate personalized recommendations, does not fully utilize the application of item relationships in sequence recommendation, and has less than ideal recommendation performance when facing cold start problems or sparse data.

[0004] A sequence recommendation method and system based on contrast learning is disclosed in Chinese Patent Application No. 202311366797.2, which includes collecting historical interaction behavior sequence data of users and items, encoding according to embedding layers to obtain initial embedding representations of users and items that have interaction records with the users; inputting the initial embedding representations into a filter bank for filtering to obtain denoised embedding representations; obtaining user intent information through a clustering algorithm according to the behavior sequence data; using a sequence encoder based on a feedforward neural network as the main body in combination with the denoised embedding representations as the first training input; combining the user intent information and the output sequence representation of the sequence encoder through a contrast self-supervised algorithm as the second training input; using the first training input and the second training input simultaneously for training of the sequence recommendation model; obtaining a predicted score for final sequence prediction; the method has the disadvantage that the clustering algorithm cannot effectively distinguish fine-grained information of users, and when using the contrast learning technology, the construction of positive and negative sample pairs for items is easily affected by noise information of a single item, resulting in low recommendation accuracy.

[0005] A sequence recommendation data enhancement method based on graph contrast learning is disclosed in Chinese Patent Application No. 202211604474.8, which includes obtaining historical behavior sequences of users in real application scenarios and pre-processing the original data; according to the user behavior sequences obtained by pre-processing, the multi-order connection information between each item pair is counted, and then the weight of the edge between each item in the global graph is calculated and normalized according to the statistical information, and a threshold is set to filter noise information, and finally a global graph is constructed; for each item, neighborhood sampling is carried out to obtain two enhanced subgraphs with a certain probability, and LightGCN is used as an encoder, and shared parameters are used in different views; negative sampling is performed using the bucket clustering method to alleviate the negative effects of uneven distribution of item representations; interest extraction is performed using a capsule network; the prediction task and the contrast learning task are jointly optimized; the method has the disadvantage that when using the contrast learning technology, the construction of positive and negative sample pairs for items is easily affected by noise information of a single item, and when fusing embedding vectors output by multiple modules, a direct merging operation is performed, which cannot build connections between multiple semantic information, and the recommended item sequence provided for users is not accurate enough.

[0006] Multi-modal learning promotes the integration of semantic information and recommendation systems, providing a new way to solve the problem of sequence recommendation being limited by data sparsity, which limits performance. By integrating rich semantic information such as text and images, multi-modal recommendation significantly enhances user preference modeling capability. However, existing methods still face two major challenges: 1) severe modal noise interference; traditional methods have difficulty effectively filtering noise in multi-modal data, resulting in dilution of useful information; 2) neglecting temporal dynamics; existing feature fusion strategies often ignore fine-grained cross-modal relationships, resulting in semantic loss and feature bias; at the same time, the model relies too much on the behavior sequence itself, ignoring the temporal context information, making it difficult to capture the dynamic changes in user interest. SUMMARY

[0007] The purpose of the present application is to solve the problems in the prior art, and to provide a multi-modal sequence recommendation method based on a double-gated hybrid expert model and Fourier noise reduction, which can resist noise interference and data sparsity problems through time-frequency dual-driven learning.

[0008] To achieve the above purpose, the present application provides a multi-modal sequence recommendation method based on a double-gated hybrid expert model and Fourier noise reduction, comprising the following steps: S1, multi-modal feature encoding: using BERT / ViT pre-training model to extract text / visual features; S2, frequency domain feature denoising: using Fourier transform to filter frequency domain noise; S3, double-gated hybrid expert model: using parallel paths to realize adaptive multi-modal fusion and time-series interest modeling; S4, multi-task joint optimization: integrating triple auxiliary contrast learning to jointly optimize the model.

[0009] As a preferred, the multi-modal representation of each item in the step S1 multi-modal feature encoding is constructed through the following three-stage process: S1-1, ID modal embedding encoding: initializing an ID embedding matrix to map discrete item IDs to dense vectors; S1-2, text modal feature extraction: given the text description associated with the item, add a special token "CLS" in front of it to input the BERT encoder to learn the text embedding; S1-3, image modal feature extraction: block the image associated with the item, serialize and splice the blocks, and add a special token "CLS" in front of it to input the ViT model to get the visual embedding.

[0010] As a preferred, in the step S1-3 image modal feature extraction, a learnable linear projection layer is introduced to convert both text and image embeddings to the ID embedding space for cross-modal feature alignment.

[0011] As preferred, the step S2 frequency domain feature denoising comprises the following steps: S2-1, give the embedding representation of each modality; S2-2, convert the feature embedding from time domain to frequency domain using fast Fourier transform; S2-3, calculate the frequency cutoff threshold according to the predefined reservation ratio, and construct a binary mask matrix for spectral filtering; S2-4, reconstruct the denoised frequency domain representation back to the time domain by inverse fast Fourier transform; S2-5, get the multi-modal feature embedding by layer normalization and residual connection.

[0012] As preferred, the step S3 double-gated hybrid expert model is constructed by the following three-stage process: S3-1, input-dependent Top-k expert: realize adaptive multi-modal fusion through dynamic expert routing; S3-2, time-aware shared expert: set a fixed number of shared experts, and explicitly model the evolution of user interest through time coding; S3-3, user preference modeling and score prediction: use the sequence modeling framework SASRec as the backbone network to model user preference, where an independent SASRec encoder is used to process the feature sequence of each modality; aggregate the multi-modal prediction results to get the predicted score between the user and the candidate item.

[0013] As preferred, the step S3-1 input-dependent Top-k expert comprises the following steps: S3-1-1, concatenate the feature vectors of the three modalities along the feature dimension; S3-1-2, construct a MoE architecture, where each expert is an independent feedforward network with the same structure but different parameters; S3-1-3, calculate the expert weight using the gating mechanism; S3-1-4, activate the Top-k experts with the highest weight to realize fine-grained feature interaction modeling.

[0014] As preferred, the step S3-2 time-aware shared expert comprises the following steps: S3-2-1, generate two types of complementary time embedding codes for a given interaction time sequence: absolute time coding: capture the user's preference at a specific time point; time interval coding: model the periodic characteristics of user behavior; S3-2-2, combine the above two embedding representations of time context; S3-2-3, calculate the weight of each shared expert using the time gating mechanism; S3-2-4, collaborative modeling by all shared experts, and obtaining the final output; S3-2-5, integrating the modeling results of the above two paths to obtain the final project embedding.

[0015] As a preferred, in the step S4 multi-task joint optimization, the main task optimizes the interaction prediction through the cross-entropy loss, and the following three auxiliary contrast learning tasks are introduced to enhance the representation learning and alleviate the data sparsity: S4-1, ID contrast, constructing an ID contrast loss for each batch to maximize the similarity between the user sequence embedding and the interaction item embedding thereof; S4-2, difficult negative sample generation, synthesizing difficult negative samples by linearly interpolating the normalized representations of the ID and text modalities to enhance the discriminability of the model, obtaining an augmented training sample set, and constructing a contrast loss to maximize the similarity between the sequence representations of the same user in the ID and text modalities; S4-3, cross-modal sequence contrast, given the text sequence embedding and the image sequence embedding, randomly masking the features of the text sequence embedding and replacing them with the features at the same position in the image sequence embedding to obtain an augmented text sequence embedding, and constructing a contrast loss to maximize the similarity between the augmented text sequence embedding and the original text sequence embedding of the same user to enhance the quality of cross-modal representation, and similarly, the contrast loss of the image modality is calculated, and the total loss is the average of the two, and the auxiliary task and the main task are integrated to optimize the model parameters.

[0016] The present application has the following beneficial effects: the present application uses the pre-training model BERT / ViT to extract text and image features, introduces Fourier transform for frequency domain noise reduction of multi-modal features, significantly improves the robustness of modal representation, integrates the double-gated MoE architecture of input-dependent non-shared experts and time-aware shared experts, the input-dependent experts realize fine-grained modal interaction through dynamic routing, the shared experts model the periodic evolution of user interest by using time coding with gating mechanism, optimize the feature fusion quality, and the FT-MSR integrates three auxiliary learning tasks: ID contrast learning to strengthen project representation, difficult negative sample generation to optimize the decision boundary, and cross-modal sequence contrast learning to promote modal semantic alignment, which alleviates the problem of data sparsity.

[0017] The features and advantages of the present application will be described in detail in conjunction with the embodiments and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the overall architecture diagram of the multi-modal sequence recommendation method based on the double-gated hybrid expert model and Fourier noise reduction of the present application; Figure 2This is a graph showing the effect of frequency domain retention ratio on recommendation performance in a parameter sensitivity analysis experiment of the multimodal sequence recommendation method based on a dual-gated hybrid expert model and Fourier denoising in the present invention; Figure 3 This is a graph showing the effect of the number of input perception experts on recommendation performance in a parameter sensitivity analysis experiment of the multimodal sequence recommendation method based on a dual-gated hybrid expert model and Fourier denoising in the present invention; Figure 4 This is a graph showing the effect of the number of time-series-aware experts on recommendation performance in a parameter sensitivity analysis experiment of the multimodal sequence recommendation method based on a dual-gated hybrid expert model and Fourier denoising in the present invention. Figure 5 This is a result diagram of the impact of loss weight on recommendation performance in the parameter sensitivity analysis experiment of the multimodal sequence recommendation method based on the dual-gated hybrid expert model and Fourier denoising in the present invention. DETAILED DESCRIPTION

[0019] Overall framework This paper proposes a multimodal sequence recommendation method based on a dual-gated hybrid expert model and Fourier denoising. Its overall architecture is shown in Figure 1. The model consists of four core components: multimodal feature encoding: using the BERT / ViT pre-trained model to extract text / visual features; frequency domain feature denoising: using Fourier transform to filter frequency domain noise; dual-gated hybrid expert model: using parallel paths to achieve adaptive multimodal fusion and temporal interest modeling; multi-task joint optimization: integrating triple-assisted contrastive learning to jointly optimize the model.

[0020] Multimodal feature encoding Each project The multimodal representation of is constructed through a three-stage process: ID modality embedding encoding: Initialize an ID embedding matrix , mapping discrete project IDs to dense vectors ; Textual modality feature extraction: given a text description associated with an item , add the special token "CLS" in front of it and input it into the BERT encoder to learn text embedding: (1), in Represents string concatenation operation; Image modality feature extraction: Divide the image associated with the item into blocks , serialize and concatenate the blocks, add a special tag "CLS" in front, and input the ViT model to obtain visual embedding: (2), To achieve cross-modal feature alignment, a learnable linear projection layer is introduced to convert both text and image embeddings into an ID embedding space: (3), where and are weight matrices, and are bias terms.

[0021] Frequency domain feature denoising Multimodal features are often susceptible to noise interference: text modality may distort semantic expression due to redundant advertisements or spelling errors, while visual modality may cause image distortion due to cluttered background or low resolution. To enhance the robustness of feature embedding, the present application uses Fourier transform to realize frequency domain noise filtering, the specific process is as follows: Given the embedding representation of each modality where , B denotes the batch size, L denotes the sequence length, D denotes the embedding dimension; first, the feature embedding is converted from time domain to frequency domain using Fast Fourier Transform (FFT): (4), where denotes the spectrum matrix, denotes the frequency vector; Then, according to the pre-defined reservation ratio r the frequency truncation threshold is calculated to construct a binary mask matrix for spectrum filtering: (5), (6), (7), where is the unit indicator function (all elements equal to 1), denotes element-wise multiplication; Next, the denoised frequency domain representation is reconstructed back to the time domain by Inverse FFT (IFFT): (8), Finally, the multimodal feature embedding is obtained through layer normalization and residual connection: (9).

[0022] Dual-gating hybrid expert model (1) Input-dependent Top-k expert This module realizes adaptive multi-modal fusion through dynamic expert routing, and the process is as follows: First, the feature vectors of the three modalities are spliced along the feature dimension: (10), Then, a MoE architecture is constructed, and each expert is an independent feedforward network with the same structure but different parameters: (11), Among them, and represent the trainable parameters of the i expert network, represent the activation function; Next, the expert weights are calculated using the gating mechanism: (12), Among them and are the trainable parameters of the i gating network; Finally, the Top-k expert with the highest weight is activated to realize fine-grained feature interaction modeling: (13), Among them represent the number of input-dependent experts, i.e. k value in Top-k; (2) Time-aware shared expert This module contains a fixed number of shared experts, which explicitly model the evolution of user interest through time encoding; for a given interaction time sequence , two types of complementary time embedding encodings are generated: Absolute time encoding: capture the user's preference at a specific time point (14), Among them represent the wavelength of dimension j , T is the maximum timestamp, d is the embedding dimension; Time interval encoding: modeling the periodic characteristics of user behavior ​ (15), where denotes the time interval between events, is a smoothing factor, is a scaling factor; The temporal context combines the above two embedding representations: (16), The weight of each shared expert is calculated using a temporal gating mechanism: (17), where and denote the trainable parameters of the i th gating network; The final output is obtained by jointly modeling from all shared experts: (18), where denotes the i th shared expert network, denotes the number of shared experts; The modeling results of the above two paths are integrated to obtain the final item embedding: (19).

[0023] User preference modeling and rating prediction: The sequence modeling framework SASRec is used as the backbone network to model user preferences, where an independent SASRec encoder is used to process the feature sequence of each modality: (20), where denotes the feature sequence of a specific modality, denotes the encoder parameters, is the user preference vector under the modality m ; The predicted ratings between users and candidate items are obtained by aggregating the multi-modal prediction results: (21), where denotes the candidate item embedding, is the Sigmoid activation function.

[0024] Multi-task joint optimization In the FT-MSR model, the main task optimizes the interaction prediction through cross-entropy loss: (22), where denotes the predicted interaction probability of item v , denotes the true interaction label, denotes the candidate item set.

[0025] To enhance representation learning and alleviate the data sparsity problem, three auxiliary contrastive learning tasks are introduced: (1) ID contrast For each batch , an ID contrast loss is constructed to maximize the similarity between the user sequence embedding and its interaction item embedding: (23), where denotes the ID vector of user i , denotes the ID embedding of its interaction item, denotes the cosine similarity, is the temperature scaling factor; (2) Difficult negative sample generation To enhance the discriminability of the model, difficult negative samples are synthesized by linearly interpolating the ID and text modal normalized representations: (24), where denotes L2 normalization, is a random mixing coefficient, denotes stop gradient update; Now, an augmented training sample set is obtained: (25), where and denote the original samples in the ID and text modal, respectively; A contrastive loss is constructed to maximize the similarity between the sequence representations of the same user in the ID and text modal: (26) ; (3) Cross-modal sequence contrast Given the text sequence embedding and the image sequence embedding , some features of the random mask are replaced using the features at the same position in to obtain the augmented text sequence embedding .

[0026] To enhance the quality of cross-modal representation, a contrastive loss is constructed to maximize the similarity between augmented text sequence embeddings and original text sequence embeddings of the same user: (27), Contrastive loss for image modality The total loss is the average of the two: (28), Finally, auxiliary tasks are integrated with the main task to optimize model parameters: (29), where , and denote the normalized weights of auxiliary tasks.

[0027] Experiments This study conducts experimental tests based on the RecBole framework to address the following research questions: RQ1: How does the performance of FT-MSR compare to the current state-of-the-art models? RQ2: How do the core components of FT-MSR contribute to performance improvement? RQ3: How do the key hyperparameters of FT-MSR affect model performance? (1) Datasets Three public Amazon datasets, Beauty, Games, and Home, are used in the experiments, which cover different application scenarios; the dataset statistics are shown in Table 1: Table 1. Statistics of datasets ; (2) Baseline models Eight representative baseline models are selected, including traditional sequence recommendation models: SASRec, GRU4Rec, LRURec, and UniSRec, and multi-modal sequence recommendation models: MISSRec, IISAN, TedRec, and HM4SR; (3) Evaluation metrics and experimental settings The normalized discounted cumulative gain NDCG@K and the mean reciprocal rank MRR@K are used as evaluation indexes, wherein K e {5, 10, 20}; the experiment process follows the leave-one-out method, that is, the last interaction of each sequence is reserved for testing; all models are trained on an NVIDIA RTX 4090 GPU 24GB, and the experimental configuration is as follows: the batch size is 1024, the hidden dimension is 64, the maximum sequence length is 50, the Adam optimizer is used, the learning rate is 1x10 -3 , the dropout rate is 0.5; the hyperparameter tuning range is as follows: the frequency domain reservation ratio r e [0.1, 0.9] with a step of 0.1, the number of input dependent experts k1 e {4, 6, 8, 10, 12}, the number of time-aware experts k2 e {2, 4, 6, 8, 10}, and the loss weight with a step of 0.1; (4) Experimental results a. Comparative experiment RQ1 As shown in Table 2, the traditional sequence recommendation models GRU4Rec, SASRec and LRURec have poor performance due to the dependence on a single ID modality; the multi-modal models UniSRec and TedRec have limited performance improvement due to the modal noise and coarse-grained feature fusion although they fuse text / visual information; the model FT-MSR of the application is significantly better than all baseline models, with an NDCG@10 improvement of 5.94% on Games and an MRR@10 improvement of 3.80% on Home; the reason is that the frequency domain denoising effectively filters the noise in the modal representation, the double-gated MoE realizes fine-grained modal fusion and interest evolution modeling, and the auxiliary contrastive learning task alleviates the data sparsity problem, further improving the recommendation performance; Table 2 Performance comparison table of different methods ; b. Ablation experiment RQ2 In order to verify the necessity and contribution of the core components of FT-MSR, five variant models are constructed and evaluated: w / o Denoise: remove the frequency domain denoising module; w / o MoE: remove the double-gated MoE module; w / o ID_CL: remove the ID contrastive learning task; w / o Hard_CL: remove the difficult negative sample generation and contrastive task; w / o Seq_CL: remove the cross-modal sequence contrastive learning task; As shown in Table 3, each core component plays a key role in driving significant recommendation performance; removing the double-gated MoE module results in the most severe performance drop, with a 50.5% drop in MRR@10 on Home, confirming its status as a core pillar for fine-grained feature interaction modeling; removing the frequency domain denoising results in a 7.6% and 4.9% loss in NDCG@10 on Games and Beauty, respectively, highlighting the value of noise filtering in multi-modal encoding, ID contrastive learning plays a key role in ultra-sparse scenarios, for example, on the Home dataset (with a sparsity of 99.97%), removing it results in a 16.5% drop in MRR@10, and only an 8.3% drop on Games, difficult negative sample generation and cross-modal sequence contrastive learning contribute stable gains, with an average increase of 3.8%; Table 3 Ablation experiment results table ; c. Parameter sensitivity analysis RQ3 As shown in FIG. 2, the recommendation performance shows a unimodal pattern with respect to the frequency domain retention ratio r; too low r<0.3 and too high r>0.6 values will cause information loss and noise retention, respectively, to reduce performance; it should be noted that the optimal r depends on the dataset: as shown in Figure 2 (a) the results on the Games dataset, Games reaches a peak NDCG@10=0.0724 at r=0.5, as shown in Figure 2 (b) the results on the Beauty dataset, Beauty maintains high performance NDCG@10>0.0500 within the range of r=0.4-0.6, as shown in Figure 2 (c) the results on the Home dataset, the sparse Home dataset requires a smaller retention ratio r=0.1, NDCG@10=0.0202, which proves the fundamental difference in noise characteristics in different domains.

[0028] As shown in FIG. 3, the optimal value of the number of input-dependent experts k1 in the hybrid MoE module shows significant dataset dependence: as shown in Figure 3 (a) the results on the Games dataset and Figure 3 (b) the results on the Beauty dataset, Games and Beauty datasets reach peak performance at k1=8 Games: MRR@10=0.0549, NDCG@10=0.0724; Beauty: MRR@10=0.0405, NDCG@10=0.0506, indicating a preference for uniform capacity for dense datasets; in contrast, as shown in Figure 3(c) shows the results on the Home dataset. The sparse Home dataset requires significantly fewer experts, reaching a peak when k1=6; this difference highlights that the input complexity of different tasks directly determines the capacity requirement of experts.

[0029] Figure 4 reveals the general principle of the number of time-aware shared experts k2 in the hybrid MoE module: Figure 4 (a) shows the results on the Games dataset. Figure 4 (b) shows the results on the Beauty dataset and Figure 4 (c) Shows the results on the Home dataset. All datasets peak at k2=2, Games: NDCG@10=0.0731; beyond this threshold, performance decreases monotonically, and k2>4 leads to a significant drop of up to 12.3% for Home; critically, sparse datasets suffer disproportionately from excessively large k2 values ​​because redundant experts introduce noise that undermines the generalization of behavioral patterns.

[0030] As shown in Figure 5, the normalized weight λ of the auxiliary learning task has a significant impact on the effectiveness of the model: refer to formula (29), where 、 and represents the normalized weight of the auxiliary task, such as Figure 5 (a) Performance results on the Games dataset, Figure 5 (b) Performance results on the Games dataset and Figure 5 (c) Performance results on the Games dataset show that a weight λ > 1.0 that is too large will dominate and inhibit the learning of the main task, while a weight λ < 0.3 that is too small will fail to utilize the benefits of the auxiliary task, both of which will lead to performance degradation; the optimal range proven by experiments is λ∈[0.3, 0.8]; it is worth noting that the three auxiliary tasks show different degrees of optimization effects, as follows: ID comparison is the most effective > difficult negative sample generation is the most effective > cross-modal sequence comparison has the least impact.

[0031] The application proposes FT-MSR, a multi-modal sequence recommendation framework integrating frequency domain noise reduction and hybrid MoE, which realizes the most advanced performance; the key innovations of the research include: ID, text, image multi-modal feature noise reduction based on Fourier transform to enhance the robustness of feature coding; double-gated MoE architecture integrating input-dependent non-shared experts for adaptive multi-modal fusion and time-aware shared experts for modeling interest evolution through time coding, for fine-grained feature interaction and multi-modal feature adaptive fusion; triple auxiliary learning tasks ID contrast, difficult negative sample generation, cross-modal sequence contrast, jointly solving the data sparsity problem; a large number of experiments on three benchmark datasets Games, Beauty, Home prove the superiority of FT-MSR, which improves 3.29%-6.24% compared with the current most advanced model HM4SR, TedRec in NDCG@10 and MRR@10 indicators, and ablation experiments further verify the necessity of each module contained in FT-MSR.

[0032] The above examples are illustrative of the application and are not limiting of the application, any simple modifications of the application are within the scope of the application.

Claims

1. A multimodal sequence recommendation method based on a dual-gated hybrid expert model and Fourier denoising, characterized by: The following steps are involved: S1. Multimodal feature encoding: Extract text / visual features using BERT / ViT pre-trained models; S2, frequency domain feature denoising: frequency domain noise filtering using Fourier transform; S3, dual-gated hybrid expert model: uses parallel paths to achieve adaptive multimodal fusion and temporal interest modeling; S4. Multi-task joint optimization: Integrate triple-assisted contrastive learning to jointly optimize the model.

2. The multimodal sequence recommendation method based on a dual-gated hybrid expert model and Fourier denoising according to claim 1, characterized in that: The multimodal representation of each item in the multimodal feature encoding step S1 is constructed through the following three-stage process: S1-1. ID modal embedding encoding: Initialize an ID embedding matrix to map discrete item IDs into dense vectors; S1-2, Text Modality Feature Extraction: Given a text description associated with an item, add a special token "CLS" in front of it and input it into the BERT encoder to learn text embedding; S1-3. Image modality feature extraction: Divide the images associated with the project into blocks, serialize and concatenate the blocks, add a special tag "CLS" in front of them, and input them into the ViT model to obtain visual embedding.

3. The multimodal sequence recommendation method based on dual-gated hybrid expert model and Fourier denoising according to claim 2, characterized in that: In the image modality feature extraction step S1-3, a learnable linear projection layer is introduced to convert both text and image embeddings into the ID embedding space for cross-modal feature alignment.

4. The multimodal sequence recommendation method based on dual-gated hybrid expert model and Fourier denoising according to claim 1, characterized in that: The step S2 of frequency domain feature denoising comprises the following steps: S2-1, given the embedding representation of each modality; S2-2, use fast Fourier transform to convert feature embedding from time domain to frequency domain; S2-3, calculating the frequency truncation threshold according to the predefined retention ratio and constructing a binary mask matrix for spectrum filtering; S2-4, reconstructing the denoised frequency domain representation back to the time domain through inverse fast Fourier transform; S2-5. Multimodal feature embedding is obtained through layer normalization and residual connection.

5. The multimodal sequence recommendation method based on dual-gated hybrid expert model and Fourier denoising according to claim 1, characterized in that: The dual-gated hybrid expert model in step S3 is constructed through the following three-stage process: S3-1, Input-dependent Top-k Experts: Adaptive multimodal fusion through dynamic expert routing; S3-2, time-aware shared experts: set a fixed number of shared experts and explicitly model the evolution of user interests through time encoding; S3-3. User preference modeling and rating prediction: The sequence modeling framework SASRec is used as the backbone network to model user preferences, where an independent SASRec encoder is used to process the feature sequence of each modality; the multimodal prediction results are aggregated to obtain the predicted rating between the user and the candidate item.

6. The multimodal sequence recommendation method based on dual-gated hybrid expert model and Fourier denoising according to claim 5, characterized in that: The step S3-1 input-dependent Top-k expert includes the following steps: S3-1-1. Concatenate the characteristic vectors of the three modes along the characteristic dimension; S3-1-2. Build a MoE architecture where each expert is an independent feedforward network with the same structure but different parameters. S3-1-3, calculate expert weights using a gating mechanism; S3-1-4. Activate the top-k experts with the highest weights to achieve fine-grained feature interaction modeling.

7. The multimodal sequence recommendation method based on dual-gated hybrid expert model and Fourier denoising according to claim 5, characterized in that: The step S3-2 of sharing time-aware experts includes the following steps: S3-2-1. Generate two complementary time embedding codes for a given interaction time series: absolute time code: captures user preferences at a specific time point; time interval code: models the periodic characteristics of user behavior; S3-2-2, combines the above two embeddings to represent the temporal context; S3-2-3. Use the time gating mechanism to calculate the weight of each shared expert; S3-2-4, all shared experts collaborate on modeling to obtain the final output; S3-2-5. Integrate the modeling results of the above two paths to obtain the final project embedding.

8. The multimodal sequence recommendation method based on dual-gated hybrid expert model and Fourier denoising as claimed in claim 1, characterized in that: In the multi-task joint optimization in step S4, the main task optimizes interactive prediction through cross entropy loss, and introduces the following three auxiliary contrastive learning tasks to enhance representation learning and alleviate data sparsity: S4-1, ID comparison, for each batch, an ID comparison loss is constructed to maximize the similarity between the user sequence embedding and its interaction term embedding; S4-2, difficult negative sample generation: By linearly interpolating the normalized representations of the ID and text modalities, we synthesize difficult negative samples to enhance the discriminative power of the model, obtain an augmented training sample set, and construct a contrastive loss to maximize the similarity between the sequence representations of the ID and text modalities of the same user; S4-3. Cross-modal sequence comparison. Given a text sequence embedding and an image sequence embedding, randomly mask the features of the text sequence embedding and replace them with the features at the same position in the image sequence embedding to obtain an augmented text sequence embedding. Construct a contrast loss to maximize the similarity between the augmented text sequence embedding and the original text sequence embedding of the same user to enhance the quality of cross-modal representation. Similarly, calculate the contrast loss of the image modality, and take the average of the two for the total loss. Integrate the auxiliary task with the main task to optimize the model parameters.

Images