Multi-modal sequence recommendation method based on double-layer gating fusion
By acquiring multimodal features from users' historical interaction records, and utilizing lightweight graph convolutional networks and two-layer gated fusion technology, the problem of unstable feature fusion in multimodal recommendation is solved, achieving stability and personalization of recommendation results and improving recommendation accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-03-27
AI Technical Summary
Existing recommendation methods struggle to establish close connections in multimodal content information processing, and are difficult to integrate visual and textual features, resulting in unstable recommendation results that fail to reflect users' true preferences. Furthermore, they lack sufficient correlation and are not sensitive to time sequence during the ranking process.
By acquiring multimodal features from users' historical interaction records, lightweight graph convolutional networks are used to generate user embedding data, construct temporal preference vectors for visual and textual modalities, and generate personalized modal fusion features through two-layer gating fusion. Finally, predicted preference scores are calculated and ranked.
It achieves stability and continuity of recommendation results in complex multimodal scenarios, reflects user behavior characteristics, maintains consistency between the recommendation list and user preferences, and improves the accuracy and personalization of recommendations.
Smart Images
Figure CN121743582A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of information recommendation, and in particular to a multi-modal sequence recommendation method based on double-layer gate fusion. BACKGROUND
[0002] With the continuous enrichment of user interaction behaviors with items in the recommendation scene, multi-modal information such as image content and text content of items gradually becomes an important factor affecting the preference modeling effect in the recommendation task. With the characteristics of multi-time sequence, multi-modal, and cross-content source of user behaviors, the recommendation process needs to capture content semantic features, interaction structure features, and behavior sequence features at the same time, so that the content expression based on visual original features and text original features can jointly act on preference inference together with the collaborative relationship based on user interaction structure. However, in actual scenarios, multi-modal content often has problems such as semantic differences, modal noise, and inconsistent cross-modal expression, which makes the recommendation method need to have structured extraction ability and dynamic expression ability when dealing with complex item content information.
[0003] Currently, in the process of using visual original features and text original features to express item content and combining user embedding data for recommendation, it is often difficult to establish a close relationship between content information and collaborative relationship. Because the content features come from different modalities and have a large semantic span, it is often difficult to form a stable and consistent expression in the embedding space after multi-modal mapping, which makes it difficult for user embedding data generated based on user interaction structure to form a corresponding relationship with content side features, so that the recommendation result is easily affected by the differences or expression imbalance between modalities.
[0004] Secondly, when constructing the time preference of users for historical interaction item sequences in visual modalities and text modalities, it is often difficult to dynamically integrate the time preference formed by users in different modalities into a unified expression. Because the time variation features of visual modalities and text modalities are not consistent, and the sensitivity of behavior changes to different modalities is different, the time preference vector is prone to modal dominant deviation, modal contribution imbalance, or invalid time features being amplified in the fusion process, thereby weakening the ability to depict the real modal preference of users in the recommendation process.
[0005] In addition, in the process of generating predicted preference scores based on user embedding data and content expression and performing ranking on candidate items, it is generally difficult to simultaneously reflect the combined effect of collaborative relationship, modal content expression, and user time preference. Because features from a single source cannot cover multi-dimensional information of user behaviors, it leads to insufficient content correlation, weak time sensitivity, or insufficient user preference feature expression in the ranking process, so that the final recommendation list is difficult to maintain stability and continuity consistent with the real preference when facing multi-modal content differences and user behavior changes. SUMMARY
[0006] To solve the above technical problems, the application provides a multi-modal sequence recommendation method based on double-layer gate fusion, which comprises the following steps: S11, obtaining original multi-modal features of interactive items in user historical interaction records and performing mapping processing to obtain modal embedding data; S12, constructing a user-item bipartite graph and performing high-order collaborative signal propagation through a light graph convolution network to generate user embedding data and item embedding data; S13, performing sorting processing based on interaction timestamps to obtain a historical interactive item sequence, and constructing a time sequence preference vector of a user in a visual mode and a text mode to generate a behavior adaptation vector; S14, performing double-layer gate fusion based on the behavior adaptation vector of the user and the modal embedding data of the item to generate personalized modal fusion features; S15, calculating a predicted preference score based on the user embedding data and the personalized modal fusion features, sorting candidate items according to the predicted preference score, selecting the top K items with the highest scores to generate a recommendation result, and outputting a recommendation list.
[0007] Further, the original multi-modal features include visual original features and text original features, and the modal embedding data includes visual feature intermediate embedding data and text feature intermediate embedding data.
[0008] Further, the step of obtaining modal embedding data comprises: S111, obtaining historical interaction records corresponding to the user identifier based on the user identifier, extracting interactive items in the historical interaction records and constructing an interactive item set; S112, performing feature extraction on each item image in the interactive item set based on a pre-trained visual ViT model to obtain visual original features; S113, performing feature extraction on each item text in the interactive item set based on a pre-trained text BERT model to obtain text original features; S114, mapping the original multi-modal features to a latent space with a unified dimension through linear transformation and a ReLU activation function to obtain modal embedding data.
[0009] Further, the step of constructing a user-item bipartite graph comprises: S121, initializing user embedding vectors and item embedding vectors based on user identifiers and corresponding interactive items; S122, constructing a user-item bipartite graph according to a user identifier set and a corresponding interactive item set.
[0010] Furthermore, the steps for generating user embedded data and item embedded data are as follows: S123, On the user-item bipartite graph, a lightweight graph convolutional network is used to perform high-order collaborative feature propagation to update the user embedding vector and the item embedding vector. S124 uses the propagation output of the last layer of the graph convolutional network as the enhanced user embedding data and item embedding data.
[0011] Furthermore, the logic for generating the historical interaction item sequence is as follows: Based on the user identifier set and the interactive item set, for any user identifier, the corresponding partial set of interactive items is retrieved, and the items are sorted from the earliest to the most recent interaction according to the interaction timestamp of each item to form a historical interactive item sequence.
[0012] Furthermore, the steps for generating behavior adaptation vectors are as follows: S131, Construct historical feature sequences for visual and textual modalities; S132, input the historical feature sequences of the visual modality and the historical feature sequences of the text modality into the sequence encoding structure to generate the user's temporal preference vector in the visual modality and the text modality; S133, perform a fusion operation on the visual modality temporal preference vector and the text modality temporal preference vector to generate a behavior adaptation vector.
[0013] Furthermore, the steps for generating personalized modality fusion features are as follows: S141, Based on a given static gating network, generate static fusion features; S142, Based on a preset adaptive gating network, generate adaptive fusion features of user-item pairs; S143 integrates static fusion features and adaptive fusion features based on a linear combination method to generate personalized modal fusion features.
[0014] Furthermore, the method also includes: S16 is based on the constructed user-positive-negative sample triplet, and optimizes all model parameters together through BPR loss and L2 regularization.
[0015] Furthermore, the steps to optimize the model parameters are as follows: S161, Record items that the user has actually interacted with as positive sample items, and items that the user has not interacted with as negative sample items, and construct user-positive sample-negative sample triplet based on user identifier, positive sample items and negative sample items; S162, calculate and obtain the predicted preference scores corresponding to positive sample items and negative sample items, and construct the BPR loss function based on the predicted preference scores; S163 introduces an L2 regularization term to optimize the BPR loss function, resulting in the final loss function. The Adam optimizer is then used to update the parameters of the final loss function.
[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention obtains original visual and textual features based on user historical interaction records and generates modal embedding data, enabling the recommendation process to form a close relationship between item content information and collaborative relationships. Since the higher-order collaborative relationships established by further utilizing user embedding data and the set of interactive items can continuously reflect changes in the user's interest structure, the recommendation preference expression based on personalized information can maintain stability and continuity in complex usage scenarios. Furthermore, this invention constructs a behavior adaptation vector based on temporal preference vectors of visual and textual modalities, and drives the synergistic effect of static fusion features and adaptive fusion features, so that the temporal preferences formed by users under different modalities can be effectively expressed in a unified latent space. This enables personalized modal fusion features to have targeted adaptation capabilities to user interest shifts, semantic differences between modalities and user historical behavior structures, thus forming a more user-feature-related expression basis for predicting preference scores. Furthermore, this invention calculates predicted preference scores based on personalized modal fusion features and user embedded data, and sorts candidate items using the predicted preference scores. This allows the recommendation results to directly reflect the preference structure under the combined influence of cross-modal content, user collaborative relationships, and temporal preferences. Through the output of this result, the recommendation list can maintain its recommendation capability that matches user behavior characteristics even under complex changes in multimodal information. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0018] Figure 1 This is a flowchart of a multimodal sequence recommendation method based on two-layer gated fusion provided in Embodiment 1 of the present invention; Figure 2 The flowchart shows a multimodal sequence recommendation method based on two-layer gating fusion provided in Embodiment 2 of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Example 1 Please see Figure 1 As shown in the figure, this embodiment discloses a multimodal sequence recommendation method based on two-layer gating fusion, the method comprising: S11, Obtain the original multimodal features of interactive items in the user's historical interaction records, and perform mapping processing to obtain modal embedding data; Specifically, the original multimodal features include original visual features and original text features, and the modal embedding data includes intermediate embedding data of visual features and intermediate embedding data of text features; In one specific embodiment, the steps for obtaining modality embedding data include: S111, Based on the user identifier, obtain the historical interaction records corresponding to the user identifier, extract the interactive items in the historical interaction records, and construct an interactive item set; In one specific embodiment, the user identifier is retrieved based on a set of user identifiers stored in the cloud, wherein the set of user identifiers is represented as follows: In the formula, For a set of user identifiers, The total number of user identifiers stored in the user identifier set. For the first Individual user identifier; It should be noted that each user identifier corresponds to a unique user; Based on the For each user identifier, a corresponding historical interaction record is constructed, creating a partial set of items that record that user's interactions, represented as follows: In the formula, User Identifier A partial collection of user-interactive items. The user identifier is The first user interaction One item; Construct an interactive item set by using local collections of interactive items from different users; Represented as: in, A collection of interactive items.
[0021] S112, Based on the pre-trained visual ViT model, feature extraction is performed on the images of each item in the interactive item set to obtain the original visual features; In a preferred embodiment, the ViT model is a ViT-Base model, where each item image is scaled to 224×224 as input image and normalized according to the standard ImageNet mean and standard deviation. Represented as: In the formula, These are the normalized pixel values. The input image contains pixel values in a specific channel (R / G / B). The standard ImageNet mean. The standard deviation of the corresponding channel in the ImageNet dataset; The ViT model employs a 16×16 image patch splitting method, using the output of the last Transformer layer of the CLS token as the original visual features. The dimension of each image in the original visual features is fixed at 768. It is important to understand that in this embodiment, the parameters of the pre-trained ViT model remain frozen during the training of the recommendation model and do not participate in gradient updates. It should be noted that: Typically an integer between 0 and 255, or a floating-point number scaled to 0 to 1; S113, Based on the pre-trained text BERT model, feature extraction is performed on the text of each item in the interactive item set to obtain the original text features; In a preferred embodiment, the BERT model is a BERT-Base model, which segments the text of each item according to the model requirements and converts it into a corresponding token sequence; then, the token sequence is padded or truncated according to a fixed maximum sequence length, and the text tokens are processed using a standardized numerical encoding method to form a text vector sequence that can be input into the BERT model. Represented as: In the formula, The token sequence is the result of text segmentation and encoding. To input text content, Standard word segmentation and encoding functions that match BERT-Base; The BERT model uses the CLStoken representation from the output of its last Transformer layer as the global semantic feature vector of the text, denoted as the original text feature. In this embodiment, the dimension of the global semantic feature vector is fixed at 768. It is important to understand that in this embodiment, the pre-trained BERT model parameters remain frozen during the recommendation model training process and do not participate in gradient updates. It should be noted that the text token sequence needs to be supplemented with a corresponding attention mask before being input into the BERT model to indicate the valid tokens and the padding positions, and to ensure that the Transformer structure can accurately capture the semantic information of the valid text.
[0022] S114 uses linear transformation and ReLU activation function to map the original multimodal features to a unified dimension latent space to obtain modality embedding data; In a preferred embodiment, the dimension of the unified latent space is set to 128; two independent linear mapping structures are used to process the original visual features and the original text features respectively. Represented as: In the formula, Intermediate embedding data for visual features in modal embedding data. ∈ , Intermediate data is embedded to embed text features in modal embedding data. , These are learnable linear transformation matrices for visual features and text features, respectively, with an input dimension of 768 and an output dimension of 128. , These are the bias terms used for visual features and text features, respectively. For activation functions; It is important to understand that two independent linear mapping structures are used to process the original visual features and the original text features respectively to ensure that the expressive characteristics of each modality are effectively preserved and that the embedding dimension after projection remains consistent. The dimension of the unified latent space is set to 128, which balances the expressive power of the model with the computational efficiency.
[0023] S12, construct a user-item bipartite graph, and use a lightweight graph convolutional network to propagate high-order collaborative signals to generate user embedding data and item embedding data; In a specific embodiment, the steps for constructing a user-item bipartite graph are as follows: S121, Based on the user identifier and the corresponding interactive item, initialize the user embedding vector and the item embedding vector; Based on the user identifier set in step S11 and interactive item collection ; Let the user ID be , ∈ For each user identifier Initialize learnable user embedding vectors And an identifier for each interactive item in the collection of interactive items. Initialize learnable item embedding vectors ; In a preferred embodiment: The initial ID embedding vector dimension is set to 128; ID embedding uses the Xavier initialization method; The initial ID embedding captures only content-independent collaborative filtering signals to provide a foundation for structured learning in subsequent graph convolution propagation.
[0024] It is important to understand that the ID embedding vector mentioned above is a learnable parameter that is updated during model training using gradient descent.
[0025] S122, Construct a user-item bipartite graph based on the user identifier set and the corresponding interactive item set; It's important to understand that the user-item bipartite diagram is used to represent the collaborative relationship between users and items. The user-item two-part diagram is represented as follows: Among them, the user-item two-part diagram The vertex set is composed of user sets Collection of interactive items Composition, edge set
[0026] This includes all historical user-item interaction events, such as clicks, browsing, favorites, adding to cart, and purchases. If user With items If there is interaction between them, then an edge is constructed. ; In a preferred embodiment, the initial edge weight of all interactive edges is set to 1; It should be noted that the bipartite graph structure is used to model potential user preference patterns in collaborative filtering and is the basic structure for subsequent graph convolutional propagation.
[0027] In a specific embodiment, the steps for generating user embedded data and item embedded data are as follows: S123, On the user-item bipartite graph, a lightweight graph convolutional network is used to perform high-order collaborative feature propagation to update the user embedding vector and the item embedding vector. In a preferred embodiment, this step employs a lightweight graph convolutional network LightGCN with L=3 layers to perform multi-layer embedding propagation on the bipartite graph constructed in step S122 to capture high-order collaborative signals across users and items. Among them, for any layer The update rules for user embedding and item embedding are expressed as follows: In the formula, User Identifier Users in User embedding vectors of the layer, For items In the Item embedding vectors of the layer, User Identifier Users after the update The user embedding vector of the layer is the representation of the user node obtained by aggregating features from all neighboring item nodes. For items After the update The item embedding vector of the layer is the representation of the item node after aggregating features from all neighboring user nodes; For users The set of neighboring items, containing all items related to the user. Item nodes that have been interacted with; For items The user's neighbor set, containing all items User nodes that have interacted with the user; For users The number of adjacent items, For items The number of adjacent items; It is a symmetric normalization coefficient used to balance the impact of nodes of different degrees on information propagation and prevent gradient amplification caused by nodes of different heights during propagation. Represents items User ID is The user's neighbors, i.e., those representing the user With items It is interactive; Indicates user identifier as The user is the item The neighbors of the object, i.e., the objects that represent the property With users It is interactive.
[0028] S124, the propagation output of the last layer of the graph convolutional network is used as the enhanced user embedding data and item embedding data; In a specific embodiment: The user embedding vector output from the Lth layer propagation of LightGCN is represented as... And output it as user-embedded data; The item embedding vector output from the Lth layer of LightGCN is represented as: And output it as embedded data of the item; The enhanced embedded data will serve as input for subsequent sequence preference modeling and two-layer gating fusion.
[0029] It should be noted that higher-order collaborative embeddings can effectively capture user preference migration paths and cross-item association structures, thereby enhancing the expressive power of sequence recommendation models.
[0030] S13, sort the data based on the interaction timestamps to obtain the sequence of historical interactive items, and construct the user's temporal preference vector in visual and textual modal modes to generate behavior adaptation vectors; This step involves rigorously structuring the user's historical interaction item sequence to construct historical feature sequences for both visual and textual modalities. Furthermore, it generates a cumulative preference vector across time using a sequence encoding structure, and then performs a fusion operation on the two types of cumulative preference vectors to obtain a behavior adaptation vector, providing the user's personalized preference signal for the subsequent two-layer gating fusion structure. In a specific embodiment, the logic for generating the historical interaction item sequence is as follows: Based on user identifier set and interactive item collection For any user identifier Retrieve the corresponding partial set of interactive items Based on the interaction timestamp of each item, they are sorted from the earliest interaction to the most recent interaction to form a historical sequence of interacted items; Represented as: In the formula, User Identifier The historical sequence of interactive items, For the sorted number The identifier of an interactive item. User Identifier The total number of interactive items at that time; In this embodiment, the maximum length of the historical sequence is set to 50; like Zero-padding is performed at the end of the sequence; if Only the 50 most recent interactive items are retained, and the sequence after length normalization is continued to be denoted as... .
[0031] In a specific embodiment, the steps for generating behavior adaptation vectors are as follows: S131, Construct historical feature sequences for visual and textual modalities; Based on the obtained visual feature intermediate embedding data Intermediate data embedded with text features Based on the historical interaction item sequence, the modality embedding corresponding to each item is arranged into two historical feature sequences, represented as follows:
[0032] In the formula, User Identifier Historical feature sequences in the visual modality; Historical feature sequences of text modalities; It should be noted that the sequence length is 50, which is consistent with the sequence processed by S131. The above two modal feature sequences are used as input to characterize the dynamics of users' time preferences under different modalities. S132, input the historical feature sequences of the visual modality and the historical feature sequences of the text modality into the sequence encoding structure to generate the user's temporal preference vector in the visual modality and the text modality; To characterize users' temporal preferences in visual and textual modalities, two independent gated recurrent unit (GRU) models are constructed: Used for visual modality history feature sequences ; Used for text modality history feature sequences ; Input the modality history feature sequence into the corresponding GRU network to obtain the temporal preference vectors of visual modality and text modality respectively; Represented as: In the formula, User Identifier Temporal preference vector in visual modality; User ID is Temporal preference vector in text modality; It should be noted that the hidden layer dimension of both GRUs is D=128, which is consistent with the dimension of the modal embedding data obtained in step S114, so as to ensure the dimensionality consistency of subsequent fusion operations; during the training process, the GRU network updates the gradient so that the temporal preference vector can accurately express the modal temporal preferences formed by the user in their historical interaction behavior.
[0033] S133, perform a fusion operation on the visual modality temporal preference vector and the text modality temporal preference vector to generate a behavior adaptation vector; To unify the modeling of users' temporal preference information in visual and textual modalities, the two temporal preference vectors obtained in step S133 are first concatenated; represented as: “ " is a symbol for splicing processing; The concatenated vector is input into a fully connected structure, and the tanh activation function is used to generate behavior adaptation vectors. Represented as: In the formula, For behavior adaptation vectors; These are the weight parameters for the fully connected layer, with an input dimension of 2D and an output dimension of D. For bias terms; Behavior adaptation vector User ID The temporal preference information in the visual and textual modalities is fused and expressed in a unified space, and serves as the core input data for the subsequent two-layer gating fusion step.
[0034] S14, Perform two-layer gated fusion based on user behavior adaptation vector and item modal embedding data to generate personalized modal fusion features; This step establishes modal weighting relationships at the item level and the user-item pair level using static gating networks and adaptive gating networks, respectively, and further combines them with a learnable linear combination structure to generate unified personalized modal fusion features, providing an input basis for subsequent preference prediction. In a specific embodiment, the steps for generating personalized modality fusion features are as follows: S141, Based on a given static gating network, generate static fusion features; Based on the visual feature intermediate embedding data obtained in S114 Intermediate data embedded with text features Perform vector concatenation to form a joint feature representation: The joint feature representation is input into a static gating network to generate a dimension-level gating vector. Represented as: For items The static gate vector; , These are the learnable parameters for a statically gated network. Use the Sigmoid activation function; In a preferred embodiment, the static gated network has an input dimension of 2D and an output dimension of D; It should be noted that the items in this step... Corresponding interactive Wuping set For ease of understanding, the visual features in the data are not embedded in the middle. Intermediate data embedded with text features and static gate vector In Written The form; Based on static gate vector Element-wise weighting is performed on the two types of modal embeddings to obtain static fusion features: In the formula, For items Static fusion characteristics; It should be noted that: static fusion features It reflects the global quality structure of the object in both visual and textual modalities, providing a basic modal representation for subsequent dynamic fusion steps; S142, Based on a preset adaptive gating network, generate adaptive fusion features of user-item pairs; Data embedded based on visual features Intermediate data embedded with text features and the behavior adaptation vector obtained in step S134 ; The joint input after concatenating the three vectors is represented as: The joint vector is input into an adaptive gating network to generate behavior-driven gating vectors. Represented as: In the formula, User Identification and items Behavior-driven gating vectors; , For the learnable parameters of the adaptive gating network, in a preferred embodiment, the input dimension of the adaptive gating network is 3D and the output dimension is D; Based on this gate vector, element-wise weighting is performed on the two types of modality embeddings to obtain adaptive fusion features: In the formula, For adaptive fusion features; It should be noted that adaptive fusion features can autonomously adjust modal contributions based on the differences in users' temporal preferences in visual and textual modalities, giving the fused expression significant personalized features.
[0035] S143, based on a linear combination method, integrates static fusion features and adaptive fusion features to generate personalized modality fusion features; To balance the global modal quality of the item with the user's personalized preferences, the static fusion features obtained in step S141 are... Represented as: In the formula, For personalized modal fusion features, For learnable parameters, Learnable parameters The initial value is 0.5, and the learnable parameter is... Update using gradient descent; The final personalized fusion features It is used in the recommendation preference score inference stage and is the representation result of user-item pairs in a unified latent space.
[0036] S15: Based on user embedded data and personalized modality fusion features, calculate and generate predicted preference scores. Sort the candidate items according to the predicted preference scores, select the top K items with the highest scores to generate recommendation results, and output the recommendation list. Based on the GCN-enhanced user embedding data obtained in step S24 The personalized modality fusion features obtained in step S14 For users and items Perform vector inner product calculations to generate predicted preference scores; Represented as: To predict preference scores, " is the vector dot product operator; Construct a set of predicted preference scores based on multiple sets of predicted preference scores; Represented as: In the formula, For users For the Items Predicted preference score; Sort all items according to their preference scores from highest to lowest. It should be noted that no normalization or threshold filtering is performed on preference scores during the ranking process, ensuring that the ranking results are entirely based on user-embedded data. Features of personalized modality fusion Make a decision.
[0037] From the sorted sequence of items, select the top-scoring items. The items in the specified positions are used as the final recommendation results, among which To pre-determine the recommended quantity, preferably in this embodiment... Set to 10; The result The items are sorted by score to form a recommendation list, and the recommendation list is used as the final output of the model to show personalized recommendation results to users; It should be noted that the recommendation list is generated strictly based on the predicted preference scores, without relying on additional filtering rules or post-processing steps, to ensure that the output results are consistent with the user preference modeling process.
[0038] Example 2 To further improve the accuracy of recommendations and thus enhance the ranking ability of predicted preference scores; This example further optimizes the recommendation method of Example 1. Specifically, a multimodal sequence recommendation method based on two-layer gated fusion also includes: S16 is based on the constructed user-positive-negative sample triplet, and optimizes all model parameters through BPR loss and L2 regularization. In a specific embodiment, the steps for optimizing model parameters are as follows: S161, Record items that the user has actually interacted with as positive sample items, and items that the user has not interacted with as negative sample items, and construct user-positive sample-negative sample triplet based on user identifier, positive sample items and negative sample items; Represented as: In the formula, User Identifier User-positive-negative triplet For positive sample items, Negative sample items; S162, calculate and obtain the predicted preference scores corresponding to positive sample items and negative sample items, and construct the BPR loss function based on the predicted preference scores; This involves calculating and obtaining the predicted preference scores for positive and negative sample items; Represented as: In the formula, The predicted preference score is the score corresponding to the positive sample item. The predicted preference score for the negative sample item; , users respectively and items ,thing Personalized modal fusion features.
[0039] Based on the calculated predicted preference score , Construct the BPR loss function; Represented as: In the formula, The Bayesian personalized ranking loss function is also known as the BPR loss function. S163, introduce L2 regularization to optimize the BPR loss function to obtain the final loss function, and use the Adam optimizer to perform parameter updates on the final loss function; To prevent overfitting during model training, all learnable parameters are processed. Introducing L2 regularization term, In the formula, For the final loss function, The coefficients of the L2 regularization term. It includes learnable parameters in the model, such as embedding vectors, weight matrices, and bias vectors; During model training, the Adam optimizer is used to optimize the final loss function. Perform gradient descent to update the set of learnable parameters In a preferred embodiment, the learning rate Choose from the following candidate set, denoted as: The training batch size is set to 2048, for each positive sample item pair ( Randomly sample one negative sample item. To form a triple ( () Participate in the current batch training; preferably, the L2 regularization term coefficient is set to: ; During training, the Recall@20 metric on the validation set is monitored. If the metric does not improve within 10 consecutive training epochs, an early stopping strategy is triggered to stop further training in order to prevent overfitting and reduce unnecessary computational overhead.
[0040] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.
Claims
1. A multimodal sequence recommendation method based on two-layer gated fusion, characterized in that, The method includes: S11, Obtain the original multimodal features of interactive items in the user's historical interaction records, and perform mapping processing to obtain modal embedding data; S12, construct a user-item bipartite graph, and use a lightweight graph convolutional network to propagate high-order collaborative signals to generate user embedding data and item embedding data; S13, sort the data based on the interaction timestamps to obtain the sequence of historical interactive items, and construct the user's temporal preference vector in visual and textual modal modes to generate behavior adaptation vectors; S14, Perform two-layer gated fusion based on user behavior adaptation vector and item modal embedding data to generate personalized modal fusion features; S15: Based on user embedded data and personalized modality fusion features, calculate and generate predicted preference scores. Sort the candidate items according to the predicted preference scores, select the top K items with the highest scores to generate recommendation results, and output the recommendation list.
2. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 1, characterized in that, The original multimodal features include original visual features and original text features, and the modal embedding data includes intermediate embedding data of visual features and intermediate embedding data of text features.
3. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 2, characterized in that, The steps to obtain modal embedding data include: S111, Based on the user identifier, obtain the historical interaction records corresponding to the user identifier, extract the interactive items in the historical interaction records, and construct an interactive item set; S112, Based on the pre-trained visual ViT model, feature extraction is performed on the images of each item in the interactive item set to obtain the original visual features; S113, Based on the pre-trained text BERT model, feature extraction is performed on the text of each item in the interactive item set to obtain the original text features; S114 uses linear transformation and the ReLU activation function to map the original multimodal features to a latent space of a unified dimension, thus obtaining modal embedding data.
4. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 3, characterized in that, The steps to construct a user-item bipartite graph are as follows: S121, Based on the user identifier and the corresponding interactive item, initialize the user embedding vector and the item embedding vector; S122, construct a user-item bipartite graph based on the user identifier set and the corresponding interactive item set.
5. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 4, characterized in that, The steps to generate user embedded data and item embedded data are as follows: S123, On the user-item bipartite graph, a lightweight graph convolutional network is used to perform high-order collaborative feature propagation to update the user embedding vector and the item embedding vector. S124 uses the propagation output of the last layer of the graph convolutional network as the enhanced user embedding data and item embedding data.
6. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 5, characterized in that, The logic for generating the historical interaction item sequence is as follows: Based on the user identifier set and the interactive item set, for any user identifier, the corresponding partial set of interactive items is retrieved, and the items are sorted from the earliest to the most recent interaction according to the interaction timestamp of each item to form a historical interactive item sequence.
7. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 6, characterized in that, The steps to generate behavior adaptation vectors are as follows: S131, Construct historical feature sequences for visual and textual modalities; S132, input the historical feature sequences of the visual modality and the historical feature sequences of the text modality into the sequence encoding structure to generate the user's temporal preference vector in the visual modality and the text modality; S133, perform a fusion operation on the visual modality temporal preference vector and the text modality temporal preference vector to generate a behavior adaptation vector.
8. The multimodal sequence recommendation method based on two-layer gated fusion according to claim 7, characterized in that, The steps for generating personalized modality fusion features are as follows: S141, Based on a given static gating network, generate static fusion features; S142, Based on a preset adaptive gating network, generate adaptive fusion features of user-item pairs; S143 integrates static fusion features and adaptive fusion features based on a linear combination method to generate personalized modal fusion features.
9. A multimodal sequence recommendation method based on two-layer gated fusion according to claim 8, characterized in that, The method further includes: S16 is based on the constructed user-positive-negative sample triplet, and optimizes all model parameters together through BPR loss and L2 regularization.
10. A multimodal sequence recommendation method based on dual-layer gated fusion according to claim 9, characterized in that, The steps to optimize model parameters are as follows: S161, Record items that the user has actually interacted with as positive sample items, and items that the user has not interacted with as negative sample items, and construct user-positive sample-negative sample triplet based on user identifier, positive sample items and negative sample items; S162, calculate and obtain the predicted preference scores corresponding to positive sample items and negative sample items, and construct the BPR loss function based on the predicted preference scores; S163 introduces an L2 regularization term to optimize the BPR loss function, resulting in the final loss function. The Adam optimizer is then used to update the parameters of the final loss function.