Recipe recommendation method based on multi-mode anti-fact learning

Abstract

The invention relates to the technical field of information recommendation, in particular to a recipe recommendation method based on multi-modal anti-fact learning, which comprises the following steps: constructing a heterogeneous graph; using a graph neural network to obtain network fusion embedding characteristics of each user node as target embedding characteristics of each user node, and network fusion embedding characteristics of each recipe node in each mode; based on network fusion embedding characteristics of each recipe node in each mode, a multi-mode characteristic decoupling module is utilized to obtain mode sharing embedding characteristics, mode specific embedding characteristics and mode fusion embedding characteristics of each recipe node, and then an anti-fact learning network is utilized to obtain final anti-fact embedding characteristics of each recipe node; fusing the modal sharing embedding feature, the modal specific embedding feature and the final anti-fact embedding feature of each recipe node as the target embedding feature of each recipe node; and inputting the target embedded features of each user node and each recipe node into a prediction module, and outputting a prediction score between each user and each recipe. The recipe recommendation precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information recommendation, and in particular to a recipe recommendation method based on multimodal counterfactual learning. Background Art

[0002] The recipe recommendation problem aims to analyze the user's true preference for recipes based on the user's historical interactions and the characteristics of the user and the recipe, so as to give personalized recipe suggestions. People's preferences for recipes are complex, and their liking for recipes is affected by many factors, such as recipe type, taste, ingredients, pictures, cooking complexity, etc. Therefore, a comprehensive recipe recommendation system cannot only consider the historical interaction information between users and recipes to make recommendations.

[0003] To solve this problem, it is necessary to consider as many different characteristics of users and recipes as possible, and quantify the impact of these characteristics on users' recipe selection by establishing a mathematical model. At present, multimodal recommendation systems often consist of two parts: multimodal feature extraction and user preference modeling.

[0004] In the multimodal feature extraction part, existing methods usually use pre-trained deep neural networks or graph convolutional networks to extract information from different modalities, and then fuse the information from different modalities into the recommendation framework for recommendation. However, existing methods ignore the impact of modal common features and modal specific features on user interaction, so that the model cannot grasp the common factors that affect users' preferences for recipes as a whole, resulting in a one-sided understanding of user preferences. When fusing heterogeneous information from different modalities, accurate alignment of cross-modal data is crucial for accurately understanding recipe features and grasping user preferences. However, existing methods focus on how to extract common knowledge across modalities, and modal specific features are often ignored. In practical applications, even for items that users have interacted with, users may like this recipe because of the specific features of a single modality. Modal common features are features that are irrelevant to user preferences. If the modal specific features are ignored, the model will not be able to capture these important details that affect user choices, making it difficult to accurately recommend recipes that meet users' true preferences, reducing the accuracy of recipe recommendations, homogenizing recommendation results, and failing to meet personalized needs.

[0005] In the user preference modeling part, most of the existing models are modeled from the perspective of data correlation, trying to make recommendations by mining relevant patterns in the data. However, the real world is composed of causal relationships rather than correlation relationships, and correlation relationships cannot replace causal relationships. However, there are some features in the multimodal information of the project (modal common features and modal specific features) that are irrelevant to user preferences. This may be a false correlation with user preferences, thereby reducing the performance of the recommendation model and damaging the diversity of the recommendation model. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the influence of modal common features and modal specific features on user interaction is ignored, so that the model cannot grasp the common factors that affect users' recipe preferences as a whole, resulting in a one-sided understanding of user preferences; modal specific features are ignored, resulting in reduced accuracy of recipe recommendations, homogeneity of recommendation results and inability to meet personalized needs; and the causal relationship between users and recipes is ignored, resulting in reduced performance of the recommendation model and lack of diversity in the recommendation model.

[0007] In order to solve the above technical problems, the present invention provides a recipe recommendation method based on multimodal counterfactual learning, comprising:

[0008] Build a recipe recommendation model, including a heterogeneous graph building module, a graph neural network building module, a multimodal feature decoupling module, a counterfactual learning network, and a prediction module;

[0009] Based on the user data and the recipe data, a user-recipe-ingredient heterogeneous graph is constructed; wherein the node types in the heterogeneous graph are user nodes, recipe nodes and ingredient nodes respectively; and the edge types in the heterogeneous graph are user-recipe edges, recipe-recipe edges, recipe-ingredient edges and ingredient-ingredient edges respectively;

[0010] Perform an initial embedding operation on each node to obtain the initial feature representation of each node; construct a graph neural network based on the user-recipe-ingredient heterogeneous graph; use the initial feature representation of all nodes as the input of the graph neural network, use a two-level attention mechanism to process the initial feature representation of each node, and then pass it through the propagation layer to output the network fusion embedding features of each user node and the network fusion embedding features of each recipe node in each mode;

[0011] The network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature values ​​of each modality in each recipe node; based on the important feature values ​​of each modality in each recipe node, the modality shared embedding features of each recipe node are calculated; the network fusion embedding features of each recipe node in each modality are subtracted from the modality shared embedding features of each recipe node to obtain the modality-specific representation of each recipe node in each modality; based on the modality-specific representation of each recipe node in each modality, the modality-specific embedding features of each recipe node are obtained; multimodal fusion operations are performed on the network fusion embedding features of each recipe node in various modalities to obtain the modality fusion embedding features of each recipe node;

[0012] Using the counterfactual learning network, the modality-shared embedding features and modality-specific embedding features of each recipe node are combined with its modality-fusion embedding features to obtain the new modality-shared embedding features and new modality-specific embedding features of each recipe node; through matrix transformation, the query vector, key vector and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node are calculated respectively, and then the preference distribution of the modality-shared content and modality-specific content of each recipe node is calculated respectively; based on the causal difference theory, the preference distribution of the modality-shared content and modality-specific content of each recipe node is combined to obtain the preliminary counterfactual embedding features of each recipe node; the preliminary counterfactual embedding features of each recipe node are input into a feedforward network with residual connection, and the final counterfactual embedding features of each recipe node are output;

[0013] Based on the modal shared embedding features, modal specific embedding features and final counterfactual embedding features of each recipe node, the target embedding features of each recipe node are obtained; the network fusion embedding features of each user node output by the graph neural network are used as the target embedding features of each user node; the target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output; the recipes corresponding to the top ten prediction scores between the current user and each recipe are sorted from high to low and are recommended to the current user.

[0014] Preferably, the network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature values ​​of each modality in each recipe node; based on the important feature values ​​of each modality in each recipe node, the modality-shared embedding features of each recipe node are calculated, which includes:

[0015] Using the softmax function, the important feature value of each mode in each recipe node is calculated, and its expression is:

[0016]

[0017] in, Represents the i-th recipe node The important feature value under modality m; q1 represents the attention vector; W represents the importance weight; b represents the importance bias; Represents the i-th recipe node The network fusion embedding features under mode m; softmax(·) represents the softmax function;

[0018] Based on the important feature values ​​of each modality in each recipe and the network fusion embedding features of each recipe node in each modality, the modality shared embedding features of each recipe node are calculated, and the expression is:

[0019]

[0020] in, Represents the i-th recipe node Modality-shared embedding features of ; Represents the modal information corresponding to the recipe node, including text modality and visual modality.

[0021] Preferably, the subtracting the network fusion embedding feature of each recipe node in each modality from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in each modality; and obtaining the modality-specific embedding feature of each recipe node based on the modality-specific representation of each recipe node in each modality includes:

[0022] The modal information corresponding to the recipe node includes text modality and visual modality;

[0023] The network fusion embedding feature of each recipe node in the text modality is subtracted from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the text modality, which is expressed as:

[0024]

[0025] in, Represents the i-th recipe node modality-specific representation in textual modality; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Modality-shared embedding features of ;

[0026] The network fusion embedding feature of each recipe node in the visual modality is subtracted from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the visual modality, which is expressed as:

[0027]

[0028] in, Represents the i-th recipe node modality-specific representations in the visual modality; Represents the i-th recipe node Network fusion embedding features in visual modality;

[0029] Based on the modality-specific representation of each recipe node in the textual modality and the modality-specific representation in the visual modality, the modality-specific embedding feature of each recipe node is obtained, and its expression is:

[0030]

[0031] in, Represents the i-th recipe node The modality-specific embedding features of W s Represents the modality specific weight.

[0032] Preferably, the multimodal fusion operation is performed on the network fusion embedding features of each recipe node in various modalities to obtain the modal fusion embedding features of each recipe node, which is expressed as:

[0033]

[0034] in, Represents the i-th recipe node The modality fusion embedding feature of W g represents the modal fusion weight; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Network fusion embedding features in visual modality.

[0035] Preferably, based on the query vector, key vector and value vector corresponding to the new modality shared embedding feature and the new modality specific embedding feature of each recipe node, the preference distribution of the modality shared content and modality specific content of each recipe node is calculated respectively; based on the causal difference theory, combined with the preference distribution of the modality shared content and modality specific content of each recipe node, the preliminary counterfactual embedding features of each recipe node are obtained, including:

[0036] The expressions for the preference distribution of the modal shared content and modal specific content of each recipe node are:

[0037]

[0038] Among them, P i Represents the i-th recipe node Preference distribution of modal shared content; Represents the i-th recipe node The preference distribution of modality-specific content of ; softmax(·) represents the softmax function; Represents the i-th recipe node The query vector corresponding to the new modality shared embedding feature; Represents the i-th recipe node The new modality shares the key vector corresponding to the embedding feature; Represents the i-th recipe node The query vector corresponding to the new modality-specific embedding features; Represents the i-th recipe node The key vector corresponding to the new modality-specific embedding feature of d; z Represents the dimensions of the query vector and key vector;

[0039] The expression of the preliminary counterfactual embedding feature of each recipe node is:

[0040]

[0041] in, Represents the i-th recipe node The initial counterfactual embedding features of P i Represents the i-th recipe node Preference distribution of modal shared content; Represents the i-th recipe node Preference distribution of modality-specific content; Represents the i-th recipe node The new modality shares the value vector corresponding to the embedded features.

[0042] Preferably, the step of inputting the preliminary counterfactual embedding features of each recipe node into a feed-forward network with residual connections and outputting the final counterfactual embedding features of each recipe node comprises:

[0043] The feed-forward network with residual connection includes: two linear layers and an activation function, as well as a residual connection and layer normalization;

[0044] The final counterfactual embedding feature expression of each recipe node is:

[0045]

[0046] in, Represents the i-th recipe node The final counterfactual embedding features of ; LN(·) represents the layer normalization operation; GELU(·) represents the nonlinear activation function; Represents the i-th recipe node The initial counterfactual embedding features of Represents the i-th recipe node The modal shared embedding features of ; W1 represents the weight corresponding to the nonlinear activation function; b1 represents the bias corresponding to the nonlinear activation function; W2 represents the weight corresponding to the layer normalization; b2 represents the bias corresponding to the layer normalization.

[0047] Preferably, the target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output, which is expressed as:

[0048]

[0049] in, Represents the u-th user node With the i-th recipe node The prediction score between u Represents the u-th user node The target embedding feature of z i Represents the i-th recipe node The target embedding feature.

[0050] Preferably, the loss of the recipe recommendation model includes:

[0051] The contrast loss function is constructed with the goal of minimizing the difference between the network fusion embedding features of each recipe node in a single modality and the modality shared embedding features of each recipe node. Its expression is:

[0052]

[0053] in, Represents a collection of recipes, Indicates the number of all recipes; Indicates the modal information corresponding to the recipe. Represents the types of all modes; Represents the i-th recipe node The network fusion embedding features under modality m; Represents the i-th recipe node The modality shared embedding feature of ; F(·) represents the contrast evaluation function, which is expressed as:

[0054]

[0055] Wherein, θ(·) represents the cosine function; τ represents the temperature parameter; Indicates the i * Recipe Node The network fusion embedding features under modality m are represented as the i-th recipe node Negative samples under modality m;

[0056] Soft orthogonality constraints are constructed with the goal of maximizing the difference between different modality-specific representations of each recipe node. Its expression is:

[0057]

[0058] in, Represents the i-th recipe node modality-specific representation in textual modality; Represents the i-th recipe node modality-specific representations in the visual modality;

[0059] The BPR loss function is constructed with the goal that the prediction score between the user node and its interacted recipe node is higher than the prediction score between the user node and its non-interacted recipe node. Its expression is:

[0060]

[0061] Among them, Ω represents the set of user nodes and recipe nodes; the preset η represents the recipe node that interacts with the user, that is, the interactive recipe node, Represents a recipe node that has no interaction with the user, that is, an uninteracted recipe node. Represents the u-th user node The prediction score between the interactive recipe node η; Represents the u-th user node Not interacting with recipe nodes The prediction score between

[0062] Based on the contrast loss function, soft orthogonal constraint and BPR loss function, the total target loss function is obtained Its expression is:

[0063]

[0064] Among them, λ C , S They represent the hyperparameters corresponding to the contrast loss function and the soft orthogonal constraint respectively.

[0065] Preferably, the two-level attention mechanism is used to process the initial feature representation of each node, including:

[0066] The two-level attention mechanism includes: a node-level attention mechanism and a relation-level attention mechanism;

[0067] The node-level attention mechanism includes:

[0068] By using the shared weight matrix, the output features corresponding to each node in the lth layer of the graph neural network in each mode are linearly transformed to obtain the linear output features corresponding to each node in the lth layer in each mode, which is expressed as:

[0069]

[0070] in, represents the kth node h in the lth layer k The corresponding linear output characteristics under mode m; Represents the weight corresponding to the relationship r and mode m in the lth layer; represents the kth node h in the lth layer k Input features under mode m;

[0071] Based on the linear output features corresponding to each node in the lth layer in each mode, the attention score between two nodes in each mode in the lth layer is calculated, and its expression is:

[0072]

[0073] in, represents the kth node h in the lth layer under mode m k and the jth node h j The attention score between them; LeakyReLU(·) represents the improved ReLU activation function; represents the kth node h in the lth layer k and the jth node h j The ones between them represent the attention score weights; represents the jth node h in the lth layer j The corresponding linear output characteristics under mode m;

[0074] The calculated attention scores are normalized to obtain the normalized attention scores between two nodes in each mode of the lth layer, which is expressed as:

[0075]

[0076] in, represents the kth node h in the lth layer under mode m k and the jth node h j The normalized attention score between ; Represents the lth node h k The neighbor node under relation r, i.e. the kth * Nodes; L k,r represents the kth node h k The set of neighbor nodes under relation r; Indicates the kth * Nodes and the jth node h j Attention scores between

[0077] Based on the normalized attention scores of each pair of nodes and between them in each modality of the lth layer, the first fused embedding feature of each node in the l+1th layer in each relationship and each modality is calculated, and its expression is:

[0078]

[0079] in, represents the kth node h in the l+1th layer k The first fused embedding feature under relation r and modality m; N represents the number of attention heads; n represents the index of the attention head; ReLU(·) represents the ReLU function; represents the jth node h in the lth layer j Input features under mode m; W represents the weight vector corresponding to the mode m in the lth layer; f represents a trainable weight vector;

[0080] The relation-level attention mechanism includes:

[0081] Using the shared nonlinear weights and intermediate vectors, the first fused embedding features of each node in the l+1th layer in each relationship and each modality are transformed to the new feature space; based on the intermediate vectors, the embedding similarity between the first fused embedding features of each node in the l+1th layer in each relationship and each modality before and after the feature space transformation is calculated, and its expression is:

[0082]

[0083] in, represents the kth node h k The corresponding embedding similarity under relation r and modality m; represents the set of nodes connected to relation r, represents the number of all nodes connected to the relationship r; q represents the intermediate vector; W R represents the relationship weight; represents the kth node h in the l+1th layer k The first fusion embedding feature under relation r and modality m; b * represents similarity bias;

[0084] The embedding similarity is normalized to obtain the relation-level attention weight of each node in each relation and each modality, which is expressed as:

[0085]

[0086] in, represents the kth node h k Relation-level attention weights under relation r and modality m; Represents the relationship with the kth node h k A collection of connected relationships;

[0087] Based on the first fused embedding feature of each node in the l+1th layer in each relationship and each modality, and the relation-level attention weight of each node in each relationship and each modality, the second fused embedding feature of each node in the l+1th layer in each modality is calculated, and its expression is:

[0088]

[0089] in, represents the kth node h in the l+1th layer k The second fused embedding features under modality m.

[0090] Preferably, the initializing embedding operation on each node to obtain an initial feature representation of each node includes:

[0091] Using the Xavier initialization method, generate the common random initialization features corresponding to each user; use the common random initialization features corresponding to each user as the initial feature representation of each user node;

[0092] The pre-trained jump instruction embedding is used as the representation of each instruction in the cooking steps of each recipe. The average of all instruction representations in the cooking steps of each recipe is calculated to obtain the average instruction corresponding to each recipe. The average instruction corresponding to each recipe is used as the cooking instruction embedding of each recipe; where the average instruction corresponding to any recipe is The expression is:

[0093]

[0094] Among them, z ins,a It represents the ath instruction in the cooking step of the current recipe; O represents the number of instructions in the current recipe;

[0095] Use the pre-trained model to encode each recipe image and obtain the image embedding of each recipe; project the cooking instruction embedding and image embedding of each recipe into the same shared input space as the initial feature representation of each recipe node;

[0096] The nutrients of the components are projected into vectors to obtain the vector corresponding to each component; the vector corresponding to each component is used as the initial feature representation of each component node.

[0097] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0098] The recipe recommendation method based on multimodal counterfactual learning described in the present invention constructs a user-recipe-ingredient heterogeneous graph and integrates relevant information of the recipe into the heterogeneous graph, and considers the information of the recipe node in different modes in the graph, which helps to understand the recipe features more comprehensively; the node attributes and structural information in the heterogeneous graph are learned through the graph neural network algorithm, the complex relationship between each node and the difference in importance of different relationships are considered, the feature representation of the node is enriched, and the network fusion embedding feature of each user node and the network fusion embedding of each recipe node in each mode are obtained; the designed multimodal feature decoupling learning algorithm is used, and comparative learning is used to ensure that modal shared information can be adaptively extracted from the network fusion embedding of the recipe node in each mode, and the modal specific representation of different modalities is ensured to be as different as possible from the modal shared representation through orthogonal constraints to extract the modal specific representation, thereby decoupling the network fusion embedding features of the recipe node in various modes into modal common features and modal specific features, comprehensively considering the influence of modal shared information and modal specific information of different modalities in the multimodal model on user interaction, and improving the model's understanding of user preference factors for recipes. The overall grasp of the recipes is achieved while avoiding the homogeneity of recipe recommendations, which can meet the personalized needs of users. The designed counterfactual learning network is used to learn the preference distribution of multimodal content calculated by user interactive recipes by comparing the causal differences in the real world and the counterfactual world, that is, the preference distribution of modal shared information and modal specific information, so as to eliminate the data bias in the decoupled multimodal features, thereby filtering out representations that are irrelevant to user preferences and better learning the user's true preferences, thereby improving the accuracy of recipe recommendations and improving the performance and diversity of the recipe recommendation model. Based on the recipe embedding enhanced by multimodal feature decoupling learning and counterfactual learning, the target embedding feature of the recipe node is obtained, and the user embedding learned in the graph neural network is used as the target embedding feature of the user node. Based on the target embedding feature of the recipe node and the target embedding feature of the user node, the final user's recommendation score for the recipe is obtained, and personalized recipes are recommended to the user according to the recommendation score of the recipe, and finally a multimodal recipe recommendation model with high precision, high robustness and high scalability is obtained, which can be used to provide different users with recipes that are highly relevant to their preferences and have personalization and diversity. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0100] Figure 1 is a schematic diagram of a recipe recommendation method based on multimodal counterfactual learning provided by the present invention;

[0101] Figure 2It is a specific framework diagram of a recipe recommendation method based on multimodal counterfactual learning provided by the present invention;

[0102] Figure 3 It is a schematic diagram of a cause-effect diagram;

[0103] Figure 4 It is a flow chart of a recipe recommendation method based on multimodal counterfactual learning provided by the present invention;

[0104] Figure 5 It is a detailed statistical chart of the large multimodal recipe dataset LargeRG;

[0105] Figure 6 This is a comparison chart of the experimental results. DETAILED DESCRIPTION

[0106] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0107] Reference Figure 1 , Figure 2 As shown, the present invention provides a recipe recommendation method based on multimodal counterfactual learning, comprising:

[0108] S1: Build a recipe recommendation model, including a heterogeneous graph building module, a graph neural network building module, a multimodal feature decoupling module, a counterfactual learning network, and a prediction module;

[0109] S2: Based on the user data and the recipe data, construct a user-recipe-ingredient heterogeneous graph; wherein the node types in the heterogeneous graph are user nodes, recipe nodes and ingredient nodes respectively; and the edge types in the heterogeneous graph are user-recipe edges, recipe-recipe edges, recipe-ingredient edges and ingredient-ingredient edges respectively;

[0110] In a specific embodiment of the present invention, the user and recipe data come from Food.com, the world's largest recipe sharing website, which has rich recipe data. The recipe data includes the recipe name, ingredients, nutrients, food pictures and cooking methods. Under each recipe, there are user comments and scores, with a score of 1-5. Through these related attributes of user data and recipe data, a heterogeneous graph with three types of nodes and four types of edges is constructed as a user-recipe-ingredient heterogeneous graph (URI-Graph);

[0111] The construction process of each type of edge in a heterogeneous graph includes:

[0112] Through the user's rating of the recipe, the user-recipe relationship is established, and the user-recipe edge in the user-recipe-ingredient heterogeneous graph is obtained;

[0113] Through a public recipe knowledge graph FoodKG, after determining the similarity between recipes, we construct recipe-recipe relationships and obtain recipe-recipe edges in the user-recipe-ingredient heterogeneous graph;

[0114] Through the ingredient list of the recipe, the recipe-ingredient relationship is established, and the recipe-ingredient edge in the user-recipe-ingredient heterogeneous graph is obtained;

[0115] Based on the normalized point-by-point mutual information method, after obtaining the common occurrence probability between various components, the component-component relationship is constructed to obtain the component-component edge in the user-recipe-component heterogeneous graph;

[0116] S3: The processing in the graph neural network building module includes:

[0117] S31: Perform an initial embedding operation on each node to obtain the initial feature representation of each node, including:

[0118] Since most users in Food.com do not fill in complete personal information (e.g., description, location, preferences, etc.), the Xavier initialization method is used to generate common random initialization features corresponding to each user; the common random initialization features corresponding to each user are used as the initial feature representation of each user node, which can be expressed by z user express;

[0119] Previous studies have shown that cooking steps are very important for providing distinctiveness of the cooking process. Therefore, in the present invention, a pre-trained skip instruction embedding (an embedding after two stages of LSTM) is used as the representation of each instruction in the cooking steps of each recipe. All instruction representations in the cooking steps of each recipe are averaged to obtain the average instruction corresponding to each recipe, and the average instruction corresponding to each recipe is used as the cooking instruction embedding of each recipe; wherein, the average instruction corresponding to any recipe The expression is:

[0120]

[0121] Among them, z ins,a It represents the ath instruction in the cooking step of the current recipe; O represents the number of instructions in the current recipe;

[0122] The pre-trained ResNet model is used to encode each recipe image to obtain the image embedding of each recipe; the cooking instruction embedding and image embedding of each recipe are projected into the same shared input space as the initial feature representation of each recipe node. The initial feature representation of the text of each recipe node can be used Indicates that the visual initial feature representation of each recipe node can be used express;

[0123] Project the nutrients of each component into a vector to obtain the vector corresponding to each component; use the vector corresponding to each component as the initial feature representation of each component node, which can be expressed by z ing express;

[0124] S32: Based on the user-recipe-ingredient heterogeneous graph, a graph neural network is constructed; the initial feature representation of all nodes is used as the input of the graph neural network, and a two-level attention mechanism is used to process the initial feature representation of each node, and then pass through the propagation layer to output the network fusion embedding features of each user node and the network fusion embedding features of each recipe node in each mode; where for the u-th user node Its network fusion embedding feature is z u ; For the i-th recipe node The corresponding text modality feature is and the visual modality features are

[0125] The two-level attention mechanism is used to process the initial feature representation of each node, including:

[0126] The two-level attention mechanism includes: a node-level attention mechanism and a relation-level attention mechanism;

[0127] The node-level attention mechanism includes:

[0128] By using the shared weight matrix, the output features corresponding to each node in the lth layer of the graph neural network in each mode are linearly transformed to obtain the linear output features corresponding to each node in the lth layer in each mode, which is expressed as:

[0129]

[0130] in, represents the kth node h in the lth layer k The corresponding linear output characteristics under mode m; represents the weight corresponding to the relationship r and mode m in the lth layer, d l represents the dimension of the input embedding of the lth layer, d l+1 represents the dimension of the input embedding of the l+1th layer; represents the kth node h in the lth layer k Input features under mode m;

[0131] Based on the linear output features corresponding to each node in the lth layer in each mode, the attention score between two nodes in each mode in the lth layer is calculated, and its expression is:

[0132]

[0133] in, represents the kth node h in the lth layer under mode m k and the jth node h j The attention score between them; LeakyReLU(·) represents the improved ReLU activation function; represents the kth node h in the lth layer k and the jth node h j The ones between them represent the attention score weights; represents the jth node h in the lth layer j The corresponding linear output characteristics under mode m;

[0134] The calculated attention scores are normalized to obtain the normalized attention scores between two nodes in each mode of the lth layer, which is expressed as:

[0135]

[0136] in, represents the kth node h in the lth layer under mode m k and the jth node h j The normalized attention score between ; represents the kth node h k The neighbor node under relation r, i.e. the kth * Nodes; L k,r represents the kth node h k The set of neighbor nodes under relation r; Indicates the kth * Nodes and the jth node h j Attention scores between

[0137] Based on the normalized attention scores of each pair of nodes and between them in each modality of the lth layer, the first fused embedding feature of each node in the l+1th layer in each relationship and each modality is calculated, and its expression is:

[0138]

[0139] in, represents the kth node h in the l+1th layer k The first fused embedding feature under relation r and modality m; N represents the number of attention heads; n represents the index of the attention head; d n represents the dimension of the attention head, which satisfies d n =d l+1 / N; ReLU(·) represents the ReLU function; represents the jth node h in the lth layer j Input features under mode m; W represents the weight vector corresponding to the mode m in the lth layer; f represents a trainable weight vector;

[0140] The relation-level attention mechanism includes:

[0141] Using the shared nonlinear weights and intermediate vectors, the first fused embedding features of each node in the l+1th layer in each relationship and each modality are transformed to the new feature space; based on the intermediate vectors, the embedding similarity between the first fused embedding features of each node in the l+1th layer in each relationship and each modality before and after the feature space transformation is calculated, and its expression is:

[0142]

[0143] in, represents the kth node h k The corresponding embedding similarity under relation r and modality m; represents the set of nodes connected to relation r, Represents the number of all nodes connected to the relationship r; represents the intermediate vector; W R represents the relationship weight; represents the kth node h in the l+1th layer k The first fusion embedding feature under relation r and modality m; b * represents similarity bias;

[0144] The embedding similarity is normalized to obtain the relation-level attention weight of each node in each relation and each modality, which is expressed as:

[0145]

[0146] in, represents the kth node h k Relation-level attention weights under relation r and modality m; Represents the relationship with the kth node h k A collection of connected relationships;

[0147] Based on the first fused embedding feature of each node in the l+1th layer in each relationship and each modality, and the relation-level attention weight of each node in each relationship and each modality, the second fused embedding feature of each node in the l+1th layer in each modality is calculated, and its expression is:

[0148]

[0149] in, represents the kth node h in the l+1th layer k The second fusion embedding feature under modality m;

[0150] In summary, for URI-Graph, a two-level attention mechanism graph representation learning method is needed to mine the attribute and structural information on the graph; for each node, the node-level attention mechanism fuses the adjacent nodes under each specific relationship and module; then the relationship-level attention mechanism combines all relationship feature embeddings in different modes; for the node-level attention mechanism, the kth node h in the l+1th layer is calculated. k The first fused embedding features under a specific relation r and a specific modality m; for the relation-level attention mechanism, further attention is paid to the relation level to integrate all relation-specific embeddings and evaluate the importance of each relation;

[0151] In addition, in heterogeneous graphs and graph neural networks, only recipe nodes have embedding features under different modalities, while user nodes and ingredient nodes do not have embedding features under different modalities. For example, assuming any user node, the representation of its corresponding first fused embedding feature and second fused embedding feature should not have modality m as a superscript. However, if the neighbor node of the current user node under relationship r is a recipe node, the representation of the neighbor node in the expression of the first fused embedding feature and the second fused embedding feature of the user node has modality m as a superscript. Therefore, in order to unify the formula representation in S2, modality m is added as a superscript in all formula expressions in S2.

[0152] S4: The network fusion embedding features of each recipe node in each mode are input into the multimodal feature decoupling module to calculate the important feature values ​​of each mode in each recipe node, and the expression is:

[0153]

[0154] in, Represents the i-th recipe node The important feature value under modality m; q1 represents the attention vector; W represents the importance weight; b represents the importance bias; Represents the i-th recipe node The network fusion embedding features under mode m; softmax(·) represents the softmax function;

[0155] Based on the important feature values ​​of each modality in each recipe node, the modality-shared embedding feature of each recipe node is calculated, and its expression is:

[0156]

[0157] in, Represents the i-th recipe node Modality-shared embedding features of ; Indicates the modal information corresponding to the recipe node, including text modality and visual modality;

[0158] Contrastive learning has been shown to be effective in various tasks, including graph representation learning and recommendation systems. Therefore, the present invention uses the self-supervisory signal in the contrastive learning framework to maximize the consistency between the recipe representation under a single modality and the fused multimodal representation to ensure that the fused multimodal representation can adaptively extract modality sharing information from the multimodal embedding, namely:

[0159] The contrast loss function is constructed with the goal of minimizing the difference between the network fusion embedding features of each recipe node in a single modality and the modality shared embedding features of each recipe node. Its expression is:

[0160]

[0161] in, Represents a collection of recipes, Indicates the number of all recipes; Indicates the modal information corresponding to the recipe. Represents the types of all modes; Represents the i-th recipe node The network fusion embedding features under modality m; Represents the i-th recipe node The modality shared embedding feature of ; F(·) represents the contrast evaluation function, which is expressed as:

[0162]

[0163] Wherein, θ(·) represents the cosine function; τ represents the temperature parameter; Indicates the i * Recipe Node The network fusion embedding features under modality m are represented as the i-th recipe node Negative samples under mode m; In a specific embodiment of the present invention, InfoNCE is used as a comparative evaluation to estimate and The mutual information consistency between the same recipe nodes and is defined as a positive sample, and the embedding corresponding to different recipe pairs of the same modality is defined as a negative sample, so as to obtain the comparative evaluation function value;

[0164] Existing multimodal recommendation models mainly focus on extracting common knowledge from different modalities, but often ignore the specific features of each modality; these modality-specific information plays a vital role in users' decision to choose recipes; therefore, in order to capture these feature representations, the network fusion embedding features of each recipe node in each modality are subtracted from the modality-shared embedding features of each recipe node to obtain the modality-specific representation of each recipe node in each modality, namely:

[0165] Based on the modal information corresponding to the recipe node, including text modality and visual modality, the network fusion embedding feature of each recipe node in the text modality is subtracted from the modality shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the text modality, which is expressed as:

[0166]

[0167] in, Represents the i-th recipe node modality-specific representation in textual modality; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Modality-shared embedding features of ;

[0168] The network fusion embedding feature of each recipe node in the visual modality is subtracted from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the visual modality, which is expressed as:

[0169]

[0170] in, Represents the i-th recipe node modality-specific representations in the visual modality; Represents the i-th recipe node Network fusion embedding features in visual modality;

[0171] Based on the modality-specific representation of each recipe node in the textual modality and the modality-specific representation in the visual modality, the modality-specific embedding feature of each recipe node is obtained, and its expression is:

[0172]

[0173] in, Represents the i-th recipe node The modality-specific embedding features of W s represents the modality-specific weight;

[0174] In order to ensure that the modality-specific representations of different modalities are as different as possible from the modality-shared representations, a soft orthogonal constraint is adopted, that is, the soft orthogonal constraint is constructed with the goal of maximizing the difference between the different modality-specific representations of each recipe node. Its expression is:

[0175]

[0176] in, Represents the i-th recipe node modality-specific representation in textual modality; Represents the i-th recipe node modality-specific representations in the visual modality;

[0177] The multimodal fusion operation is performed on the network fusion embedding features of each recipe node in various modes to obtain the modal fusion embedding features of each recipe node, which is expressed as:

[0178]

[0179] in, Represents the i-th recipe node The modality fusion embedding feature of W g represents the modal fusion weight; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Network fusion embedding features in visual modality;

[0180] In summary, in order to provide complementary insights for the recipe recommendation model, we decouple multimodal embeddings and propose contrastive fusion, which extracts modality-shared representations through contrastive learning, and then deploys orthogonal constraints to extract modality-specific representations. Given that recipe nodes have both text and visual embeddings, we design a cross-modal aggregator as a multi-module decoupling module to effectively merge these modalities, thereby enhancing the shared information between them.

[0181] S5: Based on the modality-shared embedding features, modality-specific embedding features, and modality-fusion embedding features of each recipe node, the counterfactual learning network is used to obtain the final counterfactual embedding features of each recipe node, including:

[0182] Since most of the current recommendation system-related methods are modeled from the perspective of data correlation, they try to mine the relevant patterns in the data for recommendation. However, the real world is composed of causal relationships rather than correlation relationships, and correlation relationships cannot replace causal relationships. Therefore, the present invention introduces a causal graph and introduces the relationship between each node in the causal graph, and maps the recommendation task to the causal world. At the same time, causal differences are calculated in the recipe recommendation process through causal inference analysis, and then a counterfactual learning network is designed to filter representations that are irrelevant to user preferences.

[0183] To systematically understand the difference between modality-shared and modality-specific features, a causal diagram is proposed like Figure 3 As shown, represents a set of variable nodes, ε o Represents a set of edges that represent causal relationships, and the direction of the arrows represents the causal dependency; Figure 3 The causal graph includes five variable nodes: I (multimodal content of user interaction recipes), C (modal shared information of multimodal content in user interaction recipes), S (modal specific information of multimodal content in user interaction recipes), U (user) and Y (prediction score); Figure 3 The tables in the causal diagram include:

[0184] Edge C→Y: Most multimodal recommendation frameworks emphasize extracting common knowledge across modalities, resulting in the prediction score being directly affected by the shared information of modalities without considering the more fine-grained specific modality, which leads to a spurious correlation between I and Y;

[0185] Edge U&I→Y: From the perspective of traditional recommendation tasks, the prediction score represents the user’s preference for recipes.

[0186] Edge C&S→O→Y: Y represents the user’s preferences for different dimensions of features of their interactive recipes, which are divided into modality-shared features and modality-specific features, thus providing a finer granularity in describing user preferences.

[0187] according to Figure 3 For the causal graph shown, the predicted score Y can be calculated from the value of one of its ancestor nodes U,I,C, and its expression is:

[0188] Y u,i,c =f Y (U=u,i=i,C=c);

[0189] Among them, f Y Represents a prediction function in a model, such as the inner product function.

[0190] After constructing the multimodal causal graph, the intervention operation in causal theory is used to analyze the causal relationship between variables; the intervention design of causal reasoning determines the value of the variable to make it independent of its parent node; this operation can remove the edge pointing to the node from the causal graph; in order to alleviate the bias in multimodal data that is unrelated to user preferences, it is necessary to build a world that is close to unbiased. Therefore, counterfactual reasoning can be used from the proposed multimodal causal graph to create a counterfactual world, such as Figure 3 As shown; in this counterfactual world, variable I is fixed to I * According to the d-separation criterion, this setting blocks the intermediary path from C to Y; variable C * and S * From the variable I * The recipe-shared features and recipe-specific features learned in C * and S * Not affected by the current variable I, the calculation expression of the predicted score is:

[0191]

[0192] The impact of user preference distribution on the final prediction score can be evaluated by the difference between the actual world and the counterfactual world. The expression of the final prediction score is:

[0193]

[0194] By computing the causal difference between the two worlds, the recipe recommendation model provided by the present invention is able to leverage modality-shared and modality-specific features to mitigate the side effects of irrelevant content in user interaction recipes (spurious correlations in multimodal recipe representations) on the final prediction score.

[0195] Based on the above causal difference analysis, we designed a counterfactual learning network to calculate the multimodal content preference distribution for user interaction recipes. The specific process is as follows:

[0196] S51: Using the counterfactual learning network, the modality-shared embedding features and modality-specific embedding features of each recipe node are combined with its modality-fusion embedding features to obtain the new modality-shared embedding features and new modality-specific embedding features of each recipe node; through matrix transformation, the query vector, key vector and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node are calculated respectively, and the preference distribution of the modality-shared content and modality-specific content of each recipe node is calculated respectively, and the expressions are:

[0197]

[0198] Among them, P iRepresents the i-th recipe node Preference distribution of modal shared content; Represents the i-th recipe node The preference distribution of modality-specific content of ; softmax(·) represents the softmax function; Represents the i-th recipe node The query vector corresponding to the new modality shared embedding feature; Represents the i-th recipe node The new modality shares the key vector corresponding to the embedding feature; Represents the i-th recipe node The query vector corresponding to the new modality-specific embedding features; Represents the i-th recipe node The key vector corresponding to the new modality-specific embedding feature of d; z Represents the dimensions of the query vector and key vector;

[0199] S52: Based on the causal difference theory, combined with the preference distribution of modal shared content and modal specific content of each recipe node, the preliminary counterfactual embedding feature of each recipe node is obtained, which is expressed as:

[0200]

[0201] in, Represents the i-th recipe node The initial counterfactual embedding features of P i Represents the i-th recipe node Preference distribution of modal shared content; Represents the i-th recipe node Preference distribution of modality-specific content; Represents the i-th recipe node The value vector corresponding to the new modality shared embedding feature;

[0202] S53: In order to stabilize the training process and solve the gradient vanishing problem, a feedforward network with residual connection is adopted; the feedforward network with residual connection includes: two linear layers and an activation function, as well as a residual connection and layer normalization;

[0203] The preliminary counterfactual embedding features of each recipe node are input into a feed-forward network with residual connections, and the final counterfactual embedding features of each recipe node are output, which are expressed as:

[0204]

[0205] in, Represents the i-th recipe node The final counterfactual embedding features of ; LN(·) represents the layer normalization operation; GELU(·) represents the nonlinear activation function; Represents the i-th recipe node The initial counterfactual embedding features of Represents the i-th recipe node To enhance the representation of the model, an L' layer of counterfactual learning layer is superimposed, namely W1 represents the weight corresponding to the nonlinear activation function; b1 represents the bias corresponding to the nonlinear activation function; W2 represents the weight corresponding to the layer normalization; b2 represents the bias corresponding to the layer normalization;

[0206] S6: Based on the modality-shared embedding features, modality-specific embedding features, and final counterfactual embedding features of each recipe node, the target embedding features of each recipe node are obtained; the network fusion embedding features of each user node output by the graph neural network are used as the target embedding features of each user node; the target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output, which is expressed as:

[0207]

[0208] in, Represents the u-th user node With the i-th recipe node The prediction score between u Represents the u-th user node The target embedding feature of z i Represents the i-th recipe node Target embedding features;

[0209] The BPR loss function is constructed with the goal that the prediction score between the user node and its interacted recipe node is higher than the prediction score between the user node and its non-interacted recipe node. Its expression is:

[0210]

[0211] Among them, Ω represents the set of user nodes and recipe nodes; the preset η represents the recipe node that interacts with the user, that is, the interactive recipe node, Represents a recipe node that has no interaction with the user, that is, an uninteracted recipe node. Represents the u-th user node The prediction score between the interactive recipe node η; Represents the u-th user node Not interacting with recipe nodes The prediction score between

[0212] Based on the contrast loss function, soft orthogonal constraint and BPR loss function, the total target loss function is obtained Its expression is:

[0213]

[0214] Among them, λ C , S They represent the hyperparameters corresponding to the contrast loss function and the soft orthogonal constraint respectively;

[0215] The recipes corresponding to the top ten predicted scores between the current user and each recipe are recommended to the current user after the predicted scores between the current user and the recipes are sorted from high to low.

[0216] In summary, based on S2 and S31, the present invention sets Represents a collection of users, Represents a collection of recipes, which contain information in two different modes Where t represents the textual modality and v represents the visual modality; the user-recipe-ingredient heterogeneous graph (URI-Graph) is represented as It contains three types of nodes and four types of edges; the recipe recommendation problem can be formalized as follows: Given such a heterogeneous graph Predicting user behavior by jointly modeling user behavior and multimodal recipe information Recipes used Possibility (probability) Based on S32, S4-S6, we can solve The value of; The flowchart of the recipe recommendation method provided by the present invention is as follows Figure 4 shown.

[0217] In summary, the recipe recommendation method based on multimodal counterfactual learning provided by the present invention includes five steps: heterogeneous graph construction, heterogeneous graph neural network learning node embedding, disentanglement learning of recipe multimodal embedding, and multimodal data correction, prediction and optimization of decoupled recipes using counterfactual learning networks. Compared with the prior art, the present invention proposes a multimodal recipe recommendation framework, which introduces modal decoupling learning and counterfactual learning networks on the basis of the basic multimodal recommendation model, effectively decoupling multimodal embedding into modal shared embedding and modal specific embedding; effectively solving the defects of the existing multimodal model that over-focuses on cross-modal knowledge of different modalities and ignores modal specific information, and at the same time, in order to alleviate the data bias in multimodal data, a counterfactual learning network is designed, which learns the user interaction recipe to calculate the multimodal content preference distribution by comparing the causal differences in the real world and the counterfactual world, so as to better learn the user's real preferences.

[0218] In order to verify the effectiveness of this method, a specific embodiment is provided through simulation experiments. The experiment is set up as follows:

[0219] LargeRG, a large multimodal recipe dataset, with detailed statistics such as Figure 5 As shown in the figure, the recipe data comes from the real recipe sharing website Food.com, which has recipe data shared by many users, such as recipe name, ingredients, methods, pictures, etc. Users will rate and comment on these recipes; the nutritional information about the ingredients comes from the nutrition data of the United States Department of Agriculture USDA; finally, the similarity data between recipes comes from the recipe knowledge graph FoodKG; the dataset is divided into training set, validation set and test set, with a ratio of 7:1.5:1.5.

[0220] The performance of the prediction set is evaluated using indicators widely used in the recommendation field: area under the ROC curve (AUC), hit rate (HR), normalized discounted cumulative gain (NDCG), and average precision (MAP); AUC measures the overall ranking performance of the test set, HR evaluates the relevance of the recommended content, NDCG evaluates the ranking performance from the user's perspective, and MAP evaluates the accuracy of the ranking position; an early stopping strategy is adopted, and the training process is stopped if the total loss of the validation set does not decrease in three consecutive cycles. The model parameters are set as follows: the embedding size is 128, the batch size is set to 1024, the learning rate is set to 0.005, and the Adam optimizer is used.

[0221] In order to evaluate the performance of the proposed method, several representative benchmark methods were experimentally compared:

[0222] 1) Traditional recommendation method BPR;

[0223] 2) Graph-based recommendation methods: GAT, RGCN, HAN, NGCF, and Light-GCN;

[0224] 3) Reciptor and Recipe2Vec, methods based on recipe representation learning;

[0225] 4) Recipe recommendation methods HAFR, SCHGN, RecipeRec;

[0226] 5) Recommended methods related to causal inference: InvRL and MCLN.

[0227] The experimental results compared with the method of the present invention are as follows Figure 6As shown, the method proposed in the present invention achieves the best recommendation effect under all indicators, proving the superiority of the method proposed in the present invention. The graph-based recommendation method is better than the traditional method, and among the graph recommendation methods, the method based on heterogeneous graphs is better than the recommendation method based on homogeneous graphs, so heterogeneous graphs and heterogeneous graph-related recommendation methods can effectively improve the quality of embedding; recipe representation learning performs poorly because these methods were originally designed for tasks such as recipe classification, clustering, and recipe retrieval, rather than for learning complex interactions between users and recipes; recipe-based recommendation methods and causal recommendation methods are usually better than other baselines, especially related methods based on multimodal recommendation, but the performance of these methods is still not as good as the method proposed in the present invention, which proves that modal disentanglement learning and counterfactual learning can better mitigate the side effects caused by information irrelevant to user preferences in recipe embeddings, thereby more accurately predicting user preferences.

[0228] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A recipe recommendation method based on multimodal counterfactual learning, characterized in that: include: Build a recipe recommendation model, including a heterogeneous graph building module, a graph neural network building module, a multimodal feature decoupling module, a counterfactual learning network, and a prediction module; Based on the user data and the recipe data, a user-recipe-ingredient heterogeneous graph is constructed; wherein the node types in the heterogeneous graph are user nodes, recipe nodes and ingredient nodes respectively; and the edge types in the heterogeneous graph are user-recipe edges, recipe-recipe edges, recipe-ingredient edges and ingredient-ingredient edges respectively; Perform an initial embedding operation on each node to obtain the initial feature representation of each node; construct a graph neural network based on the user-recipe-ingredient heterogeneous graph; use the initial feature representation of all nodes as the input of the graph neural network, use a two-level attention mechanism to process the initial feature representation of each node, and then pass it through the propagation layer to output the network fusion embedding features of each user node and the network fusion embedding features of each recipe node in each mode; The network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature values ​​of each modality in each recipe node; based on the important feature values ​​of each modality in each recipe node, the modality shared embedding features of each recipe node are calculated; the network fusion embedding features of each recipe node in each modality are subtracted from the modality shared embedding features of each recipe node to obtain the modality-specific representation of each recipe node in each modality; based on the modality-specific representation of each recipe node in each modality, the modality-specific embedding features of each recipe node are obtained; multimodal fusion operations are performed on the network fusion embedding features of each recipe node in various modalities to obtain the modality fusion embedding features of each recipe node; Using the counterfactual learning network, the modality-shared embedding features and modality-specific embedding features of each recipe node are combined with its modality-fusion embedding features to obtain the new modality-shared embedding features and new modality-specific embedding features of each recipe node; through matrix transformation, the query vector, key vector and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node are calculated respectively, and then the preference distribution of the modality-shared content and modality-specific content of each recipe node is calculated respectively; based on the causal difference theory, the preference distribution of the modality-shared content and modality-specific content of each recipe node is combined to obtain the preliminary counterfactual embedding features of each recipe node; the preliminary counterfactual embedding features of each recipe node are input into a feedforward network with residual connection, and the final counterfactual embedding features of each recipe node are output; Based on the modal shared embedding features, modal specific embedding features and final counterfactual embedding features of each recipe node, the target embedding features of each recipe node are obtained; the network fusion embedding features of each user node output by the graph neural network are used as the target embedding features of each user node; the target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output; the recipes corresponding to the top ten prediction scores between the current user and each recipe are sorted from high to low and are recommended to the current user.

2. According to claim 1, a recipe recommendation method based on multimodal counterfactual learning is characterized in that: The network fusion embedding features of each recipe node in each mode are input into the multimodal feature decoupling module to calculate the important characteristic values ​​of each mode in each recipe node; Based on the important feature values ​​of each modality in each recipe node, the modality-shared embedding features of each recipe node are calculated including: Using the softmax function, the important feature value of each mode in each recipe node is calculated, and its expression is: in, Represents the i-th recipe node The important feature value under modality m; q1 represents the attention vector; W represents the importance weight; b represents the importance bias; Represents the i-th recipe node The network fusion embedding features under mode m; softmax(·) represents the softmax function; Based on the important feature values ​​of each modality in each recipe and the network fusion embedding features of each recipe node in each modality, the modality shared embedding features of each recipe node are calculated, and the expression is: in, Represents the i-th recipe node Modality-shared embedding features of ; Represents the modal information corresponding to the recipe node, including text modality and visual modality.

3. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The network fusion embedding feature of each recipe node in each modality is subtracted from the modality-shared embedding feature of each recipe node to obtain a modality-specific representation of each recipe node in each modality; Based on the modality-specific representation of each recipe node in each modality, the modality-specific embedding features of each recipe node include: The modal information corresponding to the recipe node includes text modality and visual modality; The network fusion embedding feature of each recipe node in the text modality is subtracted from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the text modality, which is expressed as: in, Represents the i-th recipe node modality-specific representation in textual modality; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Modality-shared embedding features of ; The network fusion embedding feature of each recipe node in the visual modality is subtracted from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the visual modality, which is expressed as: in, Represents the i-th recipe node modality-specific representations in the visual modality; Represents the i-th recipe node Network fusion embedding features in visual modality; Based on the modality-specific representation of each recipe node in the textual modality and the modality-specific representation in the visual modality, the modality-specific embedding feature of each recipe node is obtained, and its expression is: in, Represents the i-th recipe node The modality-specific embedding features of W s Represents the modality specific weight.

4. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The multimodal fusion operation is performed on the network fusion embedding features of each recipe node in various modes to obtain the modal fusion embedding features of each recipe node. Its expression is: in, Represents the i-th recipe node The modality fusion embedding feature of W g represents the modal fusion weight; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Network fusion embedding features in visual modality.

5. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: Based on the query vector, key vector, and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node, the preference distribution of the modality-shared content and modality-specific content of each recipe node is calculated respectively; Based on the causal difference theory, combined with the preference distribution of modal shared content and modal specific content of each recipe node, the preliminary counterfactual embedding features of each recipe node are obtained, including: The expressions for the preference distribution of the modal shared content and modal specific content of each recipe node are: Among them, P i Represents the i-th recipe node Preference distribution of modal shared content; Represents the i-th recipe node The preference distribution of modality-specific content of ; softmax(·) represents the softmax function; Represents the i-th recipe node The query vector corresponding to the new modality shared embedding feature; Represents the i-th recipe node The new modality shares the key vector corresponding to the embedding feature; Represents the i-th recipe node The query vector corresponding to the new modality-specific embedding features; Represents the i-th recipe node The key vector corresponding to the new modality-specific embedding feature of d; z Represents the dimensions of the query vector and key vector; The expression of the preliminary counterfactual embedding feature of each recipe node is: in, Represents the i-th recipe node The initial counterfactual embedding features of P i Represents the i-th recipe node Preference distribution of modal shared content; Represents the i-th recipe node Preference distribution of modality-specific content; Represents the i-th recipe node The new modality shares the value vector corresponding to the embedded features.

6. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The preliminary counterfactual embedding features of each recipe node are input into a feed-forward network with residual connections, and the final counterfactual embedding features of each recipe node are outputted, including: The feed-forward network with residual connection includes: two linear layers and an activation function, as well as a residual connection and layer normalization; The final counterfactual embedding feature expression of each recipe node is: in, Represents the i-th recipe node The final counterfactual embedding features of ; LN(·) represents the layer normalization operation; GELU(·) represents the nonlinear activation function; Represents the i-th recipe node The initial counterfactual embedding features of Represents the i-th recipe node The modal shared embedding features of ; W1 represents the weight corresponding to the nonlinear activation function; b1 represents the bias corresponding to the nonlinear activation function; W2 represents the weight corresponding to the layer normalization; b2 represents the bias corresponding to the layer normalization.

7. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output, and its expression is: in, Represents the u-th user node With the i-th recipe node The prediction score between u Represents the u-th user node The target embedding feature of z i Represents the i-th recipe node The target embedding feature.

8. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The losses of the recipe recommendation model include: The contrast loss function is constructed with the goal of minimizing the difference between the network fusion embedding features of each recipe node in a single modality and the modality shared embedding features of each recipe node. Its expression is: in, Represents a collection of recipes, Indicates the number of all recipes; Indicates the modal information corresponding to the recipe. Represents the types of all modes; Represents the i-th recipe node The network fusion embedding features under modality m; Represents the i-th recipe node The modality shared embedding feature of ; F(·) represents the contrast evaluation function, which is expressed as: Wherein, θ(·) represents the cosine function; τ represents the temperature parameter; Indicates the i * Recipe Node The network fusion embedding features under modality m are represented as the i-th recipe node Negative samples under modality m; Soft orthogonality constraints are constructed with the goal of maximizing the difference between different modality-specific representations of each recipe node. Its expression is: in, Represents the i-th recipe node modality-specific representation in textual modality; Represents the i-th recipe node modality-specific representations in the visual modality; The BPR loss function is constructed with the goal that the prediction score between the user node and its interacted recipe node is higher than the prediction score between the user node and its non-interacted recipe node. Its expression is: Among them, Ω represents the set of user nodes and recipe nodes; the preset η represents the recipe node that interacts with the user, that is, the interactive recipe node, Represents a recipe node that has no interaction with the user, that is, a non-interactive recipe node. Represents the u-th user node The prediction score between the interactive recipe node η; Represents the u-th user node Not interacting with recipe nodes The prediction score between Based on the contrast loss function, soft orthogonal constraint and BPR loss function, the total target loss function is obtained Its expression is: Among them, λ C , S They represent the hyperparameters corresponding to the contrast loss function and the soft orthogonal constraint respectively.

9. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The two-level attention mechanism is used to process the initial feature representation of each node, including: The two-level attention mechanism includes: a node-level attention mechanism and a relation-level attention mechanism; The node-level attention mechanism includes: Using the shared weight matrix, the output features corresponding to each node in the lth layer of the graph neural network in each mode are linearly transformed to obtain the linear output features corresponding to each node in the lth layer in each mode, which is expressed as: in, represents the kth node h in the lth layer k The corresponding linear output characteristics under mode m; represents the weight corresponding to the relationship r and mode m in the lth layer; represents the kth node h in the lth layer k Input features under mode m; Based on the linear output features corresponding to each node in the lth layer in each mode, the attention score between each node in each mode in the lth layer is calculated, and its expression is: in, represents the kth node h in the lth layer under mode m k and the jth node h j The attention score between them; LeakyReLU(·) represents the improved ReLU activation function; represents the kth node h in the lth layer k and the jth node h j The ones between them represent the attention score weights; represents the jth node h in the lth layer j The corresponding linear output characteristics under mode m; The calculated attention scores are normalized to obtain the normalized attention scores between two nodes in each mode of the lth layer, which is expressed as: in, represents the kth node h in the lth layer under mode m k and the jth node h j The normalized attention score between ; represents the kth node h k The neighbor node under relation r, i.e. the kth * Nodes; L k,r represents the kth node h k The set of neighbor nodes under relation r; Indicates the kth * Nodes and the jth node h j Attention scores between Based on the normalized attention scores of each pair of nodes and between them in each modality of the lth layer, the first fused embedding feature of each node in the l+1th layer in each relationship and each modality is calculated, and its expression is: in, represents the kth node h in the l+1th layer k The first fused embedding feature under relation r and modality m; N represents the number of attention heads; n represents the index of the attention head; ReLU(·) represents the ReLU function; represents the jth node h in the lth layer j Input features under mode m; W represents the weight vector corresponding to the mode m in the lth layer; f represents a trainable weight vector; The relation-level attention mechanism includes: Using the shared nonlinear weights and intermediate vectors, the first fused embedding features of each node in the l+1th layer in each relationship and each modality are transformed to the new feature space; based on the intermediate vectors, the embedding similarity between the first fused embedding features of each node in the l+1th layer in each relationship and each modality before and after the feature space transformation is calculated, and its expression is: in, represents the kth node h k The corresponding embedding similarity under relation r and modality m; represents the set of nodes connected to relation r, represents the number of all nodes connected to the relationship r; q represents the intermediate vector; W R represents the relationship weight; represents the kth node h in the l+1th layer k The first fusion embedding feature under relation r and modality m; b * represents similarity bias; The embedding similarity is normalized to obtain the relation-level attention weight of each node in each relation and each modality, which is expressed as: in, represents the kth node h k Relation-level attention weights under relation r and modality m; Represents the relationship with the kth node h k A collection of connected relationships; Based on the first fused embedding feature of each node in the l+1th layer in each relationship and each modality, and the relation-level attention weight of each node in each relationship and each modality, the second fused embedding feature of each node in the l+1th layer in each modality is calculated, and its expression is: in, represents the kth node h in the l+1th layer k The second fused embedding features under modality m.

10. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that: The initializing embedding operation for each node to obtain the initial feature representation of each node includes: Using the Xavier initialization method, generate the common random initialization features corresponding to each user; use the common random initialization features corresponding to each user as the initial feature representation of each user node; The pre-trained jump instruction embedding is used as the representation of each instruction in the cooking steps of each recipe. The average of all instruction representations in the cooking steps of each recipe is calculated to obtain the average instruction corresponding to each recipe. The average instruction corresponding to each recipe is used as the cooking instruction embedding of each recipe; where the average instruction corresponding to any recipe is The expression is: Among them, z ins,a It represents the ath instruction in the cooking step of the current recipe; O represents the number of instructions in the current recipe; Use the pre-trained model to encode each recipe image and obtain the image embedding of each recipe; project the cooking instruction embedding and image embedding of each recipe into the same shared input space as the initial feature representation of each recipe node; The nutrients of the components are projected into vectors to obtain the vector corresponding to each component; the vector corresponding to each component is used as the initial feature representation of each component node.

Images