A recipe recommendation method based on multi-modal counterfactual learning

Abstract

The present application relates to the technical field of information recommendation, and particularly relates to a recipe recommendation method based on multi-modal counterfactual learning, which comprises the following steps: constructing a heterogeneous graph; obtaining network fusion embedding features of each user node as target embedding features of each user node, and network fusion embedding features of each recipe node in each mode by using a graph neural network; obtaining mode-shared embedding features, mode-specific embedding features and mode fusion embedding features of each recipe node by using a multi-modal feature decoupling module based on the network fusion embedding features of each recipe node in each mode, and then obtaining final counterfactual embedding features of each recipe node by using a counterfactual learning network; fusing the mode-shared embedding features, the mode-specific embedding features and the final counterfactual embedding features of each recipe node as target embedding features of each recipe node; and inputting the target embedding features of each user node and recipe node into a prediction module to output predicted scores between each user and each recipe. The present application improves the recipe recommendation accuracy.

Description

Technical Field

[0001] This invention relates to the field of information recommendation technology, and in particular to a recipe recommendation method based on multimodal counterfactual learning. Background Technology

[0002] Recipe recommendation aims to analyze users' historical interactions and recipe characteristics to determine their true preferences for recipes, thereby providing personalized recipe suggestions. People's recipe preferences are complex, influenced by factors such as recipe type, flavor, ingredients, images, and cooking complexity. Therefore, a comprehensive recipe recommendation system cannot rely solely on historical user-recipe interactions.

[0003] To address this issue, it is necessary to consider as many different characteristics of users and recipes as possible, and to quantify the impact of these characteristics on users' recipe choices by establishing mathematical models. Currently, multimodal recommendation systems typically consist of two parts: multimodal feature extraction and user preference modeling.

[0004] In the multimodal feature extraction part, existing methods typically use pre-trained deep neural networks or graph convolutional networks to extract information from different modalities, and then fuse the information from different modalities into the recommendation framework for recommendation. However, existing methods ignore the impact of modal common features and modality-specific features on user interaction, so the model cannot grasp the common factors that influence users' preferences for recipes as a whole, resulting in a one-sided understanding of user preferences. When fusing heterogeneous information from different modalities, accurate alignment of cross-modal data is crucial for accurately understanding recipe features and grasping user preferences. However, existing methods focus on how to extract common knowledge across modalities, and modality-specific features are often ignored. In practical applications, even for items that users have already interacted with, users may like a recipe because of a specific feature of a single modality. Modality common features are actually features that are irrelevant to user preferences. If modality-specific features are ignored, the model will not be able to capture these important details that influence user choices, making it difficult to accurately recommend recipes that match users' true preferences, reducing the accuracy of recipe recommendations, homogenizing recommendation results, and failing to meet personalized needs.

[0005] In the user preference modeling part, most existing models model from the perspective of data relevance, attempting to make recommendations by mining relevant patterns in the data. However, the real world is composed of causal relationships rather than relevance relationships, and relevance relationships cannot replace causal relationships. Furthermore, some features in the multimodal information of items (modal common features and modal specific features) are unrelated to user preferences. This may be a false relevance to user preferences, which will reduce the performance of the recommendation model and damage the diversity of the recommendation model. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problems in the prior art that ignore the influence of modal common features and modal specific features on user interaction, so that the model cannot grasp the common factors affecting users' preferences for recipes as a whole, resulting in a one-sided understanding of user preferences; ignore modal specific features, resulting in reduced recipe recommendation accuracy, homogenized recommendation results and inability to meet personalized needs; and ignore the causal relationship between users and recipes, resulting in reduced recommendation model performance and lack of recommendation model diversity.

[0007] To address the aforementioned technical problems, this invention provides a recipe recommendation method based on multimodal counterfactual learning, comprising:

[0008] A recipe recommendation model is constructed, including a heterogeneous graph construction module, a graph neural network construction module, a multimodal feature decoupling module, a counterfactual learning network, and a prediction module;

[0009] Based on user data and recipe data, a user-recipe-ingredient heterogeneous graph is constructed; wherein, the node types in the heterogeneous graph are user nodes, recipe nodes, and ingredient nodes, and the edge types in the heterogeneous graph are user-recipe edges, recipe-recipe edges, recipe-ingredient edges, and ingredient-ingredient edges, respectively.

[0010] An initial embedding operation is performed on each node to obtain the initial feature representation of each node; a graph neural network is constructed based on the user-recipe-ingredient heterogeneous graph; the initial feature representations of all nodes are used as input to the graph neural network, and a two-level attention mechanism is used to process the initial feature representation of each node, and then the data is passed through a propagation layer to output the network fusion embedding features of each user node and the network fusion embedding features of each recipe node in each modality.

[0011] The network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature value of each modality in each recipe node. Based on the important feature value of each modality in each recipe node, the modality shared embedding features of each recipe node are calculated. The network fusion embedding features of each recipe node in each modality are subtracted from the modality shared embedding features of each recipe node to obtain the modality-specific representation of each recipe node in each modality. Based on the modality-specific representation of each recipe node in each modality, the modality-specific embedding features of each recipe node are obtained. Multimodal fusion operation is performed on the network fusion embedding features of each recipe node in various modalities to obtain the modality fusion embedding features of each recipe node.

[0012] Using a counterfactual learning network, the modality-shared embedding features and modality-specific embedding features of each recipe node are combined with their modality-fused embedding features to obtain new modality-shared embedding features and new modality-specific embedding features for each recipe node. Through matrix transformation, the query vector, key vector, and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node are calculated. Then, the preference distributions of the modality-shared content and modality-specific content of each recipe node are calculated. Based on causal difference theory, the preliminary counterfactual embedding features of each recipe node are obtained by combining the preference distributions of the modality-shared content and modality-specific content of each recipe node. The preliminary counterfactual embedding features of each recipe node are input into a feedforward network with residual connections to output the final counterfactual embedding features of each recipe node.

[0013] Based on the modality-shared embedding features, modality-specific embedding features, and final counterfactual embedding features of each recipe node, the target embedding features of each recipe node are obtained. The network fusion embedding features of each user node output by the graph neural network are used as the target embedding features of each user node. The target embedding features of each user node and recipe node are input into the prediction module, which outputs the prediction score between each user and each recipe. The top ten recipes corresponding to the prediction scores between the current user and each recipe, sorted from high to low, are recommended to the current user.

[0014] Preferably, the step of inputting the network fusion embedding features of each recipe node in each modality into the multimodal feature decoupling module to calculate the important feature values ​​of each modality in each recipe node; and calculating the modality-shared embedding features of each recipe node based on the important feature values ​​of each modality in each recipe node includes:

[0015] The important characteristic value of each modality in each recipe node is calculated using the softmax function, and its expression is as follows:

[0016]

[0017] in, Represents the i-th recipe node The important feature values ​​in modality m; q1 represents the attention vector; W represents the importance weight; b represents the importance bias; Represents the i-th recipe node Network fusion embedding features under modality m; softmax(·) denotes the softmax function;

[0018] Based on the important feature values ​​of each modality in each recipe and the network fusion embedding features of each recipe node in each modality, the modality shared embedding features of each recipe node are calculated, and their expression is as follows:

[0019]

[0020] in, Represents the i-th recipe node Modal sharing embedding features; This represents the modal information corresponding to the recipe node, including text modality and visual modality.

[0021] Preferably, the step of subtracting the modality-shared embedding feature of each recipe node in each modality from the network fusion embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in each modality; and obtaining the modality-specific embedding feature of each recipe node based on the modality-specific representation of each recipe node in each modality includes:

[0022] The modal information corresponding to the recipe node includes text modality and visual modality;

[0023] Subtracting the network fusion embedding feature of each recipe node in the text modality from the modality-shared embedding feature of each recipe node yields the modality-specific representation of each recipe node in the text modality, expressed as:

[0024]

[0025] in, Represents the i-th recipe node Modality-specific representation in text modality; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Modal sharing embedding features;

[0026] Subtracting the network fusion embedding feature of each recipe node in the visual modality from the modality-shared embedding feature of each recipe node yields the modality-specific representation of each recipe node in the visual modality, expressed as:

[0027]

[0028] in, Represents the i-th recipe node Modality-specific representation in visual modality; Represents the i-th recipe node Network fusion and embedding features in visual modalities;

[0029] Based on the modality-specific representations of each recipe node in the text modality and the visual modality, the modality-specific embedding features of each recipe node are obtained, and their expressions are as follows:

[0030]

[0031] in, Represents the i-th recipe node Modality-specific embedding features; W s This represents modality-specific weights.

[0032] Preferably, the multimodal fusion operation is performed on the network fusion embedding features of each recipe node under various modalities to obtain the modal fusion embedding features of each recipe node, the expression of which is:

[0033]

[0034] in, Represents the i-th recipe node Modal fusion embedding features; W g Represents the modal fusion weights; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Network fusion embedding features in the visual modality.

[0035] Preferably, based on the new modality-shared embedding features and the query vector, key vector, and value vector corresponding to the new modality-specific embedding features of each recipe node, the preference distributions of the modality-shared content and modality-specific content of each recipe node are calculated respectively; based on the causal difference theory, combined with the preference distributions of the modality-shared content and modality-specific content of each recipe node, the preliminary counterfactual embedding features of each recipe node are obtained, including:

[0036] The expressions for the preference distribution of modal shared content and modal specific content for each recipe node are as follows:

[0037]

[0038] Among them, P i Represents the i-th recipe node The distribution of preferences for modal shared content; Represents the i-th recipe node The modality-specific content preference distribution; softmax(·) denotes the softmax function; Represents the i-th recipe node The query vector corresponding to the new modality shared embedded features; Represents the i-th recipe node The key vector corresponding to the new modality shared embedded features; Represents the i-th recipe node The query vector corresponding to the new modality-specific embedded features; Represents the i-th recipe node The key vector corresponding to the new modality-specific embedding features; d z The dimensions representing the query vector and key vector;

[0039] The expression for the initial counterfactual embedding feature of each recipe node is:

[0040]

[0041] in, Represents the i-th recipe node Preliminary counterfactual embedding features; P i Represents the i-th recipe node The distribution of preferences for modal shared content; Represents the i-th recipe node The distribution of preferences for modality-specific content; Represents the i-th recipe node The new modality shares the value vector corresponding to the embedded features.

[0042] Preferably, the step of inputting the preliminary counterfactual embedding features of each recipe node into a feedforward network with residual connections and outputting the final counterfactual embedding features of each recipe node includes:

[0043] The feedforward network with residual connections includes: two linear layers and an activation function, as well as a residual connection and layer normalization;

[0044] The expression for the final counterfactual embedding feature of each recipe node is:

[0045]

[0046] in, Represents the i-th recipe node The final counterfactual embedding feature; LN(·) denotes the layer normalization operation; GELU(·) denotes the nonlinear activation function; Represents the i-th recipe node Preliminary counterfactual embedding features, Represents the i-th recipe node The modality-shared embedding features; W1 represents the weights corresponding to the nonlinear activation function; b1 represents the biases corresponding to the nonlinear activation function; W2 represents the weights corresponding to the layer normalization; b2 represents the biases corresponding to the layer normalization.

[0047] Preferably, the step of inputting the target embedding features of each user node and recipe node into the prediction module and outputting the prediction score between each user and each recipe is expressed as follows:

[0048]

[0049] in, Represents the u-th user node With the i-th recipe node Predicted scores between; z u Represents the u-th user node Target embedding features; z i Represents the i-th recipe node The target embedding features.

[0050] Preferably, the loss of the recipe recommendation model includes:

[0051] A contrastive loss function is constructed with the goal of minimizing the difference between the network fusion embedding features of each recipe node in a single modality and the modality-shared embedding features of each recipe node. Its expression is:

[0052]

[0053] in, Represents a set of recipes. Indicates the quantity of all recipes; This represents the modal information corresponding to the recipe. Indicates the types of all modes; Represents the i-th recipe node Network fusion embedding features under modality m; Represents the i-th recipe node The modality-shared embedding features; F(·) represents the contrastive evaluation function, whose expression is:

[0054]

[0055] Where θ(·) represents the cosine function; τ represents the temperature parameter; Indicates the i-th * Recipe Nodes The network fusion embedding features in modality m are represented as the i-th recipe node. Negative samples in mode m;

[0056] To maximize the difference between different modality-specific representations of each recipe node, soft orthogonal constraints are constructed. Its expression is:

[0057]

[0058] in, Represents the i-th recipe node Modality-specific representation in text modality; Represents the i-th recipe node Modality-specific representation in visual modality;

[0059] The BPR loss function is constructed with the objective that the predicted score between a user node and the recipe node it interacts with is higher than the predicted score between a user node and the recipe node it does not interact with. Its expression is:

[0060]

[0061] Where Ω represents the set of user nodes and recipe nodes; the preset η represents the recipe nodes that interact with the user, i.e., interactive recipe nodes. This indicates a recipe node that has not interacted with the user, i.e., a non-interactive recipe node. Represents the u-th user node The predicted score between the interactive recipe node η; Represents the u-th user node With non-interactive recipe nodes Predicted scores between;

[0062] Based on the contrastive loss function, soft orthogonality constraints, and BPR loss function, the total objective loss function is obtained. Its expression is:

[0063]

[0064] Where, λ C , λ S These represent the hyperparameters corresponding to the contrastive loss function and the soft orthogonal constraint, respectively.

[0065] Preferably, the two-level attention mechanism for processing the initial feature representation of each node includes:

[0066] The two-level attention mechanism includes: node-level attention mechanism and relation-level attention mechanism;

[0067] The node-level attention mechanism includes:

[0068] Using the shared weight matrix, a linear transformation is performed on the output features of each node in the l-th layer of the graph neural network under each modality, resulting in the linear output features of each node in the l-th layer under each modality, expressed as:

[0069]

[0070] in, This represents the k-th node h in the l-th layer. k The linear output feature corresponding to mode m; This represents the weights in the l-th layer corresponding to relation r and mode m; This represents the k-th node h in the l-th layer. k Input features in modality m;

[0071] Based on the linear output features of each node in layer l in each modality, the attention score between each pair of nodes in layer l in each modality is calculated, and its expression is as follows:

[0072]

[0073] in, This represents the k-th node h in the l-th layer under mode m. k and the j-th node h j Attention scores between; LeakyReLU(·) represents the improved ReLU activation function; This represents the k-th node h in the l-th layer. k and the j-th node h j The weights between the attention scores represent the values; This represents the j-th node h in the l-th layer. j The linear output feature corresponding to mode m;

[0074] Normalize the calculated attention scores to obtain the normalized attention scores between pairs of nodes in each modality of layer l, as expressed by:

[0075]

[0076] in, This represents the k-th node h in the l-th layer under mode m. k and the j-th node h j Normalized attention scores between them; Represents the l-th node h k The neighboring nodes under relation r, i.e., the k-th node. * One node; L k,r Represents the k-th node h k The set of neighboring nodes under relation r; This indicates that in the l-th layer, the k-th mode under mode m... * Nodes and the j-th node h j Attention score between;

[0077] Based on the normalized attention scores of each pair of nodes and between them in each modality of layer l, the first fused embedding feature of each node in layer l+1 under each relationship and modality is calculated, and its expression is:

[0078]

[0079] in, This represents the k-th node h in the (l+1)-th layer. k The first fused embedding feature under relation r and modality m; N represents the number of attention heads; n represents the index of the attention head; ReLU(·) represents the ReLU function; This represents the j-th node h in the l-th layer. j Input features in modality m; W represents the weight vector in the l-th layer corresponding to mode m; f This represents a trainable weight vector;

[0080] The relation-level attention mechanism includes:

[0081] Using shared nonlinear weights and intermediate vectors, the first fused embedding features of each node in layer l+1 under each relation and modality are transformed to a new feature space. Based on the intermediate vectors, the embedding similarity between the first fused embedding features of each node in layer l+1 before and after the feature space transformation under each relation and modality is calculated, and its expression is as follows:

[0082]

[0083] in, Represents the k-th node h k Embedding similarity under relation r and modality m; This represents the set of nodes connected to relation r. The number of nodes connected to relation r is represented by q; the intermediate vector is represented by W. R Indicates relation weight; This represents the k-th node h in the (l+1)-th layer. k The first fused embedding feature under relation r and modality m; b * Indicates similarity bias;

[0084] The embedding similarity is normalized to obtain the relation-level attention weight of each node under each relation and each modality, which is expressed as:

[0085]

[0086] in, Represents the k-th node h k Relation-level attention weights under relation r and modality m; Indicates the relationship with the k-th node h k A set of connected relationships;

[0087] Based on the first fused embedding features of each node in layer l+1 under each relation and each modality, and the relation-level attention weights of each node under each relation and each modality, the second fused embedding features of each node in layer l+1 under each modality are calculated, and their expressions are as follows:

[0088]

[0089] in, This represents the k-th node h in the (l+1)-th layer. k The second fusion embedding feature under modality m.

[0090] Preferably, the initial embedding operation for each node to obtain the initial feature representation of each node includes:

[0091] Using the Xavier initialization method, generate ordinary random initialization features for each user; use the ordinary random initialization features for each user as the initial feature representation for each user node;

[0092] A pre-trained jump instruction embedding is used as the representation of each instruction in the cooking steps of each recipe. The average of all instruction representations in the cooking steps of each recipe is calculated to obtain the average instruction for each recipe. This average instruction is then embedded as the cooking description for each recipe. The average instruction for any given recipe... The expression is:

[0093]

[0094] Among them, z ins,a This represents the a-th instruction in the cooking steps of the current recipe; O represents the number of instructions in the current recipe.

[0095] Each recipe image is encoded using a pre-trained model to obtain the image embedding for each recipe; the cooking instructions embedding and the image embedding for each recipe are projected into the same shared input space as the initial feature representation for each recipe node;

[0096] The nutrients of each component are projected into vectors to obtain the vector corresponding to each component; the vector corresponding to each component is used as the initial feature representation of each component node.

[0097] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0098] This invention discloses a recipe recommendation method based on multimodal counterfactual learning. It constructs a user-recipe-ingredient heterogeneous graph and integrates relevant recipe information into this graph. Simultaneously, it considers the information of recipe nodes under different modalities, contributing to a more comprehensive understanding of recipe features. Through a graph neural network algorithm, it learns the node attributes and structural information in the heterogeneous graph, considering the complex relationships between each node and the differences in the importance of different relationships, enriching the feature representation of the nodes and obtaining the network fusion embedding features of each user node and the network fusion embedding of each recipe node under each modality. Utilizing a designed multimodal feature decoupling learning algorithm, and employing contrastive learning to ensure adaptive extraction of modality-shared information from the network fusion embeddings of recipe nodes under each modality, it uses orthogonal constraints to ensure that the modality-specific representations of different modalities are as different as possible from the modality-shared representations, thereby extracting modality-specific representations. This decouples the network fusion embedding features of recipe nodes under various modalities into modality-shared features and modality-specific features. By comprehensively considering the influence of modality-shared information and modality-specific information of different modalities on user interaction in the multimodal model, it improves the model's ability to understand user preferences for recipes. This approach achieves a holistic understanding while avoiding homogenization in recipe recommendations, thus meeting users' personalized needs. Utilizing a counterfactual learning network, it learns the multimodal content preference distribution of user interaction recipes by comparing causal differences between the real and counterfactual worlds. This distribution includes preferences for modal shared information and modal-specific information, eliminating data bias in the decoupled multimodal features. This filters representations irrelevant to user preferences and better learns the user's true preferences, thereby improving the accuracy, performance, and diversity of recipe recommendations. Based on the recipe embeddings enhanced by multimodal feature decoupling and counterfactual learning, target embedding features for recipe nodes are obtained. User embeddings learned in the graph neural network are used as target embedding features for user nodes. Based on the target embedding features of recipe nodes and user nodes, the final user's recommendation score for the recipe is obtained. Personalized recipes are recommended to users based on these scores, resulting in a high-precision, robust, and scalable multimodal recipe recommendation model. This model can provide different users with highly relevant, personalized, and diverse recipes tailored to their preferences. Attached Figure Description

[0099] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0100] Figure 1 This is a schematic diagram of a recipe recommendation method based on multimodal counterfactual learning provided by the present invention;

[0101] Figure 2This is a detailed framework diagram of a recipe recommendation method based on multimodal counterfactual learning provided by the present invention;

[0102] Figure 3 This is a schematic diagram of a cause-and-effect graph;

[0103] Figure 4 This is a flowchart of a recipe recommendation method based on multimodal counterfactual learning provided by the present invention;

[0104] Figure 5 It contains detailed statistical charts of the large multimodal recipe dataset LargeRG;

[0105] Figure 6 This is a chart comparing the experimental results. Detailed Implementation

[0106] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0107] Reference Figure 1 , Figure 2 As shown, the recipe recommendation method based on multimodal counterfactual learning provided by this invention includes:

[0108] S1: Construct a recipe recommendation model, including a heterogeneous graph construction module, a graph neural network construction module, a multimodal feature decoupling module, a counterfactual learning network, and a prediction module;

[0109] S2: Based on user data and recipe data, construct a user-recipe-ingredient heterogeneous graph; wherein, the node types in the heterogeneous graph are user nodes, recipe nodes, and ingredient nodes; and the edge types in the heterogeneous graph are user-recipe edges, recipe-recipe edges, recipe-ingredient edges, and ingredient-ingredient edges.

[0110] In one specific embodiment of the present invention, the user and recipe data comes from Food.com, the world's largest recipe sharing website, which has a wealth of recipe data. The recipe data includes the recipe name, ingredients, nutrients, food images, and preparation methods. Each recipe has user comments and ratings ranging from 1 to 5. Based on the relevant attributes of the user and recipe data, a heterogeneous graph containing three types of nodes and four types of edges is constructed as a User-Recipe-Ingredient Heterogeneous Graph (URI-Graph).

[0111] The construction process for each type of edge in a heterogeneous graph includes:

[0112] By rating recipes by users, user-recipe relationships are established, resulting in user-recipe edges in the user-recipe-ingredient heterogeneity graph.

[0113] By using a public recipe knowledge graph, FoodKG, after determining the similarity between recipes, recipe-recipe relationships are constructed to obtain recipe-recipe edges in a user-recipe-ingredient heterogeneous graph.

[0114] By establishing the recipe-ingredient relationship through the ingredient list of the recipe, the recipe-ingredient edge in the user-recipe-ingredient heterogeneous graph is obtained;

[0115] Based on the normalized pointwise mutual information method, after obtaining the probability of common occurrence between various components, component-component relationships are constructed to obtain component-component edges in the user-recipe-component heterogeneous graph.

[0116] S3: The processing steps in the graph neural network building module include:

[0117] S31: Perform initial embedding operations on each node to obtain the initial feature representation of each node, including:

[0118] Since most users on Food.com have not filled in complete personal information (e.g., description, location, preferences, etc.), the Xavier initialization method is used to generate ordinary random initialization features for each user. These ordinary random initialization features are then used as the initial feature representation for each user node, which can be represented by z. user express;

[0119] Previous research has shown that cooking steps are crucial for providing distinctiveness in the cooking process. Therefore, this invention employs a pre-trained skip instruction embedding (an embedding of a two-stage LSTM) as the representation of each instruction in the cooking steps of each recipe. The average of all instruction representations in the cooking steps of each recipe is calculated to obtain the average instruction for each recipe. This average instruction is then used as the cooking description embedding for each recipe. The average instruction for any given recipe... The expression is:

[0120]

[0121] Among them, z ins,a This represents the a-th instruction in the cooking steps of the current recipe; O represents the number of instructions in the current recipe.

[0122] A pre-trained ResNet model is used to encode each recipe image, resulting in an image embedding for each recipe. The cooking instructions embedding and the image embedding for each recipe are projected into the same shared input space as the initial feature representation for each recipe node. Therefore, the initial text feature representation for each recipe node can be obtained using... This means that the initial visual feature representation of each recipe node can be used... express;

[0123] Projecting the nutrients of each component into a vector yields the vector corresponding to each component; using this vector as the initial feature representation for each component node, we can use z... ing express;

[0124] S32: Construct a graph neural network based on the user-recipe-ingredient heterogeneous graph; use the initial feature representations of all nodes as input to the graph neural network, employ a two-level attention mechanism to process the initial feature representations of each node, and then pass them through a propagation layer to output the network fusion embedding features of each user node and the network fusion embedding features of each recipe node in each modality; where, for the u-th user node... Its network fusion embedding feature is z u For the i-th recipe node Its corresponding text modal features are and visual modal features

[0125] The two-level attention mechanism is used to process the initial feature representation of each node, including:

[0126] The two-level attention mechanism includes: node-level attention mechanism and relation-level attention mechanism;

[0127] The node-level attention mechanism includes:

[0128] Using the shared weight matrix, a linear transformation is performed on the output features of each node in the l-th layer of the graph neural network under each modality, resulting in the linear output features of each node in the l-th layer under each modality, expressed as:

[0129]

[0130] in, This represents the k-th node h in the l-th layer. k The linear output feature corresponding to mode m; d represents the weights in the l-th layer corresponding to relation r and mode m. l d represents the dimension of the input embedding at layer l. l+1 This represents the dimension of the input embedding at layer l+1; This represents the k-th node h in the l-th layer. k Input features in modality m;

[0131] Based on the linear output features of each node in layer l in each modality, the attention score between each pair of nodes in layer l in each modality is calculated, and its expression is as follows:

[0132]

[0133] in, This represents the k-th node h in the l-th layer under mode m. k and the j-th node h j Attention scores between; LeakyReLU(·) represents the improved ReLU activation function; This represents the k-th node h in the l-th layer. k and the j-th node h j The weights between the attention scores represent the values; This represents the j-th node h in the l-th layer. j The linear output feature corresponding to mode m;

[0134] Normalize the calculated attention scores to obtain the normalized attention scores between pairs of nodes in each modality of layer l, as expressed by:

[0135]

[0136] in, This represents the k-th node h in the l-th layer under mode m. k and the j-th node h j Normalized attention scores between them; Represents the k-th node h k The neighboring nodes under relation r, i.e., the k-th node. * One node; L k,r Represents the k-th node h k The set of neighboring nodes under relation r; This indicates that in the l-th layer, the k-th mode under mode m... * Nodes and the j-th node h j Attention score between;

[0137] Based on the normalized attention scores of each pair of nodes and between them in each modality of layer l, the first fused embedding feature of each node in layer l+1 under each relationship and modality is calculated, and its expression is:

[0138]

[0139] in, This represents the k-th node h in the (l+1)-th layer. k The first fused embedding feature under relation r and modality m; N represents the number of attention heads; n represents the index of the attention head; d n Denotes the dimension of the attention head, which satisfies d n =d l+1 / N; ReLU(·) represents the ReLU function; This represents the j-th node h in the l-th layer. j Input features in modality m; W represents the weight vector in the l-th layer corresponding to mode m; f This represents a trainable weight vector;

[0140] The relation-level attention mechanism includes:

[0141] Using shared nonlinear weights and intermediate vectors, the first fused embedding features of each node in layer l+1 under each relation and modality are transformed to a new feature space. Based on the intermediate vectors, the embedding similarity between the first fused embedding features of each node in layer l+1 before and after the feature space transformation under each relation and modality is calculated, and its expression is as follows:

[0142]

[0143] in, Represents the k-th node h k Embedding similarity under relation r and modality m; This represents the set of nodes connected to relation r. This represents the number of all nodes connected to relation r; W represents the intermediate vector; R Indicates relation weight; This represents the k-th node h in the (l+1)-th layer. k The first fused embedding feature under relation r and modality m; b * Indicates similarity bias;

[0144] The embedding similarity is normalized to obtain the relation-level attention weight of each node under each relation and each modality, which is expressed as:

[0145]

[0146] in, Represents the k-th node h k Relation-level attention weights under relation r and modality m; Indicates the relationship with the k-th node h k A set of connected relationships;

[0147] Based on the first fused embedding features of each node in layer l+1 under each relation and each modality, and the relation-level attention weights of each node under each relation and each modality, the second fused embedding features of each node in layer l+1 under each modality are calculated, and their expressions are as follows:

[0148]

[0149] in, This represents the k-th node h in the (l+1)-th layer. k The second fusion embedding feature under modality m;

[0150] In summary, for URI-Graph, a graph representation learning method with a two-level attention mechanism is needed to mine the attribute and structural information on the graph. For each node, the node-level attention mechanism fuses adjacent nodes under each specific relation and module. Subsequently, the relation-level attention mechanism combines the embeddings of all relation features under different patterns. For the node-level attention mechanism, the h of the k-th node in the (l+1)-th layer is calculated. k The first fusion embedding features under a specific relation r and a specific modality m; for the relation-level attention mechanism, further attention is paid to relation-level attention to integrate all relation-specific embeddings and evaluate the importance of each relation;

[0151] Furthermore, in heterogeneous graphs and graph neural networks, only recipe nodes have embedding features under different modalities, while user nodes and component nodes do not have embedding features under different modalities. For example, assuming that for any user node, the representation of its corresponding first fusion embedding feature and second fusion embedding feature should not have modality m as a superscript, but if the neighbor node of the current user node under relation r is a recipe node, the representation of the neighbor node in the expression of the first fusion embedding feature and second fusion embedding feature of the user node has modality m as a superscript. Therefore, in order to unify the formula representation in S2, modality m is added as a superscript in all formula expressions in S2.

[0152] S4: The network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature value of each modality in each recipe node. The expression is as follows:

[0153]

[0154] in, Represents the i-th recipe node The important feature values ​​in modality m; q1 represents the attention vector; W represents the importance weight; b represents the importance bias; Represents the i-th recipe node Network fusion embedding features under modality m; softmax(·) denotes the softmax function;

[0155] Based on the important feature values ​​of each modality in each recipe node, the modality shared embedding feature of each recipe node is calculated, and its expression is as follows:

[0156]

[0157] in, Represents the i-th recipe node Modal sharing embedding features; This represents the modal information corresponding to the recipe node, including text modality and visual modality;

[0158] Contrastive learning has proven effective in various tasks, including graph representation learning and recommendation systems. Therefore, this invention employs self-supervised signals within a contrastive learning framework to maximize the consistency between recipe representations in a single modality and fused multimodal representations. This ensures that the fused multimodal representation can adaptively extract modality-shared information from the multimodal embeddings, i.e.:

[0159] A contrastive loss function is constructed with the goal of minimizing the difference between the network fusion embedding features of each recipe node in a single modality and the modality-shared embedding features of each recipe node. Its expression is:

[0160]

[0161] in, Represents a set of recipes. Indicates the quantity of all recipes; This represents the modal information corresponding to the recipe. Indicates the types of all modes; Represents the i-th recipe node Network fusion embedding features under modality m; Represents the i-th recipe node The modality-shared embedding features; F(·) represents the contrastive evaluation function, whose expression is:

[0162]

[0163] Where θ(·) represents the cosine function; τ represents the temperature parameter; Indicates the i-th * Recipe Nodes The network fusion embedding features in modality m are represented as the i-th recipe node. Negative samples in mode m; in one specific embodiment of the invention, InfoNCE is used as a contrastive evaluation to estimate. and Mutual information consistency between them, that is, the consistency of information between nodes of the same recipe. and Positive samples are defined as those embedded with different recipe pairs corresponding to the same modality, while negative samples are defined as those embedded with negative samples, thus obtaining the comparison evaluation function value;

[0164] Existing multimodal recommendation models primarily focus on extracting common knowledge across different modalities, often neglecting the specific features of each modality. This modality-specific information plays a crucial role in users' recipe selection decisions. Therefore, to capture these feature representations, the network fusion embedding features of each recipe node in each modality are subtracted from the modality-shared embedding features of each recipe node to obtain the modality-specific representation of each recipe node in each modality, i.e.:

[0165] Based on the modal information corresponding to recipe nodes, including text modality and visual modality, the network fusion embedding feature of each recipe node in the text modality is subtracted from the modality-shared embedding feature of each recipe node to obtain the modality-specific representation of each recipe node in the text modality, which is expressed as follows:

[0166]

[0167] in, Represents the i-th recipe node Modality-specific representation in text modality; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Modal sharing embedding features;

[0168] Subtracting the network fusion embedding feature of each recipe node in the visual modality from the modality-shared embedding feature of each recipe node yields the modality-specific representation of each recipe node in the visual modality, expressed as:

[0169]

[0170] in, Represents the i-th recipe node Modality-specific representation in visual modality; Represents the i-th recipe node Network fusion and embedding features in visual modalities;

[0171] Based on the modality-specific representations of each recipe node in the text modality and the visual modality, the modality-specific embedding features of each recipe node are obtained, and their expressions are as follows:

[0172]

[0173] in, Represents the i-th recipe node Modality-specific embedding features; W s Indicates modality-specific weights;

[0174] To ensure that the modality-specific representations of different modalities are as different from the modality-shared representations as possible, soft orthogonality constraints are employed. Specifically, soft orthogonality constraints are constructed with the goal of maximizing the differences between the different modality-specific representations of each recipe node. Its expression is:

[0175]

[0176] in, Represents the i-th recipe node Modality-specific representation in text modality; Represents the i-th recipe node Modality-specific representation in visual modality;

[0177] Multimodal fusion is performed on the network fusion embedding features of each recipe node under various modalities to obtain the modal fusion embedding features of each recipe node, which are expressed as follows:

[0178]

[0179] in, Represents the i-th recipe node Modal fusion embedding features; W g Represents the modal fusion weights; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Network fusion and embedding features in visual modalities;

[0180] In summary, to provide supplementary insights for recipe recommendation models, we decouple multimodal embeddings and propose contrastive fusion. We extract modality-shared representations through contrastive learning and then deploy orthogonal constraints to extract modality-specific representations. Given that recipe nodes have embeddings in both text and vision, we design a cross-modal aggregator as a multimodal decoupling module to effectively merge these modalities, thereby enhancing the shared information among them.

[0181] S5: Based on the modality-shared embedding features, modality-specific embedding features, and modality-fusion embedding features of each recipe node, a counterfactual learning network is used to obtain the final counterfactual embedding features of each recipe node, including:

[0182] Since most current recommendation system methods model from the perspective of data relevance, attempting to discover relevant patterns in the data for recommendation, the real world is composed of causal relationships rather than correlational relationships. Correlational relationships cannot replace causal relationships. Therefore, this invention introduces a causal graph and describes the relationships between the nodes in the causal graph, and maps the recommendation task to the causal world. At the same time, it calculates causal differences in the recipe recommendation process through causal inference analysis, and then designs a counterfactual learning network to filter representations that are irrelevant to user preferences.

[0183] To systematically understand the difference between modality sharing and modality-specific features, a causal graph is proposed. like Figure 3 As shown, where, Represents a set of variable nodes, ε o This represents a set of edges indicating a causal relationship; the direction of the arrow indicates the causal dependency. Figure 3 The causal graph includes five variable nodes: I (multimodal content of user interaction recipes), C (modal sharing information of multimodal content in user interaction recipes), S (modal specific information of multimodal content in user interaction recipes), U (user), and Y (predicted score). Figure 3 The tables in the cause-effect graph include:

[0184] Edge C→Y: Most multimodal recommendation frameworks emphasize extracting common knowledge across modalities, causing prediction scores to be directly affected by modal shared information without considering more granular specific modalities. This leads to spurious correlations between I and Y.

[0185] U&I→Y: From the perspective of traditional recommendation tasks, the predicted score represents the user's preference for the recipe.

[0186] Edge C&S→O→Y: Y represents the user's preference for different dimensions of the interactive recipe, which is divided into modality-shared features and modality-specific features, thus providing a finer granularity when describing user preferences.

[0187] according to Figure 3 The causal graph shown shows that the prediction score Y can be calculated from the value of one of its ancestor nodes U, I, C, and its expression is:

[0188] Y u,i,c =f Y (U = u, i = i, C = c);

[0189] Among them, f Y This represents the prediction function in the model, such as the inner product function.

[0190] After constructing the multimodal causal graph, intervention operations from causal theory are used to analyze the causal relationships between variables. The intervention design of causal inference determines the values ​​of variables, making them independent of their parent nodes. This operation can remove edges pointing to a node from the causal graph. To mitigate bias in multimodal data unrelated to user preferences, a near-unbiased world needs to be constructed. Therefore, counterfactual inference can be used from the proposed multimodal causal graph to create a counterfactual world, such as... Figure 3 As shown; in this counterfactual world, variable I is fixed at I. * According to the d-separation criterion, this setting blocks the intermediate path from C to Y; variable C * and S * From variable I * The recipe-sharing and recipe-specific features learned in the process; therefore, when I is fixed, C * and S * Unaffected by the current variable I, the expression for calculating the predicted score is:

[0191]

[0192] The impact of user preference distribution on the final predicted score can be assessed by the difference between the real world and the counterfactual world. The expression for the final predicted score is:

[0193]

[0194] By calculating the causal differences between two worlds, the recipe recommendation model provided by this invention can utilize modality-shared and modality-specific features to mitigate the side effects of irrelevant content in user-interacted recipes (spurious relevance in multimodal recipe representation) on the final prediction score.

[0195] Based on the above causal difference analysis, we designed a counterfactual learning network to calculate the multimodal content preference distribution for user-interacted recipes. The specific process is as follows:

[0196] S51: Using a counterfactual learning network, the modality-shared embedding features and modality-specific embedding features of each recipe node are combined with their modality-fused embedding features to obtain new modality-shared embedding features and new modality-specific embedding features for each recipe node. Through matrix transformation, the query vector, key vector, and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node are calculated. Then, the preference distributions of the modality-shared content and modality-specific content of each recipe node are calculated, and their expressions are as follows:

[0197]

[0198] Among them, P iRepresents the i-th recipe node The distribution of preferences for modal shared content; Represents the i-th recipe node The modality-specific content preference distribution; softmax(·) denotes the softmax function; Represents the i-th recipe node The query vector corresponding to the new modality shared embedded features; Represents the i-th recipe node The key vector corresponding to the new modality shared embedded features; Represents the i-th recipe node The query vector corresponding to the new modality-specific embedded features; Represents the i-th recipe node The key vector corresponding to the new modality-specific embedding features; d z The dimensions representing the query vector and key vector;

[0199] S52: Based on the theory of causal differences, and combining the modal shared content and modal-specific content preference distribution of each recipe node, the preliminary counterfactual embedding features of each recipe node are obtained, and their expression is as follows:

[0200]

[0201] in, Represents the i-th recipe node Preliminary counterfactual embedding features; P i Represents the i-th recipe node The distribution of preferences for modal shared content; Represents the i-th recipe node The distribution of preferences for modality-specific content; Represents the i-th recipe node The new modality shares the value vector corresponding to the embedded features;

[0202] S53: To stabilize the training process and solve the gradient vanishing problem, a feedforward network with residual connections is used; the feedforward network with residual connections includes: two linear layers and an activation function, as well as a residual connection and layer normalization;

[0203] The initial counterfactual embedding features of each recipe node are input into a feedforward network with residual connections, and the final counterfactual embedding features of each recipe node are output, expressed as follows:

[0204]

[0205] in, Represents the i-th recipe node The final counterfactual embedding feature; LN(·) denotes the layer normalization operation; GELU(·) denotes the nonlinear activation function; Represents the i-th recipe node Preliminary counterfactual embedding features, Represents the i-th recipe node To enhance the model's representation, an L'-layer counterfactual learning layer is superimposed to leverage modal-shared embedding features. W1 represents the weights corresponding to the non-linear activation function; b1 represents the bias corresponding to the non-linear activation function; W2 represents the weights corresponding to the layer normalization; b2 represents the bias corresponding to the layer normalization.

[0206] S6: Based on the modality-shared embedding features, modality-specific embedding features, and final counterfactual embedding features of each recipe node, the target embedding features of each recipe node are obtained; the network fusion embedding features of each user node output by the graph neural network are used as the target embedding features of each user node; the target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output, the expression of which is:

[0207]

[0208] in, Represents the u-th user node With the i-th recipe node Predicted scores between; z u Represents the u-th user node Target embedding features; z i Represents the i-th recipe node Target embedding features;

[0209] The BPR loss function is constructed with the objective that the predicted score between a user node and the recipe node it interacts with is higher than the predicted score between a user node and the recipe node it does not interact with. Its expression is:

[0210]

[0211] Where Ω represents the set of user nodes and recipe nodes; the preset η represents the recipe nodes that interact with the user, i.e., interactive recipe nodes. This indicates a recipe node that has not interacted with the user, i.e., a non-interactive recipe node. Represents the u-th user node The predicted score between the interactive recipe node η; Represents the u-th user node With non-interactive recipe nodes Predicted scores between;

[0212] Based on the contrastive loss function, soft orthogonality constraints, and BPR loss function, the total objective loss function is obtained. Its expression is:

[0213]

[0214] Where, λ C , λ S These represent the hyperparameters corresponding to the contrastive loss function and the soft orthogonality constraint, respectively.

[0215] The top ten recipes corresponding to the current user's predicted scores, sorted from highest to lowest, are recommended to the current user.

[0216] In summary, based on S2 and S31, the present invention sets... Represents a set of users. This represents a set of recipes, which contains information in two different modalities. Where t represents the text modality and v represents the visual modality; the User-Recipe-Ingredient Heterogeneity Graph (URI-Graph) is represented as follows: It contains three types of nodes and four types of edges; the recipe recommendation problem can be formalized as: given such a heterogeneous graph Predicting user behavior by jointly modeling user behavior and multimodal recipe information. The recipe used The possibility (probability) Based on S32 and S4-S6, the solution can be obtained. The value; the flowchart of the recipe recommendation method provided by this invention is as follows: Figure 4 As shown.

[0217] In summary, the recipe recommendation method based on multimodal counterfactual learning provided by this invention includes five steps: heterogeneous graph construction, learning node embeddings using a heterogeneous graph neural network, deentanglement learning of multimodal recipe embeddings, multimodal data correction, prediction, and optimization of the decoupled recipes using a counterfactual learning network. Compared with existing technologies, this invention proposes a multimodal recipe recommendation framework that introduces modality decoupling learning and counterfactual learning networks on the basis of a basic multimodal recommendation model. This effectively decouples multimodal embeddings into modality-shared embeddings and modality-specific embeddings. It effectively solves the defect in existing multimodal models that overemphasize cross-modal knowledge of different modalities while ignoring modality-specific information. At the same time, to alleviate data bias in multimodal data, a counterfactual learning network is designed to learn the distribution of user interaction recipe calculation multimodal content preferences by comparing causal differences in the real world and the counterfactual world, thereby better learning the user's true preferences.

[0218] To verify the effectiveness of this method, a specific implementation example is provided through simulation experiments. The experiment is set up as follows:

[0219] LargeRG, a large multimodal recipe dataset, contains detailed statistics such as... Figure 5 As shown, the recipe data comes from the real recipe sharing website Food.com, which has a large number of recipes shared by users, such as recipe names, ingredients, methods, pictures, etc. Users rate and comment on these recipes; the nutritional information about the ingredients comes from the USDA nutrition data; and the similarity data between recipes comes from the recipe knowledge graph FoodKG; the dataset is divided into training set, validation set and test set in a ratio of 7:1.5:1.5.

[0220] The performance of the prediction set is evaluated using metrics widely used in the recommendation field: Area Under the ROC Curve (AUC), Hit Rate (HR), Normalized Diminished Cumulative Gain (NDCG), and Mean Precision (MAP). AUC measures the overall ranking performance on the test set, HR assesses the relevance of the recommended content, NDCG evaluates ranking performance from the user's perspective, and MAP assesses the accuracy of the ranking position. An early stopping strategy is employed: if the total loss on the validation set does not decrease within three consecutive epochs, the training process is stopped. The model parameters are set as follows: embedding size is 128, batch size is 1024, learning rate is 0.005, and the Adam optimizer is used.

[0221] To evaluate the performance of the method of this invention, several representative benchmark methods were experimentally compared:

[0222] 1) The traditional recommendation method, Business Process Reengineering (BPR);

[0223] 2) Graph-based recommendation methods: GAT, RGCN, HAN, NGCF, Light-GCN;

[0224] 3) Recipe representation learning-based methods: Reciptor, Recipe2Vec;

[0225] 4) Recommended recipe methods: HAFR, SCHGN, RecipeRec;

[0226] 5) Causal inference related recommendation methods InvRL and MCLN.

[0227] Experimental results compared with the method of the present invention are as follows: Figure 6As shown, the method proposed in this invention achieves the best recommendation performance across all metrics, demonstrating its superiority. Graph-based recommendation methods outperform traditional methods, and among graph-based recommendation methods, heterogeneous graph-based methods outperform homogeneous graph-based methods. Therefore, heterogeneous graph-based and related heterogeneous graph-based recommendation methods can effectively improve the quality of embeddings. Recipe representation learning performs poorly because these methods were originally designed for tasks such as recipe classification, clustering, and recipe retrieval, rather than for learning the complex interactions between users and recipes. Recipe-based and causal recommendation methods generally outperform other baselines, especially related methods based on multimodal recommendation; however, their performance is still inferior to the method proposed in this invention. This demonstrates that modal deentanglement learning and counterfactual learning can better mitigate the side effects caused by information irrelevant to user preferences in recipe embeddings, thereby more accurately predicting user preferences.

[0228] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A recipe recommendation method based on multimodal counterfactual learning, characterized in that, include: A recipe recommendation model is constructed, including a heterogeneous graph construction module, a graph neural network construction module, a multimodal feature decoupling module, a counterfactual learning network, and a prediction module; Based on user data and recipe data, a user-recipe-ingredient heterogeneous graph is constructed; wherein, the node types in the heterogeneous graph are user nodes, recipe nodes, and ingredient nodes, and the edge types in the heterogeneous graph are user-recipe edges, recipe-recipe edges, recipe-ingredient edges, and ingredient-ingredient edges, respectively. An initial embedding operation is performed on each node to obtain the initial feature representation of each node; a graph neural network is constructed based on the user-recipe-ingredient heterogeneous graph; the initial feature representations of all nodes are used as input to the graph neural network, and a two-level attention mechanism is used to process the initial feature representation of each node, and then the data is passed through a propagation layer to output the network fusion embedding features of each user node and the network fusion embedding features of each recipe node in each modality. The network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature value of each modality in each recipe node. Based on the important feature value of each modality in each recipe node, the modality shared embedding feature of each recipe node is calculated. The network fusion embedding feature of each recipe node in each modality is subtracted from the modality shared embedding feature of each recipe node to obtain the modality specific representation of each recipe node in each modality. Based on the modality specific representation of each recipe node in each modality, the modality specific embedding feature of each recipe node is obtained. Multimodal fusion operation is performed on the network fusion embedding features of each recipe node in various modalities to obtain the modality fusion embedding feature of each recipe node. Using a counterfactual learning network, the modality-shared embedding features and modality-specific embedding features of each recipe node are combined with their modality-fused embedding features to obtain new modality-shared embedding features and new modality-specific embedding features for each recipe node. Through matrix transformation, the query vector, key vector, and value vector corresponding to the new modality-shared embedding features and new modality-specific embedding features of each recipe node are calculated. Then, the preference distributions of the modality-shared content and modality-specific content of each recipe node are calculated. Based on causal difference theory, the preliminary counterfactual embedding features of each recipe node are obtained by combining the preference distributions of the modality-shared content and modality-specific content of each recipe node. The preliminary counterfactual embedding features of each recipe node are input into a feedforward network with residual connections to output the final counterfactual embedding features of each recipe node. Based on the modality-shared embedding features, modality-specific embedding features, and final counterfactual embedding features of each recipe node, the target embedding features of each recipe node are obtained. The network fusion embedding features of each user node output by the graph neural network are used as the target embedding features of each user node. The target embedding features of each user node and recipe node are input into the prediction module, which outputs the prediction score between each user and each recipe. The top ten recipes corresponding to the prediction scores between the current user and each recipe, sorted from high to low, are recommended to the current user.

2. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The network fusion embedding features of each recipe node in each modality are input into the multimodal feature decoupling module to calculate the important feature values ​​of each modality in each recipe node; Based on the important feature values ​​of each modality in each recipe node, the modality-shared embedding features of each recipe node are calculated, including: Using the softmax function, the importance value of each modality in each recipe node is calculated, and its expression is: in, Represents the i-th recipe node The important feature values ​​in modality m; q1 represents the attention vector; W represents the importance weight; b represents the importance bias; Represents the i-th recipe node Network fusion embedding features under modality m; softmax(·) denotes the softmax function; Based on the important feature values ​​of each modality in each recipe and the network fusion embedding features of each recipe node in each modality, the modality shared embedding features of each recipe node are calculated, and their expression is as follows: in, Represents the i-th recipe node Modal sharing embedding features; This represents the modal information corresponding to the recipe node, including text modality and visual modality.

3. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The modality-specific representation of each recipe node is obtained by subtracting the network fusion embedding feature of each recipe node in each modality from the modality-shared embedding feature of each recipe node. Based on the modality-specific representation of each recipe node in each modality, the modality-specific embedding features of each recipe node are obtained, including: The modal information corresponding to the recipe node includes text modality and visual modality; Subtracting the network fusion embedding feature of each recipe node in the text modality from the modality-shared embedding feature of each recipe node yields the modality-specific representation of each recipe node in the text modality, expressed as: in, Represents the i-th recipe node Modality-specific representation in text modality; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Modal sharing embedding features; Subtracting the network fusion embedding feature of each recipe node in the visual modality from the modality-shared embedding feature of each recipe node yields the modality-specific representation of each recipe node in the visual modality, expressed as: in, Represents the i-th recipe node Modality-specific representation in visual modality; Represents the i-th recipe node Network fusion and embedding features in visual modalities; Based on the modality-specific representations of each recipe node in the text modality and the visual modality, the modality-specific embedding features of each recipe node are obtained, and their expressions are as follows: in, Represents the i-th recipe node Modality-specific embedding features; W s This represents modality-specific weights.

4. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The process involves performing a multimodal fusion operation on the network fusion embedding features of each recipe node under various modalities to obtain the modal fusion embedding features of each recipe node. Its expression is: in, Represents the i-th recipe node Modal fusion embedding features; W g Represents the modal fusion weights; Represents the i-th recipe node Network fusion embedding features in text modality; Represents the i-th recipe node Network fusion embedding features in the visual modality.

5. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, Based on the new modality shared embedding features, new modality specific embedding features corresponding to the query vector, key vector and value vector of each recipe node, calculate the preference distribution of modality shared content and modality specific content of each recipe node respectively; Based on the theory of causal differences, and combining the modality-shared content and modality-specific content preference distribution of each recipe node, the preliminary counterfactual embedding features of each recipe node are obtained as follows: The expressions for the preference distribution of modal shared content and modal specific content for each recipe node are as follows: Among them, P i Represents the i-th recipe node The distribution of preferences for modal shared content; Represents the i-th recipe node The modality-specific content preference distribution; softmax(·) denotes the softmax function; Represents the i-th recipe node The query vector corresponding to the new modality shared embedded features; Represents the i-th recipe node The key vector corresponding to the new modality shared embedded features; Represents the i-th recipe node The query vector corresponding to the new modality-specific embedded features; Represents the i-th recipe node The key vector corresponding to the new modality-specific embedding features; d z The dimensions representing the query vector and key vector; The expression for the initial counterfactual embedding feature of each recipe node is: in, Represents the i-th recipe node Preliminary counterfactual embedding features; P i Represents the i-th recipe node The distribution of preferences for modal shared content; Represents the i-th recipe node The distribution of preferences for modality-specific content; Represents the i-th recipe node The new modality shares the value vector corresponding to the embedded features.

6. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The process of inputting the preliminary counterfactual embedding features of each recipe node into a feedforward network with residual connections, and outputting the final counterfactual embedding features of each recipe node, includes: The feedforward network with residual connections includes: two linear layers and an activation function, as well as a residual connection and layer normalization; The expression for the final counterfactual embedding feature of each recipe node is: in, Represents the i-th recipe node The final counterfactual embedding feature; LN(·) denotes the layer normalization operation; GELU(·) denotes the nonlinear activation function; Represents the i-th recipe node Preliminary counterfactual embedding features, Represents the i-th recipe node The modality-shared embedding features; W1 represents the weights corresponding to the nonlinear activation function; b1 represents the biases corresponding to the nonlinear activation function; W2 represents the weights corresponding to the layer normalization; b2 represents the biases corresponding to the layer normalization.

7. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The target embedding features of each user node and recipe node are input into the prediction module, and the prediction score between each user and each recipe is output, expressed as follows: in, Represents the u-th user node With the i-th recipe node Predicted scores between; z u Represents the u-th user node Target embedding features; z i Represents the i-th recipe node The target embedding features.

8. The recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The loss of the recipe recommendation model includes: A contrastive loss function is constructed with the goal of minimizing the difference between the network fusion embedding features of each recipe node in a single modality and the modality-shared embedding features of each recipe node. Its expression is: in, Represents a set of recipes. Indicates the quantity of all recipes; This represents the modal information corresponding to the recipe. Indicates the types of all modes; Represents the i-th recipe node Network fusion embedding features under modality m; Represents the i-th recipe node The modality-shared embedding features; F(·) represents the contrastive evaluation function, whose expression is: Where θ(·) represents the cosine function; τ represents the temperature parameter; Indicates the i-th * Recipe Nodes The network fusion embedding features in modality m are represented as the i-th recipe node. Negative samples in mode m; To maximize the difference between different modality-specific representations of each recipe node, soft orthogonal constraints are constructed. Its expression is: in, Represents the i-th recipe node Modality-specific representation in text modality; Represents the i-th recipe node Modality-specific representation in visual modality; The BPR loss function is constructed with the objective that the predicted score between a user node and the recipe node it interacts with is higher than the predicted score between a user node and the recipe node it does not interact with. Its expression is: Where Ω represents the set of user nodes and recipe nodes; the preset η represents the recipe nodes that interact with the user, i.e., interactive recipe nodes. This indicates a recipe node that has not interacted with the user, i.e., a non-interactive recipe node. Represents the u-th user node The predicted score between the interactive recipe node η; Represents the u-th user node With non-interactive recipe nodes Predicted scores between; Based on the contrastive loss function, soft orthogonality constraints, and BPR loss function, the total objective loss function is obtained. Its expression is: Where, λ C , λ S These represent the hyperparameters corresponding to the contrastive loss function and the soft orthogonal constraint, respectively.

9. A recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The two-level attention mechanism is used to process the initial feature representation of each node, including: The two-level attention mechanism includes: node-level attention mechanism and relation-level attention mechanism; The node-level attention mechanism includes: Using the shared weight matrix, a linear transformation is performed on the output features of each node in the l-th layer of the graph neural network under each modality, resulting in the linear output features of each node in the l-th layer under each modality, expressed as: in, This represents the k-th node h in the l-th layer. k The linear output feature corresponding to mode m; This represents the weights in the l-th layer corresponding to relation r and mode m; This represents the k-th node h in the l-th layer. k Input features in modality m; Based on the linear output features of each node in layer l in each modality, the attention score between each pair of nodes in layer l in each modality is calculated, and its expression is as follows: in, This represents the k-th node h in the l-th layer under mode m. k and the j-th node h j Attention scores between; LeakyReLU(·) represents the improved ReLU activation function; This represents the k-th node h in the l-th layer. k and the j-th node h j The weights between the attention scores represent the values; This represents the j-th node h in the l-th layer. j The linear output feature corresponding to mode m; Normalize the calculated attention scores to obtain the normalized attention scores between pairs of nodes in each modality of layer l, as expressed by: in, This represents the k-th node h in the l-th layer under mode m. k and the j-th node h j Normalized attention scores between them; Represents the k-th node h k The neighboring nodes under relation r, i.e., the k-th node. * One node; L k,r Represents the k-th node h k The set of neighboring nodes under relation r; This indicates that in the l-th layer, the k-th mode under mode m... * Nodes and the j-th node h j Attention scores between them; Based on the normalized attention scores of each pair of nodes and between them in each modality of layer l, the first fused embedding feature of each node in layer l+1 under each relationship and modality is calculated, and its expression is: in, This represents the k-th node h in the (l+1)-th layer. k The first fused embedding feature under relation r and modality m; N represents the number of attention heads; n represents the index of the attention head; ReLU(·) represents the ReLU function; This represents the j-th node h in the l-th layer. j Input features in modality m; W represents the weight vector in the l-th layer corresponding to mode m; f This represents a trainable weight vector; The relation-level attention mechanism includes: Using shared nonlinear weights and intermediate vectors, the first fused embedding features of each node in layer l+1 under each relation and modality are transformed to a new feature space. Based on the intermediate vectors, the embedding similarity between the first fused embedding features of each node in layer l+1 before and after the feature space transformation under each relation and modality is calculated, and its expression is as follows: in, Represents the k-th node h k Embedding similarity under relation r and modality m; This represents the set of nodes connected to relation r. The number of nodes connected to relation r is represented by q; the intermediate vector is represented by W. R Indicates relation weight; This represents the k-th node h in the (l+1)-th layer. k The first fused embedding feature under relation r and modality m; b * Indicates similarity bias; The embedding similarity is normalized to obtain the relation-level attention weight of each node under each relation and each modality, which is expressed as: in, Represents the k-th node h k Relation-level attention weights under relation r and modality m; Indicates the relationship with the k-th node h k A set of connected relationships; Based on the first fused embedding features of each node in layer l+1 under each relation and each modality, and the relation-level attention weights of each node under each relation and each modality, the second fused embedding features of each node in layer l+1 under each modality are calculated, and their expressions are as follows: in, This represents the k-th node h in the (l+1)-th layer. k The second fusion embedding feature under modality m.

10. A recipe recommendation method based on multimodal counterfactual learning according to claim 1, characterized in that, The initial embedding operation for each node, resulting in an initial feature representation for each node, includes: Using the Xavier initialization method, generate ordinary random initialization features for each user; use the ordinary random initialization features for each user as the initial feature representation for each user node; A pre-trained jump instruction embedding is used as the representation of each instruction in the cooking steps of each recipe. The average of all instruction representations in the cooking steps of each recipe is calculated to obtain the average instruction for each recipe. This average instruction is then embedded as the cooking description for each recipe. The average instruction for any given recipe... The expression is: Among them, z ins,a This represents the a-th instruction in the cooking steps of the current recipe; O represents the number of instructions in the current recipe. Each recipe image is encoded using a pre-trained model to obtain the image embedding for each recipe; the cooking instructions embedding and the image embedding for each recipe are projected into the same shared input space as the initial feature representation for each recipe node; The nutrients of each component are projected into vectors to obtain the vector corresponding to each component; the vector corresponding to each component is used as the initial feature representation of each component node.

Images