A multimodal recommendation method based on information bottleneck principle

Abstract

The present invention relates to the field of data processing technology, and in particular to a multimodal recommendation method based on the information bottleneck principle. The steps are as follows: construct a data set based on the user-item historical interaction records in the Amazon data set; construct a user-item graph for each modality based on the data set, and calculate the embedding of users and items through multi-layer graph convolution to obtain the final representation of the user and the fusion embedding of the item; construct an item-item similarity graph for each modality of the item, retain the neighbors of the top k items with the highest similarity, and then aggregate the item semantic signal according to the contribution of the item modality to the downstream recommendation task to obtain the final representation of the item; optimize through denoising, loss calculation and Adam optimizer. This method can extract key information from multimodal data, remove redundancy and noise, and can calculate the contribution of item representation to downstream recommendation tasks in different modalities.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multimodal recommendation method based on the information bottleneck principle. Background Art

[0002] In today's era of information explosion, with the rapid increase in the number of Internet users and the massive accumulation of data, recommendation systems, as an important tool for personalized information push, are increasingly playing a key role. The core goal of recommendation systems is to provide tailored content recommendations by analyzing user behavior and preferences, thereby improving user experience and satisfaction. Traditional recommendation systems usually rely on single-modal data, such as user browsing history, ratings, purchase history, etc. However, with the popularization of multimedia information, the forms of user-generated data are becoming more and more diverse, including multimodal data such as text, images, and videos. These multimodal data not only provide direct information about user behavior, but also contain rich semantic and emotional information. Therefore, how to effectively utilize these multimodal information has become a key direction in the research of recommendation systems. In recent years, multimodal recommendation systems have received widespread attention and have made certain progress in academia and industry. However, existing methods still face significant limitations in practical applications. First, the heterogeneity of multimodal information poses a huge challenge. Differences between different modalities may lead to difficulties in information fusion, thereby affecting the effectiveness of the recommendation system. For example, there are significant differences in the way images and texts are expressed, which may lead to insufficient correlation between modalities, thereby affecting the accuracy of recommendations. Secondly, mining the user's true preferences and removing noise information from a large amount of multimodal data is a difficult point in current research. Redundant and noisy information in multimodal data may interfere with the extraction of user preferences, resulting in a decrease in the accuracy of recommendation results. Finally, many existing multimodal recommendation systems fail to effectively distinguish the different contributions of different modal information to downstream recommendation tasks, which limits the performance of the model. For example, in some cases, text information may have a greater impact on the recommendation results than image information, but if the system cannot correctly measure this difference, it may lead to inaccurate recommendation results.

[0003] Therefore, the present invention provides a multimodal recommendation method based on the information bottleneck principle. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention develops a multimodal recommendation method based on the information bottleneck principle, which can not only extract key information from multimodal data and remove redundancy and noise, but also calculate the contribution of item representations in different modes to downstream recommendation tasks.

[0005] The technical solution of the present invention to solve the technical problem is: a multimodal recommendation method based on the information bottleneck principle, comprising the following steps:

[0006] a) Build a dataset based on the user-item historical interaction records in the Amazon dataset ;

[0007] b) Build a user-item graph for each modality through historical interaction records, and calculate the embedding of users and items through multi-layer graph convolution to obtain the final representation of the user and the fused embedding of the item;

[0008] c) Build an item-item similarity graph for each modality of the item, retain the neighbors of the top k items with the highest similarity, capture the semantic signals between items, and then aggregate the semantic signals of items according to the contribution of the item modality to the downstream recommendation task to obtain the final representation of the item;

[0009] d) Use contrastive learning to achieve information bottleneck denoising of modal features, further maximize the mutual information of user embeddings in different modalities, and obtain users' cross-modal preferences;

[0010] e) Comparison of learning loss by BPR loss and information bottleneck , cross-modal preference contrastive learning loss And Adam optimizer optimizes user-item embedding.

[0011] Step a) is as follows:

[0012] Extract multimodal information of users and items in Amazon dataset and build dataset , , dataset Included The user set consists of , A collection of items consisting of items Each item contains two forms: picture and text. The picture corresponds to the visual embedding and the text corresponds to the text embedding. The item data extracted from the Amazon dataset is specifically the 4096-dimensional item visual modality embedding of the image data. And the 372-dimensional embedding of text data, and then the 372-dimensional text embedding is converted into a 384-dimensional item text modality embedding through sentence-transformers for generating sentence embeddings ;

[0013] in, Represents the user, Indicates items, Represents a user collection Middle Users, Represents a collection of items Middle items;

[0014] And for the dataset Each positive sample in the dataset has a corresponding negative sample. The positive sample is a real interaction pair between the user and the item, and the negative sample is a non-interaction pair between the user and the item.

[0015] Step b) is as follows:

[0016] b-1) The images and texts in the items correspond to the visual mode and text mode respectively. The interaction relationship between users and items in the graph is constructed for the visual mode and text mode of the item, and then the interaction graph of the visual mode is obtained. Interaction diagram with text modal ;

[0017] b-2) For users, based on the collaborative filtering method, the user ID is mapped to a low-dimensional dense vector through the ID index to obtain the user's embedding. The embedding matrix of the user's visual modality is recorded as , the embedding matrix of user text modality is denoted as ;

[0018] For items, use the corresponding item visual modality embedding and item text modal embedding Initialize;

[0019] Then align the dimensions of user and item embeddings by reducing the dimensions of item visual modality embedding and item text modality embedding respectively. The specific calculation is as follows:

[0020] ,

[0021] ,

[0022] in, and Represent the weight matrices of visual modality and textual modality respectively, represents transpose, represents the embedding matrix of the item text modality, Embedding matrix representing the visual modality of items;

[0023] The visual modality embedding of user-item is obtained through the embedding matrix of user visual modality and textual modality and the embedding matrix of item visual modality and textual modality. and text-modal embeddings of user-items , the specific calculation is as follows:

[0024] ,

[0025] ;

[0026] b-3) The first User-item embedding of layer graph convolution. The specific process is as follows:

[0027] For visual modality:

[0028] ,

[0029] , ;

[0030] For text modal:

[0031] , ;

[0032] , ;

[0033] Among them, according to the interaction diagram of visual modality Interaction diagram with text modal To determine and , and Represented in the user-item graph Medium User and items The first-order neighbor of and Respectively represent the visual mode and text mode in the User-item embedding with layer graph convolution, and Respectively represent the visual mode and text mode in the User-item embedding with layer graph convolution, , graph convolution layer, ;

[0034] Then calculate the final user-item embedding in the visual mode and text mode respectively, the calculation formula is as follows:

[0035] ,

[0036] ,

[0037] in, and denote the final user-item embeddings in visual and textual modalities, respectively. , , represents the final embedding of the user’s visual modality, represents the final embedding of the item’s visual modality, represents the final embedding of the user's text modality, The final embedding representing the item text modality;

[0038] b-4) The final embeddings of multiple modalities are fused together by concatenation. The specific process is as follows:

[0039] ,

[0040] ,

[0041] in, represents the fusion of the final embedding of the two modalities of the item, Represents the final user representation.

[0042] Step c) is as follows:

[0043] c-1) Use the K-nearest neighbor (KNN) method to build an item-item similarity graph for each modality of the item and calculate the item Visual modality embedding With items Visual modality embedding Similarity scores between visual modalities , and items Text modal embedding With items Text modal embedding Similarity scores between text modalities ,thing For items The specific calculation of neighbors is as follows:

[0044] ,

[0045] ,

[0046] Keep the top most similarity in visual mode and text mode respectively. The specific calculation is as follows:

[0047] ,

[0048] ,

[0049] in, Indicates items in visual mode Before Neighbors, Indicates items in text mode Before Neighbors;

[0050] c-2) By polymerizing multi-layer objects Neighbor Items information to capture deeper semantic relationships between visual and textual modalities and , the specific calculation is as follows:

[0051] ,

[0052] ,

[0053] in, Represents semantic relations in visual modality, Represents the semantic relationship in text mode, Indicates items of Neighbor Items A collection of;

[0054] c-3) The contribution of each modality of the item to the recommendation task is calculated through the attention mechanism. The specific calculation is as follows:

[0055] ,

[0056] ,

[0057] in, Represents the attention mechanism operation, through a multi-layer perceptron accomplish, represents the contribution in the visual mode, Indicates the contribution in text mode;

[0058] The semantic signals of each modality are then used to enhance the item embedding, and the final item representation is generated that comprehensively considers multimodal information. , the specific calculation is as follows:

[0059] .

[0060] Step d) is as follows:

[0061] d-1) Calculate the information bottleneck contrastive learning loss Remove modal noise and encourage differences between the embedding matrix of items and the final embedding of items through the information bottleneck principle, while maximizing the amount of information related to the recommendation task. The embedding matrix of items includes the embedding matrix of the visual modality of items. and the embedding matrix of the item text modality , the final embedding of the item includes the final embedding of the item’s visual modality and the final embedding for the item text modal , the specific calculation is as follows:

[0062] ,

[0063] in, represents the recommendation loss based on Bayesian personalized ranking BPR, represents the mutual information of two embeddings, represents the information bottleneck contrastive learning loss in the visual modality, represents the information bottleneck contrastive learning loss in text modality;

[0064] The mutual information between the embedding matrices of items in the visual and textual modalities and the final embedding of items is calculated as follows:

[0065] ,

[0066] ,

[0067] in, It represents the similarity between two vectors. Indicates the temperature coefficient, item For users Interacted positive samples, items For users Negative samples that have not been interacted with, Indicates user In the embedding matrix of the visual modality, Indicates user In the text mode, the embedding matrix can be obtained by the same logic as step b-2). and ;

[0068] d-2) The dependencies between different modalities are captured by contrastive learning methods, and the final embeddings in the user visual modality and text modality are mapped to a unified feature space to obtain the embeddedness in the user visual modality and text modality after mapping. The specific calculation is as follows:

[0069] ,

[0070] in represents a multi-layer perceptron MLP, represents the embedding of the user’s visual modality after mapping, represents the embedding in the user text mode after mapping, represents a fusion operation;

[0071] according to and Calculating users' cross-modal preferences , the specific calculation is as follows:

[0072] ,

[0073] in, Indicates the maximum value operation.

[0074] Step e) is as follows:

[0075] e-1) Optimize the parameters through BPR loss, the specific calculation is as follows:

[0076] ,

[0077] in, represents the BPR loss, Indicates user Interacted positive samples, Indicates user Negative samples that have not been interacted with;

[0078] e-2) Compare BPR loss and information bottleneck with learning loss and cross-modal preference contrast learning loss Combined, the final loss is , the specific calculation is as follows:

[0079] ,

[0080] in, and Represents two different hyperparameters for balancing various losses;

[0081] e-3) Optimize user-item embeddings through Adam optimizer.

[0082] The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0083] The present invention proposes a multimodal recommendation method based on the information bottleneck principle, which mainly includes three core modules: a modal feature graph convolution encoder, an attention semantic graph convolution module, and a contrastive learning information bottleneck module;

[0084] First, the modal feature graph convolutional encoder module realizes the effective propagation and fusion of information in each modality through the user-item interaction graph, thereby generating multimodal representations of users and items. In this way, the model can not only capture the direct interaction relationship between users and items, but also reveal the potential relationship network through the graph structure, enhance the recommendation system's ability to understand and capture user preferences, and thus achieve a deep fusion of multimodal features of users and items;

[0085] Secondly, the attention semantic graph convolution module uses the K-nearest neighbor method to construct an item-item modal content similarity graph, and performs semantic graph convolution operations on this basis. Through this adaptive weighting mechanism, the model can flexibly adjust the weight of each modal information according to the needs of specific tasks, significantly improving the accuracy and robustness of the recommendation system;

[0086] Finally, the contrastive learning information bottleneck module uses the information bottleneck principle to capture the minimum sufficient modal information while ensuring the sufficiency of information for downstream recommendation tasks, thereby achieving the effect of modal feature denoising. This module designs a contrastive learning loss function, which maximally compresses modal information, reduces redundancy and noise, and improves the overall performance of the recommendation system. In particular, in the processing of cross-modal preferences, this method promotes the comprehensive use of multimodal information by maximizing the complementarity between user representations of different modalities, thereby not only improving the accuracy of recommendation results, but also enhancing the robustness of the system in dealing with noisy data.

[0087] In summary, the multimodal representation learning personalized recommendation method based on information bottleneck proposed in the present invention effectively improves the performance of the recommendation system by cleverly integrating graph convolution, attention mechanism and contrastive learning, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0089] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0090] Figure 2 is the user-item interaction graph.

[0091] Figure 3 are discrete graphs under two embeddings, Figure 3 (a) is the discrete graph of the original modal embedding. Figure 3 (b) is the embedded discrete graph after being processed by the present invention. DETAILED DESCRIPTION

[0092] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below.

[0093] Example 1

[0094] like Figure 1 and Figure 2As shown in Figure 1, a multimodal recommendation method based on the information bottleneck principle is as follows:

[0095] a) Build a dataset based on the user-item historical interaction records in the Amazon dataset ;

[0096] b) Build a user-item graph for each modality through historical interaction records, and calculate the embedding of users and items through multi-layer graph convolution to obtain the final representation of the user and the fused embedding of the item;

[0097] c) Build an item-item similarity graph for each modality of the item, retain the neighbors of the top k items with the highest similarity, capture the semantic signals between items, and then aggregate the semantic signals of items according to the contribution of the item modality to the downstream recommendation task to obtain the final representation of the item;

[0098] d) Use contrastive learning to achieve information bottleneck denoising of modal features, further maximize the mutual information of user embeddings in different modalities, and obtain users' cross-modal preferences;

[0099] e) Comparison of learning loss by BPR loss and information bottleneck , cross-modal preference contrastive learning loss And Adam optimizer optimizes user-item embedding.

[0100] Step a) is as follows:

[0101] Extract multimodal information of users and items in Amazon dataset and build dataset , , dataset Included The user set consists of , A collection of items consisting of items Each item contains two forms: picture and text. The picture corresponds to the visual embedding and the text corresponds to the text embedding. The item data extracted from the Amazon dataset is specifically the 4096-dimensional item visual modality embedding of the image data. And the 372-dimensional embedding of text data, and then the 372-dimensional text embedding is converted into a 384-dimensional item text modality embedding through sentence-transformers for generating sentence embeddings ;

[0102] in, Represents the user, Indicates items, Represents a user collection Middle Users, Represents a collection of items Middle items;

[0103] And for the dataset Each positive sample in the dataset has a corresponding negative sample. The positive sample is a real interaction pair between the user and the item, and the negative sample is a non-interaction pair between the user and the item.

[0104] Step b) is as follows:

[0105] b-1) The images and texts in the items correspond to the visual mode and text mode respectively. The interaction relationship between users and items in the graph is constructed for the visual mode and text mode of the item, and then the interaction graph of the visual mode is obtained. Interaction diagram with text modal ;

[0106] Depend on Figure 2 As shown, Figure 2 It is an item-user interaction diagram in visual mode or text mode. The white grid indicates that there is no interaction between the user and the item, and the orange square indicates that there is interaction between the item and the user. The items that interact are the first-order neighbors of the current user.

[0107] b-2) For users, based on the collaborative filtering method, the user ID is mapped to a low-dimensional dense vector through the ID index to obtain the user's embedding. The embedding matrix of the user's visual modality is recorded as , the embedding matrix of user text modality is denoted as ;

[0108] For items, use the corresponding item visual modality embedding and item text modal embedding Initialize;

[0109] Then align the dimensions of user and item embeddings by reducing the dimensions of item visual modality embedding and item text modality embedding respectively. The specific calculation is as follows:

[0110] ,

[0111] ,

[0112] in, and Represent the weight matrices of visual modality and textual modality respectively, represents transpose, represents the embedding matrix of the item text modality, Embedding matrix representing the visual modality of items;

[0113] The visual modality embedding of user-item is obtained through the embedding matrix of user visual modality and textual modality and the embedding matrix of item visual modality and textual modality. and text-modal embeddings of user-items , the specific calculation is as follows:

[0114] ,

[0115] ;

[0116] b-3) The first User-item embedding of layer graph convolution. The specific process is as follows:

[0117] For visual modality:

[0118] ,

[0119] , ;

[0120] For text modal:

[0121] , ;

[0122] , ;

[0123] Among them, according to the interaction diagram of visual modality Interaction diagram with text modal To determine and , and Represented in the user-item graph Medium User and items The first-order neighbor of and Respectively represent the visual mode and text mode in the User-item embedding with layer graph convolution, and Respectively represent the visual mode and text mode in the User-item embedding with layer graph convolution, , graph convolution layer, ;

[0124] Then calculate the final user-item embedding in the visual mode and text mode respectively, the calculation formula is as follows:

[0125] ,

[0126] ,

[0127] in, and denote the final user-item embeddings in visual and textual modalities, respectively. , , represents the final embedding of the user’s visual modality, represents the final embedding of the item’s visual modality, represents the final embedding of the user's text modality, The final embedding representing the item text modality;

[0128] b-4) The final embeddings of multiple modalities are fused together by concatenation. The specific process is as follows:

[0129] ,

[0130] ,

[0131] in, represents the fusion of the final embedding of the two modalities of the item, Represents the final user representation.

[0132] Step c) is as follows:

[0133] c-1) Use the K-nearest neighbor (KNN) method to build an item-item similarity graph for each modality of the item and calculate the item Visual modality embedding With items Visual modality embedding Similarity scores between visual modalities , and items Text modal embedding With items Text modal embedding Similarity scores between text modalities ,thing For items The specific calculation of neighbors is as follows:

[0134] ,

[0135] ,

[0136] Keep the top most similarity in visual mode and text mode respectively. The specific calculation is as follows:

[0137] ,

[0138] ,

[0139] in, Indicates items in visual mode Before Neighbors, Indicates items in text mode Before Neighbors;

[0140] c-2) By polymerizing multi-layer objects Neighbor Items information to capture deeper semantic relationships between visual and textual modalities and , the specific calculation is as follows:

[0141] ,

[0142] ,

[0143] in, Represents semantic relations in visual modality, Represents the semantic relationship in text mode, Indicates items of Neighbor Items A collection of;

[0144] c-3) The contribution of each modality of the item to the recommendation task is calculated through the attention mechanism. The specific calculation is as follows:

[0145] ,

[0146] ,

[0147] in, Represents the attention mechanism operation, through a multi-layer perceptron accomplish, represents the contribution in the visual mode, Indicates the contribution in text mode;

[0148] The semantic signals of each modality are then used to enhance the item embedding, and the final item representation is generated that comprehensively considers multimodal information. , the specific calculation is as follows:

[0149] .

[0150] Step d) is as follows:

[0151] d-1) Calculate the information bottleneck contrastive learning loss Remove modal noise and encourage differences between the embedding matrix of items and the final embedding of items through the information bottleneck principle, while maximizing the amount of information related to the recommendation task. The embedding matrix of items includes the embedding matrix of the visual modality of items. and the embedding matrix of the item text modality , the final embedding of the item includes the final embedding of the item’s visual modality and the final embedding for the item text modal , the specific calculation is as follows:

[0152] ,

[0153] in, represents the recommendation loss based on Bayesian personalized ranking BPR, represents the mutual information of two embeddings, represents the information bottleneck contrastive learning loss in the visual modality, represents the information bottleneck contrastive learning loss in text modality;

[0154] The mutual information between the embedding matrices of items in the visual and textual modalities and the final embedding of items is calculated as follows:

[0155] ,

[0156] ,

[0157] in, It represents the similarity between two vectors. Indicates the temperature coefficient, item For users Interacted positive samples, items For users Negative samples that have not been interacted with, Indicates user In the embedding matrix of the visual modality, Indicates user In the text mode, the embedding matrix can be obtained by the same logic as step b-2). and ;

[0158] d-2) The dependencies between different modalities are captured by contrastive learning methods, and the final embeddings in the user visual modality and text modality are mapped to a unified feature space to obtain the embeddedness in the user visual modality and text modality after mapping. The specific calculation is as follows:

[0159] ,

[0160] in represents a multi-layer perceptron MLP, represents the embedding of the user’s visual modality after mapping, represents the embedding in the user text mode after mapping, represents a fusion operation;

[0161] according to and Calculating users' cross-modal preferences , the specific calculation is as follows:

[0162] ,

[0163] in, Indicates the maximum value operation.

[0164] Step e) is as follows:

[0165] e-1) Optimize the parameters through BPR loss, the specific calculation is as follows:

[0166] ,

[0167] in, represents the BPR loss, Indicates user Interacted positive samples, Indicates user Negative samples that have not been interacted with;

[0168] e-2) Compare BPR loss and information bottleneck with learning loss and cross-modal preference contrast learning loss Combined, the final loss is , the specific calculation is as follows:

[0169] ,

[0170] in, and Represents two different hyperparameters for balancing various losses;

[0171] e-3) Optimize user-item embeddings through Adam optimizer.

[0172] In order to prove that the information bottleneck principle in the present invention can resist modal noise pollution, the distribution of representations in the baby dataset is visualized. 500 items are randomly selected from the baby dataset, and then their embeddings are mapped to a two-dimensional space using t-SNE, as shown in Figure 2 As shown, the impact of the information bottleneck component of contrast learning in a two-dimensional perspective can be seen. Figure 2The left figure is the original modality embedding discrete graph, and the right figure is the embedded discrete graph after being processed by this method. The blue and green points represent the visual modality and textual modality respectively. Figure 2 It can be observed that compared with the original modal embedding, the modal embedding processed by the method of the present invention is more discrete, which shows that the modal noise pollution can be resisted after being processed by the method of the present invention.

[0173] Example 2

[0174] As shown in Table 1, the performance of the method in the present invention is compared with other methods on three different data sets. Through the data analysis in the table, the following conclusions can be drawn:

[0175] (1) The significant advantage of the method of the present invention is that its recommendation performance is significantly better than that of traditional recommendation models and existing multimodal recommendation models, which shows that the design of the method of the present invention in multimodal recommendation scenarios is particularly reasonable. Specifically, the method of the present invention captures the high-order relationship between users and items through the modal feature encoder, extracts strong collaborative signals, and effectively utilizes modal information; at the same time, the modal similarity graph encoding enriches the semantic signal, and the aggregation of the attention mechanism further enhances the expressive power of the modal representation. Traditional modal denoising methods often focus on collaborative signals and ignore the characteristics of the modal features themselves, resulting in poor noise elimination effects. In contrast, the method of the present invention fully considers the modal characteristics of items in the denoising process, and based on the information bottleneck principle, retains the minimum sufficient modal information, effectively reduces noise interference, and improves the robustness and recommendation accuracy of the model; finally, the method of the present invention integrates information from multiple modalities through a cross-modal preference module, so that user preferences are more comprehensively reflected in the recommendation results, ensuring that the model performs better than existing methods on all test data sets;

[0176] (2) The information bottleneck principle shows significant effects in dealing with modal noise pollution. Although the MGCN method selects modal features through collaborative signal representation to achieve denoising, its limitation is that it only pushes the modal features close to the collaborative representation and fails to fully consider the uniqueness of the modal features. In contrast, the method of the present invention uses the information bottleneck principle to denoise modal features and obtains the minimum sufficient modal information required for downstream recommendation tasks through comparative learning. The model significantly improves the user's cross-modal preference by maximizing the user representation under different modalities, thereby enhancing the accuracy and stability of recommendations.

[0177] Table 1 Performance comparison of different recommendation models

[0178]

[0179] In Table 1, Baby, Sports, and Clothing are the existing infant datasets, sports product datasets, and clothing datasets, respectively. The evaluation indicators include Recall@10, Recall@20, NDCG@10, and NDCG@20. Recall@10 and Recall@20 represent the ability of the measurement method to correctly identify relevant items in the top 10 and 20 retrieval results, respectively. NDCG@10 and NDCG@20 represent the indicators for measuring the quality of the top 10 and 20 information retrieval rankings, respectively. The methods include MF (matrix decomposition algorithm), LightGCN (simplified and enhanced graph convolutional network for recommendation systems), VBPR (a model that incorporates visual features into personalized recommendations), MMGCN (multimodal graph convolutional network), LATTICE (for searching for optimal solutions in limited discrete datasets), FREEDOM (an algorithm for optimizing large-scale linear programming problems), and MGCN (multi-graph convolutional network).

[0180] Example 3

[0181] During the training process, the embedding representations of users and items are first learned on the training set through the loss function. Then, the hyperparameters are adjusted through the performance on the verification code to ensure the effectiveness of the model. Finally, the performance indicators of the model are calculated using the verification set. Through the learned user and item embeddings, the method of the present invention can predict the interaction possibility between each user and each item, and sort these possibilities from high to low. Finally, the top K items are selected to generate a recommendation list for each user.

[0182] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal recommendation method based on the information bottleneck principle, characterized in that: The following steps are involved: a) Build a dataset based on the user-item historical interaction records in the Amazon dataset ; b) Build a user-item graph for each modality through historical interaction records, and calculate the embedding of users and items through multi-layer graph convolution to obtain the final representation of the user and the fused embedding of the item; c) Build an item-item similarity graph for each modality of the item, retain the neighbors of the top k items with the highest similarity, capture the semantic signals between items, and then aggregate the semantic signals of items according to the contribution of the item modality to the downstream recommendation task to obtain the final representation of the item; d) Use contrastive learning to achieve information bottleneck denoising of modal features, further maximize the mutual information of user embeddings in different modalities, and obtain users' cross-modal preferences; e) Comparison of learning loss by BPR loss and information bottleneck , cross-modal preference contrastive learning loss And Adam optimizer optimizes user-item embedding.

2. According to claim 1, a multimodal recommendation method based on the information bottleneck principle is characterized in that: Step a) is as follows: Extract multimodal information of users and items in Amazon dataset and build dataset , , dataset Included The user set consists of , A collection of items consisting of items Each item contains two forms: picture and text. The picture corresponds to the visual embedding and the text corresponds to the text embedding. The item data extracted from the Amazon dataset is specifically the 4096-dimensional item visual modality embedding of the image data. And the 372-dimensional embedding of text data, and then the 372-dimensional text embedding is converted into a 384-dimensional item text modality embedding through sentence-transformers for generating sentence embeddings ; in, Represents the user, Indicates items, Represents a user collection Middle Users, Represents a collection of items Middle items; And for the dataset Each positive sample in the dataset has a corresponding negative sample. The positive sample is a real interaction pair between the user and the item, and the negative sample is a non-interaction pair between the user and the item.

3. According to claim 2, a multimodal recommendation method based on the information bottleneck principle is characterized in that: Step b) is as follows: b-1) The images and texts in the items correspond to the visual mode and text mode respectively. The interaction relationship between users and items in the graph is constructed for the visual mode and text mode of the item, and then the interaction graph of the visual mode is obtained. Interaction diagram with text modal ; b-2) For users, based on the collaborative filtering method, the user ID is mapped to a low-dimensional dense vector through the ID index to obtain the user's embedding. The embedding matrix of the user's visual modality is recorded as , the embedding matrix of user text modality is denoted as ; For items, use the corresponding item visual modality embedding and item text modal embedding Initialize; Then align the dimensions of user and item embeddings by reducing the dimensions of item visual modality embedding and item text modality embedding respectively. The specific calculation is as follows: ， ， in, and Represent the weight matrices of visual modality and textual modality respectively, represents transpose, represents the embedding matrix of the item text modality, Embedding matrix representing the visual modality of items; The visual modality embedding of user-item is obtained through the embedding matrix of user visual modality and textual modality and the embedding matrix of item visual modality and textual modality. and text-modal embeddings of user-items , the specific calculation is as follows: ， ； b-3) The first User-item embedding of layer graph convolution. The specific process is as follows: For visual modality: ， ， ； For text modal: ， ； ， ； Among them, according to the interaction diagram of visual modality Interaction diagram with text modal To determine and , and Represented in the user-item graph Medium User and items The first-order neighbor of and Respectively represent the visual mode and text mode in the User-item embedding with layer graph convolution, and Respectively represent the visual mode and text mode in the User-item embedding with layer graph convolution, , graph convolution layer, ; Then calculate the final user-item embedding in the visual mode and text mode respectively, the calculation formula is as follows: ， ， in, and denote the final user-item embeddings in visual and textual modalities, respectively. , , represents the final embedding of the user’s visual modality, represents the final embedding of the item’s visual modality, represents the final embedding of the user's text modality, The final embedding representing the item text modality; b-4) The final embeddings of multiple modalities are fused together by concatenation. The specific process is as follows: ， ， in, represents the fusion of the final embedding of the two modalities of the item, Represents the final user representation.

4. The multimodal recommendation method based on the information bottleneck principle according to claim 3, characterized in that: Step c) is as follows: c-1) Use the K-nearest neighbor (KNN) method to build an item-item similarity graph for each modality of the item and calculate the item Visual modality embedding With items Visual modality embedding Similarity scores between visual modalities , and items Text modal embedding With items Text modal embedding Similarity scores between text modalities ,thing For items The specific calculation of neighbors is as follows: ， ， Keep the top most similarity in visual mode and text mode respectively. The specific calculation is as follows: ， ， in, Indicates items in visual mode Before Neighbors, Indicates items in text mode Before Neighbors; c-2) By polymerizing multi-layer objects Neighbor Items information to capture deeper semantic relationships between visual and textual modalities and , the specific calculation is as follows: ， ， in, Represents semantic relations in visual modality, Represents the semantic relationship in text mode, Indicates items of Neighbor Items A collection of; c-3) The contribution of each modality of the item to the recommendation task is calculated through the attention mechanism. The specific calculation is as follows: ， ， in, Represents the attention mechanism operation, through a multi-layer perceptron accomplish, represents the contribution in the visual mode, Indicates the contribution in text mode; The semantic signals of each modality are then used to enhance the item embedding, and the final item representation is generated that comprehensively considers multimodal information. , the specific calculation is as follows: 。 5. The multimodal recommendation method based on the information bottleneck principle according to claim 4, characterized in that: Step d) is as follows: d-1) Calculate the information bottleneck contrastive learning loss Remove modal noise and encourage differences between the embedding matrix of items and the final embedding of items through the information bottleneck principle, while maximizing the amount of information related to the recommendation task. The embedding matrix of items includes the embedding matrix of the visual modality of items. and the embedding matrix of the item text modality , the final embedding of the item includes the final embedding of the item’s visual modality and the final embedding for the item text modal , the specific calculation is as follows: ， in, represents the recommendation loss based on Bayesian personalized ranking BPR, represents the mutual information of two embeddings, represents the information bottleneck contrastive learning loss in the visual modality, represents the information bottleneck contrastive learning loss in text modality; The mutual information between the embedding matrices of items in the visual and textual modalities and the final embedding of items is calculated as follows: ， ， in, It represents the similarity between two vectors. Indicates the temperature coefficient, item For users Interacted positive samples, items For users Negative samples that have not been interacted with, Indicates user In the embedding matrix of the visual modality, Indicates user In the text mode, the embedding matrix can be obtained by the same logic as step b-2). and ; d-2) The dependencies between different modalities are captured by contrastive learning methods, and the final embeddings in the user visual modality and text modality are mapped to a unified feature space to obtain the embeddedness in the user visual modality and text modality after mapping. The specific calculation is as follows: ， in represents a multi-layer perceptron MLP, represents the embedding of the user’s visual modality after mapping, represents the embedding in the user text mode after mapping, represents a fusion operation; according to and Calculating users' cross-modal preferences , the specific calculation is as follows: ， in, Indicates the maximum value operation.

6. The multimodal recommendation method based on the information bottleneck principle according to claim 5, characterized in that: Step e) is as follows: e-1) Optimize the parameters through BPR loss, the specific calculation is as follows: ， in, represents the BPR loss, Indicates user Interacted positive samples, Indicates user Negative samples that have not been interacted with; e-2) Compare BPR loss and information bottleneck with learning loss and cross-modal preference contrast learning loss Combined, the final loss is , the specific calculation is as follows: ， in, and Represents two different hyperparameters for balancing various losses; e-3) Optimize user-item embeddings through Adam optimizer.

Images