Method for multi-modal recommendation, electronic equipment and readable medium

Abstract

The invention relates to a method for multi-modal recommendation, electronic equipment and a readable medium, and the method comprises the steps: designing a graph purification strategy on an interaction graph, so as to eliminate a noise edge caused by non-uniform attention of a user; then, the three GCNs are applied in parallel to process multi-modal features and ID embedding respectively, and purified high-order representation of users and articles is obtained. For a second restriction, messaging is performed on the frozen user-user and item-item affinity maps to aggregate features between similar nodes. Summing learning results of the interaction graph and the affinity graph; afterwards, mutual information between ID embedding and modal features is maximized by introducing a contrast learning task and utilizing an inherent self-supervising signal of a multi-modal attribute, so that self-supervising learning without data enhancement is realized;

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method, electronic device and readable medium for multimodal recommendation. Background Art

[0002] With the rapid development of Internet technology and the advent of the information age, the amount of data has increased dramatically. As an effective information filtering tool, recommendation systems have been widely used in e-commerce, social networks, content distribution platforms and other occasions. Traditional recommendation systems usually rely on the interaction history between users and items, and predict users' potential preferences by analyzing these historical interaction data. However, such systems often face the problem of data sparsity, especially when new users or new items are introduced.

[0003] To address this problem, researchers began to explore the use of multimodal data (such as images, text, and audio) to enrich the information sources of recommendation systems. These multimodal data can provide more comprehensive descriptions of items and help the system capture multiple dimensions of user preferences more accurately. Although multimodal recommender systems (MMRS) have improved traditional recommendation techniques to a certain extent, they also face some new challenges and limitations.

[0004] A major challenge is how to effectively integrate and utilize information from different modalities. Early multimodal recommendation methods usually adopted simple linear fusion or weighted fusion strategies based on attention mechanisms, but these methods failed to fully utilize the potential correlation between modalities. With the successful application of graph neural networks (GNNs) in complex systems, researchers introduced GNNs to process multimodal data, using their powerful graph structure data processing capabilities to extract high-order features of users and items, thereby improving recommendation accuracy.

[0005] In addition, as an emerging learning paradigm, self-supervised learning (SSL) has demonstrated its powerful performance in fields such as computer vision and natural language processing by constructing and utilizing the data's own supervisory signals. In recommendation systems, SSL can generate multiple data views through techniques such as data augmentation and contrastive learning, and maximize the consistency between positive samples and minimize the inconsistency between negative samples, thereby effectively improving the generalization ability and robustness of the model.

[0006] However, existing multimodal recommendation methods still have shortcomings in dealing with modality-specific noise and utilizing self-supervisory signals. On the one hand, fixed interaction graph models may introduce noise edges caused by uneven user attention, which are continuously amplified during the graph convolution process, affecting the accuracy of learning effects. On the other hand, existing self-supervised learning strategies often rely on random data augmentation in multimodal recommendation, which may introduce bias and affect the final recommendation performance. Summary of the invention

[0007] The present invention provides a method, an electronic device and a readable medium for multimodal recommendation, which aims to solve the deficiencies of existing multimodal recommendation systems in processing modality-specific noise and effectively utilizing self-supervisory signals.

[0008] To achieve the above object, the present invention provides a method for multimodal recommendation in a first aspect, comprising the following steps:

[0009] Generate initial ID embeddings for each user and item, extract the original modal features of the items through the pre-trained model, and project the original modal features into a unified modal embedding space;

[0010] Using the initial ID embedding and the projected modality embedding, the attention scores between users and items are calculated according to the differences in users' attention to different modalities, and the interaction edges are pruned according to the differences in attention scores to generate a purified modality-specific interaction graph.

[0011] Apply the graph convolutional network to the purified modality-specific interaction graph and the observed original interaction graph to obtain high-order ID embeddings of users and items and corresponding high-order modality features through information propagation;

[0012] Construct a user co-occurrence graph and select users who share similar features through the top-k strategy to obtain enhanced representation between users; construct an item modal perception graph using similarity metrics and connect items with similar semantics through the top-k strategy to obtain enhanced representation of items;

[0013] The acquired high-order ID embedding and high-order modal features of users and items are weighted and summed with the acquired enhanced representations between users and items to form the final global user representation and global item representation.

[0014] Based on the final global user representation and global item representation, the inner product operation is used to calculate the preference scores between users and items, and the items with the highest scores are recommended to users.

[0015] Furthermore, the method also includes: introducing a contrastive learning task, calculating the contrastive loss by maximizing the similarity between the same nodes under the global representation, and optimizing the model parameters in combination with the Bayesian personalized ranking loss, and then using the inner product operation to calculate the preference scores between users and items using the final optimized global user representation and global item representation, and recommending the highest-scoring item to each user.

[0016] Furthermore, the method for calculating the attention score between the user and the item includes:

[0017] Project item features of each modality into a unified embedding space middle:

[0018]

[0019] in, is the projection feature under mode m, d is the embedding dimension, and Represent the transformation matrix and bias vector respectively, d m represents the dimension of the feature; σ1(·) is the LeakyReLU activation function; is the original modal feature of the item;

[0020] Use the inner product operation combined with the node degree to calculate the similarity between user u and item i under mode m

[0021] Where d(·) represents the degree of the node in the interaction graph G; is a multimodal user feature; T represents transposition; the softmax function is used to obtain the attention score of user u on item i

[0022]

[0023] Among them, N u represents the items interacted with by user u, and exp(·) is an exponential operation.

[0024] Further, methods for pruning interaction edges based on attention score differences include:

[0025] According to the attention score, determine whether to retain or prune the interaction edges between entities. The pruning rules are as follows:

[0026] If the normalized attention score is lower than the preset threshold, the interaction edge between the entities is pruned;

[0027] If the normalized attention score is higher than or equal to the preset threshold, the interaction edge between the entities is retained.

[0028] Further, methods for obtaining high-level ID embeddings of users and items through information propagation include:

[0029] Using the LightGCN network in the interaction graph G and interaction matrix G m Message propagation is performed in , and the weight transformation matrix and activation function are deleted;

[0030] During the convolution calculation at the lth layer, the embeddings of users and items are updated through matrices:

[0031]

[0032] Among them, U represents the set of users, I represents the set of items, represents the diagonal matrix of M, which is a symmetric adjacency matrix derived from the interaction graph G; the diagonal elements D aa Equal to ∑ b M ab ;

[0033] Embedding matrix of users and items in layer l Update from layer l-1, where the initial matrix of layer 0 is set to:

[0034]

[0035] in, represents the identity id embedding vector specific to user u1, represents the identity id embedding vector specific to item i1;

[0036] The ID embeddings at each layer are integrated using the summation where:

[0037]

[0038] in, is the final high-order id embedding of users and items, L in Represents the number of convolutional layers on the interaction graph G.

[0039] Furthermore, methods for obtaining corresponding high-order modal features through information propagation include:

[0040] The following convolution operation is performed on the purified interaction graph:

[0041]

[0042] Where D is the degree matrix, M is the enhanced adjacency matrix, and M m By G m Export, the initial feature matrix of layer 0 is set to:

[0043]

[0044] The final high-order modal characteristics The calculation is as follows:

[0045]

[0046] Furthermore, the method for obtaining the enhanced representation between users includes:

[0047] The user co-occurrence matrix is ​​constructed as follows:

[0048] S=G·G T ;

[0049] Where T represents transpose, The elements in represent the number of co-occurrences between users;

[0050] A top-n strategy is used to retain users with a certain number of co-occurrences:

[0051]

[0052] Among them, S a,. represents the element of the ath row in the matrix S; S is the constructed user co-occurrence matrix, is the matrix transformed by the above rules. According to the matrix Get the user co-occurrence graph A = {U, C}; The set of edges representing co-occurrences between users;

[0053] Embed using the user's high-level ID and modal characteristics Perform graph convolution on the user co-occurrence graph, and the calculation formula is:

[0054]

[0055] in, yes The diagonal matrix, N aa =n u ;

[0056] Use L af The output of the -c layer is used as the user's enhanced representation:

[0057]

[0058] in and They represent user-enhanced id embedding and modality features respectively;

[0059] Methods for getting an enhanced representation of an item include:

[0060] Under mode m, calculate i a and i b Similarity between

[0061]

[0062] in, is a fully connected matrix, cos(·,·) is the cosine function, and ||·|| is the length of the vector; and Represent the modal embeddings of items a and b respectively;

[0063] Adopt the top-n strategy as follows:

[0064]

[0065] in is a sparse matrix, and the item modal perception map is A m = {I,C m},in is the edge set representing the semantic relationship between items;

[0066] Using Higher Order Mode Features as follows:

[0067]

[0068] in yes The diagonal matrix of For ID embedding Perform the above operation in the same way to obtain Therefore, the enhancement of items is expressed as follows:

[0069]

[0070] in and Represent the enhanced ID embedding and modal features of the items, α m is the importance score of the item feature under mode m, α is a hyperparameter, α t =(1-α).

[0071] Furthermore, the method of weighted summing the obtained high-order ID embeddings of users and items with the high-order modal features, the obtained enhanced representations of users and items includes:

[0072] The operation of embedding the user ID is described as follows:

[0073]

[0074] in, is the final id embedding of the user; by performing the above operation on all ID embeddings and modal features of users and items, we get and

[0075] Use the attention weight β to control the fusion of user nodes, and use the importance score α to control the fusion of item nodes:

[0076]

[0077] in, and Represent the global representation of users and items respectively, represents weighted sum; β is a trainable parameter, α is a defined hyperparameter, v represents vision, and t represents text;

[0078] Use the resulting enhanced representation to perform a recommendation system with a preference score y ui , where the preference score y ui By H u and H I The inner product of is calculated as:

[0079] y ui =(h u ) T h i

[0080] Methods for calculating total contrast loss include:

[0081] InfoNCE is used to maximize the similarity between the same nodes under different representations and minimize the similarity between nodes of different users or items. The contrast loss L of user nodes CL,U The construction is as follows:

[0082]

[0083] Among them, U′ represents a group of users extracted from a batch of training data, and τ represents the temperature coefficient; the contrast loss of the item node is obtained in the same way Add the above results under all modes to get the total contrast loss L CL as follows:

[0084]

[0085] The method for calculating the final loss function L includes:

[0086] The Bayesian personalized ranking loss is used to make the score of positive pairs higher than that of negative pairs; the loss function L BPR It is expressed as:

[0087]

[0088] in, is a set of triples, item i b is randomly sampled; the function σ2(·) is the Sigmoid activation function;

[0089] The total contrast loss L CL And the loss function L BPR The final loss function L is merged as follows:

[0090]

[0091] Among them, λ1 and λ2 are hyperparameters for measuring contrast loss and L2 regularization, respectively, and Θ is a model parameter.

[0092] To achieve the above object, the second aspect of the present invention provides an electronic device, comprising: at least one memory and at least one processor;

[0093] The at least one memory is used to store a readable program;

[0094] The at least one processor is configured to call the readable program to execute the method.

[0095] To achieve the above objective, the third aspect of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method.

[0096] Beneficial effects of the present invention:

[0097] Compared with the prior art, the method, electronic device and readable medium for multimodal recommendation provided by the present invention effectively solve the inadequacy of modality-specific noise and self-supervisory signal utilization. First, by generating initial ID embedding and extracting the original modality features of the items, they are uniformly projected into the modality embedding space, thereby achieving effective alignment between different modalities. Then, the attention scores are calculated according to the differences in users' attention to different modalities, and the interaction edges are pruned based on these attention scores to generate a purified modality-specific interaction graph, which significantly reduces the impact of noise. Thereafter, the purified interaction graph and the original interaction graph are processed by a graph convolutional network for information propagation, thereby extracting high-order embedding features of users and items. Furthermore, by constructing a user co-occurrence graph and an item modality perception graph, users with similar features and items with similar semantics are selected based on a top-k strategy, thereby enhancing the representation of users and items. Finally, the high-order ID embedding, high-order modality features, enhanced representations between users and enhanced representations of items are weighted and summed to form the final global user and item representation. Based on this, the preference score is calculated using an inner product operation, thereby recommending the highest-scoring items to the user. This process not only improves the accuracy and reliability of recommendations, but also effectively utilizes self-supervision signals and avoids bias problems that may be introduced in data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0099] Figure 1 It is a flowchart of a method for multimodal recommendation disclosed in an embodiment of the present invention.

[0100] Figure 2It is an overview diagram of a PEARL framework disclosed in an embodiment of the present invention.

[0101] Figure 3 It is a performance comparison diagram of merging different modes on three data sets disclosed in an embodiment of the present invention.

[0102] Figure 4 It is a model performance diagram disclosed in an embodiment of the present invention, with pruning percentages ranging from 0.1 to 1.0 on three data sets.

[0103] Figure 5 It is a model performance graph of a user graph and an item graph under different top-n values ​​on three data sets disclosed in an embodiment of the present invention.

[0104] Figure 6 It is a model performance diagram of a contrast loss weight and a regularization coefficient under [0.1, 0.01, 0.001, 0.0001] disclosed in an embodiment of the present invention.

[0105] Figure 7 It is a distribution diagram of item representations learned by different models on an infant dataset disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0106] like Figure 1 As shown, the present invention provides a method for multimodal recommendation, which includes the following steps:

[0107] Step S100: Generate an initial ID embedding for each user and item, extract the original modal features of the item through the pre-trained model, and project the original modal features into a unified modal embedding space;

[0108] For each user and item, generate initial identifier (ID) embeddings. This step creates basic embedding vectors for users and items. The embeddings are randomly initialized and the dimensions are determined by the system design.

[0109] Use pre-trained models (for example, visual models such as VGG or text models such as BERT) to extract modal features for each item. Pre-trained models can extract rich feature expressions from different modalities of items (such as images, text, etc.), providing an additional level of information for the recommendation system.

[0110] The extracted original modal features are mapped to a unified embedding space through a series of transformations (linear transformations, such as matrix multiplication plus bias, accompanied by nonlinear activation functions such as ReLU or LeakyReLU). The purpose of this step is to unify modal features from different sources into a common feature space for effective comparison and fusion.

[0111] Through these steps, a comprehensive representation can be established for each user and item, including both identifier-based embedding and rich information extracted through modal features. This comprehensive representation provides the necessary input for the subsequent recommendation system to support more accurate and personalized recommendations. In the subsequent steps of the present invention, these features will be used to construct interaction graphs and perform graph convolutions to mine and utilize the complex relationships and interactions between users and items.

[0112] Step S200: using the initial ID embedding and the projected modality embedding, according to the difference in the user's attention to different modalities, calculate the attention score between the user and the item, and prune the interaction edge according to the difference in attention score, so as to generate a purified modality-specific interaction graph;

[0113] First, a comprehensive feature representation is constructed for each user and item by combining the initial ID embedding and the unified modality embedding generated in step S100. These features include identifier embeddings based on users and items as well as representations of their multimodal content.

[0114] Based on the above comprehensive features, the attention mechanism is used to calculate the user's attention to each item. This process relies on the calculation of attention scores, which reflect the difference in the user's attention to the different modal features of the items. The attention score can be calculated by methods such as the inner product followed by the softmax function to determine the user's interest in each item.

[0115] Based on the difference of calculated attention scores, interaction edges are pruned. This step aims to remove misleading interactions that may be caused by noise or inconsistent user attention. Specifically, it can be decided which edges should be retained or pruned by setting a threshold or using score differences. The pruning probability is based on the relative size of the attention score and the user's behavior pattern, and lower-scoring edges are more likely to be pruned.

[0116] After pruning, a purified modality-specific interaction graph is constructed, which more accurately reflects the user's real preferences and behaviors. This purification process helps to reduce the impact of noise in subsequent graph convolutional network learning and improve the accuracy and efficiency of the recommendation system.

[0117] It is understandable that through this meticulous attention score calculation and score-based edge pruning, the interaction graph is effectively optimized to focus more on the user's actual interests and behaviors. It not only improves the accuracy of information processing in the recommendation system, but also lays a solid foundation for in-depth mining of the complex relationship between users and items. In addition, by eliminating low-relevance interactions, large-scale data can be processed more effectively, improving the efficiency of calculations and the quality of recommendations.

[0118] Step S300: Apply the graph convolutional network to the purified modality-specific interaction graph and the observed original interaction graph to obtain high-order ID embeddings of users and items and corresponding high-order modality features through information propagation;

[0119] In this step, graph convolution operations are performed on two interaction graphs: the purified modality-specific interaction graph and the observed original interaction graph. Graph convolutional networks (GCNs) can capture high-order relationships between users and items by transferring and aggregating node information in the graph, thereby extracting more complex and deep feature expressions.

[0120] During the graph convolution process, the initial ID embedding and modality embedding are further updated and optimized. Through multi-layer information transmission and feature aggregation, the graph convolutional network (GCN) helps to generate more expressive high-order ID embedding and modality features. These high-order features contain more contextual information and potential connections between users and items, thus providing a more accurate and comprehensive description of users and items.

[0121] At each layer of a graph convolutional network (GCN), nodes are updated based on the features of their neighboring nodes. Through the weight matrix and nonlinear activation functions (such as ReLU), each iteration "mixes" the neighbor's information into the current node's features. This neighbor-based information update can gradually spread and enrich the feature representation of each node.

[0122] Through repeated graph convolution operations, features from different layers can be extracted and merged from the original and purified interaction graphs to form the final high-order user and item embeddings. These high-order embeddings can reflect the multi-dimensional relationships and interaction patterns between users and items, providing key information for subsequent recommendation decisions.

[0123] Step S400: construct a user co-occurrence graph, select users who share similar features through the top-k strategy, and obtain enhanced representation between users; construct an item modal perception graph using similarity metrics, connect items with similar semantics through the top-k strategy, and obtain enhanced representation of items;

[0124] Constructing a user co-occurrence graph: The user co-occurrence graph is constructed based on the co-occurrence relationship between users. The user co-occurrence relationship is defined based on the number of items that users interact with in common, that is, if two users have interacted with the same or similar set of items, the two users will be connected in the co-occurrence graph. A Top-k strategy is used to select the k other users that are most similar to each user. This selection is based on the frequency of co-occurrence between users, that is, the number of times users have selected the same item in common, and the user with the highest frequency is selected as the neighbor of each node. Applying a graph convolutional network on the user co-occurrence graph can further refine and enhance the representation of users. This is achieved by aggregating the features of each user node with the features of its neighbors (co-occurring users), which enhances the collaborative information in the user representation.

[0125] Constructing the Item Modal Perception Graph: The Item Modal Perception Graph is constructed based on the modal features of the items, such as text descriptions and images. This graph uses the similarity between modal features (using metrics such as cosine similarity) to connect semantically similar items. Similarly, connecting each item to its k most similar other items through a Top-k strategy can help reduce noisy and irrelevant connections, ensuring that only high-quality, semantically relevant links are retained in the graph. Performing graph convolution operations on the Item Modal Perception Graph allows items to obtain information from other semantically similar items, thereby enhancing the modal feature representation of items.

[0126] After performing graph convolution on the user co-occurrence graph and the item modality perception graph, the representations of users and items will contain rich information from their similar entities, so that the enhanced representation captures deeper relationships and patterns between users and items. The final global representations of users and items are formed by aggregating the outputs of the graph convolutional network (such as through weighted summation). These representations not only reflect the individual characteristics of users and items, but also incorporate their relative positions and contextual relationships in the network.

[0127] It should be noted that the Top-k strategy is a selection method used to select the most relevant k items from a set of candidate items in a recommendation system. Specifically, the Top-k strategy sorts candidate users or items based on a similarity metric (such as co-occurrence frequency, cosine similarity, etc.), and then selects the k users or items with the highest similarity. By focusing on the most relevant neighbors, it can effectively reduce noise and irrelevant information, and improve the accuracy and efficiency of the subsequent recommendation process.

[0128] Step S500: weighted sum the acquired high-order ID embeddings and high-order modal features of users and items, and the acquired enhanced representations of users and items, to form final global user representations and global item representations;

[0129] The high-order ID embeddings of users and items obtained from the interaction graph learning module are fused with the corresponding high-order modality features. The fusion process involves weighted summation, where the weights are adjusted according to the importance of different features.

[0130] Fusion of enhanced representations between users and items: In addition to ID embedding and modality features, enhanced representations obtained from user co-occurrence graphs and item modality perception graphs are also fused. These enhanced representations reflect the relative positions and similarities of users and items in their corresponding groups.

[0131] Comprehensive fusion: The final global user representation and global item representation are formed by combining high-order features with enhanced features. These global representations integrate information from different data sources and analysis dimensions to provide a comprehensive overview of users or items. During the fusion process, additional techniques need to be applied to optimize the quality of the representation, such as regularization, standardization, or the use of more complex fusion mechanisms (such as feature fusion layers in deep learning models).

[0132] Step S600: Based on the final global user representation and global item representation, the preference scores between the user and the item are calculated using an inner product operation;

[0133] Step S700: recommend the item with the highest score to the user based on the preference score data.

[0134] This is achieved by performing a dot product of the user's global representation and the item's global representation, where the global representation includes fused high-order ID embeddings and modal features. The resulting score reflects the user's potential preference for each item, and the higher the score, the greater the user's interest in the item. According to the calculated preference score, all recommended items are sorted. For example, each user and the list of items that may be of interest to them are sorted according to the preference score. Based on the sorting results, the item with the highest score is recommended to each user. In this embodiment, Top-N items are selected as recommendations, where N is set according to system design or user preference.

[0135] In actual applications, the recommendation algorithm will be adjusted based on user feedback or system monitoring results to optimize the accuracy of recommendations and user satisfaction. When making recommendations, in addition to considering preference scores, factors such as item diversity and novelty can also be introduced to increase the attractiveness of the recommendation list and the user's interest in exploration.

[0136] In order to more clearly understand the technical solution of the present invention, we first describe the overview of the multimodal recommendation task and graph learning (PEARL), and then introduce each component in detail.

[0137] The ID embeddings of each user u∈U and item i∈I are initialized as Where d is the embedding dimension. The original modal features of the items extracted by the pre-trained model are where m∈M, d m In addition, the multimodal user features are randomly initialized In the embodiment, only visual and textual modes are considered, that is, M = {v, t}. The historical interaction between the user and the item is represented as a sparse matrix The observed interaction graph G = {V, E} is constructed using this sparse matrix. V = {U∪I} is a node set representing users and items, and E = {<u,i> |G u,i =1} represents the edge set of interactions between users and items. Under the above setting, the multimodal recommendation task is to accurately predict the possibility of user u selecting item i by leveraging representation learning of user behavior and multimodal features of items.

[0138] like Figure 2 As shown in Figure 5, the graph learning (PEARL) consists of four parts: (1) an initialization and graph purification module, which extracts multimodal features using a pre-trained model while randomly initializing embeddings, and performs edge denoising according to the proposed graph purification strategy; (2) an interaction graph learning module, which applies GCNs to modality-specific interaction graphs and observed interaction graphs to capture high-order modal features and ID embeddings, respectively; (3) an affinity graph learning module, which aggregates similar node information on frozen user-user and item-item affinity graphs to obtain enhanced representations of users and items; and (4) a fusion and recommendation module, which fuses high-order features and enhanced features to obtain a global representation, optimizes it through contrastive learning, and recommends items to each user based on their preference scores.

[0139] After observing the interaction graph, we purify it to generate a specific interaction graph for each modality to prevent the diffusion of noisy signals caused by uneven user attention in graph learning. Specifically, we design a purification strategy based on the modal differences of users' attention to items.

[0140] Considering the differences in dimensions and semantics of item features generated by pre-trained models for different modalities, the item features of each modality are projected into a unified embedding space. As shown below:

[0141]

[0142] in, is the projection feature under mode m, and Represent the transformation matrix and bias vector respectively. Function σ1(·) is the LeakyRelu activation function. Then, the inner product operation is combined with the node degree to calculate the similarity between user u and item i in mode m.

[0143] where d(·) represents the degree of the node in the graph G, is a multimodal user feature; T represents transposition. Then, the softmax function is used to obtain the attention score of user u on item i

[0144]

[0145] Among them, N u represents the items interacted with user u, and exp(·) is an exponential operation. Based on the difference in attention scores under different modalities, a pruning probability is defined This probability determines an edge in the graph G<u,i> Whether it is pruned, so as to obtain the purified interaction graph G under mode m m :

[0146]

[0147] Where σ2(·) is the normalized Sigmoid activation function. The probability of all edge pruning in the purified interaction graph under all modalities forms a probability distribution p = {p1,p2,…,p |M||E|-1}.

[0148] Typically, a certain percentage of edges is pruned. This ensures that each edge in E appears at least once in the purified interaction graph of all modalities. In other words, each user is influenced by at least one modality. Therefore, the pruned edges are pruned according to the probability distribution p. edge, where In this way, the same interactions with lower attention scores in different modalities are more likely to be pruned. The purified interaction graph is G m = {V,E m},in The purified interaction matrix under mode m is G m .

[0149] In order to learn high-level representations of users and items under the observation and purification of interaction graphs, the LightGCN is used to build a convolutional neural network on G and G m LightGCN is a lightweight graph convolutional network that removes the weight transformation matrix and activation function. The convolution operation on the lth layer of graph G is as follows:

[0150]

[0151] in represents the diagonal matrix of M, which is a symmetric adjacency matrix derived from G. The diagonal elements D aa Equal to ∑b M ab The embedding matrix of users and items in the lth layer Update only from layer l-1, where the initial matrix of layer 0 is set to:

[0152]

[0153] Given that there are no self-loops in the interaction graph, the information from the previous layer of the same node will not be retained during message propagation. Therefore, the ID embeddings of each layer are integrated using summation as follows:

[0154]

[0155] in is the final high-order id embedding of users and items, L in Represents the number of convolutional layers on the graph G.

[0156] Similarly, in the purified interaction graph G m The following convolution operation is performed on it:

[0157]

[0158] Among them, M m By G m Export, the initial feature matrix of layer 0 is set to:

[0159]

[0160] The final high-order modal characteristics The calculation is as follows:

[0161]

[0162] Through the above-mentioned interaction graph learning, collaborative signals and multimodal information are encoded into high-level representations of users and items.

[0163] To further mine the hidden information of implicit intra-modality affinities between similar users or items, graph learning is performed on the pre-built and frozen user co-occurrence graph and item modality perception graph.

[0164] User co-occurrence graph. Inspired by Wang et al., it is believed that there is a co-occurrence relationship between users who interact with the same item. Therefore, a user co-occurrence graph is constructed to learn similar preferences between co-occurring users. First, the user co-occurrence matrix is ​​constructed as follows:

[0165] S=G·G T ;

[0166] in, The elements in represent the number of co-occurrences between users. For each user u, a small number of users co-occur with him many times, while most users only co-occur a few times. Therefore, a top-n strategy is adopted to retain users with a certain number of co-occurrences, as shown below:

[0167]

[0168] Among them, S a,. represents the element of the ath row in matrix S, where S is the constructed user co-occurrence matrix. is the matrix transformed by the above rules. Therefore, according to the matrix Get the user co-occurrence graph A = {U, C}, where Represents the co-occurrence edge set between users. Then, the user’s high-order ID is used to embed and modal characteristics Perform graph convolution on image A, taking the former as an example:

[0169]

[0170] in yes The diagonal matrix, N aa =n u Unlike the interaction graph, each node in the affinity graph has a self-loop constructed by the top-n strategy. Therefore, the result of each convolution will retain the information of the previous layer, and directly use L af The output of the -c layer is used as the user's enhanced representation:

[0171]

[0172] in, and They represent user enhanced id embedding and modality features respectively.

[0173] Item modal perception map. In order to learn the implicit semantic information between item features, based on the original modal features The project modality perception map is constructed. Specifically, under modality m, calculate i a and i b Similarity between

[0174]

[0175] in, is a fully connected matrix, cos(·,·) is a cosine function, and ||·|| is the length of the vector. In order to reduce computing resources and avoid noise caused by unnecessary connections, the top-n strategy is also adopted, as shown below:

[0176]

[0177] in, is a sparse matrix. The modal perception map of an item is A m = {I,C m},in is a set of edges that represent semantic relations between items.

[0178] Using Higher Order Mode Features as follows:

[0179]

[0180] in, yes The diagonal matrix of For ID embedding Perform the above operation in the same way to obtain Therefore, the enhancement of items is expressed as follows:

[0181]

[0182] in, and denote the enhanced id embedding and modality features of the items, α m is the importance score of the item feature under modality m. In this work, a hyperparameter α is introduced as the importance of the visual modality, and α t =(1-α).

[0183] After learning high-order and enhanced representations of users and items from the interaction and affinity graphs, a summation is performed to effectively capture the interaction information and its implicit intra-modality affinity. Specifically, the operation on user ID embedding is formulated as follows:

[0184]

[0185] in, is the final id embedding of the user. By performing the above operations on all the ID embeddings and modality features of users and items, we can get and

[0186] Different from traditional self-supervised learning, this paper develops a contrastive learning paradigm without data augmentation. The design is based on the existence of sufficient self-supervisory signals between collaborative signals and multimodal information. Specifically, InfoNCE loss is used to maximize the similarity between the same nodes under different representations and minimize the similarity between nodes of different users or projects. The contrastive loss L of user nodes is CL,U The construction is as follows:

[0187]

[0188] Where U′ represents a group of users extracted from a batch of training data, and τ represents the temperature coefficient. Similarly, the contrast loss of the item node can be obtained Add the above results under all modes to get the total contrast loss L CL as follows:

[0189]

[0190] In order to obtain a global representation of users and items, a concatenation operation is used to fuse ID embeddings and modality features. Specifically, the attention weight β is used to control the fusion of user nodes, and the importance score α is used to control the fusion of item nodes.

[0191]

[0192] in, and denote the global representation of users and items respectively, is the connection operation, β is a trainable parameter, α is a hyperparameter defined previously, v represents vision, and t represents text.

[0193] According to the previous recommendation, the inner product operation is used to calculate the preference score y between the user and the product. ui , as shown below:

[0194] y ui =(h u ) T h i ;

[0195] Where item i has no interaction with user u. For each user, recommend items with top-k preference scores.

[0196] To further optimize the parameters of PEARL, the Bayesian Personalized Ranking (BPR) loss is used to make the positive pairs have higher scores than the negative pairs. The loss function L bpr It is expressed as:

[0197]

[0198] in is a set of triples, item i b is randomly sampled. Function σ2(·) is the Sigmoid activation function.

[0199] In summary, the contrast loss L CL and BPR loss L BPR The final loss L is merged as follows:

[0200]

[0201] Among them, λ1 and λ2 are hyperparameters for measuring contrast loss and L2 regularization, respectively, and Θ is a model parameter.

[0202] In order to verify the effectiveness and superiority of the proposed PEARL, experiments were conducted on three real datasets to address the following research questions:

[0203] 1. Can PEARL achieve state-of-the-art performance on multimodal recommendation tasks compared to baselines?

[0204] 2. Does each component used in PEARL contribute to improving recommendation performance?

[0205] 3. What impact does the addition of multimodality have on the performance of PEARL?

[0206] 4. What is the impact of hyperparameter perturbations on PEARL performance?

[0207] 5. Why can the multimodal learning framework achieve better recommendation performance?

[0208] Experiments are conducted on three publicly available Amazon datasets: (a) Baby, (b) Sports and Outdoors, and (c) Clothing, Shoes, and Jewelry, hereinafter referred to as Baby, Sports, and Clothing. These three datasets include visual and textual information of items. A 5-core setting is used for the original data to ensure that each user and item has at least 5 interactions. The processing results are shown in Table 1.

[0209] Table 1 Statistics of experimental datasets

[0210]

[0211] The 4096-dimensional visual features and 384-dimensional text features published in previous work are directly used, which are extracted by the pre-trained model VGG16 and the sentence converter Sentence2Vec, respectively. For each dataset, the historical interactions are randomly divided into training set, validation set and test set in a ratio of 8:1:1. In addition, the commonly used indicators Recall@K (denoted as R@K) and NDCG@K (denoted as N@K) are used to evaluate the recommendation performance of the model. Specifically, the average results of all users in the test set are selected and reported in K∈{10,20}.

[0212] To verify the superiority of PEARL, it is compared with various recommended models and divided into the following three groups:

[0213] a) Traditional Model

[0214] BPR: is a matrix factorization (MF) based method that performs personalized ranking by comparing item pairs for each user and uses Bayesian methods for optimization.

[0215] LightGCN: Simplifies the vanilla graph convolutional network by removing the layers responsible for non-linear activation and feature transformation.

[0216] b) Recommendation via Self-Supervised Learning

[0217] SGL: Enhancing graph collaborative filtering by maximizing the mutual information among three views of the constructed user-item interaction graph using contrastive learning.

[0218] NCL: Use expectation-maximization clustering to identify semantically and structurally adjacent nodes and use these nodes to generate contrastive views that are considered positive pairs.

[0219] c) Multimodal Recommendation

[0220] VBPR: Extends the BPR method by adding visual features extracted from item images to reveal the visual dimensions that influence user preferences.

[0221] MMGCN: Performing information propagation on modality-specific user-item graphs to refine users' modality-aware representations.

[0222] GRCN: Identify false positive connections in interaction graphs and use soft denoising to reduce the impact of noise during graph convolution.

[0223] DualGNN: Introducing a user-user association graph to enhance user representation by mining hidden information between similar users

[0224] LATTICE: Condensing item-item graphs under different modalities and mining latent semantic information of items through information propagation.

[0225] SLMRec: A data augmentation method is used to construct positive and negative sample node pairs, and the self-supervisory signals are used for contrastive learning.

[0226] MGCL: Utilizes self-supervisory signals from different views to optimize the model through contrastive learning to enhance the representation of users and items.

[0227] MGCN: Adaptive fusion denoising at the modality feature level using mutual information and learning the relative importance of different modalities.

[0228] LGMRec: Designing a global hypergraph to capture many-to-many dependencies between attributes and items.

[0229] To ensure the fairness of experimental comparison, a unified framework, MMRec, is used to implement the proposed model and baseline. The embedding dimensions of users and items in all models are set to 64, and the model parameters are initialized using the Xavier method. All experiments are implemented in Pytorch and executed on an NVIDIA RTX3090Ti GPU with 24GB of memory. The Adam optimizer is used to train the model with a batch size of 2048 and a learning rate of 0.001 for a maximum of 1000 epochs. In order to reduce the hyperparameter search space of the model, the number of GCN layers in interaction and affinity graph learning is empirically set to L, L, and L, respectively. in =2 and L af =1. In addition, the temperature coefficient τ in contrastive learning is set to 0.2. In addition, the optimal values ​​of other hyperparameters are determined by grid search. Specifically, the pruning ratio ρ and the importance score α are adjusted in {0.1, 0.2, 0.3, 0.4, 0.5}. The n values ​​of the user graph are searched in {10, 20, 30, 40, 50, 60} and {5, 10, 15, 20, 25, 30}, respectively. u and n of the project graph i The weight λ1 of the contrastive loss and the regularization coefficient λ2 are both adjusted in {1e-01, 1e-02, …, 1e-05}. In particular, an early stopping strategy is used to stop the training when the R@20 metric does not improve for 20 consecutive epochs on the validation set.

[0230] The overall effect is shown in Table 2:

[0231] Table 2 Overall performance of recall and NDCG of PEARL and baselines on three datasets

[0232]

[0233] To answer RQ1, PEARL is compared with the baseline performance on three datasets. Table 3 reports the overall performance of PEARL in terms of R@10, R@20, N@10, and N@20.

[0234] (1) For the baseline models, it is clear that introducing multimodal information can improve the recommendation performance. For example, on the Baby dataset, the introduction of visual modality improves R@20 by 15.30% from BPR to VBPR. Similarly, the application of self-supervised learning improves the performance of SGL by 26.46% compared to BPR. Except for MMGCN, the multimodal methods based on GCN outperform the traditional models. Specifically, MMGCN cannot distinguish the unique features of a single modality, resulting in insufficient utilization of multimodal information. The soft denoising strategy used in GRCN performs well on dense datasets, but performs poorly on sparse datasets (e.g., Clothing dataset), which may be caused by inaccurate denoising. DualGNN and LAT-TICE mine hidden semantic information through homogeneous graph learning to enhance the representation of users and items. In addition, it can be observed that recent studies (i.e., SLMRec, MGCL) adopt self-supervised learning to further improve the recommendation performance. Among the benchmarks, MGCN performs best on the Sports and Clothing datasets, while LGMRec surpasses other methods on the Baby dataset. The reason for this difference is that the feature-level denoising method in MGCN is more effective on large-scale datasets, while the hypergraph structure adopted by LGMRec is relatively complex and more suitable for small-scale datasets.

[0235] (2) Next, these baselines are compared with PEARL. Overall, PEARL significantly outperforms all baselines across different datasets. Specifically, it achieves 7.34%, 5.99%, and 8.08% performance improvements on R@20, respectively, which are lower than the next best model on the three datasets. The performance is analyzed by the benefits of the interaction graphs of the attributes on the effective denoising of specific modes and the self-supervised signal mining by multi-graph learning. On the one hand, PEARL eliminates the information that users may not perceive based on the following reasons: the user attention of different modes is uneven. This hard denoising directly cuts off the noise propagation in graph learning, thereby achieving better performance than the soft-denoising GRCN and the feature-level denoising MGCN. On the other hand, an affinity graph learning is designed, which mines self-supervised signals more effectively than SLMRec without data augmentation. In addition, by maximizing the mutual information between ID embeddings and modality features, PEARL performs better than MGCL. Unlike LGMRec, which adopts a complex hypergraph learning method, PEARL captures historical interaction information and internal relations by learning dual representations to obtain more effective multimodal enhancement. Moreover, the proportion of performance improvement achieved by PEARL increases with the data size, which indicates that PEARL has a greater advantage in more complex scenarios.

[0236] Table 3 is the ablation experiment, as shown in Table 3:

[0237] Performance of Ablation Variants of Removing Different Components in PEARL in Table 3

[0238]

[0239] To answer RQ2 and RQ3, a detailed ablation study was conducted on the components and modal features of PEARL.

[0240] To explore the contribution of key components in PEARL to the recommendation performance, the following model variants were designed.

[0241] PEARL without GP removes graph purification, i.e., the influence of noise modal information is not considered in interactive graph learning.

[0242] PEARL without AL removes affinity graph learning, i.e., the implicit intra-modal features between similar nodes are not aggregated.

[0243] PEARL without ALuser eliminates the learning of the user co-occurrence graph and only retains the item-mode-aware graph in affinity graph learning.

[0244] PEARL without ALitem cancels the learning of the item-mode-aware graph in affinity graph learning and only retains the user co-occurrence graph.

[0245] PEARL without CL eliminates contrastive learning, i.e., the self-supervised signal between ID embeddings and modal features is ignored.

[0246] Table 4 lists the experimental results of PEARL and the above model variants on three datasets, from which the following observations can be drawn:

[0247] (1) Comparing PEARL and PEARL w / o GP, it can be seen that removing the GP module leads to a performance drop. This indicates that constructing an accurate modality-specific user-item graph in interactive graph learning is crucial, and the designed graph purification strategy is effective. In particular, different from the soft denoising of GRCN, the hard purification strategy of the present invention shows relatively stable improvement in recommendation performance on datasets of different sizes.

[0248] (2) Utilizing affinity graph learning significantly improves the model performance, especially on the Clothing dataset, which indicates that datasets with a large number of users and items contain more hidden structures. In addition, w / o ALuser < w / o ALuser indicates that item-mode-aware graph learning is the main reason for the performance improvement, because the user co-occurrence graph is constructed based on user-item interactions, and some user relationship information has been extracted through interactive graph learning. On the contrary, the item-mode-aware graph is constructed based on the original modal features and can thus capture the hidden semantic information between items, which is then limited to collaborative signals.

[0249] (3) Comparing PEARL and PEARL w / o CL, it is clear that CL (contrastive learning) effectively improves the recommendation performance, which reflects that self-supervised learning without data augmentation is feasible in multimodal domains. ID embeddings and modality features can be further enhanced by maximizing the mutual information between them. In addition, it is worth noting that CL performs relatively poorly on the Sports dataset, which may be due to the fact that the Sports dataset contains more diverse product types with significant feature differences, making feature learning more challenging.

[0250] like Figure 3 As shown in the figure, as for the influence of modality, in order to study the impact of each mode on the recommendation accuracy, different unimodal features are integrated into the ID embedding in formula (20), and the following variants are constructed: ID, containing only ID embedding; ID+Visual, containing ID embedding and visual information; ID+Textual, containing ID embedding and textual information.

[0251] like Figure 4 As shown in the figure, PEARL outperforms the above variants in terms of R@20 and N@20 indicators, proving the effectiveness of the multimodal fusion framework designed by the present invention. It can be seen that including both visual and textual modalities can improve recommendation performance, but the textual modality plays a more important role. In particular, after including visual features on the Clothing dataset, the R@20 indicator actually decreases. This obvious phenomenon can be attributed to the fact that users are more inclined to make purchase decisions based on text descriptions, and existing visual models cannot accurately identify the parts of the image that stimulate user purchasing behavior. Therefore, visual features contain a lot of noise and provide limited help in performance improvement. In order to verify this hypothesis, further research was conducted in the following content.

[0252] To answer RQ4, sensitivity experiments were conducted on the clipping percentage p, affinity map top n value, visual importance score a, loss weights λ1 and λ2.

[0253] Effect of pruning percentage p: In the graph purification module, the pruning percentage p controls the number of edges in the mode-specific interaction graph Gm. Assuming that user behavior is influenced by at least one mode (i.e., the maximum value of p is 1), each edge observed in the interaction graph G only appears once on Gm. Given the above, the p value is changed from 0.1 to 1.0 in steps of 0.1. Figure 5As shown, the experimental results clearly show that the model performance on the three datasets initially increases with the increase of p value and then decreases with the increase of p value. From this we can conclude that increasing the value of p within a certain range can effectively prune the noisy edges in the graph structure, thereby improving the model performance. However, once p exceeds a certain threshold, the effective connection between users and items will be cut off, resulting in a decrease in model performance. Necessary training data. Specifically, on the Baby and Sports datasets, the recommendation accuracy reaches the optimal value at p = 0.3, while on the Clothing dataset it is p = 0.4. This difference may be because the visual and textual features on the Clothing dataset are more significantly different, and more users are only influenced by one modality, which triggers purchasing behavior. This leads to an increase in the number of noisy edges in the specific modality interaction graph, requiring a higher p value for pruning.

[0254] Impact of top-n values ​​in affinity graph: To reduce memory consumption and prevent message propagation between irrelevant nodes in the affinity graph learning module, a top-n strategy is adopted to ensure that each user or item is only connected to the n most similar users or items. In this step, the optimal values ​​of n_u and n_i in the three datasets are explored in the range of {10, 20, 30, 40, 50, 60} and {5, 10, 15, 20, 25, 30}, respectively. Figure 6 a shows the influence of n in learning the user co-occurrence graph. It can be seen that n = 30 is the most suitable value for the number of neighbors on the Baby and Clothing datasets, while this value is 40 on the Sports dataset. Therefore, it can be considered that this is because the number of interactions in the Sports dataset is larger, resulting in more similar behaviors between users. Figure 6 b shows the effect of ni in object pattern perception learning. It can be seen that PEARL achieves the best performance when ni=10 on the Baby dataset and when ni=5 on the other two datasets. In summary, lower ni values ​​can reduce the noise of irrelevant neighbors. In addition, since the vertical axis range of the two line graphs on the same dataset is unified, it can be observed that the model performance is less sensitive to the setting of n_u than n_i. This observation is consistent with the conclusion drawn from the above content.

[0255] In equations (16) and (20), the visual importance score a controls the weight of the visual feature, while the coefficient of the corresponding text feature is 1-a. Based on the above research results, the contribution of each modality is further investigated, and the value of a is adjusted from 0.1 to 0.9 with an interval of 0.1. The experimental results are listed in Table 5, and the same trend is observed in the three datasets. In particular, regardless of whether the indicator is R@20 or N@20, the optimal value is reached when a=0.2 (i.e., a=0.2<0.8=a). This phenomenon reflects that text features contain more information than visual features. In addition, when the score value a exceeds 0.2, the accuracy of the score shows a continuous downward trend, and this trend is not obvious on the clothing dataset. Combining this observation with the conclusion in 5.2.2, it is more certain that the gap between the text mode and the visual mode is more significant in the clothing dataset.

[0256] Table 4 shows the performance of the model on three datasets with visual importance scores ranging from 0.1 to 0.9:

[0257]

[0258] Impact of loss weights λ1 and λ2: In the final loss (Eq. (23)), λ1 and λ2 control the weights of the contrastive loss and L2 regularization, respectively. A grid search is performed in {le-0l, 1e-02, …, le-05} to find the best values ​​of λ1 and λ2 on the three datasets. Figure 7 As shown, the performance of PEARL reflects the performance across different datasets. Specifically, on the Baby dataset, the best results of PEARL are λ1=1e-03 and λ2=1e-04: on the Sports dataset, it is λ1=1e-04 and λ2=2e-05: on the Clothing dataset, it is λ1=1e-03 and λ2=2e-02. In addition, it is observed that the performance of PEARL is less sensitive to the setting of the L2 regularization weights than the contrastive loss. The results show that it is very important to adjust the value of the weights according to different datasets, especially the weight of the contrastive loss. A parameter value of 1 can narrow the value scale gap between the contrastive loss and the BPR loss.

[0259] To answer RQ5, the item representations produced by SLMRec, MGCL, MGCN, and PEARL are visualized on the Baby dataset, respectively. To reduce the computation time, 500 items are randomly selected from the Baby dataset, and their final representations are mapped into 2D space using tSNE. Gaussian kernel density estimation (KDE) is used to draw circular heat maps to depict the distribution of item correspondences, and the corresponding angle-density plots are further drawn to more clearly depict the density of different angles.

[0260] Previous studies have shown that maintaining a uniformly distributed representation can preserve the intrinsic characteristics of nodes and improve their generalization ability, thereby improving recommendation performance. Both SLMRec with data augmentation and MGCL with contrastive learning exhibit multiple dense regions in terms of visualization. MGCN alleviates the distribution clustering after feature-level denoising, but still does not learn a more uniformly distributed representation like PEARL. The propagation of noise is avoided by purifying the mode-specific interaction graph, while mining the self-supervisory signal between heterogeneous and homogeneous information to improve the uniformity of the representation. In summary, this explains why PEARL outperforms other multimodal recommendation methods.

[0261] In this embodiment, a new MMRS method PEARL is proposed. Specifically, a graph purification strategy is first designed to eliminate misleading noise signals and obtain modality-specific interaction graphs. An interaction graph learning module is further designed to mine high-order representations of users and items. Then, user-user relationship graphs and item-item relationship graphs are constructed to learn implicit self-supervisory signals between similar nodes. Finally, a non-enhancement contrastive learning task is proposed in the fusion module to assist model training. Extensive experiments are carried out on three datasets to verify that PEARL outperforms various state-of-the-art baselines in terms of R@K and N@K.

[0262] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0263] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The electronic device includes a memory and a processor; wherein the memory is used to store a readable program, and the readable program is loaded and executed by the processor to implement the method as described above.

[0264] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0265] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0266] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0267] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0268] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, the term "includes", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "includes one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0269] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for multimodal recommendation, characterized in that: The steps include: Generate initial ID embeddings for each user and item, extract the original modal features of the items through the pre-trained model, and project the original modal features into a unified modal embedding space; Using the initial ID embedding and the projected modality embedding, the attention scores between users and items are calculated according to the differences in users' attention to different modalities, and the interaction edges are pruned according to the differences in attention scores to generate a purified modality-specific interaction graph. Apply the graph convolutional network to the purified modality-specific interaction graph and the observed original interaction graph to obtain high-order ID embeddings of users and items and corresponding high-order modality features through information propagation; Construct a user co-occurrence graph, select users who share similar features through the top-k strategy, and obtain enhanced representation between users; construct an item modal perception graph using similarity metrics, connect items with similar semantics through the top-k strategy, and obtain enhanced representation of items; The acquired high-order ID embedding and high-order modal features of users and items are weighted and summed with the acquired enhanced representations between users and items to form the final global user representation and global item representation. Based on the final global user representation and global item representation, the inner product operation is used to calculate the preference scores between users and items, and the items with the highest scores are recommended to users.

2. The method for multimodal recommendation according to claim 1, characterized in that: The method also includes: introducing a contrastive learning task, calculating the contrastive loss by maximizing the similarity between the same nodes under the global representation, optimizing the model parameters in combination with the Bayesian personalized ranking loss, and then using the inner product operation to calculate the preference scores between the user and the item using the finally optimized global user representation and global item representation, and recommending the item with the highest score to each user.

3. The method for multimodal recommendation according to claim 1, characterized in that: The methods for calculating the attention score between users and items include: Project item features of each modality into a unified embedding space middle: in, is the projection feature under mode m, d is the embedding dimension, and Represent the transformation matrix and bias vector respectively, d m represents the dimension of the feature; σ1(·) is the LeakyReLU activation function; is the original modal feature of the item; Use the inner product operation combined with the node degree to calculate the similarity between user u and item i under mode m Where d(·) represents the degree of the node in the interaction graph G; is a multimodal user feature; T represents transposition; Use the softmax function to get the attention score of user u on item i Among them, N u represents the items interacted with by user u, and exp(·) is an exponential operation.

4. The method for multimodal recommendation according to claim 1, characterized in that: Methods for pruning interaction edges based on attention score differences include: According to the attention score, determine whether to retain or prune the interaction edges between entities. The pruning rules are as follows: If the normalized attention score is lower than the preset threshold, the interaction edge between the entities is pruned; If the normalized attention score is higher than or equal to the preset threshold, the interaction edge between the entities is retained.

5. The method for multimodal recommendation according to claim 1, wherein: Methods for obtaining high-level ID embeddings of users and items through information propagation include: Using the LightGCN network in the interaction graph G and interaction matrix G m Message propagation is performed in , and the weight transformation matrix and activation function are deleted; During the convolution calculation at the lth layer, the embeddings of users and items are updated through matrices: Among them, U represents the set of users, I represents the set of items, represents the diagonal matrix of M, which is a symmetric adjacency matrix derived from the interaction graph G; the diagonal elements D aa Equal to ∑ b M ab ; Embedding matrix of users and items in layer l Update from layer l-1, where the initial matrix of layer 0 is set to: in, represents the identity id embedding vector specific to user u1, represents the identity id embedding vector specific to item i1; The ID embeddings at each layer are integrated using the summation where: in, is the final high-order id embedding of users and items, L in Represents the number of convolutional layers on the interaction graph G.

6. The method for multimodal recommendation according to claim 5, characterized in that: Methods for obtaining corresponding high-order modal features through information propagation include: The following convolution operation is performed on the purified interaction graph: Where D is the degree matrix, M is the enhanced adjacency matrix, and M m By G m Export, the initial feature matrix of layer 0 is set to: The final high-order modal characteristics The calculation is as follows:

7. The method for multimodal recommendation according to claim 6, characterized in that: Methods for obtaining enhanced representations between users include: The user co-occurrence matrix is ​​constructed as follows: S=G·G T ; Where T represents transpose, The elements in represent the number of co-occurrences between users; A top-n strategy is used to retain users with a certain number of co-occurrences: Among them, S a,. represents the element of the ath row in matrix S, where S is the constructed user co-occurrence matrix. is the matrix transformed by the above rules; according to the transformed matrix Get the user co-occurrence graph A = {U, C}; The set of edges representing co-occurrences between users; Embed using the user's high-level ID and modal characteristics Perform graph convolution on the user co-occurrence graph, and the calculation formula is: in, yes The diagonal matrix, N aa =n u ; Use L af The output of the -c layer is used as the user's enhanced representation: in and They represent user-enhanced id embedding and modality features respectively; Methods for getting an enhanced representation of an item include: Under mode m, calculate i a and i b Similarity between in, is a fully connected matrix, cos(·,·) is the cosine function, and ||·|| is the length of the vector; and Represent the modal embeddings of items a and b respectively; Adopt the top-n strategy as follows: in is a sparse matrix, and the item modal perception map is A m = {I,C m },in is the edge set representing the semantic relationship between items; Using higher order modal features as follows: in yes The diagonal matrix of For ID embedding Perform the above operation in the same way to obtain Therefore, the enhancement of items is expressed as follows: in and Represent the enhanced ID embedding and modal features of the items, α m is the importance score of the item feature under mode m, α is a hyperparameter, α t =(1-α).

8. The method for multimodal recommendation according to claim 7, characterized in that: The method of weighting and summing the obtained high-order ID embeddings of users and items with the high-order modal features, the obtained enhanced representations of users and the enhanced representations of items includes: The operation of embedding the user ID is described as follows: in, is the final id embedding of the user; by performing the above operation on all ID embeddings and modal features of users and items, we get and Use the attention weight β to control the fusion of user nodes, and use the importance score α to control the fusion of item nodes: in, and Represent the global representation of users and items respectively, represents weighted sum; β is a trainable parameter, α is a defined hyperparameter, v represents vision, and t represents text; Use the resulting enhanced representation to perform a recommendation system with a preference score y ui , where the preference score y ui By H u and H I The inner product of is calculated as: y ui =(h u ) T h i Methods for calculating total contrast loss include: InfoNCE is used to maximize the similarity between the same nodes under different representations and minimize the similarity between nodes of different users or items. The contrast loss L of user nodes CL,U The construction is as follows: Among them, U′ represents a group of users extracted from a batch of training data, and τ represents the temperature coefficient; the contrast loss of the item node is obtained in the same way Add the above results under all modes to get the total contrast loss L CL as follows: The method for calculating the final loss function L includes: The Bayesian personalized ranking loss is used to make the score of positive pairs higher than that of negative pairs; the loss function L BPR It is expressed as: in, is a set of triples, item i b is randomly sampled; the function σ2(·) is the Sigmoid activation function; The total contrast loss L CL And the loss function L BPR The final loss function L is merged as follows: Among them, λ1 and λ2 are hyperparameters for measuring contrast loss and L2 regularization, respectively, and Θ is a model parameter.

9. An electronic device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a readable program; The at least one processor is used to call the readable program to execute the method according to any one of claims 1-8.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 8.

Images