A method, electronic device and readable medium for multi-modal recommendation

Abstract

The present application relates to a kind of method for multimodal recommendation, electronic equipment and readable medium, the method includes: on interactive graph, designed graph purification strategy, to eliminate the noise edge caused by uneven user attention. Then three GCNs are applied in parallel to process multimodal features and ID embedding respectively, and obtain purified high-order representation of user and item. For the second restriction, message passing is performed on the frozen user-user and item-item affinity graph to aggregate features between similar nodes. Then, the learning results of the interaction graph and the affinity graph are summed. Subsequently, by introducing a contrastive learning task, the mutual information between the ID embedding and the modal features is maximized using the inherent self-supervised signal of the multimodal attributes, thereby achieving self-supervised learning without data augmentation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method, electronic device, and readable medium for multimodal recommendation. Background Art

[0002] With the rapid development of internet technology and the advent of the information age, data volumes have increased dramatically. Recommender systems, as an effective information filtering tool, have been widely used in various contexts, including e-commerce, social networks, and content distribution platforms. Traditional recommendation systems typically rely on historical interactions between users and items, analyzing this historical interaction data to predict users' potential preferences. However, such systems often face data sparsity issues, especially when new users or items are introduced.

[0003] To address this issue, researchers have begun exploring the use of multimodal data (such as images, text, and audio) to enrich the information sources of recommendation systems. This multimodal data can provide more comprehensive item descriptions, helping the system to more accurately capture the multiple dimensions of user preferences. Although multimodal recommender systems (MMRS) have improved traditional recommendation technologies to a certain extent, they also face some new challenges and limitations.

[0004] A major challenge is how to effectively integrate and utilize information from different modalities. Early multimodal recommendation methods typically adopted simple linear fusion or weighted fusion strategies based on attention mechanisms, but these methods failed to fully exploit the potential correlations between modalities. With the successful application of graph neural networks (GNNs) in complex systems, researchers introduced GNNs to process multimodal data, leveraging their powerful graph-structured data processing capabilities to extract high-level features of users and items, thereby improving recommendation accuracy.

[0005] Furthermore, self-supervised learning (SSL), an emerging learning paradigm, has demonstrated strong performance in fields such as computer vision and natural language processing by constructing and leveraging supervisory signals within the data itself. In recommender systems, SSL can effectively improve the generalization and robustness of the model by generating multiple views of the data through techniques such as data augmentation and contrastive learning, maximizing the consistency between positive samples and minimizing the inconsistency between negative samples.

[0006] However, existing multimodal recommendation methods still have shortcomings in handling modality-specific noise and leveraging self-supervisory signals. On the one hand, fixed interaction graph models may introduce noisy edges caused by uneven user attention. This noise is continuously amplified during the graph convolution process, affecting the accuracy of learning results. On the other hand, existing self-supervised learning strategies for multimodal recommendation often rely on random data augmentation, which may introduce bias and affect the final recommendation performance. Summary of the Invention

[0007] The present invention provides a method, electronic device and readable medium for multimodal recommendation, which aims to address the deficiencies of existing multimodal recommendation systems in processing modality-specific noise and effectively utilizing self-supervisory signals.

[0008] To achieve the above objectives, the present invention provides a method for multimodal recommendation in a first aspect, comprising the following steps:

[0009] Generate initial ID embeddings for each user and item, extract the original modal features of the items through the pre-trained model, and project the original modal features into a unified modal embedding space;

[0010] Using the initial ID embedding and the projected modality embedding, we calculate the attention score between users and items based on the difference in users' attention to different modalities, and prune the interaction edges based on the attention score difference to generate a purified modality-specific interaction graph.

[0011] Applying a graph convolutional network to the purified modality-specific interaction graph and the observed original interaction graph, we obtain high-order ID embeddings of users and items and corresponding high-order modality features through information propagation.

[0012] Construct a user co-occurrence graph and select users sharing similar features through a top-k strategy to obtain enhanced representations between users. Use similarity metrics to construct an item modality perception graph and connect semantically similar items through a top-k strategy to obtain enhanced representations of items.

[0013] The obtained high-level ID embeddings of users and items are weighted and summed with the high-level modal features, the enhanced representations between users, and the enhanced representations of items to form the final global user representation and global item representation.

[0014] Based on the final global user representation and global item representation, the inner product operation is used to calculate the preference score between the user and the item, and the item with the highest score is recommended to the user.

[0015] Furthermore, the method also includes: introducing a contrastive learning task, calculating the contrastive loss by maximizing the similarity between the same nodes under the global representation, and optimizing the model parameters in combination with the Bayesian personalized ranking loss, and then using the inner product operation to calculate the preference score between the user and the item using the final optimized global user representation and global item representation, and recommending the item with the highest score to each user.

[0016] Furthermore, the method for calculating the attention score between the user and the item includes:

[0017] Project item features of each modality into a unified embedding space middle:

[0018]

[0019] in, is the projection feature under mode m, d is the embedding dimension, and Denote the transformation matrix and bias vector respectively, d m represents the dimension of the feature; σ1(·) is the LeakyReLU activation function; is the original modal feature of the item;

[0020] Use the inner product operation combined with the node degree to calculate the similarity between user u and item i under mode m

[0021] Where d(·) represents the degree of the node in the interaction graph G; is a multimodal user feature; T represents transposition; the softmax function is used to obtain the attention score of user u on item i

[0022]

[0023] Among them, N u represents the items that user u interacts with, and exp(·) is the exponential operation.

[0024] Further methods for pruning interaction edges based on attention score differences include:

[0025] Based on the attention score, determine whether to retain or prune the interaction edges between entities. The pruning rules are as follows:

[0026] If the normalized attention score is lower than the preset threshold, the interaction edge between the entities is pruned;

[0027] If the normalized attention score is higher than or equal to the preset threshold, the interaction edge between the entities is retained.

[0028] Further, methods for obtaining high-level ID embeddings of users and items through information propagation include:

[0029] Using LightGCN network in interaction graph G and interaction matrix G m Perform message propagation in , delete the weight transformation matrix and activation function;

[0030] During the convolution calculation of the lth layer, the embedding of users and items is updated through the matrix:

[0031]

[0032] Among them, U represents the set of users, I represents the set of items, represents the diagonal matrix of M, which is a symmetric adjacency matrix derived from the interaction graph G; the diagonal elements D aa Equal to ∑ b M ab ;

[0033] Embedding matrix of users and items in layer l Update from layer l-1, where the initial matrix of layer 0 is set to:

[0034]

[0035] in, represents the identity id embedding vector specific to user u1, represents the identity id embedding vector specific to item i1;

[0036] The ID embeddings at each layer are integrated using a summation where:

[0037]

[0038] in, is the final high-order id embedding of users and items, L in Represents the number of convolutional layers on the interaction graph G.

[0039] Furthermore, methods for obtaining corresponding high-order modal features through information propagation include:

[0040] The following convolution operation is performed on the purified interaction graph:

[0041]

[0042] Where D is the degree matrix, M is the enhanced adjacency matrix, and M m By G m Export, the initial feature matrix of layer 0 is set to:

[0043]

[0044] The final high-order modal characteristics The calculation is as follows:

[0045]

[0046] Furthermore, the method for obtaining the enhanced representation between users includes:

[0047] Construct the user co-occurrence matrix as follows:

[0048] S=G·G T ;

[0049] Where T represents transpose, The elements in represent the number of co-occurrences between users;

[0050] A top-n strategy is used to retain users with a certain number of co-occurrences:

[0051]

[0052] Among them, S a,. Represents the element of row a in matrix S; S is the constructed user co-occurrence matrix, It is the matrix transformed by the above rules. According to the matrix Get the user co-occurrence graph A = {U, C}; The set of edges representing co-occurrences between users;

[0053] Embed using the user's high-level ID and modal features Perform graph convolution on the user co-occurrence graph, and the calculation formula is:

[0054]

[0055] in, yes The diagonal matrix, N aa =n u ;

[0056] Use L af -The output of the c layer is used as the enhanced representation of the user:

[0057]

[0058] in and represent user-enhanced id embedding and modality features respectively;

[0059] Methods for obtaining an enhanced representation of an item include:

[0060] Under mode m, calculate i a and i b The similarity between

[0061]

[0062] in, is a fully connected matrix, cos(·,·) is the cosine function, and ||·|| is the length of the vector; and Represent the modal embeddings of items a and b respectively;

[0063] Use the top-n strategy as follows:

[0064]

[0065] in is a sparse matrix, and the item modal perception map is A m ={I,C m},in is the edge set representing the semantic relationship between items;

[0066] Using higher-order modal features as follows:

[0067]

[0068] in yes The diagonal matrix of For ID embedding Perform the above operations in the same way to obtain Therefore, item enhancements are expressed as follows:

[0069]

[0070] in and Represent the enhanced id embedding and modal features of the items, α m is the importance score of the item feature under modality m, α is a hyperparameter, t =(1-α).

[0071] Furthermore, a method for weighted summing the obtained high-order ID embeddings of users and items with high-order modal features, the obtained enhanced representations of users and items includes:

[0072] The operation of embedding the user ID is described as follows:

[0073]

[0074] in, is the final id embedding of the user; by performing the above operation on all ID embeddings and modal features of users and items, we get and

[0075] Use the attention weight β to control the fusion of user nodes, and use the importance score α to control the fusion of item nodes:

[0076]

[0077] in, and denote the global representation of users and items respectively, represents weighted sum; β is a trainable parameter, α is a defined hyperparameter, v represents vision, and t represents text;

[0078] Use the resulting enhanced representation to generate a preference score y for the recommendation system ui , where the preference score y ui By H u and H I The inner product of is calculated as:

[0079] y ui =(h u ) T h i

[0080] Methods for calculating total contrast loss include:

[0081] InfoNCE is used to maximize the similarity between the same nodes under different representations and minimize the similarity between nodes of different users or items. The contrast loss L of user nodes CL,U The construction is as follows:

[0082]

[0083] Among them, U′ represents a group of users extracted from a batch of training data, and τ represents the temperature coefficient; the contrast loss of the item node is obtained in the same way Add the above results under all modes to get the total contrast loss L CL as follows:

[0084]

[0085] The method for calculating the final loss function L includes:

[0086] The Bayesian personalized ranking loss is used to make the score of positive pairs higher than that of negative pairs; the loss function L BPR Expressed as:

[0087]

[0088] in, is a set of triples, item i b is randomly sampled; the function σ2(·) is the Sigmoid activation function;

[0089] The total contrast loss L CL And the loss function L BPR The final loss function L is merged as follows:

[0090]

[0091] Among them, λ1 and λ2 are hyperparameters for measuring contrast loss and L2 regularization, respectively, and Θ is a model parameter.

[0092] To achieve the above object, the second aspect of the present invention provides an electronic device, comprising: at least one memory and at least one processor;

[0093] The at least one memory is used to store a readable program;

[0094] The at least one processor is configured to call the readable program and execute the method.

[0095] To achieve the above-mentioned object, the third aspect of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method described above.

[0096] Beneficial effects of the present invention:

[0097] Compared to existing technologies, the present invention provides a method, electronic device, and readable medium for multimodal recommendation that effectively address the shortcomings of modality-specific noise and self-supervisory signal utilization. First, initial ID embeddings are generated and the original modal features of items are extracted and uniformly projected into the modal embedding space, achieving effective alignment between modalities. Next, attention scores are calculated based on the differences in user attention across modalities. Interaction edges are pruned based on these attention scores to generate a cleansed modality-specific interaction graph, significantly reducing the impact of noise. A graph convolutional network is then used to process the cleansed interaction graph and the original interaction graph for information propagation, extracting high-order embedding features of users and items. Furthermore, by constructing a user co-occurrence graph and an item modal perception graph, a top-k strategy is used to select users with similar features and semantically similar items, enhancing the representation of users and items. Finally, a weighted summation of high-order ID embeddings, high-order modal features, enhanced user representations, and enhanced item representations is performed to form a final global user and item representation. Based on this representation, preference scores are calculated using an inner product operation to recommend the highest-scoring items to the user. This process not only improves the accuracy and reliability of recommendations, but also effectively utilizes self-supervision signals and avoids bias problems that may be introduced in data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0099] Figure 1 This is a flowchart of a method for multimodal recommendation disclosed in an embodiment of the present invention.

[0100] Figure 2This is an overview diagram of a PEARL framework disclosed in an embodiment of the present invention.

[0101] Figure 3 This is a performance comparison diagram of merging different modes on three data sets disclosed in an embodiment of the present invention.

[0102] Figure 4 This is a model performance graph disclosed in an embodiment of the present invention, with pruning percentages ranging from 0.1 to 1.0 on three data sets.

[0103] Figure 5 This is a model performance graph of a user graph and an item graph under different top-n values ​​on three data sets disclosed in an embodiment of the present invention.

[0104] Figure 6 This is a model performance diagram of a contrast loss weight and a regularization coefficient under [0.1, 0.01, 0.001, 0.0001] disclosed in an embodiment of the present invention.

[0105] Figure 7 This is a distribution diagram of item representations learned by different models on an infant dataset disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0106] like Figure 1 As shown, the present invention provides a method for multimodal recommendation, which includes the following steps:

[0107] Step S100: Generate initial ID embeddings for each user and item, extract the original modal features of the items through the pre-trained model, and project the original modal features into a unified modal embedding space;

[0108] For each user and item, generate an initial identifier (ID) embedding. This step creates a basic embedding vector for the user and item. The embedding is randomly initialized and the dimension is determined by the system design.

[0109] Use pre-trained models (for example, visual models such as VGG or text models such as BERT) to extract modal features for each item. Pre-trained models can extract rich feature expressions from different modalities of items (such as images, text, etc.), providing an additional layer of information for the recommendation system.

[0110] The extracted raw modal features are mapped to a unified embedding space through a series of transformations (linear transformations, such as matrix multiplication plus bias, accompanied by nonlinear activation functions such as ReLU or LeakyReLU). The purpose of this step is to unify modal features from different sources into a common feature space for effective comparison and fusion.

[0111] Through these steps, a comprehensive representation is built for each user and item, including both identifier-based embeddings and rich information extracted through modal features. This comprehensive representation provides the necessary input for subsequent recommendation systems, enabling more accurate and personalized recommendations. In subsequent steps of the present invention, these features will be used to construct an interaction graph and perform graph convolution to mine and exploit the complex relationships and interactions between users and items.

[0112] Step S200: Using the initial ID embedding and the projected modality embedding, the attention score between the user and the item is calculated based on the difference in the user's attention to different modalities, and the interaction edges are pruned based on the attention score difference to generate a purified modality-specific interaction graph;

[0113] First, the initial ID embedding and the unified modality embedding generated in step S100 are combined to construct a comprehensive feature representation for each user and item. These features include the identifier embedding based on the user and item as well as the representation of their multimodal content.

[0114] Based on these comprehensive features, the attention mechanism calculates the user's attention to each item. This process relies on the calculation of attention scores, which reflect the differences in the user's attention to the different modal features of the item. Attention scores can be calculated using methods such as the inner product followed by the softmax function to determine the user's interest in each item.

[0115] Based on the differences in the calculated attention scores, interaction edges are pruned. This step aims to remove misleading interactions that may be caused by noise or inconsistent user attention. Specifically, a threshold can be set or score differences can be used to determine which edges should be retained or pruned. The pruning probability is based on the relative magnitude of the attention scores and the user's behavior patterns, with lower-scoring edges being more likely to be pruned.

[0116] After pruning, a purified modality-specific interaction graph is constructed, which more accurately reflects users' true preferences and behaviors. This purification process helps reduce the impact of noise in subsequent graph convolutional network learning, improving the accuracy and efficiency of the recommendation system.

[0117] It's easy to see that this meticulous attention score calculation and score-based edge pruning effectively optimizes the interaction graph, focusing it more closely on users' actual interests and behaviors. This not only improves the accuracy of the recommendation system's information processing but also lays a solid foundation for deeply exploring the complex relationships between users and items. Furthermore, by removing low-relevance interactions, it enables more efficient processing of large-scale data, improving computational efficiency and recommendation quality.

[0118] Step S300: Apply a graph convolutional network to the purified modality-specific interaction graph and the observed original interaction graph to obtain high-order ID embeddings of users and items and corresponding high-order modal features through information propagation;

[0119] In this step, graph convolution operations are performed on two interaction graphs: the purified modality-specific interaction graph and the observed original interaction graph. Graph Convolutional Networks (GCNs) can capture high-level relationships between users and items by transferring and aggregating node information within the graph, thereby extracting more complex and in-depth feature representations.

[0120] During the graph convolution process, the initial ID embedding and modality embedding are further updated and optimized. Through multi-layer information transfer and feature aggregation, the Graph Convolutional Network (GCN) helps generate more expressive high-level ID embeddings and modality features. These high-level features contain more contextual information and potential connections between users and items, thereby providing a more accurate and comprehensive description of users and items.

[0121] At each layer of a graph convolutional network (GCN), nodes are updated based on the features of their neighboring nodes. Through a weight matrix and nonlinear activation functions like ReLU, each iteration "blends" the neighboring information into the current node's features. This neighbor-based information update gradually propagates and enriches the feature representation of each node.

[0122] Through repeated graph convolution operations, features from different layers are extracted and merged from the original and purified interaction graphs to form the final high-level user and item embeddings. These high-level embeddings can reflect the multi-dimensional relationships and interaction patterns between users and items, providing key information for subsequent recommendation decisions.

[0123] Step S400: Construct a user co-occurrence graph, select users sharing similar features through a top-k strategy, and thus obtain an enhanced representation between users; construct an item modality perception graph using a similarity metric, and connect semantically similar items through a top-k strategy to obtain an enhanced representation of items;

[0124] Constructing a user co-occurrence graph: The user co-occurrence graph is constructed based on the co-occurrence relationship between users. The user co-occurrence relationship is defined based on the number of items that users interact with in common, that is, if two users have interacted with the same or similar set of items, then the two users will be connected in the co-occurrence graph. A Top-k strategy is used to select the k other users that are most similar to each user. This selection is based on the frequency of co-occurrence between users, that is, the number of times users have selected the same item in common, and the user with the highest frequency is selected as the neighbor of each node. Applying a graph convolutional network on the user co-occurrence graph can further refine and enhance the user representation. This is achieved by aggregating the features of each user node with the features of its neighbors (co-occurring users), thereby enhancing the collaborative information in the user representation.

[0125] Constructing an item modal perception graph: The item modal perception graph is constructed based on the modal features of the items, such as text descriptions and images. This graph uses the similarity between modal features (using metrics such as cosine similarity) to connect semantically similar items. Similarly, connecting each item to its k most similar items through a Top-k strategy can help reduce noisy and irrelevant connections, ensuring that high-quality, semantically relevant links are retained in the graph. Performing graph convolution operations on the item modal perception graph allows items to obtain information from other semantically similar items, thereby enhancing the modal feature representation of the items.

[0126] After performing graph convolution on the user co-occurrence graph and item modality perception graph, the representations of users and items contain rich information from their similar entities, allowing the enhanced representations to capture deeper relationships and patterns between users and items. By aggregating the outputs of the graph convolutional network (such as through weighted summation), the final global representations of users and items are formed. These representations not only reflect the individual characteristics of users and items, but also incorporate their relative position and contextual relationships in the network.

[0127] It's important to note that the Top-k strategy is a selection method used to select the k most relevant items from a set of candidate items in a recommendation system. Specifically, the Top-k strategy ranks candidate users or items based on a similarity metric (such as co-occurrence frequency or cosine similarity) and then selects the k users or items with the highest similarity. By focusing on the most relevant neighbors, it effectively reduces noise and irrelevant information, improving the accuracy and efficiency of the subsequent recommendation process.

[0128] Step S500: The acquired high-level ID embeddings of users and items are weightedly summed with the high-level modal features, the obtained enhanced representations of users and items, and the final global user representation and global item representation are formed.

[0129] The high-order ID embeddings of users and items obtained from the interaction graph learning module are fused with the corresponding high-order modality features. The fusion process involves weighted summation, where the weights are adjusted according to the importance of different features.

[0130] Fusion of enhanced representations between users and items: In addition to ID embeddings and modality features, enhanced representations obtained from user co-occurrence graphs and item modality perception graphs are also fused. These enhanced representations reflect the relative positions and similarities of users and items in their corresponding groups.

[0131] Comprehensive fusion: By combining high-level features with enhanced features, we form final global user and item representations. These global representations integrate information from different data sources and analysis dimensions, providing a comprehensive overview of the user or item. During the fusion process, additional techniques are applied to optimize the quality of the representation, such as regularization, standardization, or the use of more complex fusion mechanisms (such as feature fusion layers in deep learning models).

[0132] Step S600: Based on the final global user representation and global item representation, the preference scores between the user and the item are calculated using an inner product operation;

[0133] Step S700: Recommend the item with the highest score to the user based on the preference score data.

[0134] This is achieved by performing a dot product between the user's global representation and the item's global representation, where the global representation includes a fused high-order ID embedding and modal features. The resulting score reflects the user's potential preference for each item, with higher scores indicating greater interest in the item. Based on the calculated preference scores, all recommended items are ranked. For example, a list of each user and their potentially interested items is sorted by preference score. Based on the ranking results, the highest-scoring item is recommended to each user. In this embodiment, the top-N items are selected for recommendation, where N is set based on system design or user preference.

[0135] In practice, recommendation algorithms are adjusted based on user feedback or system monitoring results to optimize recommendation accuracy and user satisfaction. Besides preference scores, factors such as item diversity and novelty can also be considered when making recommendations to increase the appeal of the recommendation list and encourage user exploration.

[0136] To better understand the technical solution of the present invention, we first describe the task of multimodal recommendation and graph learning (PEARL) and then introduce each component in detail.

[0137] The ID embeddings of each user u∈U and item i∈I are initialized as Where d is the embedding dimension. The original modal features of the items extracted by the pre-trained model are where m∈M, d m In addition, the multimodal user features are randomly initialized In the embodiment, only visual and textual modes are considered, that is, M = {v, t}. The historical interaction between users and items is represented as a sparse matrix The observed interaction graph G = {V, E} is constructed using this sparse matrix. Among them, V = {U∪I} is a node set representing users and items, and E = {<u,i> |G u,i =1} represents the edge set of interactions between users and items. Under the above setting, the multimodal recommendation task is to accurately predict the likelihood of user u selecting item i by leveraging representation learning of user behavior and multimodal features of items.

[0138] like Figure 2 As shown in Figure 2, graph learning (PEARL) consists of four parts: (1) an initialization and graph purification module, which extracts multimodal features using a pre-trained model while randomly initializing embeddings, and performs edge denoising according to the proposed graph purification strategy; (2) an interaction graph learning module, which applies GCNs to modality-specific interaction graphs and observed interaction graphs to capture high-order modal features and ID embeddings, respectively; (3) an affinity graph learning module, which aggregates similar node information on frozen user-user and item-item affinity graphs to obtain enhanced representations of users and items; (4) a fusion and recommendation module, which fuses high-order features and enhanced features to obtain a global representation, optimizes it through contrastive learning, and recommends items to each user based on their preference scores.

[0139] After observing the interaction graph, we purify it and generate a specific interaction graph for each modality to prevent the spread of noisy signals caused by uneven user attention in graph learning. Specifically, we design a purification strategy based on the modal differences in users' attention to items.

[0140] Taking into account the differences in dimensions and semantics of item features generated by pre-trained models for different modalities, the item features of each modality are projected into a unified embedding space. As shown below:

[0141]

[0142] in, is the projection feature under mode m, and Denote the transformation matrix and bias vector respectively. Function σ1(·) is the LeakyRelu activation function. Then, the inner product operation is combined with the node degree to calculate the similarity between user u and item i in modality m.

[0143] where d(·) represents the degree of the node in the graph G, is the multimodal user feature; T represents the transposition. Then, the softmax function is used to obtain the attention score of user u on item i

[0144]

[0145] Among them, N u represents the items that user u interacts with, and exp(·) is an exponential operation. Based on the difference in attention scores under different modalities, a pruning probability is defined This probability determines an edge in the graph G<u,i> Whether it is pruned, so as to obtain the purified interaction graph G under mode m m :

[0146]

[0147] Where σ2(·) is the normalized Sigmoid activation function. The probability of pruning all edges in the purified interaction graph under all modalities forms a probability distribution p={p1,p2,…,p |M||E|-1}.

[0148] Typically, a certain percentage of edges are pruned to ensure that each edge in E appears at least once in the purified interaction graph of all modalities. In other words, each user is influenced by at least one modality. Therefore, the pruned edges are pruned according to the probability distribution p. edge, where In this way, the same interactions with lower attention scores in different modalities are more likely to be pruned. The purified interaction graph is G m ={V,E m},in The purified interaction matrix under mode m is G m .

[0149] In order to learn high-level representations of users and items under the observation and purification of interaction graphs, the Energized Graph Convolutional Network (LightGCN) is used to build the m LightGCN is a lightweight graph convolutional network that removes the weight transformation matrix and activation function. The convolution operation on the lth layer of the graph G is as follows:

[0150]

[0151] in Denotes the diagonal matrix of M, which is a symmetric adjacency matrix derived from G. The diagonal elements D aa Equal to ∑b M ab The embedding matrix of users and items in the lth layer Update only from layer l-1, where the initial matrix of layer 0 is set to:

[0152]

[0153] Given that there are no self-loops in the interaction graph, the information from the previous layer of the same node will not be retained during message propagation. Therefore, the ID embeddings of each layer are integrated using summation as follows:

[0154]

[0155] in is the final high-order id embedding of users and items, L in Represents the number of convolutional layers on the graph G.

[0156] Similarly, in the purified interaction graph G m The following convolution operation is performed on it:

[0157]

[0158] Among them, M m By G m Export, the initial feature matrix of layer 0 is set to:

[0159]

[0160] The final high-order modal characteristics The calculation is as follows:

[0161]

[0162] Through the above-mentioned interaction graph learning, collaborative signals and multimodal information are encoded into high-level representations of users and items.

[0163] To further mine the hidden information of implicit intra-modality affinities between similar users or items, graph learning is performed on the pre-built and frozen user co-occurrence graph and item modality perception graph.

[0164] User co-occurrence graph. Inspired by Wang et al., it is believed that there is a co-occurrence relationship between users who interact with the same item. Therefore, a user co-occurrence graph is constructed to learn the similar preferences between co-occurring users. First, the user co-occurrence matrix is ​​constructed as follows:

[0165] S=G·G T ;

[0166] in, The elements in represent the number of co-occurrences between users. For each user u, a small number of users co-occur with him many times, while most users only co-occur a few times. Therefore, a top-n strategy is adopted to retain users with a certain number of co-occurrences, as shown below:

[0167]

[0168] Among them, S a,. Represents the element in row a of matrix S, where S is the constructed user co-occurrence matrix. is the matrix after transformation according to the above rules. Therefore, according to the matrix Get the user co-occurrence graph A = {U, C}, where Represents the co-occurrence edge set between users. Then, the user’s high-order ID is used to embed and modal features Perform graph convolution on image A, taking the former as an example:

[0169]

[0170] in yes The diagonal matrix, N aa =n u Unlike the interaction graph, each node in the affinity graph has a self-loop constructed by the top-n strategy. Therefore, the result of each convolution will retain the information of the previous layer, and directly use L af The output of the -c layer is used as the enhanced representation of the user:

[0171]

[0172] in, and represent user enhanced id embedding and modality features respectively.

[0173] Item modal perception map. In order to learn the implicit semantic information between item features, based on the original modal features The project modality perception map is constructed. Specifically, under modality m, calculate i a and i b The similarity between

[0174]

[0175] in, is a fully connected matrix, cos(·,·) is the cosine function, and ||·|| is the length of the vector. To reduce computing resources and avoid noise caused by unnecessary connections, the top-n strategy is also adopted, as shown below:

[0176]

[0177] in, is a sparse matrix. The modal perception map of the item is A m ={I,C m},in is a set of edges that represent semantic relationships between items.

[0178] Using higher-order modal features as follows:

[0179]

[0180] in, yes The diagonal matrix of For ID embedding Perform the above operations in the same way to obtain Therefore, the item enhancement is expressed as follows:

[0181]

[0182] in, and denote the enhanced id embedding and modality features of the items, α m is the importance score of the item feature under modality m. In this work, a hyperparameter α is introduced as the importance of the visual modality, and α t =(1-α).

[0183] After learning high-level and enhanced representations of users and items from the interaction and affinity graphs, a summation is performed to effectively capture the interaction information and its implicit intra-modality affinity. Specifically, the operation on user ID embedding is formulated as follows:

[0184]

[0185] in, is the final id embedding of the user. By performing the above operations on all the ID embeddings and modality features of users and items, we can get and

[0186] Different from traditional self-supervised learning, this paper develops a contrastive learning paradigm without data augmentation. The design is based on the existence of sufficient self-supervisory signals between collaborative signals and multimodal information. Specifically, the InfoNCE loss is used to maximize the similarity between the same nodes under different representations and minimize the similarity between nodes of different users or projects. The contrastive loss L of user nodes is CL,U The construction is as follows:

[0187]

[0188] Where U′ represents a group of users extracted from a batch of training data, and τ represents the temperature coefficient. Similarly, the contrast loss of the item node can be obtained Add the above results under all modes to get the total contrast loss L CL as follows:

[0189]

[0190] To obtain a global representation of users and items, a concatenation operation is used to fuse ID embeddings and modality features. Specifically, the attention weight β is used to control the fusion of user nodes, and the importance score α is used to control the fusion of item nodes.

[0191]

[0192] in, and denote the global representation of users and items respectively, is the connection operation, β is a trainable parameter, α is a hyperparameter defined previously, v represents vision, and t represents text.

[0193] According to the previous recommendation, the inner product operation is used to calculate the preference score y between the user and the product. ui , as shown below:

[0194] y ui =(h u ) T h i ;

[0195] Where item i has no interaction with user u. For each user, recommend items with top-k preference scores.

[0196] In order to further optimize the parameters of PEARL, the Bayesian Personalized Ranking (BPR) loss is used to make the score of positive pairs higher than that of negative pairs. bpr Expressed as:

[0197]

[0198] in is a set of triples, item i b is randomly sampled. Function σ2(·) is the Sigmoid activation function.

[0199] In summary, the contrast loss L CL and BPR loss L BPR The final loss L is combined as follows:

[0200]

[0201] Among them, λ1 and λ2 are hyperparameters for measuring contrast loss and L2 regularization, respectively, and Θ is a model parameter.

[0202] To verify the effectiveness and superiority of the proposed PEARL, experiments were conducted on three real-world datasets to address the following research questions:

[0203] 1. Can PEARL achieve state-of-the-art performance on multimodal recommendation tasks compared to baselines?

[0204] 2. Does each component used in PEARL contribute to improving recommendation performance?

[0205] 3. What impact does the addition of multimodality have on the performance of PEARL?

[0206] 4. What is the impact of hyperparameter perturbations on PEARL performance?

[0207] 5. Why can the multimodal learning framework achieve better recommendation performance?

[0208] Experiments were conducted on three publicly available Amazon datasets: (a) Baby, (b) Sports and Outdoors, and (c) Clothing, Shoes, and Jewelry (hereafter referred to as Baby, Sports, and Clothing). These datasets contain both visual and textual information about items. A 5-core setup was used on the original data to ensure at least five interactions between each user and item. The processing results are shown in Table 1.

[0209] Table 1 Statistics of experimental datasets

[0210]

[0211] We directly use the 4096-dimensional visual features and 384-dimensional textual features previously published in previous work, extracted by the pre-trained VGG16 model and the sentence transformer Sentence2Vec, respectively. For each dataset, we randomly split the historical interactions into training, validation, and test sets in an 8:1:1 ratio. Furthermore, we use the commonly used metrics Recall@K (denoted as R@K) and NDCG@K (denoted as N@K) to evaluate the model's recommendation performance. Specifically, we select K∈{10,20} and report the average results across all users in the test set.

[0212] To verify the superiority of PEARL, it is compared with various recommendation models, which are divided into the following three groups:

[0213] a) Traditional Model

[0214] BPR: is a matrix factorization (MF)-based method that performs personalized ranking by comparing item pairs for each user and uses Bayesian methods for optimization.

[0215] LightGCN: Simplifies the vanilla graph convolutional network by removing layers responsible for nonlinear activation and feature transformation.

[0216] b) Recommendation via Self-Supervised Learning

[0217] SGL: Enhancing graph collaborative filtering by maximizing the mutual information among three views of the constructed user-item interaction graph using contrastive learning.

[0218] NCL uses expectation-maximization clustering to identify semantically and structurally adjacent nodes and leverages these nodes to generate contrastive views that are considered positive pairs.

[0219] c) Multimodal Recommendation

[0220] VBPR extends the BPR method by adding visual features extracted from item images to reveal the visual dimensions that influence user preferences.

[0221] MMGCN: Performing information propagation on modality-specific user-item graphs to refine users' modality-aware representations.

[0222] GRCN: Identify false positive connections in interaction graphs and use soft denoising to reduce the impact of noise during graph convolution.

[0223] DualGNN: Introducing user-user association graph to enhance user representation by mining hidden information between similar users

[0224] LATTICE: Condensing item-item graphs under different modalities and mining latent semantic information of items through information propagation.

[0225] SLMRec: It uses data augmentation methods to construct positive and negative sample node pairs, and uses the self-supervisory signals therein for contrastive learning.

[0226] MGCL: Leveraging self-supervisory signals from different perspectives to optimize the model through contrastive learning, thereby enhancing the representation of users and items.

[0227] MGCN: Adaptive fusion denoising at the modality feature level using mutual information and learning the relative importance of different modalities.

[0228] LGMRec: Designing a global hypergraph to capture many-to-many dependencies between attributes and items.

[0229] To ensure fairness in experimental comparisons, a unified framework, MMRec, is used to implement the proposed models and baselines. The embedding dimensions of users and items in all models are set to 64, and the model parameters are initialized using the Xavier method. All experiments are implemented in Pytorch and executed on an NVIDIA RTX3090Ti GPU with 24GB of memory. The Adam optimizer is used to train the model with a batch size of 2048 and a learning rate of 0.001 for a maximum of 1000 epochs. In order to reduce the hyperparameter search space of the model, the number of GCN layers in interaction and affinity graph learning is empirically set to L, L, and L, respectively. in =2 and L af =1. In addition, the temperature coefficient τ in contrastive learning is set to 0.2. In addition, the optimal values ​​of other hyperparameters are determined by grid search. Specifically, the pruning ratio ρ and the importance score α are adjusted in {0.1, 0.2, 0.3, 0.4, 0.5}. The n values ​​of the user graph are searched in {10, 20, 30, 40, 50, 60} and {5, 10, 15, 20, 25, 30} respectively. u and n of the project graph i The weight λ1 of the contrastive loss and the regularization coefficient λ2 are both adjusted in {1e-01, 1e-02, …, 1e-05}. In particular, an early stopping strategy is used to stop training when the R@20 metric does not improve for 20 consecutive epochs on the validation set.

[0230] The overall effect is shown in Table 2:

[0231] Table 2 Overall performance of recall and NDCG of PEARL and baselines on three datasets

[0232]

[0233] To answer RQ1, we compare PEARL with the baseline performance on three datasets. Table 3 reports the overall performance of PEARL in terms of R@10, R@20, N@10, and N@20.

[0234] (1) For the baseline models, it is clear that introducing multimodal information can improve the recommendation performance. For example, on the Baby dataset, the introduction of visual modality improves R@20 by 15.30% from BPR to VBPR. Similarly, the application of self-supervised learning improves the performance of SGL by 26.46% compared to BPR. Except for MMGCN, the multimodal methods based on GCN outperform the traditional models. Specifically, MMGCN cannot distinguish the unique features of a single modality, resulting in insufficient utilization of multimodal information. The soft denoising strategy used in GRCN performs well on dense datasets, but performs poorly on sparse datasets (such as the Clothing dataset), which may be due to inaccurate denoising. DualGNN and LAT-TICE mine hidden semantic information through homogeneous graph learning to enhance the representation of users and items. In addition, it can be observed that recent studies (i.e., SLMRec, MGCL) adopt self-supervised learning to further improve the recommendation performance. In the benchmark, MGCN performs best on the Sports and Clothing datasets, while LGMRec surpasses other methods on the Baby dataset. The reason for this difference is that the feature-level denoising method in MGCN is more effective on large-scale datasets, while the hypergraph structure adopted by LGMRec is relatively complex and more suitable for small-scale datasets.

[0235] (2) Next, these baselines are compared with PEARL. Overall, PEARL significantly outperforms all baselines across different datasets. Specifically, it achieves performance improvements of 7.34%, 5.99%, and 8.08% on R@20, respectively, which are lower than the second-best model on the three datasets. The performance is attributed to the benefits of effectively denoising the interaction graph of specific modes and self-supervised signal learning mined by multi-graph. On the one hand, PEARL eliminates modal information that users may not perceive due to the following reasons: uneven user attention across different modes. This is difficult to denoise and directly cuts off the noise propagation in graph learning, thereby achieving better performance than soft-denoising GRCN and feature-level denoising MGCN. On the other hand, an affinity graph learning is designed that mines self-supervised signals more effectively than SLMRec without the need for data augmentation. In addition, by maximizing the mutual information between ID embeddings and modal features, PEARL performs better than MGCL. Unlike LGMRec, which adopts a complex hypergraph learning method, PEARL captures historical interaction information and internal relations by learning dual representations to achieve more effective multimodal enhancement. Moreover, the proportion of performance improvement achieved by PEARL increases with the data size, which indicates that PEARL has a greater advantage in more complex scenarios.

[0236] Table 3 is the ablation experiment, as shown in Table 3:

[0237] Table 3 Performance of Ablation Variants Removing Different Components in PEARL

[0238]

[0239] To answer RQ2 and RQ3, a detailed ablation study was conducted on the components and modal features of PEARL.

[0240] To explore the contribution of key components in PEARL to the recommendation performance, the following model variants were designed.

[0241] PEARL without GP, removing graph purification, i.e., not considering the influence of noisy modal information in interactive graph learning.

[0242] PEARL without AL, removing affinity graph learning, i.e., not aggregating the implicit intra-modal features between similar nodes.

[0243] PEARL without ALuser, which eliminates the learning of the user co-occurrence graph and only retains the item-mode-aware graph in affinity graph learning.

[0244] PEARL without ALitem, which cancels the learning of the item-mode-aware graph in affinity graph learning and only retains the user co-occurrence graph

[0245] PEARL without CL, eliminating contrastive learning, i.e., ignoring the self-supervised signal between ID embeddings and modal features.

[0246] Table 4 lists the experimental results of PEARL and the above model variants on three datasets, from which the following observations can be drawn:

[0247] (1) Comparing PEARL and PEARL w / o GP, it can be seen that removing the GP module leads to a performance drop. This indicates that constructing an accurate modality-specific user-item graph in interactive graph learning is crucial, and the designed graph purification strategy is effective. In particular, different from the soft denoising of GRCN, the hard purification strategy of the present invention shows relatively stable improvement in recommendation performance on datasets of different sizes.

[0248] (2) Utilizing affinity graph learning significantly improves the model performance, especially on the Clothing dataset, which indicates that datasets with a large number of users and items contain more hidden structures. In addition, w / o ALuser < w / o ALuser shows that item-mode-aware graph learning is the main reason for the performance improvement, because the user co-occurrence graph is constructed based on user-item interactions, and some user relationship information has been extracted through interactive graph learning. On the contrary, the item-mode-aware graph is constructed based on the original modal features, thus being able to capture the hidden semantic information between items, and then being limited to collaborative signals.

[0249] (3) Comparing PEARL and PEARL w / o CL, it is clear that CL (contrastive learning) effectively improves the recommendation performance, which reflects that self-supervised learning without data augmentation is feasible in the multimodal domain. ID embeddings and modality features can be further enhanced by maximizing the mutual information between them. In addition, it is worth noting that CL performs relatively poorly on the Sports dataset, which may be due to the fact that the Sports dataset contains more diverse product types with significant feature differences, making feature learning more challenging.

[0250] like Figure 3 As shown in Figure 2, in order to study the impact of each mode on the recommendation accuracy, different unimodal features are integrated into the ID embedding in formula (20), and the following variants are constructed: ID, which only contains ID embedding; ID+Visual, which contains ID embedding and visual information; ID+Textual, which contains ID embedding and textual information.

[0251] like Figure 4 As shown in the figure, PEARL outperforms the above variants in terms of R@20 and N@20 indicators, proving the effectiveness of the multimodal fusion framework designed by the present invention. It can be seen that including both visual and textual modalities can improve recommendation performance, but the textual modality plays a more important role. In particular, after including visual features on the Clothing dataset, the R@20 indicator actually decreases. This obvious phenomenon can be attributed to the fact that users are more inclined to make purchase decisions based on text descriptions, and existing visual models cannot accurately identify the parts of the image that stimulate user purchasing behavior. Therefore, visual features contain a lot of noise and provide limited help in improving performance. To verify this hypothesis, further research was conducted in the following content.

[0252] To answer RQ4, sensitivity experiments are conducted on the clipping percentage p, affinity map top n value, visual importance score a, loss weights λ1 and λ2.

[0253] Effect of pruning percentage p: In the graph purification module, the pruning percentage p controls the number of edges in the mode-specific interaction graph Gm. Assuming that user behavior is influenced by at least one mode (i.e., the maximum value of p is 1), each edge observed in the interaction graph G only appears once on Gm. Given the above situation, the p value is changed from 0.1 to 1.0 in steps of 0.1. Figure 5As shown, the experimental results clearly show that the model performance on the three datasets initially increases with the increase of p value and then decreases with the increase of p value. From this, we can conclude that increasing the value of p within a certain range can effectively prune the noisy edges in the graph structure, thereby improving the model performance. However, once p exceeds a certain threshold, the effective connection between users and items will be cut off, resulting in a decline in model performance. Necessary training data. Specifically, on the Baby and Sports datasets, the recommendation accuracy reaches the optimal value at p = 0.3, while on the Clothing dataset it is p = 0.4. This difference may be because the visual and textual features on the Clothing dataset are more significantly different, and more users are only influenced by one modality, which triggers purchasing behavior. This leads to an increase in the number of noisy edges in the specific modality interaction graph, requiring a higher p value for pruning.

[0254] Impact of top-n values ​​in affinity graphs: To reduce memory consumption and prevent message propagation between irrelevant nodes in the affinity graph learning module, a top-n strategy is adopted to ensure that each user or item is connected only to the n most similar users or items. In this step, the optimal values ​​of n_u and n_i are explored in the ranges of {10, 20, 30, 40, 50, 60} and {5, 10, 15, 20, 25, 30} for the three datasets, respectively. Figure 6 Figure a shows the effect of n on user co-occurrence graph learning. It can be seen that n = 30 is the most appropriate value for the number of neighbors on the Baby and Clothing datasets, while it is 40 on the Sports dataset. This is likely due to the higher number of interactions in the Sports dataset, which results in more similar behaviors between users. Figure 6 Figure b shows the impact of n_i on item pattern perception learning. As can be seen, PEARL achieves optimal performance when n_i = 10 on the Baby dataset and when n_i = 5 on the other two datasets. In summary, lower n_i values ​​can reduce the noise from irrelevant neighbors. Furthermore, since the vertical axis ranges of the two line graphs on the same dataset are the same, we can observe that model performance is less sensitive to the setting of n_u than to n_i. This observation is consistent with the conclusions drawn from the above.

[0255] In equations (16) and (20), the visual importance score a controls the weight of the visual features, while the corresponding coefficient of the textual features is 1-a. Based on the above research results, we further investigate the contribution of each modality and adjust the value of a from 0.1 to 0.9 with an interval of 0.1. The experimental results are listed in Table 5, and the same trend is observed in the three datasets. In particular, regardless of whether the metric is R@20 or N@20, the optimal value is reached when a=0.2 (i.e., a=0.2<0.8=a). This phenomenon reflects that textual features contain more information than visual features. In addition, when the score value a exceeds 0.2, the accuracy of the score shows a continuous downward trend, and this trend is not obvious in the clothing dataset. Combining this observation with the conclusion in 5.2.2, it is more certain that the gap between textual and visual modes is more significant in the clothing dataset.

[0256] Table 4 shows the performance of the model on three datasets with visual importance scores ranging from 0.1 to 0.9:

[0257]

[0258] Influence of loss weights λ1 and λ2: In the final loss (Eq. (23)), λ1 and λ2 control the weights of contrastive loss and L2 regularization, respectively. A grid search is performed in {le-0l, 1e-02, …, le-05} to find the best values ​​of λ1 and λ2 on the three datasets. Figure 7 As shown, the performance of PEARL reflects the performance across different datasets. Specifically, on the Baby dataset, the best results of PEARL are λ1=1e-03 and λ2=1e-04; on the Sports dataset, it is λ1=1e-04 and λ2=2e-05; on the Clothing dataset, it is λ1=1e-03 and λ2=2e-02. In addition, it is observed that the performance of PEARL is less sensitive to the setting of the L2 regularization weights than the contrastive loss. The results show that it is very important to adjust the value of the weights according to different datasets, especially the weight of the contrastive loss. A parameter value of 1 can narrow the value scale gap between the contrastive loss and the BPR loss.

[0259] To answer RQ5, we visualized the item representations generated by SLMRec, MGCL, MGCN, and PEARL on the Baby dataset. To reduce computational time, we randomly selected 500 items from the Baby dataset and mapped their final representations into 2D space using tSNE. We used Gaussian kernel density estimation (KDE) to plot circular heatmaps depicting the distribution of item correspondences, and further plotted corresponding angle-density plots to more clearly depict the density at different angles.

[0260] Previous studies have shown that maintaining a uniformly distributed representation can preserve the intrinsic characteristics of nodes, improve their generalization ability, and thus improve recommendation performance. Data-augmented SLMRec and contrastive learning MGCL both exhibit multiple dense regions in visualization. MGCN alleviates the distribution clustering after feature-level denoising, but still does not achieve a more uniformly distributed representation like PEARL. It avoids the propagation of noise by purifying the mode-specific interaction graph, while mining the self-supervisory signal between heterogeneous and homogeneous information to improve the uniformity of the representation. In summary, this explains why PEARL outperforms other multimodal recommendation methods.

[0261] In this example, a new MMRS method, PEARL, is proposed. Specifically, a graph purification strategy is first designed to remove misleading noise signals and obtain a modality-specific interaction graph. An interaction graph learning module is further designed to mine high-order representations of users and items. Then, a user-user relationship graph and an item-item relationship graph are constructed to learn implicit self-supervisory signals between similar nodes. Finally, a non-enhancement contrastive learning task is proposed in the fusion module to assist model training. Extensive experiments are conducted on three datasets, verifying that PEARL outperforms various state-of-the-art baselines in terms of R@K and N@K.

[0262] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0263] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. Software code can be stored in a memory and executed by a processor. An electronic device includes a memory and a processor; the memory is used to store a readable program, which is loaded and executed by the processor to implement the aforementioned methods.

[0264] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0265] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0266] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0267] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0268] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a ROM, a RAM, a magnetic disk, or an optical disk. It should be noted that, in this article, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0269] The one or more embodiments of this application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this application should be included in the scope of protection of this application.

Claims

1. A method for multimodal recommendation, characterized in that The steps include: Generate initial ID embeddings for each user and item, extract the original modal features of the items through the pre-trained model, and project the original modal features into a unified modal embedding space; Using the initial ID embedding and the projected modality embedding, we calculate the attention score between users and items based on the difference in users' attention to different modalities, and prune the interaction edges based on the attention score difference to generate a purified modality-specific interaction graph. Applying a graph convolutional network to the purified modality-specific interaction graph and the observed original interaction graph, we obtain high-order ID embeddings of users and items and corresponding high-order modality features through information propagation. Construct a user co-occurrence graph and select users sharing similar features through a top-k strategy to obtain enhanced representations between users. Use similarity metrics to construct an item modality perception graph and connect semantically similar items through a top-k strategy to obtain enhanced representations of items. The obtained high-level ID embeddings of users and items are weighted and summed with the high-level modal features, the enhanced representations between users, and the enhanced representations of items to form the final global user representation and global item representation. Based on the final global user representation and global item representation, the inner product operation is used to calculate the preference score between the user and the item, and the item with the highest score is recommended to the user.

2. The method for multimodal recommendation according to claim 1, wherein: The method also includes: introducing a contrastive learning task, calculating the contrastive loss by maximizing the similarity between the same nodes under the global representation, optimizing the model parameters in combination with the Bayesian personalized ranking loss, and then using the inner product operation to calculate the preference score between the user and the item using the final optimized global user representation and global item representation, and recommending the item with the highest score to each user.

3. The method for multimodal recommendation according to claim 1, wherein: Methods for calculating the attention score between users and items include: Project item features of each modality into a unified embedding space middle: in, is the projection feature under mode m, d is the embedding dimension, and Denote the transformation matrix and bias vector respectively, d m represents the dimension of the feature; σ1(·) is the LeakyReLU activation function; is the original modal feature of the item; Use the inner product operation combined with the node degree to calculate the similarity between user u and item i under mode m Where d(·) represents the degree of the node in the interaction graph G; is a multimodal user feature; T represents transposition; Use the softmax function to get the attention score of user u on item i Among them, N u represents the items that user u interacts with, and exp(·) is the exponential operation.

4. The method for multimodal recommendation according to claim 1, wherein: Methods for pruning interaction edges based on attention score differences include: Based on the attention score, determine whether to retain or prune the interaction edges between entities. The pruning rules are as follows: If the normalized attention score is lower than the preset threshold, the interaction edge between the entities is pruned; If the normalized attention score is higher than or equal to the preset threshold, the interaction edge between the entities is retained.

5. The method for multimodal recommendation according to claim 1, wherein: Methods for obtaining high-level ID embeddings for users and items through information propagation include: Using LightGCN network in interaction graph G and interaction matrix G m Perform message propagation in , delete the weight transformation matrix and activation function; During the convolution calculation of the lth layer, the embedding of users and items is updated through the matrix: Among them, U represents the set of users, I represents the set of items, represents the diagonal matrix of M, which is a symmetric adjacency matrix derived from the interaction graph G; the diagonal elements D aa Equal to ∑ b M ab ; Embedding matrix of users and items in layer l Update from layer l-1, where the initial matrix of layer 0 is set to: in, represents the identity id embedding vector specific to user u1, represents the identity id embedding vector specific to item i1; The ID embeddings at each layer are integrated using a summation where: in, is the final high-order id embedding of users and items, L in Represents the number of convolutional layers on the interaction graph G.

6. The method for multimodal recommendation according to claim 5, wherein: Methods for obtaining corresponding high-order modal features through information propagation include: The following convolution operation is performed on the purified interaction graph: Where D is the degree matrix, M is the enhanced adjacency matrix, and M m By G m Export, the initial feature matrix of layer 0 is set to: The final high-order modal characteristics The calculation is as follows:

7. The method for multimodal recommendation according to claim 6, wherein: Methods for obtaining enhanced representations between users include: Construct the user co-occurrence matrix as follows: S=G·G T ; Where T represents transpose, The elements in represent the number of co-occurrences between users; A top-n strategy is used to retain users with a certain number of co-occurrences: Among them, S a,. Represents the element in row a of matrix S, where S is the constructed user co-occurrence matrix. is the matrix transformed by the above rules; according to the transformed matrix Get the user co-occurrence graph A = {U, C}; The set of edges representing co-occurrences between users; Embed using the user's high-level ID and modal features Perform graph convolution on the user co-occurrence graph, and the calculation formula is: in, yes The diagonal matrix, N aa =n u ; Use L af -The output of the c layer is used as the enhanced representation of the user: in and represent user-enhanced id embedding and modality features respectively; Methods for obtaining an enhanced representation of an item include: Under mode m, calculate i a and i b The similarity between in, is a fully connected matrix, cos(·,·) is the cosine function, and ||·|| is the length of the vector; and Represent the modal embeddings of items a and b respectively; Use the top-n strategy as follows: in is a sparse matrix, and the item modal perception map is A m ={I,C m },in is the edge set representing the semantic relationship between items; Using higher-order modal features as follows: in yes The diagonal matrix of For ID embedding Perform the above operations in the same way to obtain Therefore, item enhancements are expressed as follows: in and Represent the enhanced id embedding and modal features of the items, α m is the importance score of the item feature under modality m, α is a hyperparameter, t =(1-α).

8. The method for multimodal recommendation according to claim 7, wherein: The method of weighted summing the obtained high-order ID embeddings of users and items with high-order modal features, the obtained enhanced representations of users and items includes: The operation of embedding the user ID is described as follows: in, is the final id embedding of the user; by performing the above operation on all ID embeddings and modal features of users and items, we get and Use the attention weight β to control the fusion of user nodes, and use the importance score α to control the fusion of item nodes: in, and denote the global representation of users and items respectively, represents weighted sum; β is a trainable parameter, α is a defined hyperparameter, v represents vision, and t represents text; Use the resulting enhanced representation to generate a preference score y for the recommendation system ui , where the preference score y ui By H u and H I The inner product of is calculated as: y ui =(h u ) T h i Methods for calculating total contrast loss include: InfoNCE is used to maximize the similarity between the same nodes under different representations and minimize the similarity between nodes of different users or items. The contrast loss L of user nodes CL,U The construction is as follows: Among them, U′ represents a group of users extracted from a batch of training data, and τ represents the temperature coefficient; the contrast loss of the item node is obtained in the same way Add the above results under all modes to get the total contrast loss L CL as follows: The method for calculating the final loss function L includes: The Bayesian personalized ranking loss is used to make the score of positive pairs higher than that of negative pairs; the loss function L BPR Expressed as: in, is a set of triples, item i b is randomly sampled; the function σ2(·) is the Sigmoid activation function; The total contrast loss L CL And the loss function L BPR The final loss function L is merged as follows: Among them, λ1 and λ2 are hyperparameters for measuring contrast loss and L2 regularization, respectively, and Θ is a model parameter.

9. An electronic device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a readable program; The at least one processor is configured to call the readable program and execute the method according to any one of claims 1 to 8.

10. A computer-readable medium, characterized in that The computer-readable medium stores computer instructions, which, when executed by a processor, cause the processor to execute the method according to any one of claims 1 to 8.

Images