A Multimodal Recommendation Method Based on Hypergraph Collaborative Filtering

By employing a hypergraph collaborative filtering method, combined with deep learning and self-supervised learning, a multimodal recommendation model is constructed, which solves the problem of insufficient utilization of multimodal information in existing technologies and achieves more accurate personalized recommendations.

CN117909599BActive Publication Date: 2026-11-17NORTHEASTERN UNIV CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410110228.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-26
Publication Date
2026-11-17
Estimated Expiration
2044-01-26

AI Technical Summary

Technical Problem

Existing recommendation algorithms fail to effectively capture fine-grained information between modalities when processing multimodal information, resulting in insufficient recommendation accuracy, especially when dealing with sparse data and changes in user preferences.

Method used

A multimodal recommendation method based on hypergraph collaborative filtering is adopted. Visual, audio and textual modal features of items are extracted through deep learning, and user-item interaction matrix and modal interaction matrix are constructed. Hypergraph convolutional network is used to aggregate high-order neighbor information, and cross-modal fusion mechanism and self-supervised learning are used to optimize feature vectors and fuse multimodal information.

Benefits of technology

It improves the accuracy and personalization of recommendation systems, enhances the ability to understand complex relationships, alleviates the problem of data sparsity, and increases user satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117909599B_ABST
    Figure CN117909599B_ABST
Patent Text Reader

Abstract

This invention provides a multimodal recommendation method based on hypergraph collaborative filtering, belonging to the field of user recommendation technology. The method first determines the number of users and items based on historical user data, and then extracts item modal features. Next, it constructs a user-item interaction matrix and a modal interaction matrix between items. Then, it calculates the k-order reachable neighbors of items, users, and each modality of an item on the interaction matrix and modal interaction matrix to obtain the corresponding hypergraph propagation matrix. Next, it initializes the feature vectors of users and items; constructs a multimodal collaborative filtering model based on a hypergraph convolutional network to optimize the user and item feature vectors; finally, it minimizes the loss function and updates the user and item feature vectors until the iteration termination condition is met. This method, by fusing information from different modalities, can improve the accuracy and personalization level of the recommendation system, while also increasing the system's ability to understand complex relationships.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of user recommendation technology, and in particular to a multimodal recommendation method based on hypergraph collaborative filtering. Background Technology

[0002] In recent years, with the development of the internet information age, while people are satisfying their information needs, they also face the problem of information overload. If this information could be filtered based on user preferences, and content that users might like could be pushed to them, it would not only save time but also improve the user experience. Therefore, recommending information that users like is very important in daily life. Existing recommendation algorithms generally use graph convolution or hypergraph convolutional networks for collaborative filtering to learn user preference information, rarely considering the correlation of multimodal information in the data. Since historical interaction records are sparse, this can lead to inaccurate recommendations.

[0003] Collaborative filtering recommendation algorithms learn from the preferences of neighboring nodes based on user and item interaction records, thus filtering information and pushing potentially appealing content to the user. The key to collaborative filtering is learning user and item preferences. Traditional collaborative filtering algorithms include user-based and item-based methods. User-based collaborative filtering constructs a user-item rating matrix based on historical behavioral data, calculates similarity scores between users, and then selects users with high similarity scores, recommending highly rated items from their interaction records to the current user. Item-based collaborative filtering is similar to user-based collaborative filtering.

[0004] Early collaborative filtering primarily used methods such as k-nearest neighbors and Pearson coefficients for computation. Later, with the rapid development of deep learning technology, many deep learning techniques were introduced into recommender systems and achieved rapid progress. Common deep learning recommender algorithms include: Multilayer Perceptron (MLP) models, Deep Factorization Machine (DFM) models, Convolutional Neural Networks (CNNs) models, and Feedforward Neural Networks (FNN) models.

[0005] In recent years, with the development of graph neural networks (GNNs), many collaborative filtering recommendation methods based on GNNs have emerged in the field of recommender systems. For example, the NGCF model was the first to apply graph convolution to recommender systems. It mainly uses a node in the graph structure to aggregate higher-order neighbor information through a graph convolutional network, thereby learning the signal information of neighboring nodes. The LightGCN model is a simplification of the NGCF model, removing some redundant parameters and simplifying model training. It can better learn the embedding representation of nodes and is often used as a baseline for method comparison. Later models introduced GAT on the basis of graph convolutional networks, using an attention mechanism to update node weights. In graph convolution, weights are calculated based on the degree of the node and remain constant. However, due to the over-smoothing problem of graph convolution, it is not easy to aggregate high-level information. Later, hypergraph convolution methods emerged for recommendation, and traditional... Figure 1 An edge can only connect two nodes, but in a hypergraph, a single hyperedge can connect multiple nodes. Therefore, using a hypergraph makes it easier to obtain higher-order relationships between nodes. With the development of hypergraph convolution, many hypergraph collaborative filtering models have emerged, such as DHCF, HCCF, and SHT.

[0006] The development of collaborative filtering recommendation algorithms has gone through three main stages. The initial stage focused on user- and item-based collaborative filtering, gradually evolving to deep learning-based collaborative filtering, and more recently, graph convolution-based collaborative filtering has been introduced. However, with the rapid increase in user data, each stage faces unique challenges and problems.

[0007] (1) Traditional Collaborative Filtering. This mainly involves user-item collaborative filtering. These two recommendation algorithms mine the relationship between users and items from users' historical behavior data to make personalized recommendations. By using cosine similarity or Pearson coefficient to calculate the similarity between users, items with high likes among users with high similarity are recommended to the user. However, since historical interaction records are sparse, it may be difficult to find similar users. Moreover, user preferences may change over time, and traditional collaborative filtering models are usually not flexible enough to adapt to such changes.

[0008] (2) Deep Learning-Based Collaborative Filtering. Deep learning-based collaborative filtering is a recommendation system method that uses deep neural networks to learn the implicit complex relationships between users and items. Traditional collaborative filtering methods rely on the similarity of user-item interaction matrices, while deep learning-based methods use deep neural networks to learn latent vectors of users and items from raw data, which can better capture the non-linear relationships and abstract features between users and items, thus improving the accuracy of recommendations. Deep learning-based collaborative filtering usually requires a large amount of data for training to avoid overfitting and improve the model's generalization ability. It is easily affected by data noise on some sparse datasets, resulting in insufficient generalization ability. For new users in the system, due to the lack of sufficient historical data, deep learning-based collaborative filtering may not be able to generate accurate recommendations.

[0009] (3) Graph Convolution-Based Collaborative Filtering. Graph convolution-based collaborative filtering utilizes graph structures to model the relationships between users and items. It employs graph neural networks to consider the adjacency relationships between users and items on the graph. Users and items can only be represented as nodes in the graph, with each node initialized with a vector, and the interaction relationships between nodes considered as edges in the graph. Then, the feature information of neighboring nodes in the graph is aggregated through graph convolution. In this way, higher-order neighbor information can be learned through multiple convolutions, which can better learn user preference information compared to traditional collaborative filtering. However, graph convolution has the problem of oversmoothing, resulting in excessively high similarity of node features, making them difficult to distinguish. Later, hypergraph convolutional networks emerged, which are more flexible than the previous graph structures. In a general graph structure, an edge can only connect two nodes, but in a hypergraph, a hyperedge can connect multiple nodes, so nodes with high similarity can be placed in a single hyperedge. This allows higher-order relationships to be obtained with fewer convolutions, acquiring the global relationships between nodes and indirectly alleviating the oversmoothing problem. Most current hypergraph collaborative filtering recommendation algorithms utilize the historical interaction information between users and items, rarely considering other modal information such as text, sound, and visual modal information. When a user likes an item, modal features often play a significant role. For example, a user might like a video because of its text or background music; therefore, the video's modal information is crucial. However, complex relationships often exist between modalities. Simply adding modal features as supplementary information to nodes only utilizes modal information at a coarse-grained level, failing to capture the fine-grained information between modalities. This limits the expressive power of recommendation models. By modeling multimodal relationships using hypergraphs, fine-grained collaborative signals between modalities can be captured globally, thereby improving the accuracy of recommendation systems. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a multimodal recommendation method based on hypergraph collaborative filtering, which mines the correlation between modalities and then fuses multimodal information in hypergraph convolution to achieve multimodal recommendation.

[0011] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multimodal recommendation method based on hypergraph collaborative filtering, comprising the following steps:

[0012] Step 1: Based on the user's historical data, determine the number of users n and the number of items m, and complete the item modal feature extraction; use deep learning technology to extract different modal information of items, including visual, sound, and text, and obtain vector representations of different modalities as the feature information of item modality, namely visual, sound, and text modal feature vectors V, A, T;

[0013] Step 2: Construct an interaction matrix based on the historical interaction records of users and items and the modal feature vectors V, A, T, including the user-item interaction matrix and the item modal interaction matrix;

[0014] Step 2.1: Construct the user-item interaction matrix; construct the graph based on the user and item interaction information. in, Let ε be the vertex, representing users and items, and ε be the edge, representing the relationship between users and items. For data where users and items interact, construct an n*m user-item interaction matrix R, where n represents the number of rows (also the number of users) and m represents the number of columns (also the number of items). If there is an edge between user u and item i in the interaction matrix R, then R... u,i If it is 1, otherwise R u,i =0;

[0015] Step 2.2: Construct the item modal interaction matrix;

[0016] (1) Calculate the modal similarity between different items;

[0017] The similarity of modal features between different items is measured by calculating the cosine value of the modal features, and the cosine value of the modal features is normalized.

[0018] For visual modal features, the cosine values ​​of the visual modal features between different items are calculated as shown in the following formula:

[0019]

[0020] Among them, M i,j Let v be the cosine value of the visual modal features of two different items i and j. i v j These represent the visual modal feature information of the two items, Indicates v j The transpose of , cos represents cosine similarity, and:

[0021]

[0022] Due to M i,j The value may be negative, so the following normalization process is performed here: M is... i,j The range is mapped to between 0 and 1, as shown in the following implementation:

[0023]

[0024] Among them, M′ i,j M represents i,j The normalized value, M′ i,j The range is between 0 and 1;

[0025] (2) Modal interaction sampling and filtering are performed based on the modal similarity between different items to obtain the visual modal interaction matrix M. V Voice modal interaction matrix M A Text modal interaction matrix M T Through the modal similarity calculation process described above, modal similarity values ​​for two different items are obtained. Then, modal similarity values ​​M are selected based on their magnitude. i,f Interaction records ≥p, where p is an initially set hyperparameter. Modal similarity values ​​meeting the condition are set to 1, otherwise to 0. The purpose is to filter items with high modal similarity to form a modal interaction matrix. After sampling and filtering, a visual modal interaction matrix M between items is obtained. V As shown in the formula below:

[0026]

[0027] The visual modal interaction matrix M between items can be obtained through the above calculations. v Voice modal interaction matrix M A Text modal interaction matrix M T ;

[0028] Step 3: Based on the user-item interaction matrix R and the modal interaction matrix M between items obtained in steps 1 and 2. V M A M T Construct a hypergraph; calculate the k-order reachable neighbors of items, users, and each modality of items on the interaction matrix and modal interaction matrix respectively to obtain the corresponding hypergraph propagation matrix;

[0029] (1) On the user-item interaction matrix R, calculate the k-th order reachable item neighbors of item i. and the super-edge group of items As shown in the formula below:

[0030]

[0031]

[0032] in, R∈{0,1} n*m , The super-edge group representing the item,

[0033] Then, the hyperedge groups of different items are concatenated together to obtain the hypergraph propagation matrix H on the item side. i As shown in the formula below:

[0034]

[0035] (2) On the user-item interaction matrix R, calculate the k-th order reachable user neighbors of user u. and the user-side hyper-edge group As shown in the formula below:

[0036]

[0037]

[0038] in, Represents the user-side hyper-edge group.

[0039] Then, the hyperedge groups of different users are concatenated together to obtain the hypergraph propagation matrix H on the user side. u As shown in the formula below:

[0040]

[0041] (3) In the visual modal interaction matrix M V Based on this, calculate the k-th order reachable neighbors of the visual modality. Hyper-edge group of object visual modal The calculation process is as follows:

[0042]

[0043]

[0044] in, The hyperedges representing the visual modalities of an object are concatenated to form the hypergraph propagation matrix H of the visual modalities of the object. i,v :

[0045]

[0046] The corresponding reachable neighbors of the sound mode of item i are calculated. Reachable neighbors of text modality Then, the hypergraph propagation matrix H of the sound modality side of the object is calculated. i,A The hypergraph propagation matrix H on the text modality side i,T As shown in the formula below:

[0047]

[0048]

[0049] Step 4: Initialize the feature vectors of users and items; randomly initialize a feature vector for each user and each item, denoted as E. u and E i Each feature is a d-dimensional vector; the initial feature vectors E for each item's text, sound, and visual modalities are... A E T E V , which is equal to the modal feature vectors A, T, V extracted in step 1;

[0050] Step 5: Construct a multimodal collaborative filtering model based on a hypergraph convolutional network to optimize user and item feature vectors; based on the user and item feature vectors E... u E i The text, sound, and visual modal feature vectors E of the item V E A E T And the hypergraph propagation matrix H u H i H i,V H i,A H i,T The model is trained using a hypergraph convolutional network and a multimodal fusion mechanism to minimize the loss function;

[0051] Step 5.1: Construct a hypergraph convolutional network;

[0052] Input the initialized feature vectors E of users and items into the hypergraph convolutional network. u E i The text, sound, and visual modal feature vectors E of the item V E T E A And the hypergraph propagation matrix H u H i H i,V H i,A H i,TThe following formula is used to aggregate higher-order neighbor node information in a hypergraph convolutional network:

[0053]

[0054] Among them, D v D represents the degree matrix of vertices in a hypergraph structure. e E represents the degree matrix of the hyperedge. l+1 E represents the features of the (l+1)th layer of the hypergraph neural network. l W represents the features of the l-th layer of the hypergraph neural network. l Here are the weight parameters of the l-th layer, LekyReLU is a non-linear activation function, and H represents the hypergraph propagation matrix. T This represents the transpose of the hypergraph propagation matrix;

[0055] Step 5.2: Employ a cross-modal fusion mechanism to integrate multimodal information into user and item interaction information;

[0056] After the hypergraph convolution above, we obtain the feature vectors of users and items at each layer. and the multimodal feature vectors of the items A cross-modal fusion mechanism is used to fuse the multimodal information and feature information of the item into E. i The specific process is shown in the following formula:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] in This represents the feature vector of the item in the l-th layer of the hypergraph convolutional network, as well as the feature vectors of the three modal information, where d represents the dimension of the modal feature vector. All represent the modal transformation parameters of the l-th layer. It is the eigenvector after the transformation of the first-level modal eigenvector. and This represents the text, sound, and visual modal feature vectors of the fused layer l, where softmax is the normalization function.

[0064] Update the item's feature information so that the fused item-side modal features are close to the item's feature vector. Specifically, the fusion coefficients of different modalities are constrained using a sign function. Specifically, each modal feature is processed once using the sign function, and the result is multiplied by the modal feature vector using a Hadamard product. Finally, the different modal feature information of the item is added together to update the item's features, as shown in the following formula:

[0065]

[0066] in, Let represent the updated item feature vector, β1 and β2 be the fusion coefficients, and sign be the sign function. The fused item's feature vector at layer l;

[0067] Step 5.3: Optimize the objective loss function; After completing steps 5.1-5.2, optimize the user and item feature vectors by minimizing the objective loss function; the loss function uses Bayesian personalized ranking, as shown below:

[0068]

[0069] in, The loss function is Bayesian, where y represents the user's rating of an item, obtained by taking the inner product of the feature vectors of the user and the item. (u, i+) represents the selected positive sample pair, and (u, i-) represents the selected negative sample pair. sigmoid is an activation function.

[0070] To employ a contrastive learning mechanism and maintain modality invariance using self-supervised learning, a modality-level contrastive loss is designed:

[0071]

[0072]

[0073]

[0074]

[0075] Where s(·) represents the calculation of cosine similarity, exp is the function for calculating softmax, and τ is a temperature hyperparameter; It is the feature vector of the l-th layer of the item after modal fusion. It is the feature vector of the item in the l-th layer. It is the visual modality feature vector of the l-th layer. It is the sound mode feature vector of the l-th state. is the text modal feature vector of the l-th layer, m is the number of items, and L is the number of convolutional layers;

[0076] Final loss function Represented as:

[0077]

[0078] Where Θ is the regularization term, which alleviates the overfitting problem, and λ1, λ2, λ3, and λ4 are hyperparameters;

[0079] Step 5.4: Convolve steps 5.1, 5.2, and 5.3 L times to obtain the vector representation of each node in the hypergraph convolutional network. The user-side vector representation is as follows: The vector representation of the item side is as follows The final user and item vectors are represented as the average of the sums of the layers, where L represents the number of convolutional layers:

[0080]

[0081] Finally, the user feature vector E′ is used. u The feature vector representation of the item E′ i The inner product is used to obtain the user's liking level y for the item, and y ui =E′ u E′ i Based on y, a list of items is recommended to the user;

[0082] Step 6: Repeat step 5 multiple times to minimize the loss function and update the feature vectors of users and items until the iteration termination condition is met.

[0083] The beneficial effects of adopting the above technical solution are as follows: The present invention provides a multimodal recommendation method based on hypergraph collaborative filtering, which utilizes historical interaction information and item modal information to mine user preference information.

[0084] (1) This invention proposes a new method for constructing a modal interaction matrix. First, modal feature information is proposed based on deep learning technology. Then, cosine similarity is calculated using the modal features of the items. After normalization, a threshold is used to filter and form a modal interaction matrix.

[0085] (2) This invention proposes a novel modal fusion method for item information. In a hypergraph convolutional network, modal features of items can be obtained through convolution of different layers. In order to explore the correlation between item modalities, this invention uses a cross-modal fusion mechanism to fuse the modal information of items into interactive features, and uses a sign function to adjust the fusion coefficients to solve the problem of differences in feature distribution between modalities, ensuring the uniformity of modal information after fusion, thereby providing more accurate recommendations.

[0086] (3) This invention proposes a novel contrastive learning method; the fused item modal information and individual modal information are compared across different modalities using self-supervised learning. Cross-modal contrastive learning integrates multimodal information, alleviating the data sparsity problem and enabling a more accurate understanding of the relationship between users and items, thereby improving the performance of the recommendation system and user satisfaction.

[0087] Compared to existing technologies, the multimodal recommendation method based on hypergraph collaborative filtering proposed in this invention not only utilizes user-item interaction information but also modal information between items. Using hypergraphs to model the modal information between users and items allows for better acquisition of implicit high-order collaborative information between modalities. Based on this, a multimodal fusion strategy is proposed to maintain the uniformity and consistency of the data after modal information fusion. Simultaneously, a modal-level cross-modal contrastive learning approach based on self-supervised contrastive learning is proposed, which, combined with collaborative filtering, can improve the recommendation performance of the collaborative filtering algorithm. Multimodal contrastive learning, by fusing information from different modalities, can improve the accuracy and personalization level of the recommendation system, while also increasing the system's ability to understand complex relationships. This has a positive impact on improving user satisfaction and the practicality of the recommendation system. Attached Figure Description

[0088] Figure 1 A flowchart illustrating a multimodal recommendation method based on hypergraph collaborative filtering provided in this embodiment of the invention;

[0089] Figure 2 This is a schematic diagram illustrating the construction process of the user / item hypergraph provided in an embodiment of the present invention;

[0090] Figure 3 This is a schematic diagram illustrating the construction process of the hypergraphs of various modalities of an item provided in an embodiment of the present invention. Detailed Implementation

[0091] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0092] In this embodiment, a multimodal recommendation method based on hypergraph collaborative filtering is described, such as... Figure 1 As shown, it includes the following steps:

[0093] Step 1: Based on the user's historical data, determine the number of users n and the number of items m, and complete the item modal feature extraction; use deep learning technology to extract different modal information of items, including visual, sound, and text, and obtain vector representations of different modalities as the feature information of item modality, namely visual, sound, and text modal feature vectors V, A, T;

[0094] In this embodiment, we take video as an example, setting the number of users to n = 9319 and the number of items (i.e., the number of videos) to m = 6710. Each video has three modalities: visual, audio, and text. The feature dimension of the modal vector is m*d (6710*64), and the dataset is divided into training and test sets in an 8:2 ratio.

[0095] Step 2: Construct an interaction matrix based on the historical interaction records of users and items and the modal feature vectors V, A, T, including the user-item interaction matrix and the item modal interaction matrix;

[0096] Step 2.1: Construct the user-item interaction matrix; construct the graph based on the user and item interaction information. in, Let ε be the vertex, representing users and items, and ε be the edge, representing the relationship between users and items. For data where users and items interact, construct an n*m user-item interaction matrix R, where n represents the number of rows (also the number of users) and m represents the number of columns (also the number of items). If there is an edge between user u and item i in the interaction matrix R, then R... u,i If it is 1, otherwise R u,i =0;

[0097] In this embodiment, an interaction matrix R with dimensions of 9319*6710 is constructed based on the number of users and items.

[0098] Step 2.2: Construct the modal interaction matrix; For the item data, deep learning techniques were used in Step 1 to obtain the multimodal feature information V, A, T for each item. To obtain the correlation information between modalities, a modal interaction matrix M was constructed here. V M A M T The specific construction process includes two steps: modal similarity calculation and screening.

[0099] (1) Calculate the modal similarity between different items; measure the modal similarity between different items by calculating the cosine value between two items. Since the value of the similarity matrix after calculation may be negative, in order to facilitate subsequent filtering calculation, the range of V is mapped to between 0 and 1.

[0100] The similarity of modal features between different items is measured by calculating the cosine value of the modal features (modal similarity refers to the degree of similarity of the same modality between two different items), and the cosine value of the modal features is normalized.

[0101] For visual modal features, the cosine values ​​of the visual modal features between different items are calculated as shown in the following formula:

[0102]

[0103] Among them, M i,j Let v be the cosine value of the visual modal features of two different items i and j. i v j These represent the visual modal feature information of the two items, Indicates v j The transpose of , cos represents cosine similarity, and:

[0104]

[0105] Due to M i,j The value may be negative, so the following normalization process is performed here: M is... i,j The range is mapped to between 0 and 1, and the specific implementation is as follows:

[0106]

[0107] Among them, M′ i,j M represents i,j The normalized value, M′ i,j The range is between 0 and 1;

[0108] (2) Modal interaction sampling and filtering are performed based on the modal similarity between different items to obtain the visual modal interaction matrix M. V Voice modal interaction matrix M A Text modal interaction matrix M T Through the modal similarity calculation process described above, modal similarity values ​​for two different items are obtained. Then, modal similarity values ​​M are selected based on their magnitude. i,j Interaction records ≥p, where p is an initially set hyperparameter. Modal similarity values ​​meeting the condition are set to 1, otherwise to 0. The purpose is to filter items with high modal similarity to form a modal interaction matrix. After sampling and filtering, a visual modal interaction matrix M between items is obtained. V As shown in the formula below:

[0109]

[0110] The visual modal interaction matrix M between items can be obtained through the above calculations. V Voice modal interaction matrix M A Text modal interaction matrix M T ;

[0111] In this embodiment, the modal interaction matrix of the items is constructed with a dimension of 6710*6710 based on the number of items.

[0112] Step 3: Based on the user-item interaction matrix R and the modal interaction matrix M between items obtained in Steps 1 and 2 V M A M T Construct a hypergraph; calculate the k-order reachable neighbors of items, users, and each modality of items on the interaction matrix and modal interaction matrix respectively to obtain the corresponding hypergraph propagation matrix;

[0113] The following are three methods for calculating reachable neighbors;

[0114] (1) On the user-item interaction matrix R, calculate the k-th order reachable item neighbors of item i. and the super-edge group of items As shown in the formula below:

[0115]

[0116]

[0117] in, R∈{0,1} n*m , The super-edge group representing the item,

[0118] Then, the hyperedge groups of different items are concatenated together to obtain the hypergraph propagation matrix H on the item side. i As shown in the formula below:

[0119]

[0120] (2) On the user-item interaction matrix R, calculate the k-th order reachable user neighbors of user u. and the user-side hyper-edge group As shown in the formula below:

[0121]

[0122]

[0123] in, Represents the user-side hyper-edge group.

[0124] Then, the hyperedge groups of different users are concatenated together to obtain the hypergraph propagation matrix H on the user side. u As shown in the formula below:

[0125]

[0126] (3) In the visual modal interaction matrix M VBased on this, calculate the k-th order reachable neighbors of the visual modality. Hyper-edge group of object visual modal As shown in the formula below:

[0127]

[0128]

[0129] in, The hyperedges representing the visual modalities of an object are concatenated to form the hypergraph propagation matrix H of the visual modalities of the object. i,V :

[0130]

[0131] The corresponding reachable neighbors of the sound mode of item i are calculated. Reachable neighbors of text modality Then, the hypergraph propagation matrix H of the sound modality side of the object is calculated. i,A The hypergraph propagation matrix H on the text modality side i,T As shown in the formula below:

[0132]

[0133]

[0134] In this embodiment, the construction process of the user / item hypergraph is as follows: Figure 2 As shown, the construction process of each modal hypergraph of the item is as follows: Figure 3 As shown.

[0135] Step 4: Initialize the feature vectors of users and items; randomly initialize the feature vectors of users and items, respectively, as E u and E i Each feature is a d-dimensional vector; the initial feature vectors E for each item's text, sound, and visual modalities are... A E T E V , which is equal to the modal feature vectors A, T, V extracted in step 1;

[0136] In this embodiment, a feature vector is randomly initialized for 9319 users and 6710 items, respectively, E u (9319*64) and E i (6710*64)), each feature is a d-dimensional vector, where d = 64. The initial modal feature vector of the item is the modal feature matrix V, A, T (6710*64) extracted in step 1;

[0137] Step 5: Construct a multimodal collaborative filtering model based on a hypergraph convolutional network to optimize user and item feature vectors; based on the user and item feature vectors E... u E i The text, sound, and visual modal feature vectors E of the item V E A E T And the hypergraph propagation matrix H u H i H i,V H i,A H i,T The model is trained using a hypergraph convolutional network and a multimodal fusion mechanism to minimize the loss function. In this embodiment, the learning rate lr = {0.01, 0.001, 0.005} and the number of convolutional layers L = 3 are set.

[0138] Step 5.1: Construct a hypergraph convolutional network;

[0139] Input the initialized feature vectors E of users and items into the hypergraph convolutional network. u E i The text, sound, and visual modal feature vectors E of the item V E T E A And the hypergraph propagation matrix H u H i H i,V H i,A H i,T The following formula is used to aggregate higher-order neighbor node information in a hypergraph convolutional network:

[0140]

[0141] Among them, D v D represents the degree matrix of vertices in a hypergraph structure. e E represents the degree matrix of the hyperedge. l+1 E represents the features of the (l+1)th layer of the hypergraph neural network. l W represents the features of the l-th layer of the hypergraph neural network. l These are the weight parameters of the l-th layer. LekyReLU is a non-linear activation function. H represents the hypergraph propagation matrix (since this invention involves many hypergraph propagation matrices, this term is used here to refer to a single hypergraph propagation matrix). T This represents the transpose of the hypergraph propagation matrix;

[0142] Step 5.2: Employ a cross-modal fusion mechanism to integrate multimodal information into user and item interaction information;

[0143] After the hypergraph convolution above, we obtain the feature vectors of users and items at each layer. and the multimodal feature vectors of the items A cross-modal fusion mechanism is used to fuse the multimodal information and feature information of the item into E. i The specific process is shown in the following formula:

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150] in This represents the feature vector of the item in the l-th layer of the hypergraph convolutional network, as well as the feature vectors of the three modal information, where d represents the dimension of the modal feature vector. All represent the modal transformation parameters of the l-th layer. It is the eigenvector after the transformation of the eigenvector of the l-th mode. and This represents the text, sound, and visual modal feature vectors of the fused layer l, where softmax is the normalization function.

[0151] Because of the differences in feature distribution between modalities after cross-modal interaction, a certain bias occurs after fusion. A more uniform distribution of modal features better represents item information. To ensure data uniformity and consistency, the features of the fused modal data should be made as close as possible to each other. This invention provides a novel strategy for updating the feature information of items.

[0152] Update the item's feature information so that the fused item-side modal features are close to the item's feature vector. Specifically:

[0153] The fusion coefficients for different modalities are constrained using a sign function. Specifically, each modal feature is processed once using the sign function, and the result is multiplied by the modal feature vector using a Hadamard product. Finally, the modal feature information of the item is added together to update the item's features, as shown in the following formula:

[0154]

[0155] in, Let represent the updated item feature vector, β1 and β2 be the fusion coefficients, and sign be the sign function. The fused item's feature vector at layer l;

[0156] Step 5.3: Optimize the objective loss function; After completing steps 5.1-5.2, optimize the user and item feature vectors by minimizing the objective loss function; the loss function used is a common loss function in recommendation algorithms called Bayesian Personalized Ranking (BPR), as shown below:

[0157]

[0158] in, The loss function is Bayesian, where y represents the user's rating of an item, obtained by taking the inner product of the feature vectors of the user and the item. (u, i+) represents the selected positive sample pair, (u, i-) represents the selected negative sample, and sigmoid is an activation function.

[0159] To employ a contrastive learning mechanism and maintain modality invariance using self-supervised learning, a modality-level contrastive loss is designed:

[0160]

[0161]

[0162]

[0163]

[0164] Where s(·) represents the calculation of cosine similarity, exp is the function for calculating softmax, and τ is a temperature hyperparameter; It is the feature vector of the l-th layer of the item after modal fusion. It is the feature vector of the item in the l-th layer. It is the visual modality feature vector of the l-th layer. It is the sound mode feature vector of the l-th state. is the text modal feature vector of the l-th layer, m is the number of items, and L is the number of convolutional layers;

[0165] Final loss function Represented as:

[0166]

[0167] Where Θ is the regularization term, which alleviates the overfitting problem, and λ1, λ2, λ3, and λ4 are hyperparameters;

[0168] Using contrastive learning at the modal level is essentially a view generation strategy without enhancement. Common view generation strategies generate a copy of a sample, then learn two different views through an encoder for contrastive learning. This invention uses the richer modal information from fused items, along with user and item interaction information, and item modal information, as enhancement samples for contrastive learning at the modal level. Combining multimodal contrastive learning with traditional collaborative filtering methods, by integrating textual, visual, and auditory modal information, enriches the graph structure and better captures potential user preferences and item features.

[0169] Step 5.4: Convolve steps 5.1, 5.2, and 5.3 L times to obtain the vector representation of each node in the hypergraph convolutional network. The user-side vector representation is as follows: The vector representation of the item side is as follows The final user and item vectors are represented as the average of the sums of the layers, where L represents the number of convolutional layers:

[0170]

[0171] Finally, the user feature vector E′ is used. u The feature vector representation of the item E′ i The inner product is used to obtain the user's liking level y for the item, and y ui =E′ u E′ i Based on y, a list of items that the user might like is recommended; the Recall@20 and NDCG@20 evaluation criteria of the recommendation system are calculated.

[0172] Step 6: Repeat Step 5 multiple times to minimize the loss function and update the feature vectors of users and items until the iteration termination condition is met. The iteration termination condition is that the loss function converges, or the evaluation metric no longer improves after multiple consecutive iterations.

[0173] In this embodiment, step 5 is repeated for 300 training rounds, continuously optimizing the loss function and updating the feature information of users and items. At the end of training, the round with the best recommendation performance during the training process is selected. To avoid randomness, the average value of 10 experiments is taken. Experimental results show that the method of this invention can improve the recommendation performance by more than 5% compared with existing recommendation methods such as LightGCN (proposed in 2020) and MMSSL (proposed in 2023), fully demonstrating the effectiveness of the method of this invention.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-modal recommendation method based on hypergraph collaborative filtering, characterized in that: Includes the following steps: Step 1: Determine the number of users based on their historical data n and the number of items m , complete the item modal feature extraction; Step 2: Construct an interaction matrix based on the historical interaction records and modal feature vectors of users and items, including the user-item interaction matrix and the modal interaction matrix between items; Step 3: Constructing a hypergraph according to the user-item interaction matrix and the modal interaction matrix between items obtained in steps 1 and 2; calculating the item, user, and modal of the item on the interaction matrix and the modal interaction matrix, respectively, to obtain the degree of the reachable neighbor of the item, the user, and the modal of the item Obtaining a corresponding hypergraph propagation matrix from the reachable neighbor of the item, the user, and the modal of the item (1) In the user-item interaction matrix , the k-th order reachable item neighbors of an item i and the hyperedge groups of an item are computed as follows: ​​ in, , The super-edge group representing the item, ; Then, the hyperedge groups of different items are concatenated together to obtain the hypergraph propagation matrix on the item side. As shown in the formula below: (2) In the user-item interaction matrix Above, calculate user u of Accessible User Neighbors and the user-side hyper-edge group As shown in the formula below: in, , Represents the user-side hyper-edge group. ; Then, the hyperedge groups of different users are concatenated together to obtain the hypergraph propagation matrix on the user side. As shown in the formula below: (3) In the visual modal interaction matrix Calculate visual modality based on Rank can reach neighbors Hyper-edge group of object visual modal The calculation process is as follows: in, The hyperedges representing the visual modalities of an object are concatenated to form a hypergraph propagation matrix representing the visual modalities of the object. : The corresponding items are calculated i reachable neighbors of sound modes Reachable neighbors of text modality Then, the hypergraph propagation matrix of the sound modal side of the object is calculated. Hypergraph propagation matrix on the text modality side As shown in the formula below: ; Step 4: Initialize the feature vectors of users and items; randomly initialize a feature vector for each user and item, respectively. and Each feature is 1D vector; the initial modal feature vector for each item is equal to the modal features extracted in step 1. ; Step 5: Construct a multimodal collaborative filtering model based on a hypergraph convolutional network to optimize user and item feature vectors; based on the user and item feature vectors... The modal feature vectors of the items and the hypergraph propagation matrix are used to train the model using a hypergraph convolutional network and a multimodal fusion mechanism, minimizing the loss function. Step 6: Repeat step 5 multiple times to minimize the loss function and update the feature vectors of users and items until the iteration termination condition is met.

2. The multimodal recommendation method based on hypergraph collaborative filtering according to claim 1, characterized in that: Step 1 uses deep learning technology to extract different modal information of the item, including visual, auditory, and textual information, and obtains vector representations of different modalities as feature information of the item modality, namely visual, auditory, and textual modal features. .

3. The multimodal recommendation method based on hypergraph collaborative filtering according to claim 2, characterized in that: The specific method for step 2 is as follows: Step 2.1: Construct the user-item interaction matrix; construct the graph based on the user and item interaction information. =( , ),in, Let be the vertices, representing users and items. Let edges represent the relationships between users and items; for data where users and items interact, construct a... User-Item Interaction Matrix , This indicates the number of rows, which is also the number of users. This represents the number of columns, which is also the quantity of items; if the interaction matrix... Chinese users and items If there is an edge between them, then =1, otherwise =0; Step 2.2: Construct the item modal interaction matrix; (1) Calculate the modal similarity between different items; measure the modal similarity between different items by calculating the cosine value of the modal features between different items, and normalize the cosine value of the modal features. (2) Modal interaction sampling and filtering are performed based on the modal similarity between different items to obtain the visual modal interaction matrix. Voice modal interaction matrix Text modal interaction matrix ; The modal similarity calculation process described above yields modal similarity values ​​for two different items. Then, the modal similarity values ​​of the two items are filtered based on their magnitude. Interaction records, This is an initial hyperparameter, and modal similarity values ​​that meet certain conditions are set to 1, otherwise to 0. The purpose is to select items with high modal similarity to form a modal interaction matrix; after sampling and filtering, a visual modal interaction matrix between items is obtained. As shown in the formula below: The visual modal interaction matrix between items is obtained through the above calculations. Voice modal interaction matrix Text modal interaction matrix .

4. The multimodal recommendation method based on hypergraph collaborative filtering according to claim 3, characterized in that: The specific method for calculating the modal similarity between different items in step 2.2 is as follows: For visual modal features, the cosine values ​​of the visual modal features between different items are calculated as shown in the following formula: in, For two different items i , j The cosine value of the visual modality features, These represent the visual modal feature information of the two items, express transpose, Let cosine similarity be expressed, and: because The value may be negative, so the following normalization process is performed here: The range is mapped to between 0 and 1, as shown in the following implementation: in, express The normalized value, The range is between 0 and 1.

5. A multimodal recommendation method based on hypergraph collaborative filtering according to claim 4, characterized in that: The specific method for step 5 is as follows: Step 5.1: Construct a hypergraph convolutional network; Input the initialized feature vectors of users and items into the hypergraph convolutional network. Text, sound, and visual modal feature vectors of items and the hypergraph propagation matrix The following formula is used to aggregate higher-order neighbor node information in a hypergraph convolutional network: in, This represents the degree matrix of the vertices in the hypergraph structure. The degree matrix represents the hyperedge. The hypergraph neural network represents the first... Layer features, The hypergraph neural network represents the first... Layer features, It is the first Layer weight parameters, It is a non-linear activation function. Represents the hypergraph propagation matrix. This represents the transpose of the hypergraph propagation matrix; Step 5.2: Employ a cross-modal fusion mechanism to integrate multimodal information into user and item interaction information; Step 5.3: Optimize the target loss function; After completing steps 5.1-5.2, optimize the user and item feature vectors by minimizing the target loss function; Step 5.4: Convolve the three steps from steps 5.1, 5.2, and 5.

3. This process yields the vector representation of each node in the hypergraph convolutional network, and then the final vector representation of the user and item.

6. The multimodal recommendation method based on hypergraph collaborative filtering according to claim 5, characterized in that: The specific method for step 5.2 is as follows: After the hypergraph convolution above, we obtain the feature vectors of users and items in each layer. and the multimodal feature vectors of items. ; A cross-modal fusion mechanism is used to fuse the multimodal information and feature information of an item into a single data structure. The specific process is shown in the following formula: , , in , , , This indicates that the item is in the hypergraph convolutional network at the 1st rank. l The feature vectors of the layered items and the feature vectors of the three modal information. The dimension of the modal eigenvector is represented. , All indicate the first l Layer mode transformation parameters, , , , , , It is the first The eigenvectors after the transformation of the layer modal eigenvectors and Indicates the fusion of the first l The text, sound, and visual modal feature vectors of the layer. It is a normalization function; Update the item's feature information so that the fused item-side modal features are close to the item's feature vector. .

7. A multimodal recommendation method based on hypergraph collaborative filtering according to claim 6, characterized in that: Step 5.2 updates the feature information of the item, so that the modal features of the fused item side are close to the feature vector of the item. The specific method is as follows: The fusion coefficients for different modalities are constrained using a sign function. Specifically, each modal feature is processed once using the sign function, and the result is multiplied by the modal feature vector using a Hadamard product. Finally, the modal feature information of the item is added together to update the item's features, as shown in the following formula: in, This represents the updated item feature vector. Let x be the fusion coefficient, sign be the sign function, and sign(x) be the fusion coefficient. , For the fused items in Layer feature vectors.

8. A multimodal recommendation method based on hypergraph collaborative filtering according to claim 7, characterized in that: The loss function described in step 5.3 uses Bayesian personalized ranking, as shown below: in, For Bayesian loss, This represents the user's rating of an item, obtained by taking the inner product of the user's and the item's feature vectors. This indicates the selected positive sample pairs. This indicates the selected negative sample pair. It is an activation function; To employ a contrastive learning mechanism and maintain modality invariance using self-supervised learning, a modality-level contrastive loss is designed: in, This indicates the calculation of cosine similarity. It is a calculation functions, It is a temperature hyperparameter; It is the first after modal fusion Layer item feature vector, It is the first Feature vectors of layered items It is the first The visual modal feature vector of the layer, It is the first Layered sound modal feature vectors It is the first The text modal feature vector of the layer, It refers to the quantity of items. It represents the number of convolutional layers; Final loss function Represented as: in, As a regularization term, it alleviates the overfitting problem. This is a hyperparameter.

9. A multimodal recommendation method based on hypergraph collaborative filtering according to claim 8, characterized in that: The user-side vector obtained in step 5.4 is represented as follows: The item's lateral vector is represented as The final user and item vectors are represented as the average of the sums of each layer, where L Indicates the number of convolutional layers: Finally, user characteristics are used. Feature vector representation of items The internal volume is used to determine the user's level of liking for the item. ,and ,according to To recommend a list of items to users.

Citation Information

Patent Citations

  • Collaborative filtering recommendation method and system based on decoupling type two-channel hypergraph neural network

    CN113505305A

  • Recommendation method based on hypergraph motif optimization multivariate user representation

    CN116340646A