Unified heterogeneous hypergraph-based incomplete multimedia recommendation method

Abstract

The invention relates to an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph, and belongs to the field of multimedia data recommendation. According to the method, a unified heterogeneous hypergraph construction framework HIRE for incomplete multimedia recommendation and a sparse constraint enhanced version HIRE S of the unified heterogeneous hypergraph construction framework HIRE are provided, and the recommendation problem of modal information missing in a multimedia sharing platform is solved. The HIRE initializes a hypergraph structure through a K-means algorithm, and captures a cross-modal high-order relationship in combination with a heterogeneous hypergraph convolution mechanism to complement missing multi-modal features. Meanwhile, a self-supervised contrast learning mechanism of text alignment is adopted, and a hypergraph structure is jointly optimized to improve multimedia recommendation performance. A sparse optimization strategy is introduced into HIRES, a hypergraph structure is refined through optimal transmission and # imgabs0 # norm constraint, noise interference is reduced, and recommendation accuracy is improved. The method is obviously superior to a baseline method, and an efficient and explainable solution is provided for multimedia recommendation in an incomplete scene.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia data recommendation, and in particular relates to an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph. Background Art

[0002] The multimedia recommendation task aims to provide users with personalized content recommendations, especially on platforms such as TikTok, Twitter, and other medical service recommendation platforms, using multimodal information (such as text, vision, and hearing) to improve the recommendation effect. User-item interaction data and multimodal features play a key role in recommendation systems. Recent studies have explored how to use multimodal information and graph structures to enhance the performance of recommendation systems. However, the current recommendation method based on collaborative filtering is limited by the complete modality assumption and cannot effectively deal with the common modality missing problem in real scenarios, resulting in a decrease in recommendation accuracy. Figure 1 As shown in Figure 2, on the TikTok platform, traditional methods process text, visual or acoustic modalities separately, while ignoring the high-order correlations between different modalities, which is often unavailable when a certain modality is missing. Summary of the Invention

[0003] The purpose of the present invention is to solve the recommendation problem of missing modal information in multimedia sharing platforms, and to provide an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph. The method proposes an incomplete multimedia recommendation framework HIRE constructed based on a unified heterogeneous hypergraph and its sparse constrained version HIRES. By capturing cross-modal high-order relationships and optimizing the hypergraph structure, the recommendation performance in incomplete scenarios is significantly improved.

[0004] To achieve the above objectives, the technical solution of the present invention is: an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph, an incomplete multimedia recommendation framework HIRE based on a unified heterogeneous hypergraph and its sparse constrained version HIRES are proposed; HIRE initializes the hypergraph structure through the K-means algorithm, and combines the heterogeneous hypergraph convolution mechanism to capture cross-modal high-order relationships to complement the missing multimodal features; at the same time, a self-supervised contrastive learning mechanism of text alignment is adopted to jointly optimize the hypergraph structure; HIRES introduces a sparse optimization strategy, through optimal transmission and l 2,1 -norm constraint refinement of hypergraph structure.

[0005] The method comprises the following steps:

[0006] Step 1: Problem description: input data, output results and other variables;

[0007] Step 2: Heterogeneous hypergraph construction: Build a structure to inject cross-modal heterogeneous high-order relations into a unified hypergraph;

[0008] Step 3: Comparative multimedia recommendation: Enhance multimodal features and make recommendations through a self-supervised comparative learning mechanism with text alignment;

[0009] Step 4: Construct the HIRE framework HIRES under sparse constraints.

[0010] The present invention also provides an incomplete multimedia recommendation system based on a unified heterogeneous hypergraph, which is characterized in that it includes a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement any of the method steps described above.

[0011] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, any of the method steps described above can be implemented.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. Based on a unified heterogeneous hypergraph structure, hypergraph convolution and self-supervised learning mechanisms are used to complete missing modalities, solving the problem that traditional recommendation methods are unable to capture complex high-order dependencies between items when processing incomplete modal data, resulting in performance degradation.

[0014] 2. By introducing sparse constraints (l 2,1 -norm) and the optimal transmission mechanism optimize the hypergraph structure and remove irrelevant noise connections, solving the inaccuracy and roughness of traditional hypergraph methods caused by dense structures, and further improving the recommendation accuracy in incomplete modal scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 For existing methods.

[0016] Figure 2 This is a framework diagram of an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to the present invention.

[0017] Figure 3 The HIRE framework HIRES under sparse constraints constructed by the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] The present invention provides an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph, and proposes an incomplete multimedia recommendation framework HIRE based on a unified heterogeneous hypergraph and its sparse constrained version HIRES. HIRE initializes the hypergraph structure through the K-means algorithm and combines the heterogeneous hypergraph convolution mechanism to capture cross-modal high-order relationships to complement the missing multimodal features. At the same time, a self-supervised contrastive learning mechanism of text alignment is adopted to jointly optimize the hypergraph structure. HIRES introduces a sparse optimization strategy to optimize the hypergraph structure through optimal transmission and l 2,1 -norm constraint refinement of hypergraph structure.

[0020] The following is a specific implementation process of the present invention.

[0021] The present invention proposes an incomplete multimedia recommendation method based on a unified heterogeneous hypergraph, and proposes an incomplete multimedia recommendation framework HIRE and its sparse constrained version HIRES based on a unified heterogeneous hypergraph. By capturing cross-modal high-order relationships and optimizing the hypergraph structure, the recommendation performance in incomplete scenarios is significantly improved. Figure 2 As shown, the specific implementation steps are as follows:

[0022] Step 1, problem description: detailed introduction of the input data, output results and other variables of the present invention;

[0023] The details are as follows:

[0024] For a given N U The user set U contains N users V The present invention uses the interaction matrix To model the purchase relationship between users and items, if the i-th user interacts with the j-th user, set R ij is 1, otherwise R ij is 0. In addition, each item contains M modalities, and the present invention uses three modalities: vision (B), text (T) and acoustic (A). The present invention also defines the original features of the i-th item under the m-th modality And all project features are combined into For the missing item features of the modality, they are set to vectors with values ​​equal to all zeros. In order to construct a heterogeneous hypergraph to capture heterogeneous information under incomplete modalities, the present invention first initializes the hypergraph structure Where K is the number of hyperedges. The ultimate goal of this invention is to obtain a scoring matrix To express the probability of recommending the j-th item to the i-th user.

[0025] Step 2: Heterogeneous hypergraph construction: Build a carefully designed structure to inject cross-modal heterogeneous high-order relations into a unified hypergraph;

[0026] The specific implementation details are as follows:

[0027] a) Hypergraph structure initialization: Initialize the hypergraph structure using the K-means algorithm where N V is the number of items, K is the number of hyperedges, and the formula is as follows:

[0028]

[0029] The K-means algorithm clusters items with similar characteristics into K groups through a clustering mechanism, which are regarded as K hyperedges here; F = Concat(F 1 ,F 2 ,…,F M ) is obtained by concatenating features from other modalities, c i represents the hyperedge embedding of the i-th cluster center, F j represents the feature of the jth supernode in F, H i Represents all super nodes contained in the i-th hyperedge of the graph structure H, j∈H i Indicates that the jth supernode belongs to cluster center c i , H ji Indicates that the jth supernode belongs to cluster center c i probability.

[0030] b) Hypergraph Convolution: Hypergraph Convolutional Network is used to capture cross-modal high-order relationships and complete missing modal features. The formula is as follows:

[0031] Q m,(l+1) =σ(D -1 HWB -1 H T Q m,(l) )

[0032] where Q m,(l) is the embedding after completion of the l-layer hypergraph convolution under the m-th mode, D and B are the diagonal matrices of the hypergraph structure H, W is the trainable weight of the hypergraph neural network, and the 0th order represents Q m,(0) By project feature F V，m Obtained, σ(·) is the activation function that introduces nonlinear factors.

[0033] Step 3: Comparative multimedia recommendation, through the self-supervised comparative learning mechanism of text alignment, further enhance the multimodal features and make recommendations;

[0034] The specific implementation details are as follows:

[0035] a) Self-supervised contrastive learning: Contrastive learning is performed between different modalities to optimize multimodal features by keeping them consistent with text modal features. The formula is as follows:

[0036]

[0037] in Q m is the i-th item embedding, τ is the temperature parameter, and θ(·,·) represents the distance between two vectors. The core goal of the loss function in the above formula is to shorten the representation distance between positive sample pairs and push the representation distance between negative sample pairs through contrastive learning, thereby improving the accuracy and robustness of recommendations.

[0038] b) Graph information aggregation: Based on the user-item interaction matrix, the graph information aggregation operation corresponding to the ID of the user and item neighbors is performed. The formula is as follows:

[0039] X U,(l+1) =σ((D U ) -1 RX V,(l) Θ U,(l) )

[0040] X V,(l+1) =σ((D V ) -1 R T X U,(l) Θ V,(l) )

[0041] in, is the ID corresponding embedding of users and items in the lth layer of the graph neural network, and the zero-layer embedding X U,(0) and X V,(0) Initialized from a trainable lookup table; D U and D V is a diagonal matrix, D U The diagonal elements are composed of the correlation between the user node and the project node, D V The diagonal elements are composed of the correlation between the project node and the user node, (D U ) -1 and (D V ) -1 Indicates the inverse operation of the matrix, Θ U,(l) and Θ V,(l) are the trainable user and item weights of the lth layer of the graph neural network.

[0042] c) The ID embedding and item features are combined as a multimodal embedding and incorporated into the recommendation framework. The formula is as follows:

[0043]

[0044] in, and is defined as the final representation of users and items, Concat(·) is the concatenation function, and Z V，m is the complete mode after self-supervision under the mth mode.U,m =(D U ) -1 RZ V,m ,in is the user-item interaction matrix and Project-User Interaction Matrix The diagonal matrix of .

[0045] d) Use a multi-layer perceptron (MLP) to project the concatenated fusion embedding onto a common latent subspace. The projection formula is as follows:

[0046]

[0047] in, and is the projection weight, and is the bias of the MLP. Therefore, the scoring matrix Can be achieved through get, in represents the probability of recommending item j to user i.

[0048] e) Multimedia Recommendation: This invention uses the Bayesian personalized ranking loss function commonly used in recommendation systems to enhance multimedia recommendations. The formula is as follows:

[0049]

[0050] Among them, j p and j n are the positive and negative samples of user i, respectively, and |ε| represents the triple (i, j p ,j n ), Represents the model prediction of user i for positive sample j p The preference score of Represents the model prediction of user i for negative sample j n The preference score of , the role of the sigmoid function is to project any real number into the (0,1) interval.

[0051] Finally, the present invention uses the combined loss to train the recommendation system to optimize HIRE, as follows:

[0052]

[0053] Among them, λ1 is the weight hyperparameter.

[0054] Step 4, HIRE framework under sparse constraints (HIRES);

[0055] Thanks to the K-means algorithm, HIRE can capture high-order relationships across modalities to complete the missing modalities in multimedia recommendations. However, the hypergraph structure directly clustered from the original project features cannot take into account prior knowledge in incomplete scenarios, which may hinder the full potential of incomplete multimedia recommendations. Figure 3 As shown in the figure, project v1 contains complete modalities and rich modal interactions, while project v2 has more sparse interactions due to the lack of a certain modality. Therefore, v1 may have more information to enhance incomplete multimedia recommendations, while v2 may have more incomplete information with noise, and the information is further aggravated by the aggregation of hypergraph-based information mechanisms. Therefore, for scenarios with incomplete modalities, dense clustering graph structures may be inaccurate and noisy. Therefore, the present invention hopes to obtain a sparse unified heterogeneous hypergraph structure to reduce unreliable interactions caused by projects with missing modalities.

[0056] a) Sparse Optimal Transfer Framework: A sparse optimal transfer mechanism is used to replace the clustering-based heterogeneous hypergraph construction in the HIRE framework.

[0057] Different from the unconstrained mechanism in clustering, the design of HIRES is inspired by two straightforward insights.

[0058] Insight 1: Modality-driven constraint: Items with complete modalities (rows in the hypergraph structure) are more trustworthy than those with missing modalities. In order to reduce the weight of low-reliability items with missing modalities, the present invention uses a l-based 2,1 -norm sparse regularization is used as a constraint to correctly rearrange the hypergraph structure. 2,1 The -norm is defined as follows:

[0059]

[0060] Where K is the number of hyperedges controlling the column constraints, and N is the number of items controlling the row sparsity. The hypergraph structure H is represented as a learnable matrix under column and row constraints, so H can be used ij To measure the similarity between the probability distribution of the i-th item and the j-th category, that is, the super node distribution i and the super edge distribution j. Therefore, l 2,1 This is equivalent to first calculating the l2-norm constraint of the columns and then finding the l1-norm constraint of the rows. To balance the influence of terms and hyperedges in the structure, this paper proposes to use optimal transmission technology combined with row sparse regularization to ensure a reliable hypergraph structure. The formula for the above process can be expressed as:

[0061]

[0062] Among them, J is the objective function, which represents the total cost in sparse optimal transmission. The optimization goal is to minimize the value of J by adjusting the hypergraph structure H. η is a hyperparameter used to control the sparsity strength, and Δ is a constraint condition in sparse optimal transmission. K or Defined as K-dimensional or N V dimensional vectors, used to calculate the sum of rows or columns in the incidence matrix H. <·,·> is the inner product of the matrix. The matrix M represents the transportation cost. Therefore, M can be calculated by measuring the distance between supernodes and hyperedges. The formula for the cost matrix M can be obtained by using the supernode embedding F and the hyperedge embedding E. The calculation method is:

[0063]

[0064] Among them, M ij It can be regarded as the distance between supernode i and hyperedge j, M ij The smaller the value of , the higher the similarity between the i-th item and the corresponding j-th hyperedge. i and j represent the rows of F and the columns of E, respectively. E is initialized using the Xavier distribution (Xavier is an initialization method for deep learning).

[0065] Insight 2: Interaction-driven constraints: Items with rich interactions should have a greater impact on incomplete multimedia recommendations, because these items may have richer semantic information for the recommendation system to complete the missing modalities. In order to assign a larger weight to items with rich interactions, the present invention uses the degree of the node as a measure of interaction, which can be expressed as in, and are the in-degree and out-degree of the i-th item respectively. In addition, the present invention also considers the item-modality interaction and uses represents the number of modes of interaction of item i, i.e. The value range of is {0,1,2,3}. Finally, the present invention adopts As the final value of the measurement interaction driver.

[0066] In order to inject interaction-driven constraints into the optimal transmission theory, the present invention modifies the distribution constraints of items in Δ from uniformly distributing equal weights to each node. Change to interaction-based (H1 K =p) distribution, where Therefore, the final formula of the sparse optimal transmission mechanism can be expressed as:

[0067]

[0068] b) Differentiable optimization strategy: Since l 2,1The -norm is not smooth, and the above equation is difficult to solve. Therefore, the present invention designs a differentiable sparse optimization strategy to calibrate the gradient through the Frank-Wolfe algorithm.

[0069] Specifically, the calibrated Frank-Wolfe gradient is considered to be the direction most consistent with the negative value of the original gradient, which can be used to move towards the optimal solution within the feasible region. During the entire optimization process, the hypergraph structure H is first optimized, the derivative of H in the above formula is calculated, and the diagonal matrix D is introduced. The formula is as follows:

[0070]

[0071] The value of D can be calculated by calculating the l2-norm of the columns in H, that is, ||H j ||2 is obtained. Then, the above equation can be simplified and the Frank-Wolfe algorithm can be used to calculate the calibration gradient to optimize the unified hypergraph structure H according to the following objectives U Parameters in:

[0072]

[0073] The idea of ​​the Frank-Wolfe method is to find the iteration point s with the largest angle with the current gradient direction. s can be obtained by calculation:

[0074]

[0075] Where vec(·) represents the process of matrix vectorization. The DeepEM (DEarth Mover's Distance) algorithm is used to learn the minimization solution of s in a differentiable way. According to the KKT condition, the above formula can be rewritten as a linear programming problem:

[0076]

[0077] in is the optimization variable, the equality constraint As = b represents all boundary conditions constructed using equality conditions, and the inequality constraint Fs ≤ 0 represents the range of all feasible solutions in the variable domain under the KKT condition. Using the Lagrangian principle of the Lagrangian problem, we can obtain:

[0078]

[0079] Where μ is the set of equality constraints, λ≥0 is the set of inequality constraints, and ω is the learnable parameter in the neural network designed by the present invention. According to the KKT condition, the optimal loss function can be calculated by g(θ, s, μ, λ) = 0.

[0080]

[0081] According to the differentiability of convex optimization, we can calculate and the implicit function of θ:

[0082]

[0083] in, express Regarding the Jacobian determinant of θ, the implicit function theorem is applied to the KKT condition to obtain the formula of the Jacobian determinant, and the parameter θ is obtained. The closed form of the gradient, that is, using the deep back propagation method to iterate the points No need to optimize the trajectory. In order to facilitate the calculation of the objective function, the present invention converts the hypergraph matrix Expand and get As a column vector. h (k) is the embedding at the kth iteration and is considered as a fixed vector, so the final formula can be obtained as follows:

[0084] h (k+1) =(1-γ)h (k) +γs

[0085] Where γ is a hyperparameter used to maintain the current iteration point. By setting the derivative to 0, we can get the formula The optimization formula of the Chinese Super League side E:

[0086]

[0087] Among them, the closed-form solution of E can also be solved by differential method.

[0088] Finally, the present invention combines the BPR recommendation loss, hypergraph differentiable sparse optimization loss, and contrastive learning loss used above to obtain the final combined loss:

[0089]

[0090] Here, λ2 is a hyperparameter of the loss function in contrastive learning, which is used to balance the strength of text alignment during training. By integrating the three losses, a joint hypergraph structure suitable for incomplete scenarios and multimedia recommendations can be learned.

[0091] To evaluate the performance of HIRE and HIRES framework in incomplete multimedia recommendation tasks, we conduct validation on four datasets: Amazon-Baby, Amazon-Sports, Amazon-Elec, and TikTok. The experiment selects data containing user-item interaction records, covering multimodal features such as text, vision, and acoustics. The total number of items and the number of interactions vary from dataset to dataset. Given the interaction record (R ij ) and the related multimodal feature set (F V,m ), predicting the probability of a user's preference for an item, with recommendations covering all items in the dataset. This study uses metrics such as Precision@K, Recall@K, and NDCG@K to evaluate recommendation performance. Precision@K uses the number of correct items in the top-K predictions as the numerator and K as the denominator to calculate the accuracy of user recommendations. Recall@K uses the number of correct items in the top-K predictions as the numerator and the total number of items actually preferred by the user as the denominator to evaluate the coverage of recommendations. NDCG@K comprehensively considers the ranking order to measure the overall quality of the recommendation results.

[0092] To verify the impact of missing modalities on the model and validate the effectiveness of the HIRE and HIRES frameworks in completing missing modalities, we compare with the following representative and state-of-the-art baseline methods, which can be divided into two categories: (1) collaborative filtering without multimedia content (MF-BPR, NGCF, LightGCN, SGL, NCL, HCCF, MMGCN, LATTICE, and CLCRec), and (2) multimedia recommendation with missing modalities (LightGCN-M, MMGCL, SLMRec, MMSSL, AGCN, DualGNN, M3care, GCIMH, T2-GNN, MoMKE, and CI2MG).

[0093] Experimental results show that HIRE and HIRES significantly outperform the aforementioned baselines by unifying heterogeneous hypergraph structures and implementing sparse optimization mechanisms. For example, HIRES achieves a 62.30% improvement in NDCG@20 over GCCF on the TikTok dataset, and a 37.14% improvement in Recall@20 over MKGAT on the Sports dataset, demonstrating its superiority in capturing high-order cross-modal relationships and handling incomplete modal scenarios.

[0094] Table 1: Comparison of the performance (%) of baseline models on the Recall@20, Precision@20, and NDCG@20 metrics on the Amazon-Baby, Amazon-Sports, Amazon-Elec, and TikTok multimedia datasets, conditioned on a 90% miss rate (the best performance is in bold, the second best performance is in bold).

[0095] (shown underlined).

[0096]

[0097]

[0098] The experimental results are summarized in Table 1. Our method outperforms state-of-the-art baselines across all evaluation metrics, demonstrating its superiority in capturing cross-modal high-order relationships and improving recommendation accuracy and diversity in incomplete multimedia recommendation scenarios. Specifically, compared to the best-performing baseline, HIRES achieves significant improvements in Precision@20, Recall@20, and NDCG@20, from a 5.95% improvement in Recall@20 on TikTok to a 9.09% improvement in Precision@20 on Amazon-Sports.

[0099] This paper proposes a novel framework that aims to jointly learn heterogeneous hypergraph structures to make accurate recommendations in modality-incomplete scenarios. The framework is named HIRE. HIRE first initializes the hypergraph structure through the K-means algorithm, and uses a unified heterogeneous hypergraph convolution mechanism to complete the missing multimodal features through high-order relationships. Subsequently, a contrastive learning multimedia recommendation method is designed, and a text alignment self-supervision mechanism is used to enhance incomplete multimedia recommendations. In addition, this paper also proposes a sparse optimization HIRE framework named HIRES. In order to optimize the hypergraph structure, the optimal transmission and l 2,1 -norm constraint and proposes a new optimization strategy. Extensive experiments demonstrate that HIRES significantly improves upon existing state-of-the-art baseline methods, resolving the issue of poor recommendation quality in traditional methods due to the lack of item modality features. This provides a powerful solution for personalization and accuracy in multimedia recommendation systems in real-world scenarios.

[0100] The present invention also provides an incomplete multimedia recommendation system based on a unified heterogeneous hypergraph, which is characterized in that it includes a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement any of the method steps described above.

[0101] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, any of the method steps described above can be implemented.

[0102] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. An incomplete multimedia recommendation method based on a unified heterogeneous hypergraph, characterized in that: An incomplete multimedia recommendation framework HIRE based on a unified heterogeneous hypergraph and its sparse constrained version HIRES are proposed.

2. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 1, characterized in that: HIRE initializes the hypergraph structure through the K-means algorithm and combines it with the heterogeneous hypergraph convolution mechanism to capture cross-modal high-order relationships to complete the missing multimodal features. At the same time, it adopts the self-supervised contrastive learning mechanism of text alignment to align and optimize the features of different modalities to capture cross-modal high-order relationships. HIRES introduces a sparse optimization strategy to achieve the best transmission and l 2,1 -norm constraint refinement of hypergraph structure.

3. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 1 or 2, characterized in that: The method comprises the following steps: Step 1: Problem description: input data, output results and other variables; Step 2: Heterogeneous hypergraph construction: Build a structure to inject cross-modal heterogeneous high-order relations into a unified hypergraph; Step 3: Comparative multimedia recommendation: Enhance multimodal features and make recommendations through a self-supervised comparative learning mechanism with text alignment; Step 4: Construct the HIRE framework HIRES under sparse constraints.

4. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 3, characterized in that: Step 1 is implemented as follows: For a given N U The user set U contains N users V The number of items in the project set V is calculated using the interaction matrix To model the purchase relationship between users and items, if the i-th user interacts with the j-th user, set R ij is 1, otherwise R ij is 0; in addition, each item contains M modes; the original features of the i-th item under the m-th mode are also defined And all project features are combined into For the missing item features of the modality, they are set to vectors with values ​​equal to all zeros; in order to construct a heterogeneous hypergraph to capture the heterogeneous information under incomplete modalities, the hypergraph structure is initialized. Where K is the number of hyperedges, and finally a scoring matrix is ​​obtained To express the probability of recommending the j-th item to the i-th user.

5. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 4, characterized in that: M is set to 3, including three modalities: vision, text, and acoustics.

6. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 4, characterized in that: Step 2 is implemented as follows: a) Hypergraph structure initialization: Initialize the hypergraph structure using the K-means algorithm where N V is the number of items, K is the number of hyperedges, and the formula is as follows: The K-means algorithm clusters items with similar characteristics into K groups through a clustering mechanism, which are regarded as K hyperedges here; F = Concat(F 1 ,F 2 ,…,F M ) is obtained by concatenating features from other modalities, c i represents the hyperedge embedding of the i-th cluster center, F j represents the feature of the jth supernode in F, H i Represents all super nodes contained in the i-th hyperedge of the graph structure H, j∈H i Indicates that the jth supernode belongs to cluster center c i , H ji Indicates that the jth supernode belongs to cluster center c i probability; b) Hypergraph Convolution: Hypergraph Convolutional Network is used to capture cross-modal high-order relationships and complete missing modal features. The formula is as follows: Q m,(l+1) =σ(D -1 HWB -1 H T Q m , (l) ) where Q m,(l) is the embedding after completion of the l-layer hypergraph convolution under the m-th mode, D and B are the diagonal matrices of the hypergraph structure H, W is the trainable weight of the hypergraph neural network, and the 0th order represents Q m,(0) By project feature F V，m Obtained, σ(·) is the activation function that introduces nonlinear factors.

7. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 6, characterized in that: Step three is implemented as follows: a) Self-supervised contrastive learning: Contrastive learning is performed between different modalities to optimize multimodal features by keeping them consistent with text modal features. The formula is as follows: in Q m The core goal of the loss function is to improve the accuracy and robustness of recommendations by shortening the representation distance between positive pairs and extending the representation distance between negative pairs through contrastive learning. b) Graph information aggregation: Based on the user-item interaction matrix, the graph information aggregation operation corresponding to the ID of the user and item neighbors is performed. The formula is as follows: X U,(l+1) =σ((D U ) -1 RX V,(l) I U,(l) ) X V,(l+1) =σ((D V ) -1 R T X U,(l) I V,(l) ) in, is the ID corresponding embedding of users and items in the lth layer of the graph neural network, and the zero-layer embedding X U,(0) and X V,(0) Initialized from a trainable lookup table; D U and D V is a diagonal matrix, D U The diagonal elements are composed of the correlation between the user node and the project node, D V The diagonal elements are composed of the correlation between the project node and the user node, (D U ) -1 and (D V ) -1 Indicates the inverse operation of the matrix, Θ U,(l) and Θ V,(l) are the trainable user and item weights of the lth layer of the graph neural network; c) The ID embedding and item features are combined as a multimodal embedding and incorporated into the recommendation framework. The formula is as follows: in, and is defined as the final representation of users and items, X U 、X V X U,(l) 、X V,(l) The last layer result, Concat(·) is the connection function, Z V,m is the complete mode after self-supervision under the mth mode, Z U,m =(D U ) -1 RZ V,m ,in is the user-item interaction matrix and Project-User Interaction Matrix The diagonal matrix of d) Use the multi-layer perceptron (MLP) to project the concatenated fusion embedding onto a common latent subspace; the projection formula is as follows: in, and is the projection weight, and is the bias of the MLP; therefore, the scoring matrix pass get, in represents the probability of recommending item j to user i; e) Multimedia Recommendation: A Bayesian personalized ranking loss function is used to enhance multimedia recommendations. The formula is as follows: Among them, j p and j n are the positive and negative samples of user i, respectively, and |ε| represents the triple (i, j p ,j n ), Represents the model prediction of user i for positive sample j p The preference score of Represents the model prediction of user i for negative sample j n The preference score of the sigmoid function is to project any real number into the interval (0,1); Finally, the combined loss is used to optimize HIRE, as follows: Among them, λ1 is the weight hyperparameter.

8. The incomplete multimedia recommendation method based on a unified heterogeneous hypergraph according to claim 7, characterized in that: Step 4 is implemented as follows: a) Sparse Optimal Transfer Framework: We use a sparse optimal transfer mechanism to replace the clustering-based heterogeneous hypergraph construction in the HIRE framework; Mode-driven constraint: To reduce the weight of low reliability items with missing modes, the e-based 2,1 -norm sparse regularization as a constraint to correctly rearrange the hypergraph structure; e 2,1 The -norm is defined as follows: Where K is the number of hyperedges controlling the column constraint, and N is the number of items controlling the row sparsity; the hypergraph structure H is represented as a learnable matrix under the column constraint and row constraint, so H is used ij to measure the similarity between the probability distribution of the i-th item and the j-th category, that is, the supernode distribution i and the superedge distribution j; therefore, e 2,1 This is equivalent to first calculating the e2-norm constraint of the column and then finding the e1-norm constraint of the row. To balance the influence of the midterm and hyperedge in the structure, it is proposed to use the optimal transmission technology combined with row sparse regularization to ensure a reliable hypergraph structure. The formula for the whole process is expressed as: Among them, J is the objective function, which represents the total cost in sparse optimal transmission. The optimization goal is to minimize the value of J by adjusting the hypergraph structure H. η is the hyperparameter used to control the sparsity strength. Δ is the constraint condition in sparse optimal transmission. K or Defined as K-dimensional or N V dimensional vector, used to calculate the sum of rows or columns in the incidence matrix H, <·,·> is the matrix inner product; the matrix M represents the transportation cost; therefore, M is calculated by measuring the distance between supernodes and hyperedges. The formula of the cost matrix M is obtained by the supernode embedding F and the hyperedge embedding E, and the calculation method is: Among them, M ij is considered as the distance between supernode i and hyperedge j, M ij The smaller the value of , the higher the similarity between the i-th item and the corresponding j-th hyperedge. i and j represent the rows of F and the columns of E, respectively. E is initialized by Xavier distribution. Interaction-driven constraint: To assign larger weights to items with rich interactions, the degree of the node is used as the measure of interaction, expressed as in, and are the in-degree and out-degree of the i-th item respectively; in addition, item-modality interaction is also considered and used represents the number of modes of interaction of item i, i.e. The value range of is {0,1,2,3}; Finally, adopt As the final value of the measurement interaction driver; To inject interaction-driven constraints into the optimal transmission theory, the distribution constraints of the items in Δ are modified from uniformly distributing equal weights to each node. Switch to an interaction-based H1 K =p distribution, where Therefore, the final formula of the sparse optimal transmission mechanism is expressed as: b) Differentiable optimization strategy: Since l 2,1 -norm is not smooth, and the above equation is difficult to solve. Therefore, a differentiable sparse optimization strategy is designed to calibrate the gradient through the Frank-Wolfe algorithm; Specifically, the calibrated Frank-Wolfe gradient is considered to be the direction most consistent with the negative value of the original gradient, which can move towards the optimal solution within the feasible region. During the entire optimization process, the hypergraph structure H is first optimized, the derivative of H in the above formula is calculated, and the diagonal matrix D is introduced. The formula is as follows: The value of D is obtained by calculating the l2-norm of the columns in H, that is, ||H j ||2 is obtained; then, the above equation is simplified and the Frank-Wolfe algorithm is used to calculate the calibration gradient, and the unified hypergraph structure H is optimized according to the following objectives U Parameters in: The idea of ​​the Frank-Wolfe method is to find the iteration point s with the largest angle with the current gradient direction: Where vec(·) represents the process of matrix vectorization. The DeepEMD algorithm is used to learn the minimization solution of s in a differentiable way. According to the KKT condition, the above formula can be rewritten as a linear programming problem: in is the optimization variable, the equality constraint As=b represents all boundary conditions constructed by the equality conditions, and the inequality constraint Fs≤0 represents the range of all feasible solutions in the variable domain under the KKT condition. By using the Lagrangian principle of the Lagrangian problem, we can obtain: Where μ is the set of equality constraints, λ≥0 is the set of inequality constraints, and ω is the learnable parameter in the neural network. According to the KKT condition, the optimal loss function is calculated by g(θ,s,μ,λ)=0. According to the differentiability of convex optimization, we can calculate and implicit functions of ω: in, express Regarding the Jacobian determinant of θ, the implicit function theorem is applied to the KKT condition to obtain the formula of the Jacobian determinant, and the parameter θ is obtained. The closed form of the gradient, that is, using the deep back propagation method to iterate the points There is no need to optimize the trajectory; to facilitate the calculation of the objective function, the hypergraph matrix Expand and get As a column vector, h (k) is the embedding at the kth iteration and is considered as a fixed vector, so the final formula is obtained as follows: h (k+1) =(1-γ)h (k) +γs Where γ is a hyperparameter used to maintain the current iteration point. By setting the derivative to 0, we get the formula The optimization formula of the Chinese Super League side E: Among them, the closed-form solution of E is also solved by the differential method; Finally, the BPR recommendation loss, hypergraph differentiable sparse optimization loss, and contrastive learning loss are combined to obtain the final combined loss: Here, λ2 is a hyperparameter of the loss function in contrastive learning, which is used to balance the strength of text alignment during training. By integrating the three losses, a joint hypergraph structure suitable for incomplete scenarios and multimedia recommendations is learned.

9. An incomplete multimedia recommendation system based on a unified heterogeneous hypergraph, characterized by: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images