A multi-modal recommendation system and method based on multi-source hypergraph

Abstract

This invention belongs to the field of recommender system technology, specifically relating to a multimodal recommender system and method based on a multi-source hypergraph. The system includes a modality consistency interaction filtering module, a hypergraph construction module, and a representation learning and recommendation module. The modality consistency interaction filtering module calculates the consistency between users and items based on the multimodal features of items and filters user-item interaction data to obtain denoised interaction data. The hypergraph construction module constructs a behavior co-occurrence hypergraph based on the denoised interaction data and a content modality hypergraph based on the multimodal features of items, and performs node representation propagation on the behavior co-occurrence hypergraph and the content modality hypergraph. The representation learning and recommendation module performs representation learning based on the behavior co-occurrence hypergraph and the content modality hypergraph, and generates recommendation results based on user representations and item representations.

Description

Technical Field

[0001] This invention belongs to the field of recommendation system technology, specifically relating to a multimodal recommendation system and method based on multi-source hypergraphs. Background Technology

[0002] Multimodal recommendation methods based on multi-source hypergraphs belong to the field of personalized recommendation. This method fuses user-item interaction data with multimodal feature information of items and uses a hypergraph structure to model higher-order relationships, thereby characterizing user preferences and making recommendations based on these preferences. The recommendation result depends on the modeling quality of user and item representations. A typical multimodal recommendation process mainly involves modeling user-item interaction relationships and fusing multimodal information. Traditional collaborative filtering-based recommendation algorithms characterize user preferences by constructing a user-item interaction matrix, primarily assuming that a user's historical interactions fully reflect their interest characteristics. Unlike methods that rely solely on interaction data, existing multimodal recommendation methods typically introduce auxiliary information such as visual and textual data and combine graph structures to model user-item relationships to improve recommendation performance. However, these methods still mainly model pairwise user-item interactions, indirectly characterizing higher-order relationships through multi-hop propagation, making it difficult to effectively preserve the overall association information between multiple items, resulting in the weakening of higher-order semantics during propagation. Furthermore, during representation propagation, low-quality or sporadic user interactions are easily amplified layer by layer, introducing noise and reducing the stability of the representation. Therefore, while existing methods improve recommendation performance to some extent, they still struggle to simultaneously achieve effective modeling of higher-order joint relations and suppression of noise interference.

[0003] In recent years, with the development of graph neural networks, graph-based recommendation methods have achieved good results in multimodal recommendation tasks. These methods typically improve recommendation performance by constructing user-item relationship graphs and combining them with multimodal features such as visual and textual data. Generally, these methods are based on user-item interaction relationships and use graph structures for information propagation to model user preferences. However, these methods mainly construct graph structures based on pairwise interaction relationships, relying on multi-hop neighborhood aggregation to indirectly characterize higher-order relationships during information propagation, making it difficult to directly express joint interaction patterns between multiple items at the structural level. In real-world recommendation scenarios, user interests are often reflected in the common interaction behaviors of multiple items. When such higher-order relationships are decomposed into independent binary relationships, their overall structural information is easily weakened, thus affecting the model's ability to model user interests. Furthermore, in the multi-layer propagation process, low-quality or sporadic user interactions are easily amplified layer by layer, leading to noise accumulation in the representation space and reducing the model's stability and robustness. To alleviate the aforementioned problems, some methods have attempted to introduce multimodal information or improve propagation mechanisms to enhance representation capabilities, but they have still not broken through the limitations of pairwise relationship modeling at the structural level. Therefore, existing methods still have shortcomings in high-order joint relationship modeling and noise suppression, making it difficult to simultaneously achieve effective fusion of multi-source information and stable representation learning. Furthermore, how to directly model high-order joint relationships between multiple items in multimodal recommendation scenarios, and on this basis achieve robust representation propagation and multi-view alignment, has become a critical problem that urgently needs to be solved.

[0004] In summary, although existing work has made some progress in multimodal recommendation tasks, there are still problems such as insufficient ability to model high-order joint relations and difficulty in suppressing noise interference during representation propagation, which means that the accuracy and stability of recommendation results need to be further improved. Summary of the Invention

[0005] The technical problem solved by this invention is that existing multimodal recommendation methods are mainly based on the pairwise interaction relationship between users and items, which makes it difficult to effectively characterize the high-order joint interaction relationship between multiple items at the structural level. At the same time, they are easily affected by noise interaction during the representation propagation process, which weakens the high-order semantic information and thus affects the accuracy and stability of the recommendation results.

[0006] This invention provides a multimodal recommendation system based on a multi-source hypergraph, comprising: a modal consistency interaction filtering module, a hypergraph construction module, and a representation learning and recommendation module. The modal consistency interaction filtering module calculates the consistency between users and items based on the multimodal features of items, and filters user-item interaction data to obtain denoised interaction data. The hypergraph construction module constructs a behavior co-occurrence hypergraph based on the denoised interaction data and a content modal hypergraph based on the multimodal features of items, and performs node representation propagation on the behavior co-occurrence hypergraph and the content modal hypergraph. The representation learning and recommendation module performs representation learning based on the behavior co-occurrence hypergraph and the content modal hypergraph, constructs an alignment loss through multi-view contrastive learning, and fuses the node representations to obtain user representations and item representations, and generates recommendation results based on the user representations and item representations.

[0007] Furthermore, the modal consistency interaction filtering module is used to calculate the consistency between users and items based on the multimodal features of the items, and to filter the user-item interaction data to obtain denoised interaction data; given item i, its visual features are represented as follows: Text features are represented as First, the visual modal representation vectors of item i are obtained through modal mapping functions. and text modality representation vector Specifically, it is expressed as: , .in, and These represent the visual modality mapping function and the text modality mapping function, respectively. The consistency filtering submodule is used to filter interaction data based on the modal consistency between the user's historical interaction items and the target item, including: given user u, whose historical interaction sequence is... First, calculate the prototype representation of user u's preferences in the visual modality. and the prototype representation of preferences in the text modality Specifically, it is expressed as: Then, for target item i, its visual consistency score and textual consistency score with the user modality preference prototype are calculated, as follows: When an item contains multiple modalities, the consistency results of different modalities are weighted and fused, and a final mask is generated by thresholding. .in , For modal weights, This represents the consistency threshold. When only a single modality exists, the mask degenerates into a threshold determination for the corresponding modality. Based on the modality consistency results, the original interaction matrix is ​​filtered to construct a denoised interaction structure: .in For the consistency mask matrix, This represents the interaction structure after noise reduction.

[0008] Furthermore, the hypergraph construction module includes a content modality hypergraph construction submodule and a behavior co-occurrence hypergraph construction submodule. The content modality hypergraph construction submodule is used to construct a content modality hypergraph based on the visual and textual features of an item. The visual and textual features of the item are projected through a learnable linear mapping to obtain a modal representation in a unified semantic space. For item i, its modal representation is calculated as follows: ,in, and Let represent the original visual feature representation and text feature representation of item i, respectively. and The projection matrix is ​​learnable. Based on the modal representation, items are assigned to multiple content hyperedges through a learnable mapping mechanism. User nodes establish connections with their corresponding content hyperedges through their historical interaction items, thereby constructing a content modal hypergraph structure. The behavior co-occurrence hypergraph construction submodule is used to construct a behavior co-occurrence hypergraph based on user-item interaction data, including: constructing a user-item interaction matrix. ,in This indicates that user u has interacted with item i; otherwise, the value is 0. Calculate the item-item co-occurrence matrix based on the interaction matrix. .in This represents the number of times item i and item j are interacted with by the same user. The co-occurrence matrix is ​​then threshold-filtered to retain only co-occurrence relationships that meet the following conditions: The co-occurrence matrix is ​​then row-normalized to obtain the normalized co-occurrence matrix: Based on the normalized co-occurrence matrix, behavioral hyperedges are constructed, and a learnable behavioral hyperedge prototype matrix is ​​introduced. As a prototype representation of potential behavioral hyperedges, the association strength between items and behavioral hyperedges is obtained through structural propagation. .in This represents the association strength between item i and the k-th potential behavior hyperedge after the behavior co-occurrence structure propagates. Smooth item-hyperedge associations are obtained through Softmax normalization, resulting in soft connections between items and hyperedges. Based on the aforementioned association matrix, a behavioral co-occurrence hypergraph structure is constructed, and further, a node-hyperedge association matrix is ​​constructed. Define the node degree matrix. With hypermarginal matrix Given the node representation of the l-th layer Representation propagation is performed using the normalized hypergraph convolution operator to obtain the node representation of the (l+1)th layer: The propagation process enables information interaction between nodes and hyperedges, thereby fusing higher-order relationships between nodes; and through the stacking of multiple hypergraph convolutions, the node representations are aggregated layer by layer with semantic information from different hyperedges to obtain node representations from different perspectives.

[0009] Furthermore, the representation learning and recommendation module includes a contrastive learning submodule, a representation fusion submodule, and a recommendation generation submodule; the contrastive learning submodule is used to construct a multi-view contrastive learning loss based on node representations in the behavior co-occurrence hypergraph and the content modality hypergraph, and to construct a visual-text alignment loss based on node representations in the visual content hypergraph and the text content hypergraph, wherein... and These represent the hypergraph representations of user u obtained through hypergraph convolution in the visual and text hypergraphs, respectively, and are defined as follows: The behavior-content alignment loss is constructed based on node representations in the behavior co-occurrence hypergraph and the content modality hypergraph, and is defined as follows: Based on the node representations under different hypergraph structures, a structural consistency loss is constructed, and the following is assumed: and These represent the user's representation from the perspectives of two hypergraph structures, and are defined as follows: And construct a contrastive learning objective function. .in and The weighting coefficients are used to represent the node representations from different sources, resulting in user and item representations, which are represented as follows: .in This represents the representation obtained from the initial embedding learning phase. This represents the representation learned through the global hypergraph structure. To integrate weight parameters, the recommendation generation submodule is used to calculate the user's preference score for items based on the user representation and item representation, and generate recommendation results based on the preference score.

[0010] This invention provides a multimodal recommendation system based on a multi-source hypergraph, comprising: a modal consistency interaction filtering module, a hypergraph construction module, and a representation learning and recommendation module. The modal consistency interaction filtering module calculates the consistency between users and items based on the multimodal features of items, and filters user-item interaction data to obtain denoised interaction data. The hypergraph construction module constructs a behavior co-occurrence hypergraph based on the denoised interaction data and a content modal hypergraph based on the multimodal features of items, and performs node representation propagation on the behavior co-occurrence hypergraph and the content modal hypergraph. The representation learning and recommendation module performs representation learning based on the behavior co-occurrence hypergraph and the content modal hypergraph, constructs an alignment loss through multi-view contrastive learning, and fuses the node representations to obtain user representations and item representations, and generates recommendation results based on the user representations and item representations.

[0011] The beneficial effects of this invention include: (1) It can effectively improve the modeling ability of high-order joint relationships in multimodal recommendation. Most existing multimodal recommendation methods are based on the pairwise interaction relationship between users and items, which makes it difficult to directly express the joint interaction pattern between multiple items at the structural level, resulting in the weakening of high-order structural information. This invention introduces a multi-source hypergraph structure to model the high-order association relationship between multiple nodes, thereby more completely preserving the global collaborative information in user interaction behavior and improving the expressive ability of user representation. (2) It can enhance the robustness of the representation learning process and improve the stability of recommendation results. Existing methods are easily affected by low-quality or occasional user interaction behavior during representation propagation, and noise information will be gradually amplified with multi-layer propagation, thereby reducing the stability of the model. This invention uses a modal consistency interaction screening mechanism to denoise the original interaction data and combines multi-view comparative learning to constrain different representations, thereby effectively suppressing noise propagation and improving the robustness of the model and the accuracy of the recommendation results. Attached Figure Description

[0012] Figure 1 This is a framework diagram of the present invention. Detailed Implementation

[0013] As the background technology shows, existing multimodal recommendation methods struggle to fully characterize high-order joint relationships between multiple items when modeling user interests. Furthermore, they are susceptible to interference from noisy interactions during representation propagation, leading to insufficient accuracy and stability of recommendation results. The applicant has researched these issues and believes the main reasons are twofold: First, most existing methods are based on pairwise user-item interactions, making it difficult to directly express joint interactions between multiple items at the structural level, thus limiting their ability to characterize complex user interest patterns. Second, during multi-layer representation propagation, low-quality or sporadic user interactions are easily amplified layer by layer, causing noise to accumulate in the representation space and reducing the model's robustness.

[0014] To address the aforementioned issues, the applicant has conducted further research and, in this invention, provides a multimodal recommendation system and method based on a multi-source hypergraph. First, the invention filters user-item interaction data using a modality consistency interaction filtering module to suppress the interference of noisy interactions on the model. Then, a hypergraph construction module constructs a content modality hypergraph and a behavior co-occurrence hypergraph based on the filtered interaction data to characterize high-order relationships between multiple nodes, thereby enhancing the expressive power of user interests. Finally, a representation learning and recommendation module performs representation propagation and multi-view comparative learning on the multi-source hypergraph, and fuses representations from different sources to obtain user and item representations. Based on these user and item representations, a user's preference score for an item is calculated, and recommendation results are generated according to the preference score.

[0015] The spirit and essence of the present invention will be further described below with reference to the accompanying drawings and embodiments. Figure 1 As shown in the figure, the multimodal recommendation system based on a multi-source hypergraph provided in this embodiment of the invention includes: a modal consistency interaction filtering module 01, a hypergraph construction module 02, and a representation learning and recommendation module 03. The modal consistency interaction filtering module 01 is used to calculate the consistency between users and items based on the multimodal features of items, filter user-item interaction data to obtain denoised interaction data, and output it to the hypergraph construction module 02. The hypergraph construction module 02 is used to construct a content modal hypergraph and a behavior co-occurrence hypergraph based on the denoised interaction data, perform node representation propagation on the hypergraph structure, and output it to the representation learning and recommendation module 03. The representation learning and recommendation module 03 is used to perform multi-view representation learning based on the content modal hypergraph and the behavior co-occurrence hypergraph, and fuse node representations from different sources to obtain user representations and item representations, thereby generating recommendation results.

[0016] In this embodiment, the dataset used is the publicly available Amazon product review dataset, which includes three sub-datasets: Baby, Sports, and Clothing. Each dataset is preprocessed using a 5-core filtering strategy. All datasets contain both visual and textual features of the items, with a visual feature dimension of 4096 and a text feature dimension of 384. The user's historical interaction data is randomly divided into a training set, a validation set, and a test set, with a ratio of 8:1:1.

[0017] In this embodiment, the modal consistency interaction filtering module is used to calculate the consistency between users and items based on the multimodal features of items, and to filter user-item interaction data to obtain denoised interaction data. Specifically, in the modal representation construction submodule, given item i, its visual features are represented as follows: Text features are represented as First, the visual modal representation vectors of item i are obtained through modal mapping functions. and text modality representation vector Specifically, it is expressed as: , .in, and These represent the visual modality mapping function and the text modality mapping function, respectively. The consistency filtering submodule is used to filter interaction data based on the modal consistency between the user's historical interaction items and the target item, including: given user u, whose historical interaction sequence is... First, calculate the prototype representation of user u's preferences in the visual modality. and the prototype representation of preferences in the text modality Specifically, it is expressed as: Then, for target item i, its visual consistency score and textual consistency score with the user modality preference prototype are calculated, as follows: When an item contains multiple modalities, the consistency results of different modalities are weighted and fused, and a final mask is generated by thresholding. .in , For modal weights, This represents the consistency threshold. When only a single modality exists, the mask degenerates into a threshold determination for the corresponding modality. Based on the modality consistency results, the original interaction matrix is ​​filtered to construct a denoised interaction structure: .in For the consistency mask matrix, This represents the interaction structure after noise reduction.

[0018] In this embodiment, the hypergraph construction module includes a content modality hypergraph construction submodule and a behavior co-occurrence hypergraph construction submodule. The content modality hypergraph construction submodule is used to construct a content modality hypergraph based on the visual and textual features of items. During the content modality hypergraph construction process, the visual and textual features of the items are first projected using a learnable linear mapping to obtain a modal representation in a unified semantic space. For item i, its modal representation is calculated as follows: ,in, and Let represent the original visual feature representation and text feature representation of item i, respectively. and This is a learnable projection matrix. After obtaining the modal representation in a unified semantic space, items are assigned to multiple content hyperedges through a learnable mapping mechanism. User nodes establish connections with the corresponding content hyperedges through their historical interaction items, thereby constructing a content modal hypergraph structure.

[0019] Furthermore, in the behavior co-occurrence hypergraph construction submodule, a behavior co-occurrence hypergraph is constructed based on user-item interaction data. This involves building a user-item interaction matrix. ,in This indicates that user u has interacted with item i; otherwise, it is 0. Then, the item-item co-occurrence matrix is ​​calculated based on the interaction matrix. .in This represents the number of times item i and item j are interacted with by the same user. Then, the co-occurrence matrix is ​​used for threshold filtering, retaining only co-occurrence relationships that meet the following conditions: The co-occurrence matrix is ​​then row-normalized to obtain the normalized co-occurrence matrix: Behavioral hyperedges are constructed based on normalized co-occurrence matrices, and a learnable prototype matrix for behavioral hyperedges is introduced. As a prototype representation of potential behavioral hyperedges, the association strength between items and behavioral hyperedges is obtained through structural propagation. .in This represents the association strength between item i and the k-th potential behavior hyperedge after the behavior co-occurrence structure propagates. Smooth item-hyperedge associations are obtained through Softmax normalization, resulting in soft connections between items and hyperedges. Based on the aforementioned association matrix, a behavioral co-occurrence hypergraph structure is constructed, and further, a node-hyperedge association matrix is ​​constructed. Define the node degree matrix. With hypermarginal matrix Given the node representation of the l-th layer Representation propagation is performed using the normalized hypergraph convolution operator to obtain the node representation of the (l+1)th layer: In the aforementioned propagation process, nodes and hyperedges interact to exchange information, thereby fusing higher-order relationships between nodes. Through the stacking of multiple hypergraph convolutions, the node representations aggregate semantic information from different hyperedges layer by layer. The above representation propagation process is performed on the visual content hypergraph, text content hypergraph, and behavior co-occurrence hypergraph respectively to obtain node representations from different perspectives.

[0020] In this embodiment, the representation learning and recommendation module includes a contrastive learning submodule, a representation fusion submodule, and a recommendation generation submodule. The contrastive learning submodule constructs a multi-view contrastive learning mechanism based on node representations in the behavior co-occurrence hypergraph and the content modality hypergraph. Specifically, in the contrastive learning submodule, a visual-text alignment loss is first constructed based on node representations in the visual content hypergraph and the text content hypergraph; wherein... and These represent the hypergraph representations of user u obtained through hypergraph convolution in the visual and text hypergraphs, respectively. The visual-text alignment loss is calculated using a similarity function: Furthermore, a behavior-content alignment loss is constructed based on the node representations in the behavior co-occurrence hypergraph and the content modality hypergraph, and this loss is calculated using a similarity function: Furthermore, structural consistency constraints are constructed between different hypergraph structures. By aligning the node representations from different structural perspectives, the structural consistency loss is obtained: and These represent the user's representation from the perspectives of two hypergraph structures, and are defined as follows: Based on the above alignment constraints, a contrastive learning objective function is constructed, and different alignment losses are weighted and combined: .in and For weighting coefficients; further, it indicates that the fusion submodule fuses node representations from different sources to obtain user and item representations: .in This represents the representation obtained from the initial embedding learning phase. This represents the representation learned through the global hypergraph structure. To integrate the weight parameters, the recommendation generation submodule then calculates the user's preference score for the item based on the user representation and the item representation, and generates a recommendation result based on the preference score.

[0021] To verify the multimodal recommendation system based on multi-source hypergraphs provided in this embodiment, the publicly available Amazon product review dataset was selected for experimentation, including three sub-datasets: Baby, Sports, and Clothing. The data statistics are shown in Table 1. During the experiment, a control experiment was set up to compare the method of this invention with various baseline models, including collaborative filtering-based methods BPR, graph neural network-based methods LightGCN, SGL, and NCL, and multimodal recommendation methods VBPR, MMGCN, and GRCN. For evaluation metrics, Recall@K and NDCG@K were used to evaluate the performance of different models on the Top-K recommendation task. Experimental results show that the recommendation performance of the method of this invention is superior to the baseline models on all datasets, achieving optimal results under different Top-K settings. The performance improvement is shown in Table 2.

[0022] Table 1 shows the statistics of the two public benchmark datasets used in the experiment.

[0023] Table 2 shows the overall performance of GHRL compared to other baseline models on three datasets.

[0024]

[0025] In the ablation experiments, the roles of each functional module were verified by constructing different variant models. In the variant that removes multimodal information (w / o MM), the model models only based on user-item interactions, and its performance decreases across all datasets, indicating that multimodal semantic information can effectively improve representation learning ability. In the variant that removes behavioral co-occurrence hypergraph branches (w / o BHC), the model performance decreases across all datasets, indicating that high-order collaborative relationships based on user behavior play an important role in model performance. In the variant that removes the modal consistency interaction filtering mechanism (w / o MID), the model is trained directly using the original interaction data, and its performance decreases, indicating that processing interaction data through a consistency filtering mechanism can effectively improve model stability. In the variant that removes the contrastive learning mechanism (w / o HCL), the model only fuses multi-source representations, and its performance decreases, indicating that the multi-view contrastive learning mechanism helps improve the consistency between different representations. Furthermore, when only single-modal information is retained, the model performance is lower than the multimodal joint modeling results, indicating that different information sources have complementary roles in structural modeling.

[0026] Table 3 Experimental Comparison Results

[0027]

[0028] In summary, this invention analyzes and explores user-item interaction behavior in multimodal recommendation scenarios and proposes a multimodal recommendation method based on multi-source hypergraphs. By constructing a content modality hypergraph and a behavior co-occurrence hypergraph, high-order relationships between multiple nodes are modeled. A modality consistency interaction filtering mechanism is used to denoise the original interaction data, improving the stability of representation learning. Furthermore, a multi-view contrastive learning mechanism is introduced to align and optimize node representations from different sources. User and item representations are obtained through representation fusion, and user preference scores for items are calculated based on these representations to generate recommendation results. The generated recommendation results outperform existing methods in both accuracy and stability.

[0029] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any person skilled in the art can make possible variations and modifications to the invention using the disclosed methods and techniques without departing from the spirit and scope of the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the invention, are all within the protection scope of the present invention.

Claims

1. A multimodal recommendation system based on multi-source hypergraphs, characterized in that, include: Modality consistency interaction filtering module, hypergraph construction module, representation learning and recommendation module; The modal consistency interaction filtering module is used to calculate the consistency between users and items based on the multimodal features of items, and to filter user-item interaction data to obtain denoised interaction data; the hypergraph construction module is used to construct a behavior co-occurrence hypergraph based on the denoised interaction data, and to construct a content modality hypergraph based on the multimodal features of items, and to perform node representation propagation on the behavior co-occurrence hypergraph and the content modality hypergraph; the representation learning and recommendation module is used to perform representation learning based on the behavior co-occurrence hypergraph and the content modality hypergraph, and to generate recommendation results based on user representation and item representation.

2. The multimodal recommendation system based on multi-source hypergraphs according to claim 1, characterized in that... The modal consistency interaction filtering module is used to calculate the consistency between users and items based on the multimodal features of items, and to filter user-item interaction data to obtain denoised interaction data; including: given item i, its visual features are represented as Text features are represented as First, the visual modal representation vectors of item i are obtained through modal mapping functions. and text modality representation vector Specifically, it is expressed as: , .in, and Let represent the visual modal mapping function and the text modal mapping function, respectively; then, calculate the user's preference prototype representation in different modalities based on the modal representation of the user's historical interaction items; given user u, its historical interaction sequence is... First, calculate the prototype representation of user u's preferences in the visual modality. and the prototype representation of preferences in the text modality Specifically, it is expressed as: Then, for target item i, its visual consistency score and textual consistency score with the user modality preference prototype are calculated, as follows: When an item contains multiple modalities, the consistency results of different modalities are weighted and fused, and a final mask is generated by thresholding. .in , For modal weights, The consistency threshold is used. When only a single modality exists, the mask degenerates into a threshold determination for the corresponding modality. Finally, the original user-item interaction data is filtered based on the consistency mask to obtain denoised interaction data.

3. The content modality hypergraph construction submodule according to claim 1 is used to construct a content modality hypergraph based on the visual and textual features of an item, comprising: Visual and textual features of objects are projected through a learnable linear mapping to obtain modal representations in a unified semantic space. For item i, its modal representation is calculated as follows: ,in, and Let represent the original visual feature representation and text feature representation of item i, respectively. and The projection matrix is ​​learnable. Based on the modal representation, items are assigned to multiple content hyperedges through a learnable mapping mechanism. User nodes establish connections with their corresponding content hyperedges through their historical interaction items, thereby constructing a content modal hypergraph structure. The content hypergraph construction process for text modality is completely consistent with that for visual modality. The behavior co-occurrence hypergraph construction submodule is used to construct a behavior co-occurrence hypergraph based on user-item interaction data, including: constructing a user-item interaction matrix. ,in This indicates that user u has interacted with item i; otherwise, it is 0. Based on this interaction matrix, calculate the item-item co-occurrence matrix: .in This represents the number of times item i and item j are interacted with by the same user. The co-occurrence matrix is ​​then threshold-filtered to retain only co-occurrence relationships that meet the following conditions: Subsequently, we performed row normalization on the co-occurrence matrix and threshold filtering on the co-occurrence matrix, retaining only co-occurrence relationships that met the following conditions: The normalized co-occurrence matrix serves as the foundational structure at the behavioral level. After obtaining the normalized co-occurrence structure, the co-occurrence relationships of items are constructed as a hypergraph structure based on the normalized co-occurrence matrix. Specifically, a learnable behavioral hyperedge prototype matrix is ​​introduced. As a prototype representation of potential behavioral hyperedges, and using a normalized co-occurrence matrix for structural propagation, the association strength between items and behavioral hyperedges is obtained: .in This represents the association strength between item i and the k-th potential behavior hyperedge after the behavior co-occurrence structure propagates. Smooth item-hyperedge associations are obtained through Softmax normalization, resulting in soft connections between items and hyperedges. The aforementioned This is the item-hyperedge association matrix for the behavior hypergraph. User nodes establish connections with behavior hyperedges through their historical interactions, thereby constructing a behavior co-occurrence hypergraph structure. For any hypergraph structure... Construct a node-hyperedge association matrix Representation. Define the node degree matrix. With hypermarginal matrix Given the node representation of the l-th layer Representation propagation is performed using the normalized hypergraph convolution operator to obtain the node representation of the (l+1)th layer: The propagation process enables bidirectional information transfer between nodes, hyperedges, and nodes, thereby fusing higher-order relationships between multiple nodes. Through the stacking of multiple layers of hypergraph convolution, the node representation aggregates semantic information from different hyperedges layer by layer. The above representation propagation process is performed on the visual content hypergraph, text content hypergraph, and behavior co-occurrence hypergraph respectively to obtain node representations from different perspectives.

4. The representation learning and recommendation module according to claim 1 is used to perform representation learning based on the behavior co-occurrence hypergraph and the content modality hypergraph and generate recommendation results, including: Based on the node representations in the visual hypergraph and the text hypergraph, a visual-text alignment loss is constructed, whereby... and These represent the hypergraph representations of user u obtained through hypergraph convolution in the visual and text hypergraphs, respectively, and are defined as follows: .in sim(·) represents the similarity function. Let be the temperature coefficient. The visual-text alignment loss at the item level is defined in the same form. Based on the node representations in the behavior co-occurrence hypergraph and the content modality hypergraph, the behavior-content alignment loss is constructed and defined as: .in This represents the representation that user u learns in the behavior co-occurrence hypergraph. This represents the user's representation from the perspective of a content hypergraph; the behavior-content alignment loss at the item level also adopts the above form. Based on node representations under different hypergraph structures, a structural consistency loss is constructed, and we set... and Let represent the representations learned by user u from two different hypergraph structure perspectives, defined as follows: The comparative learning objective function is: .in and represents the weighting coefficients. Node representations from different sources are fused to obtain user and item representations, which are represented as follows: .in This represents the representation obtained from the initial embedding learning phase. This represents the representation learned through the global hypergraph structure. Hyperparameters are used to control the contribution of global hypergraph information. Based on the user and item representations, user preference scores for items are calculated, and recommendation results are generated.

5. The multimodal recommendation method based on multi-source hypergraphs according to any one of claims 1 to 4, characterized in that, include: Step 1: The modal consistency interaction filtering module takes the multimodal features of the item and the user-item interaction data as input, calculates the consistency between the user and the item, filters the user-item interaction data, obtains denoised interaction data, and inputs it into the hypergraph construction module. Step 2: The hypergraph construction module takes the denoised interactive data and the multimodal features of the items as input, constructs a behavior co-occurrence hypergraph and a content modality hypergraph, and performs node representation propagation on the behavior co-occurrence hypergraph and the content modality hypergraph to obtain node representations, which are then input to the representation learning and recommendation module. Step 3: The representation learning and recommendation module takes the node representation as input, constructs an alignment loss based on multi-view contrastive learning, fuses the node representations to obtain user representation and item representation, calculates the user's preference score for the item, and outputs the recommendation result based on the preference score.

Images