Multi-modal recommendation method based on heterogeneous graph and homomorphic graph collaborative optimization and application

Abstract

The invention relates to a multi-modal recommendation method based on heterogeneous graph and homomorphic graph collaborative optimization and application, a user-article heterogeneous graph and an article-article homomorphic graph are respectively constructed corresponding to each modal, the user-article heterogeneous graph is used for aligning and fusing article representations in different modals, and the user-article homomorphic graph and the article-article homomorphic graph are combined. Based on the article-article isomorphic composition of different modalities, generating an article-article isomorphic composition of modal fusion; a multi-modal recommendation model is constructed, the preprocessed data are trained, and content recommendation is achieved through the trained multi-modal recommendation model; the method is applied to a multi-modal recommendation system. According to the method, preferences of users to different modes are adaptively fused through a dynamic weight distribution strategy, the characterization quality of the users and the articles is remarkably improved, the robustness of sparse interaction data is enhanced, mutual information among different modes is maximized, multi-mode characterization of the same article is forced to be highly consistent after mapping, and the user experience is improved. Semantic gaps between modes such as images and texts are effectively reduced, and cross-mode consistency of recommendation results is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computing, reckoning or counting, and particularly to a multi-modal recommendation method and application based on collaborative optimization of heterogeneous graphs and homogeneous graphs in the field of deep learning. Background Art

[0002] Recommendation systems play a key role in e-commerce websites. By providing accurate product information and suggestions to customers to assist their purchase decisions, they simulate the support functions of traditional salespersons. They can not only automatically achieve personalized recommendations, reducing the trouble for users when facing a vast amount of information, but also open up new ways for enterprises to increase revenue. With the continuous progress of algorithm technology and the rapid development of the e-commerce industry, recommendation systems have now become an important part of providing personalized services in multiple fields. For example, in scenarios such as e-commerce, financial product recommendations (including funds, etc.), and short video content recommendations, they are all indispensable. Such systems help to ensure that users obtain a more considerate service experience and at the same time promote the application and development of personalized marketing strategies in various industries.

[0003] Multi-modal recommendation is a common problem in recommendation systems. A multi-modal recommendation system refers to a recommendation system that can process and fuse multiple types of data (such as text, images, audio, and video). By integrating data from multiple modalities, multi-modal recommendations can provide richer and more personalized recommendations, making it easier for users to find content that matches their interests and meeting the diverse needs of users on multimedia platforms.

[0004] However, in the process of fusing multi-modal data to enhance the representations of users and items, multi-modal recommendations have the following defects:

[0005] (1) Traditional modality fusion methods, such as simple weighted summation, concatenation, etc., cannot dynamically capture users' preferences for different modalities, resulting in the underutilization of the complementarity of multi-modal information;

[0006] (2) The semantic differences between different modality data, such as text and images, are not effectively aligned, reducing the robustness and consistency of the recommendation results;

[0007] (3) Existing graph neural network-based methods do not fully optimize the structures of heterogeneous graphs and homogeneous graphs, and the introduced noisy edges affect the representation learning, ultimately affecting the recommendation effect. Summary of the Invention

[0008] To solve the problems in the prior art, the present invention provides a multi-modal recommendation method and application based on collaborative optimization of heterogeneous graphs and homogeneous graphs.

[0009] The technical solution adopted by the present invention is a multi-modal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs. For each modality, a user-item heterogeneous graph and an item-item homogeneous graph are constructed respectively. The user-item heterogeneous graph is used to align and fuse the item representations in different modalities, and a modality-fused item-item homogeneous graph is generated based on the item-item homogeneous graphs in different modalities. A multi-modal recommendation model is constructed and trained with the preprocessed data, and content recommendation is implemented with the trained multi-modal recommendation model.

[0010] Preferably, a user-item heterogeneous graph is constructed for each modality , where , correspond to different modalities, which are the ID modality, the image modality, and the text modality here respectively; using the user and the item as the nodes of the heterogeneous graph, , , the initial representations of the user and the item are used as the representations of the nodes, and the edges of the heterogeneous graph are set based on the interaction behaviors between the user and the item;

[0011] An item-item homogeneous graph is constructed for each modality , using the item as the nodes of the homogeneous graph, the initial representations of the item in each modality are used as the representations of the nodes, and the edge set of the homogeneous graph in the modality is composed of weighted edges connecting the items pairwise, and the weight of each edge is the similarity between the item representations.

[0012] Preferably, aligning and fusing the item representations in different modalities with the user-item heterogeneous graph includes the following steps:

[0013] S1.1 Obtain the representations of the user and the item in different modalities using a graph neural network and ;

[0014] S1.2 Perform multi-level cross-modal alignment on the item representations in different modalities;

[0015] S1.3 Perform dynamic multi-weight cross-modal fusion on the representations of the user and the item in different modalities.

[0016] Preferably, in S1.1, a graph neural network operator is used on the user-item heterogeneous graph to respectively aggregate the neighbor node information of the target user and the item in different modalities to enrich the representation of the central node, and the node representations obtained by each layer of graph convolution are respectively summed to obtain the target user and the item In the representation of each modality.

[0017] Preferably, in S1.2, a contrastive hierarchical cross-modal alignment module is constructed, and a model based on a graph neural network encodes the original modal features to obtain the unimodal representation of the item , and the unimodal representations of different modalities are mapped into feature spaces with different granularities through a mapping layer, and alignment is performed based on the item ID modality with the InfoNCE loss function.

[0018] Preferably, in S1.3, a dynamic multi-weight cross-modal fusion module is constructed to calculate the user embedding representation for each modality , calculate the unnormalized attention weights for each modality with a fully connected layer and normalize them with the softmax function, and use the normalized attention weights and the user embedding representation to calculate the weighted sum to obtain the fused user representation ; use the normalized attention weights and the user embedding representation to calculate the weighted sum to obtain the fused item embedding representation .

[0019] Preferably, generating a modality-fused item-item homogeneous graph based on the item-item homogeneous graphs of different modalities includes the following steps:

[0020] S2.1 Denoise the item-item homogeneous graph of each modality;

[0021] S2.2 Perform multi-graph representation fusion on the homogeneous graphs of different modalities to generate a modality-fused item-item homogeneous graph, and deploy a graph neural network on it to obtain the feature representation of the corresponding item.

[0022] Preferably, in S2.1, retain the k most similar items for each item, retain the edges between any two of these k items, and remove other noisy edges.

[0023] Preferably, in S2.2, based on the obtained item-item homogeneous graphs of different modalities, perform weighted sum and summation on the item-item homogeneous graphs of different modalities according to the importance score, and deploy a graph neural network operator to aggregate the neighbor node information of the target item to finally obtain the representation of the target item ; the different modalities here are generally the ID modality, the image modality, and the text modality.

[0024] An application of the described multi-modal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs, applied to a multi-modal recommendation system.

[0025] The present invention relates to a multi-modal recommendation method and application based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. For each modality, a user-item heterogeneous graph and an item-item homogeneous graph are respectively constructed. The user-item heterogeneous graph is used to align and fuse the item representations under different modalities, and a modality-fused item-item homogeneous graph is generated based on the item-item homogeneous graphs of different modalities. A multi-modal recommendation model is constructed and trained with the preprocessed data, and content recommendation is realized with the trained multi-modal recommendation model. The method is applied to a multi-modal recommendation system.

[0026] The beneficial effects of the present invention are as follows:

[0027] (1) A multi-modal recommendation framework for collaborative optimization combining heterogeneous graphs and homogeneous graphs is proposed. By means of a dynamic weight allocation strategy, the preferences of users for different modalities are adaptively fused, significantly improving the representation quality of users and items. Compared with traditional methods, this framework can exploit the complementarity of multi-modal data and jointly model through a dual-graph structure (user-item heterogeneous graph and item-item homogeneous graph), enhancing the robustness to sparse interaction data.

[0028] (2) A multi-granularity cross-modal alignment method based on contrastive learning is proposed. Through the joint projection of a fine-grained space (preserving visual details) and a coarse-grained space (capturing text semantics), the mutual information between different modalities is maximized. By introducing the InfoNCE loss function, the multi-modal representations of the same item are forced to be highly consistent after mapping, effectively narrowing the semantic gap between modalities such as images and texts and improving the cross-modal consistency of the recommendation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is the flowchart of the method of the present invention;

[0030] Figure 2 is the overall framework diagram of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] The present invention relates to a multi-modal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. For each modality, a user-item heterogeneous graph and an item-item homogeneous graph are respectively constructed. The user-item heterogeneous graph is used to align and fuse the item representations under different modalities, and a modality-fused item-item homogeneous graph is generated based on the item-item homogeneous graphs of different modalities. A multi-modal recommendation model is constructed and trained with the preprocessed data, and content recommendation is realized with the trained multi-modal recommendation model.

[0033] The objective of the present invention is to solve the problem of how to effectively enhance and fuse the multimodal features of items, such as images and texts, in the multimodal recommendation scenario, so as to obtain a better representation of users, items, and their relationships, and improve the recommendation effect of the model.

[0034] In this embodiment, the Amazon public dataset is taken as an example. In the embodiment, the steps are as follows:

[0035] (1) Data preprocessing and feature extraction;

[0036] (2) Construct a modality-specific user-item heterogeneous graph;

[0037] (3) Use graph neural networks to obtain the representations of users and items in different modalities;

[0038] (4) Perform multi-level cross-modal alignment on the item representations in different modalities;

[0039] (5) Perform dynamic multi-weight cross-modal fusion on the representations of users and items in different modalities;

[0040] (6) Construct a modality-specific item-item homogeneous graph;

[0041] (7) Denoise the item-item homogeneous graph for each modality;

[0042] (8) Perform multi-graph representation fusion on the homogeneous graphs in different modalities to generate a modality-fused item-item homogeneous graph;

[0043] (9) Deploy graph neural networks on the modality-fused item-item homogeneous graph to obtain the feature representations of the corresponding items;

[0044] (10) Train and predict the multimodal recommendation model.

[0045] The following elaborates in detail according to each part in the implementation steps.

[0046] (1) Data preprocessing and feature extraction;

[0047] (1-1) Organize all user and item interaction records in the Amazon public dataset, where users , items , when the interaction behavior between user and item is "click", , it is recorded as a positive sample , otherwise, , it is recorded as a negative sample ; The dataset organized according to the above method is denoted as , where They are positive and negative samples respectively.

[0048] Initialize the ID representation mapping tables of users and items respectively , where is the dimension of the representation vector.

[0049] (1-2) Encode the item images in the Amazon public dataset through the pre-trained ResNet-152 to obtain the initial representation of each item in the image feature modality, and denote the item 's initial representation in the image modality as , where is the dimension of the image representation.

[0050] (1-3) Encode the item descriptions in the Amazon public dataset through the pre-trained Sentence-Bert to obtain the initial representation of each item in the text feature modality, and denote the item 's initial representation in the text feature modality as , where is the dimension of the text representation.

[0051] (2) Construct a modality-specific user-item heterogeneous graph ;

[0052] Based on the user-item interaction records in the Amazon public dataset, a user-item heterogeneous graph can be constructed , where the user and the item , as the nodes of the heterogeneous graph, and the initial representations of the user and the item are used as the representations of the nodes. The edges of the heterogeneous graph are determined by the user-item interaction behavior. If the interaction behavior is "click", then connect an unweighted edge to the corresponding user and item nodes;

[0053] To enable the user and item nodes to aggregate information in different modalities, construct a modality-specific user-item heterogeneous graph based on each individual modality, denoted as , where the node representation of the item node in each modality-specific heterogeneous graph is the initial representation of the item , , represents the ID modality, represents the image modality, represents the text modality.

[0054] (3) Use graph neural networks to obtain the representations of users and items in different modalities ;

[0055] To accurately predict whether the user will click on the item , it is necessary to inject the joint filtering information of users and items into their respective representations , and a large amount of joint filtering information is contained in the user-item heterogeneous graph. Therefore, deploying a graph neural network operator on the heterogeneous graph obtained in (2) can aggregate the neighbor node information of the target user and item ;

[0056] Specifically, calculate the representations of the target user and item of the layer graph convolutional layer according to the following formula ,

[0057]

[0058]

[0059] where respectively represent the sets of neighbor nodes (also items or users, represented as vectors) of the user and item , are the node representations of the neighbor nodes of the item and the neighbor nodes of the user in the layer graph convolutional layer respectively; in order to obtain the multi-level joint filtering information representation, sum the node representations obtained by each layer of graph convolution to get

[0060]

[0061]

[0062] After performing graph convolution operations on the user-item heterogeneous graph in each modality, the representations of the target user and item in each modality are respectively .

[0063] (4) Perform multi-level cross-modal alignment on the item representations in different modalities;

[0064] Compared with the hierarchical cross-modal alignment module (CHCA), it aims to reduce the semantic gap between different modalities through contrastive learning and enhance the cross-modal consistency between modalities;

[0065] The CHCA module includes a graph neural network-based encoding model and a contrastive loss unit connected in sequence, and the training result of the contrastive loss unit is fed back to the output of the graph neural network-based encoding model ;

[0066] Specifically, the CHCA module first encodes the original modal features using a graph neural network-based model to obtain the unimodal representation of the item. , and then maps the representations of different modalities into feature spaces with different granularities through a mapping layer for alignment. Considering the internal connection between different modalities of the item, the item ID (ID modality) is used as a bridge. In this embodiment, the embeddings of the visual and text modalities are aligned with the item ID embedding in different distinguishable spaces, defined as follows:

[0067]

[0068]

[0069] The functions and are simple mapping functions, aiming to transform the item representation from one feature space to another, where represent different distinguishable spaces respectively.

[0070] For the visual modal data of the item, since it usually contains rich detailed information, such as different objects, colors, and textures in the image, the visual data is mapped to the fine-grained space . To ensure the consistency of the visual embedding and the ID embedding of the same item in this space, the contrastive loss unit is aligned with the InfoNCE loss function, and the specific formula is as follows:

[0071] L CL v &= -log exp CosSim h c id → id ⋅ v , h c v → id ⋅ v / τ ∑ c ' ∈ [ Q ] exp CosSim h c id → id ⋅ v , h c ' v → id ⋅ v / τ

[0072] where, c ' ∈ [ Q ] represent items different from in the dataset, is a predefined parameter; represents mapping the representations of the ID modality and the image modality to the fine-grained space ;

[0073] In contrast, the text modality provides a more abstract, semantically rich but less detailed description. Therefore, the text embedding is projected into the coarse-grained feature space . In this space, the contrastive loss unit also defines the InfoNCE loss function to guide the alignment between the text embedding and other modal embeddings, and the specific formula is as follows:

[0074] L CL t =-log exp CosSim h i id → id ⋅ v ⋅ t , h i t → id ⋅ v ⋅ t / τ ∑ j ∈ [ Q ] exp CosSim h i id → id ⋅ v ⋅ t , h j t → id ⋅ v ⋅ t / τ

[0075] Among them, represents mapping the representations of the text modality ID modality to the coarse-grained feature space , and the rest is the same as above;

[0076] Subsequently, and are added to obtain the final contrastive learning loss, and the specific formula is as follows:

[0077]

[0078] (5) Dynamically fuse the representations of users and items across different modalities with multiple weights;

[0079] The Dynamic Multi-Weight Cross-Modal Fusion (DyMFuse) module aims to effectively combine the user preference information of different modalities;

[0080] Specifically, this module first calculates the user embedding representation for each modality, such as the visual modality, text modality, and item ID modality, to obtain , and then uses a fully connected layer to calculate the unnormalized attention weights for each modality. The formula is as follows:

[0081]

[0082] where represents the weights of the fully connected layer;

[0083] To prevent the weights of certain modalities from fluctuating violently during training, the softmax function is applied to normalize the unnormalized attention weights. The formula is as follows:

[0084]

[0085] Finally, the weighted sum is calculated using the attention weights and modality embeddings to obtain the fused user representation ,

[0086]

[0087] Subsequently, the item embedding representation is calculated for each modality, such as the visual modality, text modality, and item ID modality, to obtain , and then a fully connected layer is used to calculate the unnormalized attention weights for each modality. The formula is as follows:

[0088]

[0089] Among them represents the weight of the fully connected layer;

[0090] To prevent the weights of some modalities from fluctuating violently during training, the softmax function is applied to normalize the unnormalized attention weights. The formula is as follows:

[0091]

[0092] Finally, the attention weights and modality embeddings are used to calculate the weighted sum to obtain the fused user representation ,

[0093]

[0094] In this way, the DyMFuse module can dynamically adjust the fusion weights of modality features to generate richer user and item representations.

[0095] (6) Construct modality-specific item-item homogeneous graphs;

[0096] Based on the item set of the Amazon public dataset , construct a modality-specific item-item homogeneous graph , where the item serves as the node of the heterogeneous graph, and the initial representation of the item in each modality serves as the representation of the node; the edge set of the homogeneous graph under modality is composed of weighted edges connecting items pairwise, and the weight of each edge is the similarity between item representations. The specific formula is as follows:

[0097]

[0098] (7) Denoise the item-item homogeneous graph for each modality;

[0099] The item-item homogeneous graph obtained in step (6) is a weighted complete connected graph, and its edge set contains many noisy edges. Including these noisy edges in the learning process may hinder the discovery of collaborative signals between items. Therefore, keep the k most similar items for each item( ), and convert the weighted graph to an unweighted graph according to the following formula:

[0100]

[0101] (8) Perform multi-graph representation fusion on the homogeneous graphs of different modalities to generate a modality-fused item-item homogeneous graph;

[0102] Here, the item-item homogeneous graphs of the visual and text modalities have been obtained , given that different modalities contribute differently to the final recommendation result, according to the importance score and perform weighted sum on the item-item graphs of the two modalities:

[0103]

[0104] where , and correspond to the two modalities of m in

[0105] (9) Deploy a graph neural network on the item-item isomorphic graph after modality fusion to obtain the feature representation of the corresponding item;

[0106] To obtain the collaborative filtering information based on the item-item isomorphic graph, use the graph neural network operator in step (3) to aggregate the neighbor node information of the target item , and the final obtained target item representation is .

[0107] Specifically, calculate the representation of the target item in the -th layer of the graph convolutional layer according to the following formula

[0108]

[0109] where represents the set of neighbor nodes of item , is the node representation of the neighbor nodes of the item in the -th layer of the graph convolutional layer; to obtain the multi-layer collaborative filtering information representation, sum up the node representations obtained by each layer of graph convolution to get

[0110]

[0111] (10) Train and predict the multi-modal recommendation model;

[0112] Obtain the representation of user , the item representation based on the user-item heterogeneous graph and the item representation based on the item-item isomorphic graph from the above steps, and calculate the score of user clicking on item according to the following formula:

[0113]

[0114] To fully train the multi-modal recommendation model to obtain more realistic click-through rate scores , the Bayesian Personalized Ranking Loss (BPR Loss) is used as the optimization function for the recommendation task, and the specific formula is as follows:

[0115]

[0116] The dataset based on positive and negative samples constructed in (1) is used as the training data for this optimization task. To prevent overfitting from occurring during model training, regularization terms are added to all model parameters ; ;

[0117] Therefore, the multi-modal recommendation model consists of the optimization function of the recommendation task and the multi-level contrastive learning loss :

[0118]

[0119] where are hyperparameters to be adjusted.

[0120] The present invention also relates to an application of the multi-modal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs described above, which is applied to a multi-modal recommendation system.

[0121] The multi-modal recommendation system finally applied by the present invention can be deployed on a server with the following configuration:

[0122] Adopt a high-performance multi-core CPU, such as the Intel Xeon series or the AMD EPYC series, to support large-scale parallel computing tasks; these processors have high clock frequencies and large caches, and can efficiently process data-intensive tasks;

[0123] Configure a large-capacity DDR4 or DDR5 RAM, usually ranging from 64GB to 512GB, to ensure that the system can run at high speed when processing massive user data;

[0124] Storage: Use solid-state drives (SSDs) or NVMe SSDs as the main storage medium to provide fast data read and write speeds; considering the large amount of data, a distributed storage solution such as Ceph or Hadoop HDFS can be adopted to achieve reliable data storage and fast access;

[0125] Adopt high-performance multi-card GPUs, such as AI processors launched by the AMD series, NVIDIA series, or Huawei Ascend series, to support large-scale matrix operations.

[0126] In the application process of the present invention, there is also involved a computer-readable storage medium, on which there is stored a multi-modal recommendation program based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. When this program is executed by a processor, it implements the multi-modal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs;

[0127] There is also involved a computer device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, it implements the multi-modal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs.

[0128] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0130] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1Steps of the functions specified in one or more boxes.

[0132] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0133] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-modal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs, characterized in that: The method constructs a user-item heterogeneous graph and an item-item homogeneous graph for each modality respectively, aligns and fuses the item representations in different modalities with the user-item heterogeneous graph, and generates a modality-fused item-item homogeneous graph based on the item-item homogeneous graphs of different modalities; Construct a multi-modal recommendation model, train it with the preprocessed data, and use the trained multi-modal recommendation model to implement content recommendation.

2. The multimodal recommendation method based on collaborative optimization of heterogeneous graph and homogeneous graph according to claim 1, wherein: Construct a user-item heterogeneous graph corresponding to each modality , where , correspond to different modalities respectively; taking user and item as the nodes of the heterogeneous graph, , , the initial representations of the user and the item are used as the representations of the nodes, and the edges of the heterogeneous graph are set based on the interaction behaviors between the user and the item; Construct an item-item isomorphic graph corresponding to each modality , using the item as the nodes of the isomorphic graph, and the initial representation of the item in each modality as the representation of the nodes. The edge set of the isomorphic graph in modality is composed of weighted edges that are pairwise connected between items, and the weight of each edge is the similarity between item representations.

3. The multimodal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs according to claim 2, wherein: Aligning and fusing the item representations in different modalities with the user-item heterogeneous graph includes the following steps: S1.1 Obtain the representations of users and items in different modalities using graph neural networks and the representations of items ; S1.2 Perform multi-level cross-modal alignment on the item representations in different modalities; S1.3 Dynamically perform multi-weight cross-modal fusion on the representations of users and items in different modalities.

4. The multimodal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs according to claim 3, wherein: In S1.1, graph neural network operators are used on the user-item heterogeneous graph to separately aggregate the neighbor node information of the target user and item , sum the node representations obtained by each layer of graph convolution respectively, and obtain the representations of the target user and item in each modality.

5. The multimodal recommendation method based on collaborative optimization of heterogeneous graph and homogeneous graph according to claim 3, characterized in that: In S1.2, a contrastive hierarchical cross-modal alignment module is constructed, and the original modal features are encoded by a graph neural network-based model to obtain the unimodal representation of the item. , and the unimodal representations of different modalities are mapped into feature spaces with different granularities, and alignment is performed based on the item ID modality with the InfoNCE loss function.

6. The multimodal recommendation method based on collaborative optimization of heterogeneous graph and homogeneous graph according to claim 3, wherein: In S1.3, a dynamic multi-weight cross-modal fusion module is constructed to calculate the user embedding representation for each modality , calculate the unnormalized attention weights for each modality with a fully connected layer and normalize them with the softmax function, and use the normalized attention weights and the user embedding representation to calculate the weighted sum to obtain the fused user representation ; use the normalized attention weights and the user embedding representation to calculate the weighted sum to obtain the fused item embedding representation .

7. A multi-modal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs according to claim 2, characterized in that: Generating a modality-fused item-item homogeneous graph based on the item-item homogeneous graphs of different modalities includes the following steps: S2.1 Denoise the item-item homogeneous graph of each modality; S2.2 Perform multi-graph representation fusion on the homogeneous graphs of different modalities to generate a modality-fused item-item homogeneous graph, and deploy a graph neural network on it to obtain the feature representations of the corresponding items.

8. A multimodal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs according to claim 7, characterized in that: In S2.1, retain the k most similar items for each item, retain the edges between any two of these k items, and remove other noisy edges.

9. A multimodal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs according to claim 7, characterized in that: In S2.2, based on the obtained item-item homogeneous graphs of different modalities, the item-item homogeneous graphs of different modalities are weighted and summed according to the importance scores, and a graph neural network operator is deployed to aggregate the neighbor node information of the target item Finally, the representation of the target item is obtained. .

10. Application of the multimodal recommendation method based on collaborative optimization of heterogeneous graph and homogeneous graph according to any one of claims 1 to 9, characterized in that: Applied to a multi-modal recommendation system.

Images