Multimodal recommendation method and application based on collaborative optimization of heterogeneous and homogeneous graphs

Abstract

The present invention relates to a multimodal recommendation method and application based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. A user-item heterogeneous graph and an item-item homogeneous graph are constructed for each modality. The user-item heterogeneous graph is used to align and fuse the item representations in different modalities. A modally fused item-item homogeneous graph is generated based on the item-item homogeneous graphs of different modalities. A multimodal recommendation model is constructed and trained with preprocessed data. Content recommendation is performed using the trained multimodal recommendation model. The method is applied to a multimodal recommendation system. The method adaptively fuses user preferences for different modalities through a dynamic weight allocation strategy, significantly improving the quality of user and item representations, enhancing robustness to sparse interaction data, maximizing mutual information between different modalities, and forcing multimodal representations of the same item to be highly consistent after mapping. This effectively narrows the semantic gap between modalities such as images and text, and improves the cross-modal consistency of recommendation results.

Description

Technical Field

[0001] The present invention relates to the technical field of calculation, inference or counting, and in particular to a multimodal recommendation method and application based on collaborative optimization of heterogeneous graphs and homogeneous graphs in the field of deep learning. Background Art

[0002] Recommender systems play a key role on e-commerce websites, providing customers with accurate product information and recommendations to assist in their purchasing decisions, mimicking the support functions of traditional salespeople. They not only automatically deliver personalized recommendations, alleviating users' frustration with overwhelming amounts of information, but also provide businesses with new avenues for increasing revenue. With the continuous advancement of algorithmic technology and the rapid development of the e-commerce industry, recommendation systems have become an essential component of personalized services in various fields, such as e-commerce, financial product recommendations (including funds), and short video content recommendations. These systems help ensure a more personalized service experience for users and promote the application and development of personalized marketing strategies across various industries.

[0003] Multimodal recommendation is a common problem in recommendation systems. A multimodal recommendation system is one that can process and integrate multiple types of data, such as text, images, audio, and video. By integrating data from multiple modalities, multimodal recommendation systems can provide richer and more personalized recommendations, making it easier for users to find content that matches their interests, thus meeting their diverse needs on multimedia platforms.

[0004] However, in the process of fusing multimodal data to enhance the representation of users and items, multimodal recommendation has the following defects:

[0005] (1) Traditional modal fusion methods, such as simple weighted summation and splicing, cannot dynamically capture users’ preferences for different modalities, resulting in the underutilization of the complementarity of multimodal information;

[0006] (2) Semantic differences between different modal data, such as text and images, are not effectively aligned, which reduces the robustness and consistency of recommendation results;

[0007] (3) Existing graph neural network-based methods do not fully optimize the structures of heterogeneous and homogeneous graphs. The introduced noise edges affect representation learning and ultimately affect the recommendation effect. Summary of the Invention

[0008] To solve the problems in the prior art, the present invention provides a multimodal recommendation method and application based on collaborative optimization of heterogeneous graphs and homogeneous graphs.

[0009] The technical solution adopted by the present invention is a multimodal recommendation method based on the collaborative optimization of heterogeneous graphs and isomorphic graphs. The method constructs a user-item heterogeneous graph and an item-item isomorphic graph for each modality, aligns and fuses the item representations under different modalities using the user-item heterogeneous graph, and generates a modally fused item-item isomorphic graph based on the item-item isomorphic graphs of different modalities; constructs a multimodal recommendation model, trains it with preprocessed data, and implements content recommendation with the trained multimodal recommendation model.

[0010] Preferably, a user-item heterogeneous graph is constructed for each modality ,in, , Corresponding to different modes, here are ID mode, image mode, and text mode; and items As a node of a heterogeneous graph, , ,The initial representation of users and items is used as the representation of nodes, and the edges of the heterogeneous graph are set based on the interaction behaviors of users and items;

[0011] Construct an item-item isomorphism graph corresponding to each modality , with items As nodes of the isomorphic graph, the initial representation of the item in each modality is used as the representation of the node. The edge set of the isomorphic graph under It consists of weighted edges connecting two items, and the weight of each edge is the similarity between the item representations.

[0012] Preferably, aligning and fusing item representations in different modalities using a user-item heterogeneous graph includes the following steps:

[0013] S1.1 Using Graph Neural Networks to Represent Users in Different Modalities and the representation of objects ;

[0014] S1.2 Perform multi-level cross-modal alignment of item representations across modalities;

[0015] S1.3 Dynamically perform multi-weight cross-modal fusion of user and item representations under different modalities.

[0016] Preferably, in S1.1, a graph neural network operator is used on the user-item heterogeneous graph to aggregate target users in different modalities separately. and items The neighbor node information is used to enrich the representation of the central node, and the node representations obtained by each layer of graph convolution are summed up to obtain the target user and items In each modality's representation.

[0017] Preferably, in S1.2, a comparative hierarchical cross-modal alignment module is constructed, and the original modal features are encoded based on the graph neural network model to obtain a unimodal representation of the object. , through the mapping layer, the single modality representation of different modalities Mapped into feature spaces with different granularities, they are aligned based on the item ID modality using the InfoNCE loss function.

[0018] Preferably, in S1.3, a dynamic multi-weight cross-modal fusion module is constructed to calculate the user embedding representation for each modality , calculate the non-normalized attention weight of each modality with a fully connected layer and normalize it with a softmax function, and use the normalized attention weight and user embedding representation Calculate the weighted sum to get the fused user representation ; Use normalized attention weights and user embedding representation Calculate the weighted sum to get the fused item embedding representation .

[0019] Preferably, generating a modality-fused item-item isomorphism graph based on item-item isomorphism graphs of different modalities comprises the following steps:

[0020] S2.1 Denoise the item-item isomorphism graph for each modality;

[0021] S2.2 Perform multi-graph representation fusion on isomorphic graphs of different modalities to generate a modality-fused item-item isomorphic graph, and deploy a graph neural network on it to obtain the feature representation of the corresponding items.

[0022] Preferably, in S2.1, the k most similar items are retained for each item, the edges between any two items among these k items are retained, and other noise edges are removed.

[0023] Preferably, in S2.2, based on the obtained item-item isomorphism graphs of different modalities, the item-item isomorphism graphs of different modalities are weighted and summed according to the importance scores, and a graph neural network operator is deployed to aggregate the target items. Neighbor node information, the final target item Characterization ; The different modes here are generally ID mode, image mode, and text mode.

[0024] An application of the multimodal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs is applied to a multimodal recommendation system.

[0025] The present invention relates to a multimodal recommendation method and application based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. A user-item heterogeneous graph and an item-item homogeneous graph are constructed for each modality. The user-item heterogeneous graph is used to align and fuse item representations under different modalities. A modality-fused item-item homogeneous graph is generated based on the item-item homogeneous graphs of different modalities. A multimodal recommendation model is constructed and trained with preprocessed data. Content recommendation is implemented using the trained multimodal recommendation model. The method is applied to a multimodal recommendation system.

[0026] The beneficial effects of the present invention are:

[0027] (1) A multimodal recommendation framework that combines the collaborative optimization of heterogeneous and homogeneous graphs is proposed. Through a dynamic weight allocation strategy, the user's preferences for different modalities are adaptively integrated, significantly improving the representation quality of users and items. Compared with traditional methods, this framework can explore the complementarity of multimodal data and enhance the robustness to sparse interaction data through joint modeling of a dual-graph structure (user-item heterogeneous graph and item-item homogeneous graph).

[0028] (2) A multi-granularity cross-modal alignment method based on contrastive learning is proposed. By jointly projecting the fine-grained space (preserving visual details) and the coarse-grained space (capturing text semantics), the mutual information between different modalities is maximized. By introducing the InfoNCE loss function, the multimodal representations of the same item are forced to be highly consistent after mapping, effectively narrowing the semantic gap between modalities such as images and text, and improving the cross-modal consistency of recommendation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific implementations described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The present invention relates to a multimodal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. The method constructs a user-item heterogeneous graph and an item-item homogeneous graph for each modality, aligns and fuses item representations under different modalities using the user-item heterogeneous graph, and generates a modally fused item-item homogeneous graph based on the item-item homogeneous graphs of different modalities; constructs a multimodal recommendation model, trains it with preprocessed data, and implements content recommendation using the trained multimodal recommendation model.

[0033] The purpose of this invention is to solve the problem of how to effectively enhance and fuse the multimodal features of items, such as images and text, in multimodal recommendation scenarios, so as to obtain a better representation of users, items and their relationships, and improve the recommendation effect of the model.

[0034] This embodiment takes the Amazon public dataset as an example. In this embodiment, the steps are as follows:

[0035] (1) Data preprocessing and feature extraction;

[0036] (2) Constructing a modality-specific user-item heterogeneous graph;

[0037] (3) Use graph neural networks to obtain representations of users and items in different modalities;

[0038] (4) Multi-level cross-modal alignment of item representations in different modalities;

[0039] (5) Dynamically multi-weight cross-modal fusion of user and item representations under different modalities;

[0040] (6) Constructing a modality-specific item-item isomorphism graph;

[0041] (7) Denoising the item-item isomorphism graph of each modality;

[0042] (8) Perform multi-graph representation fusion on isomorphic graphs of different modalities to generate modality-fused item-item isomorphic graphs;

[0043] (9) Deploy a graph neural network on the modality-fused item-item isomorphism graph to obtain the feature representation of the corresponding items;

[0044] (10) Multimodal recommendation model training and prediction.

[0045] The following is a detailed explanation of each part of the implementation steps.

[0046] (1) Data preprocessing and feature extraction;

[0047] (1-1) All user and item interaction records in Amazon's public dataset are sorted out, including user ,thing , when the user and items When the interaction behavior between is "click", , recorded as positive sample ,on the contrary, , recorded as negative samples The data set compiled according to the above method is recorded as ,in are positive and negative samples respectively.

[0048] Initialize the ID representation mapping tables for users and items respectively ,in is the dimension of the representation vector.

[0049] (1-2) Encode the item images in the Amazon public dataset through the pre-trained ResNet-152 to obtain the initial representation of each item in the image feature mode. The initialization representation in the image mode is denoted as ,in The dimension of image representation.

[0050] (1-3) Encode the item descriptions in the Amazon public dataset through the pre-trained Sentence-Bert to obtain the initial representation of each item in the text feature mode. The initial representation under the text feature mode is recorded as ,in Dimensions of text representation.

[0051] (2) Constructing a modality-specific user-item heterogeneous graph ;

[0052] Based on the user-item interaction records of Amazon's public dataset, a user-item heterogeneous graph can be constructed. , where users and items , as nodes of the heterogeneous graph, the initial representations of users and items serve as node representations. The edges of the heterogeneous graph are determined by the interaction between users and items. If the interaction behavior is "click", an unweighted edge is connected to the corresponding user and item nodes;

[0053] In order to aggregate user and item nodes to information under different modalities, a modality-specific user-item heterogeneous graph is constructed based on each individual modality, denoted as , where the node representation of the item node in each modality-specific heterogeneous graph is the initialization representation of the item , , Indicates ID modal, represents the image modality, Represents a text modal.

[0054] (3) Use graph neural networks to obtain representations of users and items in different modalities ;

[0055] In order to accurately predict user Whether to click on the item , it is necessary to inject the joint filtering information of users and items into their respective representations , and the user-item heterogeneous graph contains a large amount of joint filtering information. Therefore, deploying graph neural network operators on the heterogeneous graph obtained in (2) can aggregate target users under different modalities. and items Neighbor node information;

[0056] Specifically, the following formula is used to calculate the Target users of the graph convolution layer and items Characterization ,

[0057]

[0058]

[0059] in Representing users and items The set of neighbor nodes (also items or users, represented by vectors), They are the neighbor nodes of the item and the neighbor nodes of the user in the Node representation in the layer graph convolution layer; in order to obtain multi-level joint filtering information representation, the node representation obtained by each layer graph convolution is summed to obtain,

[0060]

[0061]

[0062] After performing graph convolution on the user-item heterogeneous graph under each modality, the target user is obtained and items The representation of each mode is .

[0063] (4) Multi-level cross-modal alignment of item representations in different modalities;

[0064] Contrastive Hierarchical Cross-Modal Alignment (CHCA) module aims to reduce the semantic gap between different modalities and enhance cross-modal consistency between modalities through contrastive learning;

[0065] The CHCA module includes a sequentially connected encoding model based on a graph neural network and a contrastive loss unit. The results of the contrastive loss unit training are fed back to the output of the encoding model based on the graph neural network. ;

[0066] Specifically, the CHCA module first uses a graph neural network-based model to encode the original modal features to obtain a unimodal representation of the item. , then the representations of different modalities are mapped to feature spaces of different granularities through the mapping layer for alignment. Considering the intrinsic connection between different modalities of items, the item ID (ID modality) is used as a bridge. In this embodiment, the embeddings of visual and textual modalities are aligned with the item ID embedding in different distinguishable spaces, which are defined as follows:

[0067]

[0068]

[0069] function and is a simple mapping function that aims to transform item representation from one feature space to another, where They represent different distinguishable spaces.

[0070] For the visual modality data of items, since it usually contains rich detailed information such as different objects, colors and textures in the image, it is necessary to map the visual data into a fine-grained space. In order to ensure the consistency of visual embedding and ID embedding of the same object in this space, the contrast loss unit is aligned with the InfoNCE loss function. The specific formula is as follows:

[0071] L CL v &= -log exp CosSim h c id → id ⋅ v , h c v → id ⋅ v / τ ∑ c ' ∈ [ Q ] exp CosSim h c id → id ⋅ v , h c ' v → id ⋅ v / τ

[0072] in, c ' ∈ [ Q ] Represents a dataset that is different from Items, are predefined parameters; Indicates that the ID modal and image modality The representations are mapped to the fine-grained space ;

[0073] In contrast, the text modality provides a more abstract, semantically rich but less detailed description, so the text embedding is projected into a coarse-grained feature space. In this space, the contrastive loss unit also defines the InfoNCE loss function to guide the alignment between text embedding and other modality embeddings. The specific formula is as follows:

[0074] L CL t = -log exp CosSim h i id → id ⋅ v ⋅ t , h i t → id ⋅ v ⋅ t / τ ∑ j ∈ [ Q ] exp CosSim h i id → id ⋅ v ⋅ t , h j t → id ⋅ v ⋅ t / τ

[0075] in, Indicates that the text mode ID Modal The representations are mapped to the coarse-grained feature space , the rest is the same as above;

[0076] Afterwards, and The final contrastive learning loss is obtained by adding them together. The specific formula is as follows:

[0077]

[0078] (5) Dynamically multi-weight cross-modal fusion of user and item representations under different modalities;

[0079] The dynamic multi-weight cross-modal fusion (DyMFuse) module aims to effectively combine user preference information from different modalities;

[0080] Specifically, the module first calculates the user embedding representation for each modality, such as visual modality, text modality and item ID modality, and obtains , and then use a fully connected layer to calculate the unnormalized attention weight of each modality as follows:

[0081]

[0082] in represents the weight of the fully connected layer;

[0083] In order to prevent the weights of some modalities from fluctuating drastically during training, the softmax function is applied to normalize the non-normalized attention weights. The formula is as follows:

[0084]

[0085] Finally, the attention weights and modality embeddings are used to calculate the weighted sum to obtain the fused user representation. ,

[0086]

[0087] Then, the item embedding representation is calculated for each modality, such as visual modality, text modality and item ID modality, to obtain , and then use a fully connected layer to calculate the unnormalized attention weight of each modality as follows:

[0088]

[0089] in represents the weight of the fully connected layer;

[0090] In order to prevent the weights of some modalities from fluctuating drastically during training, the softmax function is applied to normalize the non-normalized attention weights. The formula is as follows:

[0091]

[0092] Finally, the attention weights and modality embeddings are used to calculate the weighted sum to obtain the fused user representation. ,

[0093]

[0094] In this way, the DyMFuse module is able to dynamically adjust the fusion weights of modal features to generate richer user and item representations.

[0095] (6) Constructing a modality-specific item-item isomorphism graph;

[0096] Item collection based on Amazon public dataset , construct a modality-specific item-item isomorphism graph , where items As nodes of the heterogeneous graph, the initial representation of the item in each modality is used as the representation of the node; The edge set of the isomorphic graph under It is composed of weighted edges connecting two items. The weight of each edge is the similarity between the item representations. The specific formula is as follows:

[0097]

[0098] (7) Denoising the item-item isomorphism graph of each modality;

[0099] The item-item isomorphism graph obtained in step (6) is a weighted fully connected graph with an edge set contains many noisy edges. Including these noisy edges in the learning process may hinder the discovery of collaborative signals between items, so the most similar k items are retained for each item ( ), and convert the weighted graph into an unweighted graph according to the following formula:

[0100]

[0101] (8) Perform multi-graph representation fusion on isomorphic graphs of different modalities to generate modality-fused item-item isomorphic graphs;

[0102] Here we have obtained the item-item isomorphism graph of both visual and textual modalities , given that different modalities contribute differently to the final recommendation results, we score them according to their importance and Perform weighted summation on the item-item graphs of the two modalities:

[0103]

[0104] in, , and correspond The two modes of m.

[0105] (9) Deploy a graph neural network on the modality-fused item-item isomorphism graph to obtain the feature representation of the corresponding items;

[0106] In order to obtain joint filtering information based on the item-item isomorphism graph, the graph neural network operator in step (3) is used to aggregate the target items. Neighbor node information, the final target item Characterized by .

[0107] Specifically, the following formula is used to calculate the Target objects of the convolutional layer Characterization ,

[0108]

[0109] in Representative items The set of neighbor nodes of Is the neighbor node of the item in Node representation in the layer graph convolution layer; in order to obtain multi-level joint filtering information representation, the node representation obtained by each layer graph convolution is summed to obtain,

[0110]

[0111] (10) Multimodal recommendation model training and prediction;

[0112] Get the user from the above steps Characterization , Item representation based on user-item heterogeneous graph and item representation based on item-item isomorphism graph Calculate the user according to the following formula Click on the item Score:

[0113]

[0114] In order to fully train the multimodal recommendation model to obtain a more realistic click-through rate score , using Bayesian personalized ranking loss (BPR Loss) as the optimization function for the recommendation task. The specific formula is as follows:

[0115]

[0116] The dataset based on positive and negative samples constructed in (1) is used as the training data for this optimization task. In order to prevent overfitting from occurring during the model training process, all model parameters are Add regular terms ;

[0117] Therefore, the multimodal recommendation model is composed of the optimization function of the recommendation task and multi-level contrastive learning loss constitute:

[0118]

[0119] in are the hyperparameters that need to be tuned.

[0120] The present invention also relates to an application of the multimodal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs, which is applied to a multimodal recommendation system.

[0121] The multimodal recommendation system finally applied by the present invention can be deployed on a server with the following configuration:

[0122] Use high-performance multi-core CPUs, such as the Intel Xeon series or AMD EPYC series, to support large-scale parallel computing tasks. These processors have high clock speeds and large caches, enabling them to efficiently handle data-intensive tasks.

[0123] A large amount of DDR4 or DDR5 RAM, typically ranging from 64GB to 512GB, ensures the system can maintain high speeds while processing massive amounts of user data;

[0124] Storage: Use solid-state drives (SSDs) or NVMe SSDs as the primary storage medium to provide fast data read and write speeds. Given the large amount of data, a distributed storage solution such as Ceph or Hadoop HDFS can be used to ensure reliable data storage and fast access.

[0125] Use high-performance multi-GPU cards, such as AI processors from AMD, NVIDIA, or Huawei's Ascend series, to support large-scale matrix operations.

[0126] The application process of the present invention also involves a computer-readable storage medium, on which is stored a multimodal recommendation program based on the collaborative optimization of heterogeneous graphs and homogeneous graphs. When the program is executed by a processor, the multimodal recommendation method based on the collaborative optimization of heterogeneous graphs and homogeneous graphs is implemented;

[0127] It also relates to a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multimodal recommendation method based on collaborative optimization of heterogeneous graphs and homogeneous graphs is implemented.

[0128] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1The steps for the function specified in one or more boxes.

[0132] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0133] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs, characterized by: The method constructs a user-item heterogeneous graph and an item-item isomorphic graph for each modality, aligns and fuses the item representations under different modalities using the user-item heterogeneous graph, and generates a modality-fused item-item isomorphic graph based on the item-item isomorphic graphs of different modalities. Aligning and fusing item representations from different modalities using the user-item heterogeneous graph includes the following steps: S1.1 Using Graph Neural Networks to Represent Users in Different Modalities and the representation of objects ; S1.2 Perform multi-level cross-modal alignment of item representations in different modalities: Construct a comparative hierarchical cross-modal alignment module, and encode the original modal features based on the graph neural network model to obtain the unimodal representation of the object. , through the mapping layer, the single modality representation of different modalities Mapped into feature spaces of different granularities, aligned based on item ID modality using the InfoNCE loss function; S1.3 Dynamically multi-weight cross-modal fusion of user and item representations in different modalities: Construct a dynamic multi-weight cross-modal fusion module to calculate user embedding representation for each modality , calculate the non-normalized attention weight of each modality with a fully connected layer and normalize it with a softmax function, and use the normalized attention weight and user embedding representation Calculate the weighted sum to get the fused user representation ; Use normalized attention weights and user embedding representation Calculate the weighted sum to get the fused item embedding representation ; Build a multimodal recommendation model, train it with preprocessed data, and use the trained multimodal recommendation model to implement content recommendation.

2. The multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs according to claim 1, characterized in that: Construct a user-item heterogeneous graph for each modality ,in, , Corresponding to different modes respectively; based on user and items As a node of a heterogeneous graph, , ,The initial representation of users and items is used as the representation of nodes, and the edges of the heterogeneous graph are set based on the interaction behaviors of users and items; Construct an item-item isomorphism graph corresponding to each modality , with items As nodes of the isomorphic graph, the initial representation of the item in each modality is used as the representation of the node. The edge set of the isomorphic graph under It consists of weighted edges connecting two items, and the weight of each edge is the similarity between the item representations.

3. The multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs according to claim 1, characterized in that: In S1.1, graph neural network operators are used on the user-item heterogeneous graph to aggregate target users under different modalities separately. and items The neighbor node information of the target user is obtained by summing up the node representations obtained by each layer of graph convolution. and items In each modality's representation.

4. The multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs according to claim 2, characterized in that: Generating a modality-fused item-item isomorphism graph based on item-item isomorphism graphs of different modalities includes the following steps: S2.1 Denoise the item-item isomorphism graph for each modality; S2.2 Perform multi-graph representation fusion on isomorphic graphs of different modalities to generate a modality-fused item-item isomorphic graph, and deploy a graph neural network on it to obtain the feature representation of the corresponding items.

5. The multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs according to claim 4, characterized in that: In S2.1, for each item, the k most similar items are retained, the edges between any two items among these k items are retained, and other noise edges are removed.

6. The multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs according to claim 4, characterized in that: In S2.2, based on the obtained item-item isomorphism graphs of different modalities, the item-item isomorphism graphs of different modalities are weighted and summed according to the importance score, and the graph neural network operator is deployed to aggregate the target items. Neighbor node information, the final target item Characterization .

7. An application of the multimodal recommendation method based on collaborative optimization of heterogeneous and homogeneous graphs according to any one of claims 1 to 6, characterized in that: Application in multimodal recommendation systems.

Images