Commodity recommendation method oriented to multi-modal data

Abstract

The invention discloses a multi-modal data-oriented commodity recommendation method, which relates to the technical field of commodity recommendation, and comprises the following steps: constructing a user-project association diagram, a project modal association diagram and a user modal association diagram; constructing a modal intersection association graph of the project and a modal intersection association graph of the user; carrying out feature convolution on the association graph by adopting LightGCN, generating embedding for the project and the user, and implementing hierarchical contrast learning of the project and the user in each round of feature convolution; performing recommendation prediction through a prediction model with multi-dimensional loss joint training by using an inner product form as a score of a user on the item; and when convolution operation is carried out on the user-item association graph, the consistency of explicit preference and implicit preference is increased through a user preference optimization loss function. According to the method, the model scene generalization ability is high, the accuracy of project and user preference modeling is effectively improved, and the recommendation quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of commodity recommendation, and in particular to a commodity recommendation method oriented to multimodal data. Background Art

[0002] Recommendation systems are widely used in various e-commerce platforms. They predict users’ interests through historical interaction information between users and products (clicks, browsing, purchases, favorites, etc.) and recommend potential products to users.

[0003] In recent years, technicians have discovered that the various modal information contained in the display of goods, such as text, images, and videos, will have an important impact on the quality of recommendations. Different modal information has its own unique advantages in displaying product features. For example, text information is mainly used to describe the function and material of the product, image information can well display the appearance and color of the product, and video can more easily express the use of the product. Different user groups will show different sensitivities to the different modal characteristics of the product. The formation of user preferences is often affected by the joint action of multiple modal information of the product. Therefore, the recommendation that considers multiple modal information of the product (i.e., multimodal recommendation) has become a current hot technology.

[0004] In multimodal recommendation, high-quality modeling of items and user preferences is the key to improving the quality of recommendations. In terms of item modeling (i.e., the embedding representation of items), the current popular technology is to use item-item association graphs to assist in improving the quality of item embeddings. This approach usually combines the embedding representation of items in the item-item association graph under different modalities with the item embedding in the user-item graph to obtain the final item embedding.

[0005] When modeling user preferences, we mainly use convolution operations to perform convolutions on the user-item graph based on the item embeddings to obtain the user embeddings. At the same time, in order to improve the quality of recommendations, when training the recommendation model, we usually use modality alignment and user preference enhancement to construct a joint loss function to improve the quality of recommendations.

[0006] Rich modal information provides useful help for users to fully understand products. The above popular practices improve the quality of item and user preference modeling, but they still face three technical difficulties in the recommendation process, which limit the improvement of recommendation quality: First, when using the item-item association graph to assist in the generation of item embedding, it is difficult to determine the modal fusion weight, which affects the quality of item embedding generation. For the modal fusion weight, whether it is a simple average or a hyperparameter adjustment method, it will cause modal imbalance when embedding items, which is easy to cause overfitting of the training item affiliation scenario, resulting in the lack of scenario generalization ability of the trained model.

[0007] Second, users may shift their interests due to the display of a certain modal information of the project, thereby generating preference noise and reducing the quality of user preference modeling.

[0008] Third, existing technical solutions mainly enhance user preferences from the perspective of interest denoising and diversity, and lack the mining and alignment of users' explicit and implicit preferences. Summary of the invention

[0009] In order to overcome the above problems existing in the prior art, the present invention proposes a commodity recommendation method for multimodal data.

[0010] The technical solution adopted by the present invention to solve the technical problem is: a commodity recommendation method for multimodal data, comprising the following steps: Step 1, constructing a user-item association graph, an item modal association graph, and a user modal association graph; Step 2, constructing a modal intersection association graph of the project and a modal intersection association graph of the user; Step 3: Use LightGCN to perform feature convolution on the association graph obtained in steps 1 and 2 to generate embeddings for items and users, and implement hierarchical comparative learning of items and users in each round of feature convolution; Step 4: Use the prediction model with multi-dimensional loss joint training to make recommendation predictions by taking the inner product as the user's rating of the item; In the step 1, the project modality association diagram includes a text modality association diagram of the project and a visual modality association diagram of the project, and the user modality association diagram includes a text modality association diagram of the user and a visual modality association diagram of the user; When performing the convolution operation on the user-item association graph in step 3, the consistency of explicit preference and implicit preference is increased by optimizing the loss function based on user preference.

[0011] In the above-mentioned commodity recommendation method for multimodal data, the process of constructing the user-item association graph in step 1 specifically includes: taking all users and items as nodes of the user-item association graph, and if there is an interactive relationship between the user and the item, establishing an edge between the two, and finally obtaining the user-item association graph .

[0012] In the above-mentioned commodity recommendation method for multimodal data, the process of constructing the item modality association graph in step 1 specifically includes: taking the nodes of all items as nodes of the association graph; For Projects and , and its similarity on mode m is defined as and The cosine similarity of ,in and The projects are and Embedding representation on modality m; at the same time, build a project The set of top-k similar items on modality m; For each project node, establish the The associated edges between the nodes in the top-k similarity item set are obtained Next project modal association diagram .

[0013] In the above-mentioned commodity recommendation method for multimodal data, the process of constructing the user modality association graph in step 1 specifically includes: taking all user nodes as nodes of the association graph; Calculate the user’s initial preference embedding on modality m and freeze it; For Users and , its modality preference similarity on modality m is defined as and The cosine similarity between ,in and Is a user and Embedding representation on modality m; at the same time, construct a user The set of top-k similar items on modality m; For each user node, establish an association edge between it and the nodes in the top-k similarity item set on modality m, and obtain the user modality association graph under modality m.

[0014] In the above-mentioned commodity recommendation method for multimodal data, the construction of the modal intersection association graph of the items in step 2 specifically includes: taking all item nodes as nodes of the association graph; if two nodes and , if there is an edge in both the text modal association graph of the project and the visual modal association graph of the project, then an edge is established in the modal intersection association graph of the project; finally, the modal intersection association graph of the project can be obtained ; Constructing the user's modal intersection association graph specifically includes: taking all user nodes as nodes of the association graph; if two nodes and , if there is an edge in both the user's text modal association graph and the user's visual modal association graph, then an edge is established in the user's modal intersection association graph; finally, the user's modal intersection association graph can be obtained .

[0015] In the above-mentioned method for recommending products for multimodal data, the user embedding generation process in step 3 includes: Layer item embedding Convolution is performed through the user-item association graph to obtain the embedding ,Will The modal intersection graph sent to the user Perform convolution to get embedding ,Will and The mean value of The final item embedding of the layer ; The project embedding generation process in step 3 includes: User embedding Convolution is performed through the user-item association graph to obtain the embedding ,Will The modal intersection association graph of the input project is convolved to obtain the embedding ,Will and The mean value of End-user embedding of layers ; For L-layer user embedding and item embedding, average pooling is used to obtain the final user embedding matrix. and the item embedding matrix .

[0016] In the above-mentioned commodity recommendation method for multimodal data, the hierarchical comparative learning of users in step 3 specifically includes: comparative learning of user embedding after convolution of the user-item association graph and the user's modal intersection association graph, and the specific formula is: ; in, represents the hierarchical contrast loss of user embeddings obtained in the user-item association graph and the user's modal intersection association graph, Indicates the first The user obtained by layer convolution Embed, represents the temperature coefficient, Indicates the first The user obtained by layer convolution The embedding Indicates the first The user obtained by layer convolution The embedding represents the user set, L represents the number of convolution layers, Indicates layer, Represents a user in the user collection Represents another user in the user set; The user embedding after convolution of the user's modal intersection association graph and the user's different modal association graph is compared and learned. The specific formula is: ; in, represents the hierarchical contrast loss of the embedding obtained by the user in the user's modal intersection association graph and the association graph of each modality, m represents a certain modality, Represents a user in the user collection, represents a set of modes, Indicates the first The user obtained by layer convolution The embedding Indicates the first The user obtained by layer convolution The embedding Indicates the first The user obtained by layer convolution Embedding.

[0017] In the above-mentioned commodity recommendation method for multimodal data, the hierarchical comparative learning of items in step 3 specifically includes: comparative learning of item embedding after convolution of the user-item association graph and the modal intersection association graph of the item, and the specific formula is: ; in, represents the hierarchical contrast loss of item embeddings obtained in the user-item association graph and the modal intersection association graph of items, Indicates the modal intersection association diagram of the project. Items obtained by layer convolution Embed, represents the temperature coefficient, Indicated in the user-item association graph Items obtained by layer convolution The embedding Indicated in the user-item association graph Items obtained by layer convolution The embedding represents the item set, L represents the number of convolution layers, Indicates layer; The modal intersection association graph of the project and the project embedding after convolution of the different modal association graphs of the project are compared and learned. The specific formula is: ; in, represents the hierarchical contrast loss of the item embedding obtained in the modal intersection association graph of the item and the association graph of the item in each modality, m represents a certain modality, Represents an item in a collection of items, Represents another item in the item collection, represents a set of modes, Indicates the first Items obtained by layer convolution The embedding Indicates the modal intersection association diagram of the project. Items obtained by layer convolution The embedding Indicates the first Items obtained by layer convolution Embedding.

[0018] The beneficial effect of the present invention is that the patent of the present invention constructs a multimodal recommendation method, which uses the modal intersection graph and hierarchical contrast learning to solve the problems of modal imbalance and user interest deviation in project embedding in the prior art, and solves the problem of lack of mining and alignment of user explicit preferences and implicit preferences in the prior art by improving the consistency of user displayed preferences and implicit preferences. The recommendation method of the present invention has a strong ability to generalize model scenarios, improves the quality of project and user preference modeling, and effectively improves the accuracy of recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a schematic diagram of the user embedding generation process of the present invention; Figure 3 It is a schematic diagram of the project embedding generation process of the present invention; Figure 4 It is a schematic diagram of the loss calculation process in the user-level comparative learning of the present invention; Figure 5 It is a schematic diagram of the calculation process of each loss in the project level comparative learning of the present invention; Figure 6 It is a schematic diagram of the user preference optimization loss calculation of the present invention; Figure 7 It is a schematic diagram of the relationship between the prediction model of the present invention and the loss. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. This embodiment discloses a commodity recommendation method for multimodal data. According to the actual scenario of recommendation, the text modality (t modality) and visual modality information (v modality) of the item are considered, wherein the text modality feature can be obtained from the textual introduction of the item, and the visual modality feature is obtained from the image of the item. The present invention uses the special term "item" in the recommendation system to represent the recommended commodity.

[0021] In this embodiment, for the item text and visual modality (denoted as ), extract the representation of the project in this modality through the pre-trained model , the modal feature matrix of the project is , where visual features are extracted by CNN and text features are extracted by sentence-transformers. is the dimension of the modal feature.

[0022] like Figure 1 As shown, the specific steps include: The first step is to construct three types of association graphs, namely the user-item association graph, the item modal association graph, and the user modal association graph.

[0023] 1.1 The user-item association graph is established based on the interaction relationship between users and items. The specific steps are as follows: (1) All users and projects are regarded as nodes of the user-project graph (2) If there is an interactive relationship between the user and the item, an edge is established between them.

[0024] Based on the above steps, a user-item-interaction graph is constructed according to the user-item interaction history. The graph is formally defined as a bipartite graph. ,in: (1) is a collection of users, Indicates users.

[0025] (2) is a collection of projects, Indicates projects.

[0026] (3) It is a collection of edges, representing the interaction relationship between users and items.

[0027] For the graph , set its corresponding user-item interaction matrix to , if the user With Project If there is interaction, ,on the contrary, ,Calculate the normalized adjacency matrix of user items as shown in formula (1)-(2): (1) (2); in, is the normalized user-item adjacency matrix, is the normalized item-user adjacency matrix, is the number of neighbors of user u, is the user-item interaction matrix, For Project The number of neighbors; 1.2 The modal association diagram of the project includes the textual modal association diagram of the project and the visual modal association diagram of the project. For any modality ( The construction steps are as follows: (1) Treat all project nodes as nodes of the association graph.

[0028] (2) For projects and , and its similarity on mode m is defined as and The cosine similarity of Based on this, build a project The top-k similar items set on modality m is as follows: (3).

[0029] (3) For each project node, establish a The associated edges between the nodes in the top-k similarity item set can get the modality Below is the project-project relationship diagram.

[0030] Based on the above steps, the modal association diagram of the project can be formally defined as, in: (1) is a collection of projects, Indicates projects.

[0031] (2) , Indicates the item corresponds to the modality.

[0032] (3) It is a set of edges, representing the relationship between items under mode m.

[0033] In modal In the modal association diagram of the following project, each node and its modal The k nodes with the largest similarity on the same node establish an edge. The adjacency matrix of , the normalized adjacency matrix is .

[0034] 1.3 The user's modal association graph includes the user's textual modal association graph and the user's visual modal association graph. For any modality ( for or ) The corresponding association graph is constructed as follows: (1) All user nodes are regarded as nodes of the association graph.

[0035] (2) Calculate users first Initial preference embedding on modality m And freeze it, see formula (4): (4); in, For the adjacency matrix In, users The value of the adjacency matrix corresponding to each item, is the embedding matrix of the item under mode m.

[0036] (3) For users and , its modality preference similarity on modality m is defined as and The cosine similarity between At the same time, build a user The top-k similar item set on modality m is as follows: (5); in, For users , the set of k items with the highest similarity, where k represents the number.

[0037] (4) For each user node, establish an associated edge with the nodes in the top-k similarity item set on modality m, and then obtain the user-user association graph under modality m.

[0038] Based on the above steps, the user's modal association graph can be formally defined as ,in: (1) is a collection of users, Indicates users.

[0039] (2) , Indicates the item corresponds to the modality.

[0040] (3) It is a set of edges, representing the association relationship between users under mode m.

[0041] In the modality user association graph under modality m, each node establishes an edge with the k nodes with the greatest similarity to it in modality m. The adjacency matrix of , the normalized adjacency matrix is .

[0042] Step 2: Construct the modal intersection association graphs of users and items respectively.

[0043] 2.1 Construct the modal intersection association graph of the project. The method of constructing the modal intersection association graph of the project is as follows: (1) All project nodes are regarded as nodes of the association graph; (2) If two nodes and , if there is an edge in both the text modal association graph of the project and the visual modal association graph of the project, then an edge is established in the modal intersection association graph of the project.

[0044] Based on the above steps, the modal intersection graph of the project can be formally defined as , the corresponding adjacency matrix is , the corresponding normalized adjacency matrix is ,in: (1) is a collection of projects, Indicates projects.

[0045] (2) It is a collection of edges, representing the relationship between items.

[0046] 2.2 Constructing the user's modal intersection association graph. The method for constructing the user's modal intersection association graph is as follows: (1) All user nodes are regarded as nodes of the association graph.

[0047] (2) If two knots and , if there are edges in both the user's text modal association graph and the user's visual modal association graph, an edge is established in the user's modal intersection association graph.

[0048] Based on the above steps, the user intersection graph can be formally defined as , the corresponding adjacency matrix is , the normalized adjacency matrix is ,in: is a collection of users, Indicates users.

[0049] It is a collection of edges, representing the relationship between users.

[0050] 3. Use LightGCN to perform feature convolution under multi-dimensional association graphs to generate embeddings for items and users.

[0051] 3.1 Section The user embedding of the layer is generated as follows: Layer item embedding First pass Perform convolution to get embedding , then The t-modal association graph of each user is fed into , user's v-modal association graph And the intersection graph of users Perform convolution and get embedding respectively , and ,in , They are respectively , The matrix composed of is used in the subsequent hierarchical contrast learning shown in 4.1, and then and The mean value of End-user embedding of layers , each embedding calculation method is shown in formula (6)-(10), and the specific process is as follows Figure 2 .

[0052] (6); (7); (8); (9); (10).

[0053] 3.2 Section The layer project embedding is generated as follows: User embedding First pass Perform convolution to get embedding , then t modal association diagram for each project , the v-modal association diagram of the project And the intersection association diagram of the project Perform convolution and get embedding respectively , and ,in , They are respectively , The matrix composed of is used in the hierarchical contrastive learning shown in Section 4.2, and then and The mean value of Final item embedding of the layer , each embedding calculation method is shown in formula (11)-(15), and the specific process is as follows Figure 3 .

[0054] (11); (12); (13); (14); (15).

[0055] 3.3 For the obtained L-layer user and item embedding , , take the average pooling approach to get the final user embedding matrix and the item embedding matrix , the calculation formula is shown in (16)(17): (16); (17).

[0056] 4. Use hierarchical contrastive learning to achieve embedding comparison when aggregating features at different levels of convolution for projects and users respectively.

[0057] 4.1 In each round of feature convolution, user-level contrastive learning is implemented. Specific approach: Use formula (18) to Figure and Comparative learning of user embedding after cross-graph convolution improves user-item interaction graph and user modal intersection graph At the same time, formula (19) is used to , , The user embedding after graph convolution is used for contrastive learning to enhance the consistency of user aggregation features in different modal association graphs. The hierarchical contrastive learning of the above users can effectively solve the user preference noise problem caused by interest deviation during user modeling (solving the second technical problem). The loss calculation process is shown below: Figure 4 As shown: (18); in, represents the hierarchical contrast loss of user embeddings obtained in the user-item association graph and the user's modal intersection association graph, Indicates the first The user obtained by layer convolution Embed, represents the temperature coefficient, Indicates the first The user obtained by layer convolution The embedding Indicates the first The user obtained by layer convolution The embedding represents the user set, L represents the number of convolution layers, Indicates layer, Represents a user in the user collection Represents another user in the user set; (19); in, represents the hierarchical contrast loss of the embedding obtained by the user in the user's modal intersection association graph and the user in the association graph of each modality, m represents a certain modality, Represents a user in the user collection, represents a set of modes, Indicates the first The user obtained by layer convolution The embedding Indicates the first The user obtained by layer convolution The embedding Indicates the first The user obtained by layer convolution The embedding of and They are the embedding matrices middle , The corresponding embedding. According to the values ​​t and v of the mode m, Can correspond and , we can get from step 3.1 and .

[0058] (20).

[0059] 4.2 In each round of feature convolution, hierarchical contrastive learning of items is implemented. Specific approach: Use formula (21) to Figure and Comparative learning of item embedding after cross-graph convolution improves item embedding in user-item interaction graph and project modal intersection graph At the same time, formula (22) is used to pay, , The project embedding after graph convolution is subjected to comparative learning to enhance the consistency of the aggregated features of the project in different modal association graphs. The hierarchical comparative learning of the above projects can effectively improve the multimodal feature alignment capability during project modeling, while alleviating the problem of modal feature imbalance in project embedding caused by modal feature fusion weights (solving the first technical difficulty). The loss calculation process is shown below: Figure 5 shown.

[0060] (twenty one); in, represents the hierarchical contrast loss of item embeddings obtained in the user-item association graph and the modal intersection association graph of items, Indicates the modal intersection association diagram of the project. Items obtained by layer convolution Embed, represents the temperature coefficient, Indicated in the user-item association graph Items obtained by layer convolution The embedding Indicated in the user-item association graph Items obtained by layer convolution The embedding represents the item set, L represents the number of convolution layers, Indicates layer.

[0061] (twenty two); in, represents the hierarchical contrast loss of the embedding of the item in the modal intersection association graph of the item and the item in the association graph of each modality, m represents a certain modality, Represents an item in a collection of items, Represents another item in a collection of items. represents a set of modes, Indicates the first Items obtained by layer convolution The embedding Indicates the modal intersection association diagram of the project. Items obtained by layer convolution The embedding Indicates the first Items obtained by layer convolution of embedding, where and is the item embedding matrix Medium Project and The corresponding embedding. According to the values ​​t and v of the mode m, Can correspond and , we can get from step 3.2 and .

[0062] (twenty three).

[0063] 5. Use the loss function shown in formula (24) to improve the consistency of users' explicit preferences and implicit preferences, thereby improving the quality of user modeling.

[0064] In the user-item graph, through convolution operations, explicit preferences can be obtained from the project nodes directly associated with the user; implicit preferences can be obtained from nodes that are far away from the user (not directly associated). Implicit preferences refer to the preferences that users may have in the future. Increasing the consistency of explicit preferences and implicit preferences is conducive to more accurate acquisition of user preferences (solving the third technical problem). The loss calculation process is shown below: Figure 6 As shown.

[0065] The user preference optimization loss function is designed as follows, where is the consistency difference loss between explicit preference and implicit preference, see formula (24): (twenty four).

[0066] 6. Establish a prediction model with multi-dimensional loss joint training, and use the model shown in formula (25) for recommendation prediction. The overall loss of the model is shown in formula (27).

[0067] After obtaining the final embedding representation of the user-item, the inner product is used as the user's score on the item, see formula (25): (25); in, Indicates user About Project The predicted preference score, e i Indicates the final project Embed, u Indicates the end user Embed.

[0068] In the training of the model, Bayesian personalized ranking loss (BPR) is selected as the loss function. This loss function makes the user's score for positive items higher than the score for negative items, see formula (26): (26); in is a set of training instances, and each triplet satisfies the relationship between user u and item Interacted with the project No interaction, is the sigmoid function.

[0069] The overall loss function of this embodiment consists of the above four parts: , , , , see formula (27): + (27); in , , γ is a hyperparameter Represents the total loss of the level comparison of the project, Represents the total loss of the user's level comparison. The relationship between the prediction model and the loss is as follows Figure 7 shown.

[0070] Experimental comparison: Dataset: We selected three public datasets of Amazon’s e-commerce platform, namely Baby, Sports, and Clothing. Each dataset contains text and visual features. The text features and visual features of the items are encoded as 384-dimensional text embedding and 4096-dimensional visual embedding, respectively.

[0071] Implementation details: The embedding vector spaces of users and projects are unified to 64 dimensions. The model is built based on the pytorch framework and uses the Adam optimizer to optimize parameters. During training, the learning rate is set to 1e-3, the mini-batch size of all data sets is set to 2048, and the number of training iterations is set to 1000. In order to prevent the model from overfitting, an early stopping strategy is introduced. If the validation set does not improve on R@20 in 20 consecutive iterations, the training is automatically terminated.

[0072] Comparison method: (1) VBPR: By combining the visual features extracted by CNN with matrix decomposition technology, the user’s preferences in the visual dimension are obtained.

[0073] (2) MMGCN: Introducing a multimodal graph convolutional network to enhance user representation through information exchange between users and items, thereby capturing fine-grained preferences across modalities.

[0074] (3) LightGCN: It simplifies the structure of the recommendation model by removing the nonlinear activation layer and feature transformation layer of the original GCN.

[0075] (4) GRCN: This method introduces a graph optimized convolutional network (GRCN), which adaptively adjusts the structure of the interaction graph according to the training state of the model and effectively extracts information signals about user preferences from the optimized graph.

[0076] (5) LATTICE: This method learns the item-item relationship graph for each modality, aggregates them into a latent item graph, and uses graph convolution to inject high-order affinities, thereby improving the accuracy of collaborative filtering recommendations.

[0077] (6) DualGNN: This method designs a multimodal representation learning module to explicitly model users’ attention to different modalities and inductively learn multimodal user preferences.

[0078] (7) FREEDOM: This method introduces a degree-sensitive edge pruning technique to remove potential noisy edges during graph sampling. It simultaneously freezes the item-item graph and denoises the user-item interaction graph to achieve multimodal recommendation.

[0079] (8) SLMRec: This method designs data augmentation based on multimodal content to generate multiple views of a single item and uses contrastive learning to distinguish the view of the item from other views to extract additional supervisory signals.

[0080] (9) POWEREC: This method proposes a hint-based weak modality enhanced multimodal recommendation framework. It exploits different modality hints to model user interests and enhances the learning of user preferences in modalities whose predictions are less reliable.

[0081] (10) LGMRec: This method proposes a Local and Global Graph Learning Guided Multimodal Recommender (LGMRec) to jointly model local and global user interests. The global embedding is combined with two decoupled local embeddings to improve the accuracy and robustness of recommendations.

[0082] (11) SynerGraph: This method improves the modeling of user preferences by integrating knowledge with item information. In addition, it develops a filter to remove noise in various types of data, thereby improving the reliability of recommendations.

[0083] In the above comparison methods, classic methods from previous years and the latest methods in recent years were selected for comparison, among which VBPR and MMGCN were in 2019, LightGCN and GRCN were in 2020, LATTICE and DualGNN were in 2021, SLMRec was in 2022, FREEDOM was in 2023, POWEREC, LGMRec, and SynerGraph were in 2024. Experimental comparisons were carried out, and the comparison results are shown in Tables 1-3.

[0084] As can be seen from Tables 1 to 3, in terms of the four evaluation indicators R@10, R@20, N@10, and N@20, the method of this embodiment is superior to the above baseline method, where %Improv1. is the percentage of improvement compared to the suboptimal baseline, and %Improv2. is the percentage of improvement relative to the average of all baselines. From the comparison data, it can be seen that the product recommendation method of this embodiment is significantly superior to the comparison method.

[0085] Table 1

[0086] Table 2

[0087] Table 3

[0088] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A commodity recommendation method for multimodal data, characterized in that: The steps include: Step 1, constructing a user-item association graph, an item modal association graph, and a user modal association graph; Step 2, constructing a modal intersection association graph of the project and a modal intersection association graph of the user; Step 3: Use LightGCN to perform feature convolution on the association graph obtained in steps 1 and 2 to generate embeddings for items and users, and implement hierarchical comparative learning of items and users in each round of feature convolution; Step 4: Use the prediction model with multi-dimensional loss joint training to make recommendation predictions by taking the inner product as the user's rating of the item; In the step 1, the project modality association diagram includes a text modality association diagram of the project and a visual modality association diagram of the project, and the user modality association diagram includes a text modality association diagram of the user and a visual modality association diagram of the user; When performing the convolution operation on the user-item association graph in step 3, the consistency of explicit preference and implicit preference is increased by optimizing the loss function based on user preference.

2. The method for recommending products based on multimodal data according to claim 1, characterized in that: The construction process of the user-item association graph in step 1 specifically includes: taking all users and items as nodes of the user-item association graph, and if there is an interactive relationship between the user and the item, an edge is established between the two, and finally the user-item association graph is obtained. .

3. The method for recommending products based on multimodal data according to claim 1, characterized in that: The process of constructing the project modal association graph in step 1 specifically includes: taking the nodes of all projects as the nodes of the association graph; For Projects and , and its similarity on mode m is defined as and The cosine similarity of ,in and The projects are and Embedding representation on modality m; at the same time, build a project The set of top-k similar items on modality m; For each project node, establish the The associated edges between the nodes in the top-k similarity item set are obtained Next project modal association diagram .

4. The method for recommending products based on multimodal data according to claim 1, characterized in that: The process of constructing the user modality association graph in step 1 specifically includes: taking all user nodes as nodes of the association graph; Calculate the user’s initial preference embedding on modality m and freeze it; For Users and , its modality preference similarity on modality m is defined as and The cosine similarity between ,in and Is a user and Embedding representation on modality m; at the same time, construct a user The set of top-k similar items on modality m; For each user node, establish an association edge between it and the nodes in the top-k similarity item set on modality m, and obtain the user modality association graph under modality m.

5. The method for recommending products based on multimodal data according to claim 1, characterized in that: The process of constructing the modal intersection association graph of the project in step 2 specifically includes: taking all project nodes as nodes of the association graph; if two nodes and , if there is an edge in both the text modal association graph of the project and the visual modal association graph of the project, then an edge is established in the modal intersection association graph of the project; the modal intersection association graph of the project is obtained ; Constructing the user's modal intersection association graph specifically includes: taking all user nodes as nodes of the association graph; if two nodes and , if there is an edge in both the user's text modal association graph and the user's visual modal association graph, then an edge is established in the user's modal intersection association graph; the user's modal intersection association graph is obtained .

6. The method for recommending products based on multimodal data according to claim 1, characterized in that: The user embedding generation process in step 3 includes: Layer item embedding Convolution is performed through the user-item association graph to obtain the embedding ,Will The modal intersection graph sent to the user Perform convolution to get embedding ,Will and The mean value of Final item embedding of the layer ; The project embedding generation process in step 3 includes: User embedding Convolution is performed through the user-item association graph to obtain the embedding ,Will The modal intersection association graph of the input project is convolved to obtain the embedding ,Will and The mean value of End-user embedding of layers ; For the obtained Layer item embedding and user embedding, respectively, use average pooling to obtain the final user embedding matrix and the item embedding matrix .

7. The method for recommending products based on multimodal data according to claim 1, characterized in that: The hierarchical comparative learning of users in step 3 specifically includes: performing comparative learning on the user embedding after convolution of the user-item association graph and the user's modal intersection association graph, and the specific formula is: ; in, represents the hierarchical contrast loss of the embedding obtained by the user in the user-item association graph and the user's modal intersection association graph, Indicates the first The user obtained by layer convolution Embed, represents the temperature coefficient, Indicates the first The user obtained by layer convolution Embed, Indicates the first The user obtained by layer convolution Embed, represents the user set, L represents the number of convolution layers, Indicates layer, Represents a user in the user collection Represents another user in the user set; The user embedding after convolution of the user's modal intersection association graph and the user's different modal association graph is compared and learned. The specific formula is: ; in, represents the hierarchical contrast loss of the user embedding obtained in the user's modal intersection association graph and the user association graph of each modality, m represents a certain modality, Represents a user in the user collection, represents a set of modes, Indicates the first The user obtained by layer convolution Embedded, Indicates the first The user obtained by layer convolution Embedded, Indicates the first The user obtained by layer convolution Embedding.

8. The method for recommending products based on multimodal data according to claim 1, characterized in that: The hierarchical comparative learning of the items in step 3 specifically includes: comparative learning of the item embedding after convolution of the user-item association graph and the modal intersection association graph of the items, and the specific formula is: ; in represents the hierarchical contrast loss of item embeddings obtained in the user-item association graph and the modal intersection association graph of items, Indicates the modal intersection association diagram of the project. Items obtained by layer convolution Embed, represents the temperature coefficient, Indicated in the user-item association graph Items obtained by layer convolution Embedded, Indicated in the user-item association graph Items obtained by layer convolution Embedded, represents the item set, L represents the number of convolution layers, Indicates layer; The modal intersection association graph of the project and the project embedding after convolution of the different modal association graphs of the project are not learned. The specific formula is: ； in, represents the hierarchical contrast loss of the item embedding obtained in the modal intersection association graph of the item and the item association graph of each modality, m represents a certain modality, Represents an item in a collection of items, Represents another item in the item collection, represents a set of modes, Indicates the first Items obtained by layer convolution Embedded, Indicates the modal intersection association diagram of the project. Items obtained by layer convolution Embedded, Indicates the first Items obtained by layer convolution Embedding.

Images