Multimodal recommendation method and device based on dual-stream feature representation learning

Abstract

The present invention discloses a recommendation method, specifically a multimodal recommendation method and apparatus based on dual-stream feature representation learning. The method comprises: mapping the original modal features of multiple modalities corresponding to an item to a modality-aligned space and a modality-specific space, respectively, to obtain modality-aligned features and modality-unique features corresponding to each modality; constructing a first cross-modal item-item graph based on the modality-aligned features and obtaining a first enhanced item modal feature representation; constructing a second cross-modal item-item graph based on the modality-unique features and obtaining a second enhanced item modal feature representation corresponding to each modality; performing interactive learning on a user-item historical interaction graph to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each modality; and performing prediction based on user fusion features and item fusion features to obtain a recommendation result. This method can produce more accurate recommendation results.

Description

Technical Field

[0001] The present invention relates to a recommendation method, and specifically to a multimodal recommendation method and device based on dual-stream feature representation learning. Background Art

[0002] As an important technology for solving information overload, recommendation systems are used to filter information and provide personalized services in various fields such as e-commerce and social media. Traditional recommendation systems rely on a single type of data, such as user behavior data or content information, to analyze user interests and preferences to provide personalized recommendations. However, with the emergence of multimodal data on online platforms, users often make decisions based on a comprehensive evaluation of multiple modalities (such as images, text, and audio). Therefore, researchers are committed to developing multimodal recommendation systems that combine user behavior signals with multimodal item data to enhance the feature representation of users and items. In addition, the processing and integration of multimodal data can effectively solve the data sparsity and cold start problems that are prevalent in traditional recommendation systems, thereby improving the generalization and robustness of recommendations.

[0003] Mainstream methods in the existing technology usually use pre-trained models to extract modal features from the original modal data of items, and then integrate them into a modeling framework based on historical user-item interactions to learn comprehensive user preferences. In recent years, a series of studies have used graph neural networks to process user-item interaction bipartite graphs and item-item association graphs to obtain high-order feature representations. For example, MMGCN, LATTICE, and MMSSL are several commonly used methods in the existing technology. MMGCN uses graph convolutional networks to propagate and aggregate modal representations of users and items on modality-specific interaction graphs. LATTICE constructs an item-item affinity graph based on the similarity between item modal features, thereby mining hidden modal structure information. MMSSL enriches sparse interaction information through adversarial generative data enhancement and uses contrastive learning to model dependencies between modalities.

[0004] Although the existing methods have made some achievements, due to the gap between heterogeneous modalities, previous methods still face the challenge of insufficient multimodal feature mining. This is reflected in the following two aspects: (1) Multimodal alignment fusion loses modal uniqueness. Specifically, the existing methods are committed to aligning the semantic differences between heterogeneous modalities to obtain multimodal fusion features, which only include common features between different modalities. This leads to the neglect of unique features that only exist in a specific modality, which contain key information that may affect user preferences, thereby affecting the representation learning of item features and user preferences. (2) Insufficient inter-modal interaction between modal features. In order to capture the potential semantic relationship between items, some existing methods use low-dimensional modal features learned from the modality perception layer or raw modal features directly extracted from the pre-trained model to construct a unimodal item-item affinity graph, which is then processed by a graph convolutional neural network to propagate and aggregate modal features. However, this method only mines intra-modal interactions and ignores inter-modal interactions between modal features. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the above-mentioned methods commonly used in the prior art only learn the common features between multimodal information, but ignore the unique features and details of each individual modality. In order to solve the above problem, the present invention provides a multimodal recommendation method and device based on dual-stream feature representation learning.

[0006] The present invention includes:

[0007] In a first aspect, an embodiment of the present invention provides a multimodal recommendation method based on dual-stream feature representation learning, comprising:

[0008] For each of the multiple items, original modal features of multiple modalities corresponding to the item are mapped to a modality-aligned space and a modality-specific space, respectively, to obtain a modality-aligned feature corresponding to each modality in the modality-aligned space, and a modality-unique feature corresponding to each modality in the modality-specific space;

[0009] constructing a first cross-modal item-item graph based on the modality alignment features and obtaining a first enhanced item modality feature representation, and constructing a second cross-modal item-item graph based on the modality unique features and obtaining a second enhanced item modality feature representation corresponding to each of the modalities;

[0010] Performing a graph convolution operation on a user-item historical interaction graph using the user preference features, the first augmented item modal feature representation, and the second augmented item modal feature representation to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each modality, wherein the user-item historical interaction graph is generated based on historical interaction data between users and items, and the user preference features are obtained by random initialization;

[0011] A prediction is performed based on user fusion features and item fusion features to obtain a recommendation result. The user fusion features are obtained by fusing the first user feature representation and the second user feature representation, and the item fusion features are obtained by fusing the first item feature representation and the second item feature representation.

[0012] Optionally, constructing a first cross-modal item-item graph based on the modal alignment feature and obtaining a first enhanced item modal feature representation includes:

[0013] Calculating the semantic similarity between any two of the items based on the modality alignment features corresponding to the modalities to obtain a first semantic similarity matrix;

[0014] Converting the first semantic similarity matrix into a first item-item relationship graph edge value matrix to obtain a first unimodal graph corresponding to the modality;

[0015] Aggregating edges of the first unimodal graph corresponding to the plurality of modalities to obtain the first cross-modal item-item graph;

[0016] The first cross-modal item-item graph is input into a graph convolution module for processing to obtain a first enhanced item modality feature representation.

[0017] Optionally, constructing a second cross-modal item-item graph based on the modality-unique features and obtaining a second enhanced item modality feature representation corresponding to each modality includes:

[0018] Calculating the semantic similarity between any two of the items based on the modality unique feature corresponding to the modality to obtain a second semantic similarity matrix;

[0019] Converting the second semantic similarity matrix into a second item-item relationship graph edge value matrix to obtain a second unimodal graph corresponding to the modality;

[0020] Determining connections between a plurality of inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph;

[0021] The second cross-modal item-item graph is input into a graph convolution module for processing to obtain a second enhanced item modality feature representation corresponding to each of the modalities.

[0022] Optionally, constructing connections between inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph includes:

[0023] Using cosine similarity and top-n strategy to build connections between multiple nodes of the modalities;

[0024] Determining an edge matrix of a cross-modal item-item graph based on connections between the plurality of inter-modal nodes and the second unimodal graphs corresponding to the plurality of modalities;

[0025] The second cross-modal item-item graph is generated based on the edge matrix of the cross-modal item-item graph.

[0026] Optionally, performing a graph convolution operation on the user-item history interaction graph using the user preference feature, the first enhanced item modality feature representation, and the second enhanced item modality feature representation corresponding to each of the modalities to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each of the modalities includes:

[0027] Generate a user-item historical interaction graph based on historical interaction data between users and items, and randomly initialize to obtain user preference features, wherein the user preference features include user alignment features and user unique features corresponding to each modality;

[0028] Performing a graph convolution operation on the user-item history interaction graph using the user alignment feature and the first enhanced item modality feature representation to obtain the first user feature representation and the first item feature representation;

[0029] A graph convolution operation is performed on the user-item historical interaction graph using the user-unique features corresponding to the modality and the second enhanced item modality feature representation corresponding to the modality to obtain the second user feature representation corresponding to each modality and the second item feature representation corresponding to each modality.

[0030] Optionally, the prediction based on the user fusion features and the item fusion features to obtain the recommendation result includes:

[0031] Inputting the first user feature representation and the second user feature representation into a multi-head attention feature fusion module for fusion processing to obtain the user fusion feature, and inputting the first item feature representation and the second item feature representation into the multi-head attention feature fusion module for fusion processing to obtain the item fusion feature;

[0032] Calculating a prediction score corresponding to each of the items based on the user fusion feature and the item fusion feature of each of the items;

[0033] The recommendation result is determined based on the prediction score.

[0034] Optionally, determining a recommendation result based on the prediction score includes:

[0035] The items whose prediction scores are ranked in descending order and ranked in the top K are determined as the recommendation results.

[0036] In a second aspect, an embodiment of the present invention provides a multimodal recommendation device based on dual-stream feature representation learning, comprising:

[0037] a mapping module configured to map, for each of the plurality of items, the original modal features of the plurality of modalities corresponding to the item to a modality-aligned space and a modality-specific space, respectively, to obtain a modality-aligned feature corresponding to each of the modalities in the modality-aligned space, and to obtain a modality-unique feature corresponding to each of the modalities in the modality-specific space;

[0038] a processing module, configured to construct a first cross-modal item-item graph based on the modality alignment features and obtain a first enhanced item modality feature representation, and to construct a second cross-modal item-item graph based on the modality unique features and obtain a second enhanced item modality feature representation corresponding to each of the modalities;

[0039] a learning module configured to perform a graph convolution operation on a user-item historical interaction graph using user preference features, the first augmented item modal feature representation, and the second augmented item modal feature representation to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each modality, wherein the user-item historical interaction graph is generated based on historical interaction data between users and items, and the user preference features are obtained by random initialization;

[0040] A prediction module is configured to perform prediction based on user fusion features and item fusion features to obtain recommendation results. The user fusion features are obtained by fusion of the first user feature representation and the second user feature representation, and the item fusion features are obtained by fusion of the first item feature representation and the second item feature representation.

[0041] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps of the multimodal recommendation method based on dual-stream feature representation learning as described in the first aspect.

[0042] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the multimodal recommendation method based on dual-stream feature representation learning as described in the first aspect.

[0043] The beneficial effect of the present invention is that, in an embodiment of the present invention, the original modal features of multiple modalities of an item are mapped to a modal-aligned space and a modal-specific space, respectively, to generate modal-aligned features and modal-unique features. Then, in the modal-aligned space and the modal-specific space, a unimodal item-item graph is constructed based on the semantic relationship of the items, respectively, and the unimodal item-item graph is used to form a cross-modal item-item graph by adding edges or establishing connections, thereby mining the interactions within and between modalities. Through the above-mentioned setting of the modal-aligned space and the modal-specific space, the importance of each modality-specific feature to feature representation learning in multimodal recommendation is emphasized, and the modal-aligned and modal-specific features that affect user preferences are fully extracted to obtain accurate multimodal features of items and multimodal preferences of users, thereby obtaining more accurate recommendation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Attachment Figure 1 A flowchart of a multimodal recommendation method based on dual-stream feature representation learning provided by an embodiment of the present invention;

[0045] Attachment Figure 2 A flowchart of a multimodal recommendation model based on dual-stream feature representation learning provided by an embodiment of the present invention;

[0046] Attachment Figure 3 For attachment Figure 2 Schematic diagram of the dual-space feature representation module in the multimodal recommendation model based on dual-stream feature representation learning;

[0047] Attachment Figure 4 For attachment Figure 2 Schematic diagram of the cross-modal item-item graph learning module in the multimodal recommendation model based on two-stream feature representation learning;

[0048] Attachment Figure 5 For attachment Figure 2 Schematic diagram of the user-item interaction graph learning module in the multimodal recommendation model based on two-stream feature representation learning;

[0049] Attachment Figure 6 For attachment Figure 2 Schematic diagram of the fusion and prediction modules in the multimodal recommendation model based on two-stream feature representation learning;

[0050] Attachment Figure 7A schematic diagram of a multimodal recommendation device based on dual-stream feature representation learning provided by an embodiment of the present invention;

[0051] Attachment Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0055] The embodiments of the present application provide a multimodal recommendation method and apparatus based on dual-stream feature representation learning, aiming to more comprehensively mine users' preferences for content in different modalities by learning modality-aligned and modality-specific multimodal features that influence user preferences.

[0056] See Figure 1 , Figure 1 This is a flow chart of a multimodal recommendation method based on dual-stream feature representation learning provided by an embodiment of the present invention. This method specifically includes the following steps:

[0057] Step 101: For each of a plurality of items, the original modal features of the plurality of modalities corresponding to the item are mapped to a modal-aligned space and a modal-specific space, respectively, and a modal-aligned feature corresponding to each of the modalities is obtained in the modal-aligned space, and a modal-unique feature corresponding to each of the modalities is obtained in the modal-specific space.

[0058] Step 102: construct a first cross-modal item-item graph based on the modal alignment features and obtain a first enhanced item modal feature representation, and construct a second cross-modal item-item graph based on the modal unique features and obtain a second enhanced item modal feature representation corresponding to each of the modalities.

[0059] Step 103: Perform a graph convolution operation on the user-item historical interaction graph using the user preference features, the first enhanced item modal feature representation, and the second enhanced item modal feature representation to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each of the modalities. The user-item historical interaction graph is generated based on the historical interaction data between the user and the item, and the user preference features are obtained through random initialization.

[0060] Step 104: Predict based on the user fusion feature and the item fusion feature to obtain a recommendation result. The user fusion feature is obtained by fusing the first user feature representation and the second user feature representation, and the item fusion feature is obtained by fusing the first item feature representation and the second item feature representation.

[0061] See Figure 2 As shown, an embodiment of the present invention further provides a multimodal recommendation model based on dual-stream feature representation learning. The multimodal recommendation model based on dual-stream feature representation learning includes four modules, which can be used to execute the above steps 101 to 104 respectively, specifically including: a dual-space feature representation module for executing step 101, a cross-modal item-item graph learning module for executing step 102, a user-item interaction graph learning module for executing step 103, and a fusion and prediction module for executing step 104.

[0062] The dual-space feature representation module includes a modality-aligned space and a modality-specific space. In step 101, in order to respectively mine the modality-aligned item modal features and the modality-specific item modal features, the original modal features extracted from the pre-trained model are mapped to the modality-aligned space and the modality-specific space.

[0063] In a specific implementation, data from multiple modalities of multiple items is acquired. For each item, the original modal features of each modality are mapped to a modality-aligned space and a modality-specific space, respectively. Any item may include data from multiple different modalities, such as image modality data, text modality data, video modality data, and audio modality data. For ease of description, in subsequent embodiments, the modalities of an item are set to include text modality and image modality, and m∈v,t is set to represent the modality, where t is used to represent the text modality and v is used to represent the image modality.

[0064] like Figure 3 As shown, taking any object as an example, the specific process of mapping the original modal features of the modal m of the object to the modal alignment space and the modal-specific space is explained below.

[0065] Specifically, the original modal features of the modal m of item i are mapped into the modal alignment space, and the modal alignment features obtained are as follows:

[0066]

[0067] in, is the transformation matrix of the multilayer perceptron corresponding to the modality m in the modality-aligned space, d m is the dimension of the original modal feature, d ma is the dimension of the features in the modality alignment space, is the bias vector of the multilayer perceptron corresponding to the modality m in the modality-aligned space, d is used to represent the dimension, and ma is used to represent the modality-aligned space. is the original modal feature of modality m corresponding to item i, and σ1(·) is used to represent the LeakyRelu activation function.

[0068] For different modes, the mode alignment features corresponding to the mode can be obtained. For example, The original modal features used to characterize the text modality of the article, The original modal features used to characterize the image modality of the object, Modality alignment features for characterizing image modalities in a modality-aligned space, Modality alignment features for characterizing text modalities in the modality alignment space, W t,ma is the transformation matrix of the multilayer perceptron corresponding to the text modality in the modality-aligned space, b t,ma is the bias vector of the multilayer perceptron corresponding to the text modality in the modality-aligned space, W v,ma is the transformation matrix of the multilayer perceptron corresponding to the image modality in the modality-aligned space, b v,mais the bias vector of the multilayer perceptron corresponding to the image modality in the modality-aligned space.

[0069] Specifically, the original modal features of the modal m of item i are mapped into the modal-specific space, and the modal unique features obtained are as follows:

[0070]

[0071] in, is the transformation matrix of the multilayer perceptron corresponding to the modality m in the modality-specific space, d mu is the dimension of the modality-specific spatial features, is the bias vector of the multilayer perceptron corresponding to the modality m in the modality-specific space, d is used to represent the dimension, and mu is used to represent the modality-specific space. is the original modal feature of the modality m of the item, m is used to represent the ordinal number of the modality, and σ1(·) is used to represent the LeakyRelu activation function.

[0072] For different modes, the unique modal features corresponding to the mode can be obtained. For example, modality-unique features used to characterize the image modality in a modality-specific space, The modality-unique feature used to characterize the text modality in the modality-specific space, W t,mu is the transformation matrix of the multilayer perceptron corresponding to the text modality in the modality-specific space, b t,mu is the bias vector of the multilayer perceptron corresponding to the text modality in the modality-specific space, W v,mu is the transformation matrix of the multilayer perceptron corresponding to the image modality in the modality-specific space, b v,mu is the bias vector of the multilayer perceptron corresponding to the image modality in the modality-specific space.

[0073] Optionally, in some embodiments, in order to assist the multimodal recommendation model training based on dual-stream feature representation learning to obtain more accurate feature representations in each space, an alignment loss is introduced in the modality alignment space. Minimize the difference between each modality. Taking text modality and image modality as an example, the specific calculation method is as follows:

[0074]

[0075] Where exp(·) is the exponential function, τ is the temperature coefficient, and I is the set of items.

[0076] Optionally, in some embodiments, a specific loss is introduced in the modality-specific space The representations of each modality are subject to soft orthogonal constraints. Taking the text modality and image modality as an example, the specific calculation method is as follows:

[0077]

[0078] Among them, the function (·) T represents the matrix transpose, It is the Frobenius paradigm.

[0079] In order to further explore the semantic relationship between modal features of items and enhance the representation of modal features, existing technologies usually directly construct a single-modal item-item semantic graph, which can only obtain the semantic relationship within the modality but cannot obtain the interactive relationship between modalities. Figure 4 As shown, in step 102 of this embodiment, a first cross-modal item-item graph is constructed in the modality-aligned space, and a second cross-modal item-item graph is constructed in the modality-specific space, so as to fully obtain the interactive relationship between the modalities.

[0080] First, the space of modal alignment is described. Optionally, in some embodiments, constructing a first cross-modal item-item graph based on the modal alignment features and obtaining a first enhanced item modal feature representation based on the first cross-modal item-item graph includes:

[0081] Calculating the semantic similarity between any two of the items based on the modality alignment features corresponding to the modalities to obtain a first semantic similarity matrix;

[0082] Converting the first semantic similarity matrix into a first item-item relationship graph edge value matrix to obtain a first unimodal graph corresponding to the modality;

[0083] Aggregating edges of the first unimodal graph corresponding to the plurality of modalities to obtain the first cross-modal item-item graph;

[0084] The first cross-modal item-item graph is input into a graph convolution module for processing to obtain a first enhanced item modality feature representation.

[0085] Specifically, first, the semantic similarity between each item is calculated based on the modal alignment features of the items in the modal alignment space. In this embodiment, cosine similarity is used to represent the semantic similarity. The specific calculation method is as follows:

[0086]

[0087] in, Indicates item i a and i b Semantic similarity in the modality-aligned space, |I| is used to indicate the number of items. For item i a The modal alignment feature corresponding to the mode m of For item i b The modal alignment feature corresponding to the mode m.

[0088] It should be understood that, in different modalities, the semantic similarity between items in that modality can be derived based on the modal alignment features corresponding to that modality. For any modality, the semantic similarity between any two items in the plurality of items in that modality can be used to construct a first semantic similarity matrix corresponding to that modality.

[0089] As an optional implementation, a fully connected graph can be generated based on the first semantic similarity matrix. Generating a fully connected graph consumes a large amount of computing resources in subsequent graph convolution operations, and the presence of unnecessary connections introduces noise. Therefore, as another optional implementation, a top-n sparsification strategy is applied to the fully connected graph to obtain the first unimodal graph corresponding to each modality.

[0090] Specifically, for any modality m, only the n edges with the highest semantic similarity are retained for each item, and the first semantic similarity matrix is ​​converted into the first item-item relationship graph edge value matrix as follows:

[0091]

[0092] in, is the item i in the edge value matrix of the first item-item relationship graph a and i b Boundary value, Then item i in the first unimodal graph a and i b The corresponding nodes are connected, otherwise they are not connected. Indicates item i a The set of semantic similarities with all other items.

[0093] In the above manner, the first unimodal graph corresponding to each modality can be obtained. Since the differences between modalities are required to be eliminated as much as possible in the modal alignment space, the representations of the same item in different modalities are similar, and the first unimodal graphs formed should also be similar. In this embodiment, the edges of the first unimodal graphs corresponding to multiple modalities are aggregated to obtain the first cross-modal item-item graph. Specifically, the set of the first cross-modal item-item graph edges in the modal alignment space is the union of all the first unimodal graph edges:

[0094]

[0095] By the above method, the first cross-modal item-item graph in the modal alignment space can be obtained The first cross-modal item-item graph is input into the graph convolution module for processing, propagating and aggregating information from adjacent nodes, thereby obtaining the first enhanced item modality feature representation In some embodiments, the graph convolution module is a lightweight graph convolutional network (Light Graph Convolutional Network, LightGCN).

[0096] Optionally, in some embodiments, constructing a second cross-modal item-item graph based on the modality-unique features and obtaining a second enhanced item modality feature representation corresponding to each modality includes:

[0097] Calculating the semantic similarity between any two of the items based on the modality unique feature corresponding to the modality to obtain a second semantic similarity matrix;

[0098] Converting the second semantic similarity matrix into a second item-item relationship graph edge value matrix to obtain a second unimodal graph corresponding to the modality;

[0099] Determining connections between a plurality of inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph;

[0100] The second cross-modal item-item graph is input into a graph convolution module for processing to obtain a second enhanced item modality feature representation corresponding to each of the modalities.

[0101] Specifically, the semantic similarity between items is first calculated based on the modal alignment features of items in the modality-specific space. In this embodiment, cosine similarity is used to represent the semantic similarity. The specific calculation method is as follows:

[0102]

[0103] in, Indicates item i a and i b Semantic similarity in a modality-specific space, For item i a The unique modal feature corresponding to the mode m of For item i b The modal unique feature corresponding to the mode m.

[0104] It should be understood that, in different modalities, the semantic similarity between items in that modality can be derived based on the modal-unique features corresponding to that modality. For any modality, a second semantic similarity matrix corresponding to that modality can be constructed based on the semantic similarity between any two items in that modality.

[0105] As an optional implementation, a fully connected graph can be generated based on the second semantic similarity matrix. Generating a fully connected graph consumes a large amount of computing resources in subsequent graph convolution operations, and the presence of unnecessary connections introduces noise. Therefore, as another optional implementation, a top-n sparsification strategy is applied to the fully connected graph to obtain a second unimodal graph corresponding to each modality.

[0106] Specifically, for any modality m, only the n edges with the highest semantic similarity are retained for each item, and the second semantic similarity matrix is ​​converted into the second item-item relationship graph edge value matrix as follows:

[0107]

[0108] in, is the item i in the edge value matrix of the second item-item relationship graph a and i b Boundary value, Then item i in the second unimodal graph a and i b The corresponding nodes are connected, otherwise they are not connected.

[0109] Through the above method, a corresponding second unimodal graph can be obtained for each modality. The representations of the same item in different modalities within the modality-specific space are completely different, so the edges of the second unimodal graph cannot be simply aggregated. Optionally, in some embodiments, the connection between the nodes of the intermodal graph is constructed based on the second unimodal graphs corresponding to multiple modalities to obtain the second cross-modal item-item graph, including:

[0110] Using cosine similarity and top-n strategy to build connections between multiple nodes of the modalities;

[0111] Determining an edge matrix of a cross-modal item-item graph based on connections between the plurality of inter-modal nodes and the second unimodal graphs corresponding to the plurality of modalities;

[0112] The second cross-modal item-item graph is generated based on the edge matrix of the cross-modal item-item graph.

[0113] Specifically, cosine similarity and top-n strategy are used to build connections between modal nodes:

[0114]

[0115] in, is the similarity matrix between item modalities in the modality-specific space, is the adjacency matrix between modes in the mode-specific space.

[0116] On this basis, combined with the edges in the second unimodal graph, we can obtain the edge matrix of the second cross-modal item-item graph in the modality-specific space:

[0117]

[0118] in,

[0119] By the above method, we can obtain the second cross-modal item-item graph in the modality-specific space The second cross-modal item-item graph is input into the graph convolution module for processing, propagating and aggregating information from adjacent nodes, thereby obtaining the second enhanced item modality feature representation corresponding to each module

[0120] like Figure 4 and Figure 5 As shown, when the modality includes text modality and image modality, the second enhanced item modality feature representation corresponding to the text modality can be obtained. The second enhanced item modality feature representation corresponding to the image modality

[0121] In an embodiment of the present application, by constructing a cross-modal item-item graph, the intra-modal and inter-modal interactions between modal features are fully exploited, and the flow of cross-modal semantic information is promoted, thereby obtaining a more accurate enhanced item model.

[0122] See Figure 5 Optionally, in some embodiments, step 103 includes:

[0123] Generate a user-item historical interaction graph based on historical interaction data between users and items, and randomly initialize to obtain user preference features, wherein the user preference features include user alignment features and user unique features corresponding to each modality;

[0124] Performing a graph convolution operation on the user-item history interaction graph using the user alignment feature and the first enhanced item modality feature representation to obtain the first user feature representation and the first item feature representation;

[0125] A graph convolution operation is performed on the user-item historical interaction graph using the user-unique features corresponding to the modality and the second enhanced item modality feature representation corresponding to the modality to obtain the second user feature representation corresponding to each modality and the second item feature representation corresponding to each modality.

[0126] Specifically, LightGCN is used to perform graph convolution operations on the user-item history interaction graph using randomly initialized user preference features and the item modal feature representations obtained in step 102 to obtain high-order user feature representations and item feature representations (i.e., the first user feature representation and the first item feature representation, the second user feature representation corresponding to each modality, and the second item feature representation corresponding to each modality) to cross-align modal features.

[0127] First, based on the historical interaction data between users and items, a user-item historical interaction graph is generated to obtain the user-item historical interaction matrix. At the same time, user preference features are obtained through randomization, such as Figure 5 As shown, the user preference features include user alignment features and user unique features corresponding to each modality.

[0128] Taking the modality-aligned space as an example, we perform graph convolution on the user-item history interaction graph using the user alignment features and the first enhanced item modality feature representation to obtain the convolution result of the lth layer. as follows:

[0129]

[0130] Among them, G is the user-item historical interaction matrix, and D is a diagonal matrix whose value is the sum of the corresponding row elements of the matrix M. The results of each layer of convolution are added together to obtain the first user feature representation and the first item feature representation as follows:

[0131]

[0132] Similarly, in the modality-specific space, taking the text modality as an example, the unique user feature corresponding to the text modality is used on the user-item historical interaction graph and the second enhanced item modal feature Indicates the convolution result of the lth layer obtained by performing the graph convolution operation as follows:

[0133]

[0134] Among them, G is the user-item historical interaction matrix, and D is a diagonal matrix whose value is the sum of the corresponding row elements of the matrix M. By adding the results of each layer of convolution, we can get the second user feature representation corresponding to the text modality and the second item feature representation as follows:

[0135]

[0136] Where L is the total number of convolutional layers.

[0137] In this embodiment, by leveraging the advantages of the attention mechanism in feature fusion, a multi-head attention feature fusion module is used to fuse the user feature representation and item feature representation in the modality-aligned space and the modality-specific space in step 103.

[0138] like Figure 6 As shown, optionally, in some embodiments, step 104 includes:

[0139] Inputting the first user feature representation and the second user feature representation into a multi-head attention feature fusion module for fusion processing to obtain the user fusion feature, and inputting the first item feature representation and the second item feature representation into the multi-head attention feature fusion module for fusion processing to obtain the item fusion feature;

[0140] Calculating a prediction score corresponding to each of the items based on the user fusion feature and the item fusion feature of each of the items;

[0141] The recommendation result is determined based on the prediction score.

[0142] Taking text modality and image modality as an example, for user feature representation, the features that need to be fused include the first user feature representation Second user feature representation corresponding to text modality The second user feature representation corresponding to the image modality The specific fusion process is as follows:

[0143]

[0144] in, Fusion features for users, Represents a splicing operation, is the transformation matrix of the query vector, is the transformation matrix of the key vector, is the projection matrix of the value vector, is the weight matrix of the linear projection, MAFF(·) is used to represent the multi-head attention feature fusion module, d k is the dimension of the query vector.

[0145] For item feature representation, the features that need to be fused include the first user feature representation Second user feature representation corresponding to text modality The second user feature representation corresponding to the image modality The specific fusion process is as follows:

[0146]

[0147] in, It is the item fusion feature.

[0148] like Figure 6 As shown, the user fusion features are obtained Fusion features with items Then, an inner product operation is used to calculate a prediction score between the user and the candidate item. A high prediction score indicates that the user has a strong preference for the item. Optionally, in some embodiments, determining the recommendation result based on the prediction score includes:

[0149] The items whose prediction scores are ranked in descending order and ranked in the top K are determined as the recommendation results.

[0150] Specifically, the prediction scores are sorted in descending order, and the top K items are selected to be recommended to the user.

[0151]

[0152] in, is the predicted score of user u for item i, is the user fusion feature corresponding to user u, is the item fusion feature corresponding to item i.

[0153] Optionally, in some embodiments, Figure 2 The loss of training the multimodal recommendation model based on two-stream feature representation learning is shown Including modality alignment loss Modality-specific losses and BPR loss The details are as follows:

[0154]

[0155] Among them, λ1 is the measure of alignment loss The hyperparameters of λ2 are used to measure the modality-specific loss The hyperparameters of modality alignment loss Modality-specific losses The calculation method of BPR loss can be found in the above content, and will not be described in detail here. for:

[0156]

[0157] Among them, log represents logarithmic transformation, σ2(·) represents Sigmoid activation function, is a set of triples, each triple is used to represent a set of training data, specifically including user u, positive sample i a and negative samples i b , For user u to item i a The prediction score of For user u to item i b The prediction score of .

[0158] In an embodiment of the present application, the original modal features are first mapped into a dual-feature representation space to generate modality-aligned and modality-unique features. Then, a unimodal item-item affinity graph is constructed based on the semantic relationship of the items, and the unimodal item-item graph is used to form a cross-modal item-item graph by adding edges or establishing connections, thereby mining the interactions within and between modalities. Then, the obtained item modal features are connected with the randomly initialized user preference features, and graph learning is performed on the user-item interaction graph to generate their high-order representations. Finally, a multi-head attention feature fusion mechanism is utilized, which dynamically and adaptively integrates modality-corresponding and modality-unique item features with user features, calculates user preference scores and generates recommendations. In the above manner, the importance of modality-specific features to feature representation learning in multimodal recommendations is emphasized, and modality-aligned and modality-specific features that affect user preferences are fully extracted.

[0159] like Figure 7 As shown, an embodiment of the present invention further provides a multimodal recommendation device based on dual-stream feature representation learning, comprising:

[0160] A mapping module 701 is configured to map, for each of a plurality of items, the original modal features of the plurality of modalities corresponding to the item to a modality-aligned space and a modality-specific space, respectively, to obtain a modality-aligned feature corresponding to each modality in the modality-aligned space and a modality-unique feature corresponding to each modality in the modality-specific space;

[0161] A processing module 702 is configured to construct a first cross-modal item-item graph based on the modality alignment features and obtain a first enhanced item modal feature representation, and to construct a second cross-modal item-item graph based on the modality unique features and obtain a second enhanced item modal feature representation corresponding to each of the modalities;

[0162] A learning module 703 is configured to perform a graph convolution operation on a user-item historical interaction graph using the user preference features, the first enhanced item modal feature representation, and the second enhanced item modal feature representation to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each modality. The user-item historical interaction graph is generated based on historical interaction data between users and items, and the user preference features are obtained by random initialization.

[0163] The prediction module 704 is used to make predictions based on user fusion features and item fusion features to obtain recommendation results. The user fusion features are obtained by fusion of the first user feature representation and the second user feature representation, and the item fusion features are obtained by fusion of the first item feature representation and the second item feature representation.

[0164] Optionally, constructing a first cross-modal item-item graph based on the modal alignment feature and obtaining a first enhanced item modal feature representation includes:

[0165] Calculating the semantic similarity between any two of the items based on the modality alignment features corresponding to the modalities to obtain a first semantic similarity matrix;

[0166] Converting the first semantic similarity matrix into a first item-item relationship graph edge value matrix to obtain a first unimodal graph corresponding to the modality;

[0167] Aggregating edges of the first unimodal graph corresponding to the plurality of modalities to obtain the first cross-modal item-item graph;

[0168] The first cross-modal item-item graph is input into a graph convolution module for processing to obtain a first enhanced item modality feature representation.

[0169] Optionally, constructing a second cross-modal item-item graph based on the modality-unique features and obtaining a second enhanced item modality feature representation corresponding to each modality includes:

[0170] Calculating the semantic similarity between any two of the items based on the modality unique feature corresponding to the modality to obtain a second semantic similarity matrix;

[0171] Converting the second semantic similarity matrix into a second item-item relationship graph edge value matrix to obtain a second unimodal graph corresponding to the modality;

[0172] Determining connections between a plurality of inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph;

[0173] The second cross-modal item-item graph is input into a graph convolution module for processing to obtain a second enhanced item modality feature representation corresponding to each of the modalities.

[0174] Optionally, constructing connections between inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph includes:

[0175] Using cosine similarity and top-n strategy to build connections between multiple nodes of the modalities;

[0176] Determining an edge matrix of a cross-modal item-item graph based on connections between the plurality of inter-modal nodes and the second unimodal graphs corresponding to the plurality of modalities;

[0177] The second cross-modal item-item graph is generated based on the edge matrix of the cross-modal item-item graph.

[0178] Optionally, the learning module 703 includes:

[0179] a generating unit, configured to generate a user-item historical interaction graph based on historical interaction data between users and items, and randomly initialize to obtain user preference features, wherein the user preference features include user alignment features and user unique features corresponding to each of the modalities;

[0180] a first graph convolution unit, configured to perform a graph convolution operation on the user-item history interaction graph using the user alignment feature and the first enhanced item modality feature representation to obtain the first user feature representation and the first item feature representation;

[0181] The second graph convolution unit is used to perform a graph convolution operation on the user-item historical interaction graph using the user unique features corresponding to the modality and the second enhanced item modality feature representation corresponding to the modality to obtain the second user feature representation corresponding to each modality and the second item feature representation corresponding to each modality.

[0182] Optionally, the prediction module 704 includes:

[0183] a fusion unit, configured to input the first user feature representation and the second user feature representation into a multi-head attention feature fusion module for fusion processing to obtain the user fusion feature, and input the first item feature representation and the second item feature representation into the multi-head attention feature fusion module for fusion processing to obtain the item fusion feature;

[0184] a calculation unit, configured to calculate a prediction score corresponding to each of the items based on the user fusion feature and the item fusion feature of each of the items;

[0185] A determining unit is configured to determine the prediction score as a recommendation result.

[0186] Optionally, the determining unit is specifically configured to:

[0187] The items whose prediction scores are ranked in descending order and ranked in the top K are determined as the recommendation results.

[0188] The multimodal recommendation device 700 based on dual-stream feature representation learning provided in the embodiment of the present application can execute the above method embodiment. Its implementation principles and technical effects are similar and will not be described in detail in this embodiment.

[0189] It should be noted that the division of units in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0191] like Figure 8 As shown, an embodiment of the present application provides an electronic device 800, comprising: a memory 802, a processor 801, and a program stored on the memory 802 and executable on the processor 801; the processor 801 is configured to read the program in the memory 802 to implement the steps in the multimodal recommendation method based on dual-stream feature representation learning as described above.

[0192] An embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the various processes of the above-mentioned multimodal recommendation method embodiment based on dual-stream feature representation learning are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor memory (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid state drive (SSD)), etc.

[0193] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0194] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD-ROM), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0195] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A multimodal recommendation method based on dual-stream feature representation learning, characterized by: include: For each of the multiple items, original modal features of multiple modalities corresponding to the item are mapped to a modality-aligned space and a modality-specific space, respectively, to obtain a modality-aligned feature corresponding to each modality in the modality-aligned space, and a modality-unique feature corresponding to each modality in the modality-specific space; constructing a first cross-modal item-item graph based on the modality alignment features and obtaining a first enhanced item modality feature representation, and constructing a second cross-modal item-item graph based on the modality unique features and obtaining a second enhanced item modality feature representation corresponding to each of the modalities; Performing a graph convolution operation on a user-item historical interaction graph using the user preference features, the first augmented item modal feature representation, and the second augmented item modal feature representation to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each modality, wherein the user-item historical interaction graph is generated based on historical interaction data between users and items, and the user preference features are obtained by random initialization; A prediction is performed based on user fusion features and item fusion features to obtain a recommendation result. The user fusion features are obtained by fusing the first user feature representation and the second user feature representation, and the item fusion features are obtained by fusing the first item feature representation and the second item feature representation.

2. The method according to claim 1, wherein: The constructing a first cross-modal item-item graph based on the modal alignment feature and obtaining a first enhanced item modal feature representation includes: Calculating the semantic similarity between any two of the items based on the modality alignment features corresponding to the modalities to obtain a first semantic similarity matrix; Converting the first semantic similarity matrix into a first item-item relationship graph edge value matrix to obtain a first unimodal graph corresponding to the modality; Aggregating edges of the first unimodal graph corresponding to the plurality of modalities to obtain the first cross-modal item-item graph; The first cross-modal item-item graph is input into a graph convolution module for processing to obtain a first enhanced item modality feature representation.

3. The method according to claim 1, wherein: The constructing a second cross-modal item-item graph based on the modality-unique features and obtaining a second enhanced item modality feature representation corresponding to each modality includes: Calculating the semantic similarity between any two of the items based on the modality unique feature corresponding to the modality to obtain a second semantic similarity matrix; Converting the second semantic similarity matrix into a second item-item relationship graph edge value matrix to obtain a second unimodal graph corresponding to the modality; Determining connections between a plurality of inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph; The second cross-modal item-item graph is input into a graph convolution module for processing to obtain a second enhanced item modality feature representation corresponding to each of the modalities.

4. The method according to claim 3, wherein: The step of constructing connections between inter-modal nodes based on the second unimodal graphs corresponding to the plurality of modalities to obtain the second cross-modal item-item graph includes: Using cosine similarity and top-n strategy to build connections between multiple nodes of the modalities; Determining an edge matrix of a cross-modal item-item graph based on connections between the plurality of inter-modal nodes and the second unimodal graphs corresponding to the plurality of modalities; The second cross-modal item-item graph is generated based on the edge matrix of the cross-modal item-item graph.

5. The method according to claim 1, wherein: The method further comprises performing a graph convolution operation on the user-item history interaction graph using the user preference feature, the first enhanced item modality feature representation, and the second enhanced item modality feature representation corresponding to each of the modalities to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each of the modalities, including: Generate a user-item historical interaction graph based on historical interaction data between users and items, and randomly initialize to obtain user preference features, wherein the user preference features include user alignment features and user unique features corresponding to each modality; Performing a graph convolution operation on the user-item history interaction graph using the user alignment feature and the first enhanced item modality feature representation to obtain the first user feature representation and the first item feature representation; A graph convolution operation is performed on the user-item historical interaction graph using the user-unique features corresponding to the modality and the second enhanced item modality feature representation corresponding to the modality to obtain the second user feature representation corresponding to each modality and the second item feature representation corresponding to each modality.

6. The method according to claim 1, wherein: The prediction based on the user fusion features and the item fusion features to obtain the recommendation results includes: Inputting the first user feature representation and the second user feature representation into a multi-head attention feature fusion module for fusion processing to obtain the user fusion feature, and inputting the first item feature representation and the second item feature representation into the multi-head attention feature fusion module for fusion processing to obtain the item fusion feature; Calculating a prediction score corresponding to each of the items based on the user fusion feature and the item fusion feature of each of the items; The recommendation result is determined based on the prediction score.

7. The method according to claim 6, wherein: The determining of a recommendation result based on the prediction score includes: The items whose prediction scores are ranked in descending order and ranked in the top K are determined as the recommendation results.

8. A multimodal recommendation device based on dual-stream feature representation learning, characterized in that: include: a mapping module configured to map, for each of the plurality of items, the original modal features of the plurality of modalities corresponding to the item to a modality-aligned space and a modality-specific space, respectively, to obtain a modality-aligned feature corresponding to each of the modalities in the modality-aligned space, and to obtain a modality-unique feature corresponding to each of the modalities in the modality-specific space; a processing module, configured to construct a first cross-modal item-item graph based on the modality alignment features and obtain a first enhanced item modality feature representation, and to construct a second cross-modal item-item graph based on the modality unique features and obtain a second enhanced item modality feature representation corresponding to each of the modalities; a learning module configured to perform a graph convolution operation on a user-item historical interaction graph using user preference features, the first augmented item modal feature representation, and the second augmented item modal feature representation to obtain a first user feature representation, a first item feature representation, a second user feature representation, and a second item feature representation corresponding to each modality, wherein the user-item historical interaction graph is generated based on historical interaction data between users and items, and the user preference features are obtained by random initialization; A prediction module is configured to perform prediction based on user fusion features and item fusion features to obtain recommendation results. The user fusion features are obtained by fusion of the first user feature representation and the second user feature representation, and the item fusion features are obtained by fusion of the first item feature representation and the second item feature representation.

9. An electronic device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is configured to read the program in the memory to implement the steps of the multimodal recommendation method based on dual-stream feature representation learning as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the multimodal recommendation method based on dual-stream feature representation learning are implemented as described in any one of claims 1 to 7.

Images