Multimodal recommendation method and system based on hierarchical fusion network

Abstract

The application provides a kind of multi-modal recommendation method and system based on hierarchical fusion network, the method includes the following steps: obtaining the user information of target user and article multi-modal information;Encoding is carried out by encoder, obtains multiple article single modal characteristics, and constructs multiple single modal article-article graphs;All single modal characteristics are fused by cross-modal alignment, to obtain cross-modal fusion characteristics, construct cross-modal article-article graph according to cross-modal fusion characteristics;Combine user information, historical interaction record and article multi-modal information to construct user-article graph;Based on multiple article single modal characteristics, the feature difference score between each article single modal characteristic is calculated;Based on feature difference score and by hierarchical fusion network, output user feature representation and article global fusion feature representation;The article recommendation result for target user is calculated by TOP-K method.The application has the effect that the output is more ideal multi-modal recommendation result.

Description

Technical Field

[0001] This invention belongs to the field of information service technology, specifically relating to a multimodal recommendation method and system based on a hierarchical fusion network. Background Technology

[0002] Multimodal recommender systems are a type of content-enhanced recommendation problem that integrates user preferences obtained from various item patterns into the traditional collaborative filtering (CF) paradigm, thereby improving recommendation accuracy. In the rapidly developing field of information services, recommender systems have become crucial tools for helping users filter products or content that match their personal preferences from vast amounts of information. Given the abundance of multimodal data on various online platforms, and the fact that this multimodal data reflects users' preferences at a fine-grained modality level, research primarily focuses on integrating multimodal information into recommender systems to more accurately construct user preferences.

[0003] Existing multimodal fusion methods are mainly divided into early fusion and late fusion strategies. Early fusion occurs during the feature extraction stage, while late fusion occurs after decisions are made based on each modality. However, the differences between modalities cause these two methods to be effective only under different limited conditions. Specifically, when the differences between different modalities are small, early fusion is prone to overfitting due to information redundancy, while late fusion is easily affected by noise due to larger differences, resulting in unsatisfactory recommendation results. Summary of the Invention

[0004] This invention provides a multimodal recommendation method and system based on a hierarchical fusion network to solve the problem of unsatisfactory multimodal recommendation results.

[0005] In a first aspect, the present invention provides a multimodal recommendation method based on a hierarchical fusion network, the method comprising the following steps:

[0006] Obtain user information of the target user, and multimodal information of the target items that have historical interaction records with the target user;

[0007] The single-modal information of each item in the multimodal information of the item is encoded by an encoder to obtain multiple single-modal features of the item. Graph network technology is then used to construct multiple single-modal item-item graphs based on each of the single-modal features of the item.

[0008] All the single-modal features are fused across modality to obtain cross-modal fused features. A cross-modal item-item graph is then constructed using graph network technology based on the cross-modal fused features.

[0009] A user-item graph is constructed using graph network technology and combined with the user information, the historical interaction records, and the multimodal information of the items.

[0010] Based on the single-modal features of the multiple items, multimodal difference learning is performed through a variational autoencoder to calculate the feature difference score between each single-modal feature of the item.

[0011] Multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs are input into a hierarchical fusion network. Based on the feature difference scores, user feature representations and global fusion feature representations of items are output through the hierarchical fusion network.

[0012] The item recommendation result for the target user is obtained by combining the user feature representation and the global fusion feature representation of the item and calculating using the TOP-K method.

[0013] Optionally, the step of performing cross-modal alignment and fusion on all the single-modal features to obtain cross-modal fused features, and constructing a cross-modal item-item graph using graph network technology based on the cross-modal fused features includes the following steps:

[0014] The dimensionality of each of the single-modal features is reduced using a multilayer perceptron, so that all the single-modal features have the same feature dimension.

[0015] All the dimensionality-reduced single-modal features are connected into a multimodal composite feature;

[0016] The multimodal composite features are deeply fused through a preset linear layer to obtain cross-modal fused features;

[0017] A cross-modal item-item graph is constructed using graph network technology and based on the cross-modal fusion features.

[0018] Optionally, the step of calculating the feature difference score between each of the single-modal features of the items by performing multimodal difference learning through a variational autoencoder includes the following steps:

[0019] The mean value and variance of the single-modal feature distribution of each item are calculated by a variational autoencoder.

[0020] The variational posterior probability distribution and single-modal dataset distribution of each item's single-modal feature are calculated by combining the mean value and variance of the single-modal feature distribution;

[0021] The feature differences between the individual modal features of each item are quantified using the KL divergence metric, and the feature difference scores between the individual modal features of each item are calculated by combining the variational posterior probability distribution and the distribution of the individual modal dataset.

[0022] Optionally, the step of inputting multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs into a hierarchical fusion network, and outputting user feature representations and global fusion feature representations of items based on the feature difference scores and through the hierarchical fusion network includes the following steps:

[0023] Multiple single-modal item-item graphs are input into a hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain multiple single-modal feature representations;

[0024] The cross-modal item-item graph is input into the hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain the cross-modal feature representation;

[0025] The user-item graph is input into the hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain user feature representation and item interaction feature representation;

[0026] All the single-modal feature representations are fused into a single-modal fused feature representation;

[0027] Based on the feature difference score, the single-modal fusion feature representation and the cross-modal feature representation are fused into a hybrid fusion feature representation;

[0028] The hybrid fusion feature representation and the item interaction feature representation are merged into a global fusion feature representation for the item.

[0029] Optionally, the expression formula for the hybrid fusion feature representation is as follows:

[0030]

[0031] In the formula: d represents the hybrid fusion feature representation corresponding to the i-th target item. i This represents the feature difference score corresponding to the i-th target item. This represents the cross-modal feature representation corresponding to the i-th target item. This represents the single-modal fusion feature representation corresponding to the i-th target item.

[0032] Secondly, the present invention also provides a multimodal recommendation system based on a hierarchical fusion network, the system comprising:

[0033] The information acquisition module is used to acquire user information of the target user and multimodal information of the target items that have historical interaction records with the target user;

[0034] The first graph network construction module is used to encode each item unimodal information in the multimodal information of the item through an encoder to obtain multiple item unimodal features, and to construct multiple unimodal item-item graphs based on each item unimodal feature using graph network technology.

[0035] The second graph network construction module is used to perform cross-modal alignment and fusion of all the single-modal features to obtain cross-modal fused features, and to construct a cross-modal item-item graph using graph network technology and based on the cross-modal fused features.

[0036] The third graph network construction module is used to construct a user-item graph by utilizing graph network technology and combining the user information, the historical interaction records, and the item multimodal information;

[0037] The difference learning module is used to perform multimodal difference learning based on the single-modal features of the multiple items and through a variational autoencoder, and to calculate the feature difference score between each single-modal feature of the item.

[0038] The feature output module is used to input multiple single-modal item-item graphs, cross-modal item-item graphs, and user-item graphs into a hierarchical fusion network, and output user feature representations and global fusion feature representations of items based on the feature difference scores and through the hierarchical fusion network.

[0039] The recommendation result output module is used to combine the user feature representation and the global fusion feature representation of the item and calculate the item recommendation result for the target user using the TOP-K method.

[0040] Optionally, the second graph network building module includes:

[0041] The dimensionality reduction unit is used to perform dimensionality reduction processing on each of the single-modal features using a multilayer perceptron, so that all the single-modal features have the same feature dimension.

[0042] The feature connection unit is used to connect all the dimensionality-reduced single-modal features into a multimodal composite feature;

[0043] The feature fusion unit is used to perform deep fusion of the multimodal composite features through a preset linear layer to obtain cross-modal fused features;

[0044] The graph network construction unit is used to construct a cross-modal item-item graph using graph network technology and based on the cross-modal fusion features.

[0045] Optionally, the difference learning module includes:

[0046] The first difference learning computation unit is used to calculate the average value and variance of the single-modal feature distribution of each item's single-modal feature through a variational autoencoder.

[0047] The second difference learning computation unit is used to calculate the variational posterior probability distribution and single-modal dataset distribution of each item's single-modal feature by combining the mean value of the single-modal feature distribution and the variance of the single-modal feature distribution;

[0048] The difference score calculation unit is used to quantify the feature differences between the individual modal features of each item using the KL divergence metric, and to calculate the feature difference score between the individual modal features of each item by combining the variational posterior probability distribution and the distribution of the individual modal dataset.

[0049] Optionally, the feature output module includes:

[0050] The first convolution operation unit is used to input multiple single-modal item-item graphs into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain multiple single-modal feature representations;

[0051] The second convolution operation unit is used to input the cross-modal item-item graph into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain the cross-modal feature representation;

[0052] The third convolution operation unit is used to input the user-item graph into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain user feature representation and item interaction feature representation;

[0053] The first feature fusion unit is used to fuse all the single-modal feature representations into a single-modal fused feature representation;

[0054] The second feature fusion unit is used to fuse the single-modal fusion feature representation and the cross-modal feature representation into a hybrid fusion feature representation based on the feature difference score;

[0055] The third feature fusion unit is used to fuse the hybrid fusion feature representation and the item interaction feature representation into a global fusion feature representation of the item.

[0056] Optionally, the expression formula for the hybrid fusion feature representation is as follows:

[0057]

[0058] In the formula: d represents the hybrid fusion feature representation corresponding to the i-th target item. i This represents the feature difference score corresponding to the i-th target item. This represents the cross-modal feature representation corresponding to the i-th target item. This represents the single-modal fusion feature representation corresponding to the i-th target item.

[0059] The beneficial effects of this invention are:

[0060] The method provided by this invention includes the following steps: acquiring user information of the target user and multimodal information of items; encoding multiple unimodal features of items through an encoder and constructing multiple unimodal item-item graphs; performing cross-modal alignment and fusion of all unimodal features to obtain cross-modal fused features, and constructing cross-modal item-item graphs based on the cross-modal fused features; constructing user-item graphs by combining user information, historical interaction records, and multimodal information of items; calculating feature difference scores between each unimodal feature of items based on multiple unimodal features of items; outputting user feature representations and global fused feature representations of items based on feature difference scores and through a hierarchical fusion network; and calculating item recommendation results for the target user using the TOP-K method. This invention achieves the effect of outputting relatively ideal multimodal recommendation results. Through the above steps, multimodal difference learning is used to quantify differences and guide the hierarchical fusion network to adaptively fuse cross-modal and unimodal embeddings, thereby seamlessly integrating early and late fusion strategies of multimodal information, achieving complementary advantages of the two methods, and ultimately obtaining robust and comprehensive multimodal representations of items and improving the accuracy of multimodal recommendations. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the steps of a multimodal recommendation method based on a hierarchical fusion network in one embodiment of this application.

[0062] Figure 2 This is a schematic diagram of the framework of a multimodal recommendation method based on a hierarchical fusion network in one embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0064] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0065] Figure 1 This is a flowchart illustrating a multimodal recommendation method based on a hierarchical fusion network in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps. For example Figure 1 As shown, the multimodal recommendation method based on a hierarchical fusion network disclosed in this invention specifically includes the following steps:

[0066] S101. Obtain user information of the target user, and multimodal information of the target items that have historical interaction records with the target user.

[0067] Here, the target user refers to the user who needs to be recommended in a multimodal manner. The user information of the target user is usually the target user's personal information. The multimodal information of the item can include information of multiple modalities, such as text, images, videos, etc. In this embodiment, refer to... Figure 2 The multimodal information of an item includes both textual and image information. Let U represent the target user set containing all target users, and I represent the target item set containing all target items. A neural network embedding layer generates input ID embeddings for target user u∈U and target item i∈I, respectively. Where d is the embedding dimension. Furthermore, the multimodal features of the target item i are represented as follows: Where m∈M, M represents the modality set, m represents the modality, and d m The dimension representing the multimodal features.

[0068] S102. Encode each item monomodal information in the multimodal information of the item by an encoder to obtain multiple item monomodal features, and use graph network technology to construct multiple monomodal item-item graphs based on each item monomodal feature.

[0069] Among them, the same number of different encoders are preset according to the number of modalities of the multimodal information of the item, such as Figure 2 As shown, when the multimodal information of an item includes unimodal information such as item text and item image, the preset encoders are the text encoder and the image encoder. The text encoder and image encoder encode the item text and item image information in the unimodal information to obtain the item unimodal text features. Item single-modal image features Using graph network technology, a text-based unimodal item-item graph G is constructed based on both the unimodal text features and the unimodal image features of items. t And image-based single-modal items - item image G v .

[0070] S103. Perform cross-modal alignment and fusion on all unimodal features to obtain cross-modal fused features. Utilize graph network technology and construct a cross-modal item-item graph based on the cross-modal fused features.

[0071] In one embodiment, step S103 specifically includes the following steps:

[0072] Multilayer perceptrons are used to reduce the dimensionality of each single-modal feature so that all single-modal features have the same feature dimension.

[0073] All the dimensionality-reduced single-modal features are concatenated into multimodal composite features;

[0074] Multimodal composite features are deeply fused through a pre-defined linear layer to obtain cross-modal fused features;

[0075] We utilize graph network technology and construct a cross-modal item-item graph based on cross-modal fusion features.

[0076] In this implementation, the features obtained from different modalities exhibit significant semantic differences. To address this issue, a multilayer perceptron is first used to perform dimensionality reduction on the original single-modal features, ensuring that all modal features have a uniform dimension. The specific dimensionality reduction formula is as follows:

[0077]

[0078] In the formula: σ1(·) represents the transformation matrix and bias vector in the multilayer perceptron, respectively, and σ1(·) represents the LeakyReLU activation function.

[0079] Unlike traditional additive or averaging fusion strategies, this approach concatenates all dimensionality-reduced single-modal features into multimodal composite features, fully preserving the original modal information. Subsequently, the concatenated multimodal composite features are input into a linear layer for deep fusion, resulting in cross-modal fused features. The formula for expressing cross-modal fusion features is as follows:

[0080]

[0081] In the formula: and Let represent the transformation matrix and bias vector in the linear layer, respectively, and || denote the concatenation operation.

[0082] For the single-modal item-item graph (G) in step S102 t G v ) and the cross-modal item-item graph G in this step c These can be collectively referred to as items - item diagrams G. m , m∈{v, t, c}. Top-n sparsity can be used to construct an item-item graph G. m By calculating different target items (i.e., i) under the same modality. a and i b The cosine similarity score between different target items is used to evaluate the semantic relationship between them. The formula for calculating the cosine similarity score is as follows:

[0083]

[0084] In the formula: Represents target item i a and target item i b Cosine similarity score on mode m.

[0085] Generating a fully connected graph requires significant computational resources, and unnecessary connections introduce noise. To address these issues, top-n sparsity can be applied to fully connected graphs, maintaining only the top-n edges with the highest similarity for each node corresponding to each target item i. The formula for the top-n sparsity process is as follows:

[0086]

[0087] In the formula: Representation matrix S m All elements in row a. The value in the value is 0 or 1, where 1 indicates that there is a potential relationship between the two.

[0088] S104. Construct a user-item graph using graph network technology and combining user information, historical interaction records, and multimodal information of items.

[0089] Specifically, nodes v corresponding to the target user are generated based on user information and item multimodal information. u The node v corresponding to the target item i The historical interaction records between the target user and the target item can be represented as a sparse graph G0 = {V, E}, also known as a user-item graph. Here, V = {U∪I} represents the set of nodes in the sparse graph, and E... u，i ∈{0,1} represents the node υ corresponding to the target user. u The node corresponding to the target item. i The node edge between the target user and the target item. If the target user and the target item have interacted, there is a node edge between them, and it is assigned a value of 1; otherwise, it is assigned a value of 0.

[0090] Since popular nodes can lead to over-smoothing, thus affecting the accurate construction of user preferences, this implementation designs an edge pruning strategy based on the concept of sparse graphs. Specifically, in the user-item graph G0 = {V, E}, for each existing node edge (v... u v i Let E ∈ E, and define a pruning probability:

[0091]

[0092] In the formula: d(·) represents the degree of a node in graph G0, and the function exp(·) is the exponential operation. The probability of pruning increases with the degree of node connectivity; the higher the degree, the greater the likelihood of pruning.

[0093] Then, from the entire pruning probability distribution p = {p1, p2, ..., p...} |E|-1 Sampling is performed on the edge, and a pruning percentage ρ is defined. That is, <ρ|E|| edges need to be deleted, where <·| indicates rounding down. The resulting clean graph of the target user interaction after pruning is... in and The corresponding matrix is

[0094] S105. Based on multiple item unimodal features and through variational autoencoder, multimodal difference learning is completed, and feature difference scores between the unimodal features of each item are calculated.

[0095] S106. Input multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs into a hierarchical fusion network, and output user feature representations and global fusion feature representations of items based on feature difference scores and through the hierarchical fusion network.

[0096] S107. Combine user feature representation and item global fusion feature representation and calculate the item recommendation result for the target user using the TOP-K method.

[0097] Among them, in obtaining the user feature representation h u Global fusion feature representation of items The recommendation score between the target user and the target item can be calculated using the inner product operation method. The formula for calculating the recommendation score is as follows:

[0098]

[0099] In the formula: This represents the recommendation score between target user u and target item i. A higher recommendation score indicates a stronger preference for the item. The recommendation scores are then sorted in descending order using the TOP-K method, and the top K items are selected to recommend to the target user.

[0100] In one embodiment, step S105 specifically includes the following steps:

[0101] The mean value and variance of the single-modal feature distribution of each item are calculated by a variational autoencoder.

[0102] The variational posterior probability distribution and single-modal dataset distribution of each item's single-modal feature are calculated by combining the mean and variance of the single-modal feature distribution.

[0103] The feature difference between the unimodal features of each item is quantified by using KL divergence, and the feature difference score between the unimodal features of each item is calculated by combining the variational posterior probability distribution and the unimodal dataset distribution.

[0104] In this embodiment, it is assumed that the distribution distance between unimodal features of an item reflects the information gap between them. A variational autoencoder (VAE)-based approach can be used to understand the fixed unimodal feature distribution of each item's unimodal feature, thereby capturing the underlying structure of the unimodal features from a generative perspective. Specifically, the Kullback-Leibler (KL) divergence in the feature space can be used to quantify the distribution differences between different modal features. This quantification can more accurately assess the impact of unimodal and multimodal features on model performance, thereby optimizing the multimodal fusion strategy to improve prediction accuracy.

[0105] like Figure 2 As shown, firstly, the average value of the single-modal feature distribution is obtained by using a variational autoencoder customized for the original single modality. and the variance of the single-mode characteristic distribution Then, the variational posterior probability distribution of single-modal observations can be modeled using the following expression:

[0106]

[0107] Considering the overall dataset distribution for each modality, the distribution of a single-modality dataset can be modeled:

[0108]

[0109] Then, the difference scores between modes can be quantified by the average KL divergence between their respective unimodal distributions. Considering the asymmetry of the KL divergence, the following two formulas for calculating the difference scores are derived:

[0110]

[0111] In the formula: function D KL (·) represents the KL divergence.

[0112] Will and The average value is used to obtain the symmetric KL divergence as the feature difference score between the unimodal features of the two items. The calculation formula is as follows:

[0113]

[0114] In the formula: Tanh(·) is the Tanh activation function, used to map the difference scores to a range between 0 and 1. To ensure the generative capability of the model, and by using all... To eliminate noise, the material is aligned to a standard normal distribution, using the following formula:

[0115]

[0116] In addition, in order to minimize the output and input of the variational autoencoder The distance between them is handled using the following formula:

[0117]

[0118] In the formula: The embedding generated by the decoder in the variational autoencoder.

[0119] In one embodiment, step S106 specifically includes the following steps:

[0120] Multiple unimodal item-item graphs are input into a hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain multiple unimodal feature representations;

[0121] The cross-modal item-item graph is input into a hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain cross-modal feature representations;

[0122] The user-item graph is input into a hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain user feature representations and item interaction feature representations;

[0123] All single-modal feature representations are fused into a single-modal fused feature representation;

[0124] Based on feature difference scores, single-modal fusion feature representation and cross-modal feature representation are fused into a hybrid fusion feature representation;

[0125] The hybrid fusion feature representation and the item interaction feature representation are merged into a global fusion feature representation for the item.

[0126] In this implementation, firstly, all nodes in the item-item graph and user-item graph are initialized by embedding and mapping all target user and target item input IDs into dense vector representations, as follows:

[0127]

[0128] In the formula: and These represent the input ID embedding matrices for the target user and target item at layer 0, respectively.

[0129] Then, as Figure 2 As shown, multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs are input into a hierarchical fusion network for convolutional operations. LightGCN is used to propagate and aggregate information from neighboring nodes, and the features of each node are enriched by identifying commonalities between similar nodes.

[0130] Specifically, for multiple unimodal item-item graphs and cross-modal item-item graphs, the convolution operation can be defined as:

[0131]

[0132] In the formula: for The angle matrix and The embedding of layer l-1 is obtained by transforming the embedding of layer l. Since each target item node in the sparsed item-item graph has a self-loop, each layer retains information from the previous layer. Therefore, the embedding of layer l is directly used. iiThe result of the layer is used as the final representation.

[0133] In L ii After multiple convolutions, the modal feature representation can be obtained:

[0134]

[0135] The corresponding unimodal item-item image is the unimodal feature representation (image modality feature representation). and text modal feature representation ), The corresponding cross-modal item-item graph is the cross-modal feature representation.

[0136] Similarly, convolution operations can be performed on the user-item graph. To improve computational efficiency, and Connect them into a matrix And create a symmetric adjacency matrix, as shown below:

[0137]

[0138] In the formula: Then the convolution operation of the user-item graph is defined as:

[0139]

[0140] In the formula: Let M be the angle matrix and D aa =∑ b M ab Unlike item-item graphs, nodes in user-item graphs do not retain self-information from the previous layer during convolution. Therefore, in the Lth... ui After the layer process, the embeddings learned from each layer are integrated to construct the final user feature representation h. u The interaction features of items are represented by h. i :

[0141]

[0142] Based on the embeddings obtained from multiple information sources, these information can be effectively fused using a multi-step hierarchical fusion method.

[0143] Step 1: Represent the modal features of the image and text modal feature representation Combining these, we obtain the single-modal fusion feature representation, as shown in the following formula:

[0144]

[0145] In the formula: α is a hyperparameter representing the proportion of the image shape, which can indicate the relative importance of image information and text information.

[0146] Step 2: Based on the feature difference score, fuse the single-modal fusion feature representation and the cross-modal feature representation into a hybrid fusion feature representation. The formula for the hybrid fusion feature representation is as follows:

[0147]

[0148] In the formula: d represents the hybrid fusion feature representation corresponding to the i-th target item. i This represents the feature difference score corresponding to the i-th target item. This represents the cross-modal feature representation corresponding to the i-th target item. This represents the single-modal fusion feature representation corresponding to the i-th target item.

[0149] Step 3: Fuse the hybrid fusion feature representation and the item interaction feature representation into a global fusion feature representation for the item. The specific formula is as follows:

[0150]

[0151] In one implementation, a multi-task learning strategy is employed to optimize the overall process of the multimodal recommendation method based on hierarchical fusion networks in parallel, specifically involving the following steps:

[0152] 1. The cross-modal alignment and fusion task in step S103

[0153] The optimization method is as follows: Bayesian personalized ranking is used for optimization. Specifically, observed user-item interactions are treated as positive sample pairs (u,i), while target items j that do not interact with the target user u are randomly sampled as negative sample pairs (u,j), thus forming triples (u,i,j)∈D. Next, BPR loss is used to maximize the similarity of positive pairs while minimizing the similarity of negative pairs, with the specific formula as follows:

[0154]

[0155] In the formula: σ2(·) is the Sigmoid activation function.

[0156] 2. Multimodal difference learning task in step S105

[0157] The optimization method is as follows: In multimodal difference learning, the single-modal distribution generated during the intermediate process of the variational autoencoder is used to quantify the differences between the original single-modal features. To improve the accuracy of difference calculation, two loss functions existing in the variational autoencoder are used for training, with the specific formulas as follows:

[0158]

[0159] 3. The layered fusion task in step S106

[0160] The optimization method is as follows: The BPR loss is used to train the final feature representations of the target user and target item obtained in the hierarchical fusion network. The specific formula is as follows:

[0161]

[0162] Therefore, the joint training loss function for the overall process of the multimodal recommendation method based on hierarchical fusion networks is:

[0163] L = L bpr +λ1L cmbpr +λ2L vae

[0164] In the formula: λ1 and λ2 are both hyperparameters for controlling loss.

[0165] This invention also discloses a multimodal recommendation system based on a hierarchical fusion network, the system comprising:

[0166] The information acquisition module is used to acquire user information of the target user, as well as multimodal information of the target items that have historical interaction records with the target user;

[0167] The first graph network construction module is used to encode the single-modal information of each item in the multimodal information of the item through an encoder to obtain multiple single-modal features of the item, and to construct multiple single-modal item-item graphs based on the single-modal features of each item using graph network technology.

[0168] The second graph network construction module is used to perform cross-modal alignment and fusion of all single-modal features to obtain cross-modal fused features, and to construct a cross-modal item-item graph using graph network technology and based on the cross-modal fused features.

[0169] The third graph network construction module is used to construct a user-item graph by utilizing graph network technology and combining user information, historical interaction records, and multimodal information of items.

[0170] The difference learning module is used to perform multimodal difference learning based on multiple item single-modal features and through variational autoencoder, and calculate the feature difference score between the single-modal features of each item.

[0171] The feature output module is used to input multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs into the hierarchical fusion network, and output user feature representations and global fusion feature representations of items based on feature difference scores and through the hierarchical fusion network.

[0172] The recommendation result output module is used to combine user feature representation and item global fusion feature representation and calculate the item recommendation result for the target user using the TOP-K method.

[0173] In one implementation, the second graph network building module includes:

[0174] The dimensionality reduction unit is used to perform dimensionality reduction processing on each single-modal feature using a multilayer perceptron, so that all single-modal features have the same feature dimension.

[0175] Feature connection unit, used to connect all dimensionality-reduced single-modal features into multimodal composite features;

[0176] The feature fusion unit is used to perform deep fusion of multimodal composite features through a preset linear layer to obtain cross-modal fused features;

[0177] Graph network building unit, used to construct cross-modal item-item graphs using graph network technology and based on cross-modal fusion features.

[0178] In one implementation, the difference learning module includes:

[0179] The first difference learning computation unit is used to calculate the mean value and variance of the single-modal feature distribution of each item's single-modal feature through a variational autoencoder.

[0180] The second difference learning computation unit is used to calculate the variational posterior probability distribution and single-modal dataset distribution of each item's single-modal feature by combining the mean and variance of the single-modal feature distribution.

[0181] The difference score calculation unit is used to quantify the feature differences between the single-modal features of each item using KL divergence, and to calculate the feature difference score between the single-modal features of each item by combining the variational posterior probability distribution and the single-modal dataset distribution.

[0182] In one embodiment, the feature output module includes:

[0183] The first convolutional operation unit is used to input multiple single-modal item-item graphs into the hierarchical fusion network, and perform convolutional operations through the hierarchical fusion network to obtain multiple single-modal feature representations;

[0184] The second convolutional operation unit is used to input the cross-modal item-item graph into the hierarchical fusion network, and perform convolutional operations through the hierarchical fusion network to obtain the cross-modal feature representation;

[0185] The third convolution operation unit is used to input the user-item graph into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain user feature representation and item interaction feature representation;

[0186] The first feature fusion unit is used to fuse all single-modal feature representations into a single-modal fused feature representation;

[0187] The second feature fusion unit is used to fuse single-modal fusion feature representation and cross-modal feature representation into a hybrid fusion feature representation based on feature difference scores;

[0188] The third feature fusion unit is used to fuse the hybrid fusion feature representation and the item interaction feature representation into a global fusion feature representation of the item.

[0189] In one implementation, the expression formula for the hybrid fusion feature representation is as follows:

[0190]

[0191] In the formula: d represents the hybrid fusion feature representation corresponding to the i-th target item. i This represents the feature difference score corresponding to the i-th target item. This represents the cross-modal feature representation corresponding to the i-th target item. This represents the single-modal fusion feature representation corresponding to the i-th target item.

[0192] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0193] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A multimodal recommendation method based on a hierarchical fusion network, characterized in that, Includes the following steps: Obtain user information of the target user, and multimodal information of the target items that have historical interaction records with the target user; The single-modal information of each item in the multimodal information of the item is encoded by an encoder to obtain multiple single-modal features of the item. Graph network technology is then used to construct multiple single-modal item-item graphs based on each of the single-modal features of the item. All the single-modal features are fused across modality to obtain cross-modal fused features. A cross-modal item-item graph is then constructed using graph network technology based on the cross-modal fused features. A user-item graph is constructed using graph network technology and combined with the user information, the historical interaction records, and the multimodal information of the items. Based on the single-modal features of the multiple items, multimodal difference learning is performed through a variational autoencoder to calculate the feature difference score between each single-modal feature of the item. Multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs are input into a hierarchical fusion network. Based on the feature difference scores, user feature representations and global fusion feature representations of items are output through the hierarchical fusion network. The item recommendation result for the target user is obtained by combining the user feature representation and the global fusion feature representation of the item and calculating using the TOP-K method.

2. The multimodal recommendation method based on hierarchical fusion network according to claim 1, characterized in that, The step of performing cross-modal alignment and fusion of all the single-modal features to obtain cross-modal fused features, and constructing a cross-modal item-item graph using graph network technology based on the cross-modal fused features includes the following steps: The dimensionality of each of the single-modal features is reduced using a multilayer perceptron, so that all the single-modal features have the same feature dimension. All the dimensionality-reduced single-modal features are connected into a multimodal composite feature; The multimodal composite features are deeply fused through a preset linear layer to obtain cross-modal fused features; A cross-modal item-item graph is constructed using graph network technology and based on the cross-modal fusion features.

3. The multimodal recommendation method based on hierarchical fusion network according to claim 1, characterized in that, The step of calculating the feature difference score between each of the single-modal features of the items by performing multimodal difference learning through a variational autoencoder includes the following steps: The mean value and variance of the single-modal feature distribution of each item are calculated by a variational autoencoder. The variational posterior probability distribution and single-modal dataset distribution of each item's single-modal feature are calculated by combining the mean value and variance of the single-modal feature distribution; The feature differences between the individual modal features of each item are quantified using the KL divergence metric, and the feature difference scores between the individual modal features of each item are calculated by combining the variational posterior probability distribution and the distribution of the individual modal dataset.

4. The multimodal recommendation method based on hierarchical fusion network according to claim 1, characterized in that, The step of inputting multiple unimodal item-item graphs, cross-modal item-item graphs, and user-item graphs into a hierarchical fusion network, and outputting user feature representations and global fusion feature representations of items based on the feature difference scores and through the hierarchical fusion network includes the following steps: Multiple single-modal item-item graphs are input into a hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain multiple single-modal feature representations; The cross-modal item-item graph is input into the hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain the cross-modal feature representation; The user-item graph is input into the hierarchical fusion network, and convolution operations are performed through the hierarchical fusion network to obtain user feature representation and item interaction feature representation; All the single-modal feature representations are fused into a single-modal fused feature representation; Based on the feature difference score, the single-modal fusion feature representation and the cross-modal feature representation are fused into a hybrid fusion feature representation; The hybrid fusion feature representation and the item interaction feature representation are merged into a global fusion feature representation for the item.

5. The multimodal recommendation method based on hierarchical fusion network according to claim 4, characterized in that, The expression formula for the hybrid fusion feature representation is as follows: In the formula: d represents the hybrid fusion feature representation corresponding to the i-th target item. i This represents the feature difference score corresponding to the i-th target item. This represents the cross-modal feature representation corresponding to the i-th target item. This represents the single-modal fusion feature representation corresponding to the i-th target item.

6. A multimodal recommendation system based on a hierarchical fusion network, characterized in that, The system includes: The information acquisition module is used to acquire user information of the target user and multimodal information of the target items that have historical interaction records with the target user; The first graph network construction module is used to encode each item unimodal information in the multimodal information of the item through an encoder to obtain multiple item unimodal features, and to construct multiple unimodal item-item graphs based on each item unimodal feature using graph network technology. The second graph network construction module is used to perform cross-modal alignment and fusion of all the single-modal features to obtain cross-modal fused features, and to construct a cross-modal item-item graph using graph network technology and based on the cross-modal fused features. The third graph network construction module is used to construct a user-item graph by utilizing graph network technology and combining the user information, the historical interaction records, and the item multimodal information; The difference learning module is used to perform multimodal difference learning based on the single-modal features of the multiple items and through a variational autoencoder, and to calculate the feature difference score between each single-modal feature of the item. The feature output module is used to input multiple single-modal item-item graphs, cross-modal item-item graphs, and user-item graphs into a hierarchical fusion network, and output user feature representations and global fusion feature representations of items based on the feature difference scores and through the hierarchical fusion network. The recommendation result output module is used to combine the user feature representation and the global fusion feature representation of the item and calculate the item recommendation result for the target user using the TOP-K method.

7. The multimodal recommendation system based on a hierarchical fusion network according to claim 6, characterized in that, The second graph network construction module includes: The dimensionality reduction unit is used to perform dimensionality reduction processing on each of the single-modal features using a multilayer perceptron, so that all the single-modal features have the same feature dimension. The feature connection unit is used to connect all the dimensionality-reduced single-modal features into a multimodal composite feature; The feature fusion unit is used to perform deep fusion of the multimodal composite features through a preset linear layer to obtain cross-modal fused features; The graph network construction unit is used to construct a cross-modal item-item graph using graph network technology and based on the cross-modal fusion features.

8. The multimodal recommendation system based on a hierarchical fusion network according to claim 6, characterized in that, The differential learning module includes: The first difference learning computation unit is used to calculate the average value and variance of the single-modal feature distribution of each item's single-modal feature through a variational autoencoder. The second difference learning computation unit is used to calculate the variational posterior probability distribution and single-modal dataset distribution of each item's single-modal feature by combining the mean value of the single-modal feature distribution and the variance of the single-modal feature distribution; The difference score calculation unit is used to quantify the feature differences between the individual modal features of each item using the KL divergence metric, and to calculate the feature difference score between the individual modal features of each item by combining the variational posterior probability distribution and the distribution of the individual modal dataset.

9. The multimodal recommendation system based on a hierarchical fusion network according to claim 6, characterized in that, The feature output module includes: The first convolution operation unit is used to input multiple single-modal item-item graphs into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain multiple single-modal feature representations; The second convolution operation unit is used to input the cross-modal item-item graph into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain the cross-modal feature representation; The third convolution operation unit is used to input the user-item graph into the hierarchical fusion network, and perform convolution operations through the hierarchical fusion network to obtain user feature representation and item interaction feature representation; The first feature fusion unit is used to fuse all the single-modal feature representations into a single-modal fused feature representation; The second feature fusion unit is used to fuse the single-modal fusion feature representation and the cross-modal feature representation into a hybrid fusion feature representation based on the feature difference score; The third feature fusion unit is used to fuse the hybrid fusion feature representation and the item interaction feature representation into a global fusion feature representation of the item.

10. The multimodal recommendation system based on a hierarchical fusion network according to claim 9, characterized in that, The expression formula for the hybrid fusion feature representation is as follows: In the formula: d represents the hybrid fusion feature representation corresponding to the i-th target item. i This represents the feature difference score corresponding to the i-th target item. This represents the cross-modal feature representation corresponding to the i-th target item. This represents the single-modal fusion feature representation corresponding to the i-th target item.

Images