Multi-modal recommendation system and method based on denoising and interest alignment

Abstract

The invention discloses a multi-modal recommendation system and method based on denoising and interest alignment, and the system comprises a denoising image convolution module, a user interest alignment module and a prediction and sorting module, and the denoising image convolution module carries out the dual-stage denoising at an edge level and a node level to obtain enhanced representation; the user interest alignment module aligns the interests of users and the modality semantics of articles through comparative learning, and after feature projection and aggregation, aggregation modal features are obtained and semantic alignment is realized; the prediction and sorting module constructs an article semantic graph and outputs through a lightweight graph convolutional network to obtain article semantic embedding representations; and comprehensively enhancing the representation and aggregating the modal features to obtain the final representation of the article, and calculating the Bayesian personalized ranking (BPR) to obtain the predicted score and ranking of each user for each article. According to the method, noise is removed in the structure and representation double layers, robust embedding is obtained, and meanwhile, the phenomenon of modal alignment but interest deviation is reduced by explicitly shortening the semantic distance between user interest and commodity modal features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information recommendation, and particularly relates to a multi-modal recommendation system and method based on denoising and interest alignment. BACKGROUND

[0002] The existing recommendation technology directly splices visual / text features as auxiliary vectors into item embeddings through matrix factorization (MF), or constructs a user-item bipartite graph, aggregates and propagates multi-modal signals through graph convolution network (GCN), and simultaneously forces the image and text embeddings of the same commodity learned by the model to be as close as possible through cross-modal alignment. However, the above-mentioned existing technology is sensitive to noise samples, ignores user interest, is not suitable for multi-modal scenarios, and the way of adding noise is too simple, which can easily cover the original information with noise and reduce the recommendation performance. SUMMARY

[0003] In view of the above problems in the prior art, the present application provides a multi-modal recommendation system and method based on denoising and interest alignment, which removes noise on both structure and representation levels, obtains robust embedding, and reduces the phenomenon of modal alignment but interest deviation by explicitly narrowing the semantic distance between user interest and commodity modal features.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a multi-modal recommendation system based on denoising and interest alignment, comprising a denoising graph convolution module, a user interest alignment module and a prediction and ranking module, wherein: the denoising graph convolution module performs two-stage denoising on the edge and node levels to obtain enhanced representations; the user interest alignment module aligns the semantics of user interest and item modal through contrastive learning, obtains aggregated modal features after feature projection and aggregation, and realizes semantic alignment; the prediction and ranking module constructs an item semantic graph, outputs item semantic embedding representations through a lightweight graph convolution network, obtains the final representation of the item by integrating the enhanced representation and the aggregated modal feature, and calculates the Bayesian personalized ranking (BPR) to obtain the predicted score and ranking of each user for each item.

[0006] The two-stage denoising refers to: first, using the interaction data to construct a user-item bipartite graph , and initializing user representations and item representations , collectively referred to as representations , then performing degree-sensitive pruning denoising on the bipartite graph to reduce the edge-level noise brought by the interaction, and obtaining the pruned graph , and the representations Input to The embedded representations of users and items are obtained through a lightweight graph convolutional network (LightGCN). Then, adaptive noise perturbation is added to the embedded representation to reduce node-level noise caused by graph convolution smoothing and sparsity, resulting in an enhanced representation. .

[0007] The semantic alignment mentioned refers to aligning the semantics of user interests and item modalities through contrastive learning: First, based on the different characteristics and dimensions of modal features of items, such as text features and image features, they are aggregated after feature projection transformation to the same semantic space to obtain the aggregated modal features of the items. Finally, by using the widely adopted InfoNCE, the mutual information between the user's interests and the modalities of the items they have interacted with is maximized, thereby achieving semantic alignment between the two.

[0008] The aforementioned item semantic embedding representation is constructed by building an item semantic graph through feature cosine similarity matrices of different modalities of the item, inputting the item representation into the item semantic graph, and outputting it through a lightweight graph convolutional network.

[0009] The denoising graph convolutional module includes: a degree-sensitive pruning unit, a graph convolutional network unit, and an adaptive noise perturbation unit, wherein: the degree-sensitive pruning single-source unit first determines the noise based on user interaction history information. Construct a user-item bipartite graph For the user's node set, For the set of nodes of the items, This is the set of edges representing the interaction information between users and items. Degree-sensitive pruning determines the probability of an edge being deleted based on the degree of the two vertices connected to it; the higher the degree of a node, the greater the probability of its deletion. The pruned graph consists of all nodes and the remaining edges after pruning. The graph convolutional network unit initializes the user and item representations obtained from Xavier. Input trimming image Information propagation is performed through a lightweight graph convolutional network (lightGCN) to obtain embedded representations of users and items. The adaptive noise perturbation unit is based on the embedded representation vector. Adaptive random noise is added to the modulus and the superquadrant in which it is located, resulting in an enhanced representation. .

[0010] The user interest alignment module includes: an item modality feature aggregation unit and a user interest and item modality alignment unit, wherein: the item modality feature aggregation unit first utilizes pre-extracted item modality features from the dataset. Feature projection is performed to obtain modal features in the same feature space. Then, the modal features are aggregated to obtain aggregated modal features. The user interest and item modality alignment unit calculates the user's enhanced embedded representation. Aggregation modal features of items Information Noise Contrast Estimation (InfoNCE) aligns user interests and item modalities at the semantic level.

[0011] The prediction and ranking module includes: a noise contrast unit, an item semantic graph convolution unit, and a Bayesian personalized ranking unit. The noise contrast unit calculates the InfoNCE of the embedded representations before and after noise enhancement, ensuring that noise does not obscure the original information while enhancing the model's robustness. The item semantic graph convolution unit constructs a cosine similarity matrix based on the representations of different modalities of the item. ,right After retaining the k largest values ​​in each column, binarization is performed to obtain a 0-1 matrix with only k 1s in each column. Then, the matrices under different modes are weighted and summed to obtain the matrix. ,Will As an item semantic map The adjacency matrix of the graph Input the item representation and execute LightGCN to obtain the item's modal representation. Bayesian personalized ranking units are based on enhanced embeddings of users and items. Aggregation modal features of items Modal representation of objects Calculate the Bayesian Personalized Ranking Loss (BPR loss) to obtain each user's preference score for all items, and make recommendations and predictions based on the preference scores.

[0012] Technical effect

[0013] This invention uses adaptive noise addition technology in conjunction with contrastive learning to align user interests with item modalities. It limits the noise range based on the modulus of the embedded representation and the superquadrant, thereby improving model robustness while preserving the original information of the embedded representation to the greatest extent and avoiding the negative impact of neglecting the user side in existing multimodal alignment methods. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the system of the present invention;

[0015] Figure 2 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0016] like Figure 1As shown, this embodiment relates to a multimodal recommendation method based on the above system for denoising and interest alignment, including:

[0017] Step 1: Construct a user-item bipartite graph, specifically by extracting user IDs, item IDs, and the interaction matrix from the dataset. The set of nodes consists of all users and items. The set of edges represents user interactions with items (e.g., clicks, purchases). Construct a bipartite graph ;

[0018] Step 2, Degree-sensitive pruning and edge-level noise reduction: The edge set obtained in Step 1... Each edge (u, i) is assigned a weight. ,in For the degrees of nodes u and i, set the dropout hyperparameter, typically 0.8. Weighted random sampling (1-dropout) edges are used to retain the sampled points, resulting in a trimmed image after edge denoising. .

[0019] Step 3, Graph Convolution: In the graph The above is represented by nodes. As input to layer 0 Perform L layers of lightweight graph convolution (LightGCN), specifically: , ,in: Are nodes u and i in The neighbor set on the Lth layer. Output at the Lth layer. Embedded representation of nodes ;

[0020] Step 4: Noise perturbation and node-level denoising: Set the hyperparameter 'scale' to modify the node representation obtained in Step 3. Apply within the same superquadrant, with a modulus length not exceeding *scale random uniform noise The enhanced representation of the node is obtained. Without causing ambiguity, the following text... Enhanced characterization The user section and the item section.

[0021] Step 5, Modal Feature Mapping and Aggregation: Read the preprocessed modal features, map the features of different dimensions under different modalities to the same low-dimensional feature space, and aggregate them to obtain the aggregated modal features of the item, specifically: ,in: Modal features for items in textual and visual modalities. All are learnable parameters;

[0022] Step 6: Construct an item-item semantic graph Specifically, it includes:

[0023] 6.1 Using all items as the node set For each mode Calculate the feature cosine similarity matrix of the item in this modality. in: For items In modality Modal characteristics under [the specified conditions].

[0024] 6.2 Preserving the cosine similarity matrix Take the k largest numbers from each column, binarize them into a 0-1 matrix, and then normalize them using Laplace's normalization: The matrix is ​​obtained by weighted averaging the normalized matrices under different modes. The normalized adjacency matrix;

[0025] 6.3 Representing items on the semantic graph Using this as input, perform the same L-layer graph convolution as in step 3 to obtain item representations with semantic information. ;

[0026] Step 7: Jointly train the model, specifically including:

[0027] 7.1 Enhanced characterization obtained in step 4 For users to ultimately embed representations, item representations The item representation with semantic information obtained in step 6 The summation serves as the final embedded representation of the item, while simultaneously using the aggregated modal features obtained in step 5. As a modal representation of the item, calculate the Bayesian Personalized Ranking (BPR) loss: ,in: , That is, for users , These are positive samples that exhibit interaction. These are non-interactive negative samples obtained through random negative sampling.

[0028] 7.2 Calculate the self-supervised contrast loss , : Use self-supervised contrastive learning methods to constrain step 4 and Consistency, respectively for the user side and item side The calculation is performed using embedding pairs of the same node before and after noise addition as positive samples, and embedding pairs between different nodes before and after noise addition as negative samples. The corresponding information-noise contrastive estimates (InfoNCE) are calculated for each, resulting in the self-supervised loss. in: This is a temperature hyperparameter, typically set to 0.2.

[0029] 7.3 Calculating Interest-Mode Alignment Loss In the same training batch of data, user-item pairs with interaction are considered positive samples, and all other user-item pairs without interaction are considered negative samples. The user augmentation representation obtained in step 4 is then calculated. The aggregated modal features obtained in step 5 Information noise contrast estimation (InfoNCE) between the two is used to obtain the alignment loss. ,in: It is a temperature hyperparameter.

[0030] 7.4 Calculate the total loss function and optimize the model parameters accordingly, specifically as follows: This is a hyperparameter.

[0031] During the online simulation phase, the dataset is divided into training, validation, and test sets in an 8:1:1 ratio. The gradients of the parameters representing users and items and modal feature mappings are calculated using the aforementioned loss function. The model parameters are trained using gradient descent. The recall rate (recall@k) and normalized depreciation cumulative gain (NDCG@k) of the top k (usually 20) items in the Bayesian personalized ranking are used as validation and testing metrics for model training and testing.

[0032] Based on specific practical experiments, in an environment with an NVIDIA RTX-4080 single card, PyTorch 2.6 + CUDA 13.0, using the Amazon Baby / Sports / Clothing dataset as the dataset, with 4096 dimensions of visual features and 384 dimensions of text features, the Adam optimizer was set with embedding dimension d=64, number of graph convolutional layers L=3, pruning rate dropout=0.8, perturbation scale=0.2, temperature τ=0.2, α=1e-3, β=1e-2, learning rate 1e-3, batch=2048, epoch=1000, and an early stopping strategy was adopted when no better results were obtained after 20 consecutive epochs. The experimental results obtained by the above multimodal recommendation system based on denoising and interest alignment are shown in Table 1.

[0033] Table 1 Comparison of experimental results and baseline model

[0034] The indicator that achieves the best result among the comparison methods in the table is underlined, and the indicator that achieves the best result among all methods is displayed in bold.

[0035] Quantitative analysis shows that DAUIM outperforms all baseline models in Recall@20 across all datasets, achieving improvements of 1.91%, 2.02%, and 3.10% on the baby, sports, and clothing datasets, respectively. On the sports and clothing datasets, DAUIM outperforms all state-of-the-art baseline models across all metrics. On the sports dataset, the DAUIM model outperforms the best contrastive learning baseline model (LGMRec) by 4.29% in Recall@10 and by 6.63% in NDCG@10; on the clothing dataset, the DAUIM model outperforms the best baseline model (FREEDOM) with graph denoising by 6.41% and 5.92% in Recall@10 and NDCG@10, respectively. These significant improvements highlight the key role of the two-stage denoising and alignment of user interests with product modalities in this invention. As shown in the ablation experiments in Table 2, both modules play a crucial role in improving the metrics.

[0036] Table 2 Ablation Experiment Results

[0037] In the table, DAIIM represents the complete model, w / o AL represents the model with the alignment module removed, and w / o DN represents the model with the denoising module removed.

[0038] To further explore the model's sensitivity to the two modules, different weight hyperparameters were set. Observe the changes in the value of the indicator, such as Figure 2 As shown. Experimental data indicates that when Value from Upgraded to The system performance initially improved and then declined. This indicates that the denoising module can significantly improve recommendation performance, but if the weights are too large and denoising is too aggressive, it can inhibit the model from acquiring effective semantic information. This also indirectly confirms the importance of preserving the original semantic information while denoising in this invention. Overall, the model performance is affected by... The sensitivity to value changes is not as high as However, when properly configured The fact that the model can achieve an improvement of more than 2% indicates that aligning with user interests is important. However, because interests are somewhat subjective, the model is less sensitive to the alignment module than the denoising module.

[0039] In summary, this invention introduces a two-stage denoising process involving edges and nodes, which preserves the original information to the greatest extent while ensuring the robustness of the model. At the same time, it aligns user interests with item modalities, enhancing the interpretability of the model while making full use of user-side information in multimodal scenarios.

[0040] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A multimodal recommendation system based on denoising and interest alignment, characterized in that, include: The system includes a denoised graph convolution module, a user interest alignment module, and a prediction and ranking module. The denoised graph convolution module performs two-stage denoising at both the edge and node levels to obtain enhanced representations. The user interest alignment module aligns the semantics of user interests and item modalities through contrastive learning. After feature projection and aggregation, it obtains aggregated modal features and achieves semantic alignment. The prediction and ranking module constructs an item semantic graph and outputs the item semantic embedding representation through a lightweight graph convolutional network. Then, it integrates the enhanced representation and aggregated modal features to obtain the final representation of the item and calculates the Bayesian personalized ranking (BPR) to obtain the predicted score and ranking of each user for each item.

2. The multimodal recommendation system based on denoising and interest alignment according to claim 1, characterized in that, The aforementioned two-stage denoising refers to: first using interactive data... Construct a user-item bipartite graph and initialize user representation and item representation Collectively referred to as representation Then, regarding the two-part diagram Degree-sensitive pruning is used to reduce edge-level noise caused by interactions, and the pruned graph is obtained. , will characterize Input to The embedded representations of users and items are obtained through a lightweight graph convolutional network (LightGCN). Then, adaptive noise perturbation is added to the embedded representation to reduce node-level noise caused by graph convolution smoothing and sparsity, resulting in an enhanced representation. .

3. The multimodal recommendation system based on denoising and interest alignment according to claim 1, characterized in that, The semantic alignment mentioned refers to aligning the semantics of user interests and item modalities through contrastive learning: First, based on the different characteristics and dimensions of modal features of items, such as text features and image features, they are aggregated after feature projection transformation to the same semantic space to obtain the aggregated modal features of the items. Finally, by using the widely adopted InfoNCE, the mutual information between the user's interests and the modalities of the items they have interacted with is maximized, thereby achieving semantic alignment between the two.

4. The multimodal recommendation system based on denoising and interest alignment according to claim 1, characterized in that, The aforementioned item semantic embedding representation is constructed by building an item semantic graph through feature cosine similarity matrices of different modalities of the item, inputting the item representation into the item semantic graph, and outputting it through a lightweight graph convolutional network.

5. The multimodal recommendation system based on denoising and interest alignment according to any one of claims 1-4, characterized in that, The denoising graph convolutional module includes: a degree-sensitive pruning unit, a graph convolutional network unit, and an adaptive noise perturbation unit, wherein: the degree-sensitive pruning single-source unit first determines the noise based on user interaction history information. Construct a user-item bipartite graph For the user's node set, For the set of nodes of the items, The edge set consists of user and item interaction information. Degree-sensitive pruning determines the probability of deleting an edge based on the degree of the two nodes connected to it; the higher the degree of a node, the greater the probability of its deletion. The pruned graph is composed of all nodes and the remaining edges after pruning. The graph convolutional network unit initializes the user and item representations obtained from Xavier. Input trimming image Information propagation is performed through a lightweight graph convolutional network (lightGCN) to obtain embedded representations of users and items. The adaptive noise perturbation unit is based on the embedded representation vector. Adaptive random noise is added to the modulus and the superquadrant in which it is located, resulting in an enhanced representation. .

6. The multimodal recommendation system based on denoising and interest alignment according to any one of claims 1-4, characterized in that, The user interest alignment module includes: an item modality feature aggregation unit and a user interest and item modality alignment unit, wherein: the item modality feature aggregation unit first utilizes pre-extracted item modality features from the dataset. Feature projection is performed to obtain modal features in the same feature space. Then, the modal features are aggregated to obtain aggregated modal features. The user interest and item modality alignment unit calculates the user's enhanced embedded representation. Aggregation modal features of items Information Noise Contrast Estimation (InfoNCE) aligns user interests and item modalities at the semantic level.

7. The multimodal recommendation system based on denoising and interest alignment according to any one of claims 1-4, characterized in that, The prediction and ranking module includes: a noise contrast unit, an item semantic graph convolution unit, and a Bayesian personalized ranking unit. The noise contrast unit calculates the InfoNCE of the embedded representations before and after noise enhancement, ensuring that noise does not obscure the original information while enhancing the model's robustness. The item semantic graph convolution unit constructs a cosine similarity matrix based on the representations of different modalities of the item. ,right After retaining the k largest values ​​in each column, binarization is performed to obtain a 0-1 matrix with only k 1s in each column. Then, the matrices under different modes are weighted and summed to obtain the matrix. ,Will As an item semantic map The adjacency matrix of the graph Input the item representation and execute LightGCN to obtain the item's modal representation. Bayesian personalized ranking units are based on enhanced embeddings of users and items. Aggregation modal features of items Modal representation of objects Calculate the Bayesian Personalized Ranking Loss (BPR loss) to obtain each user's preference score for all items, and make recommendations and predictions based on the preference scores.

8. A multimodal recommendation method based on the system described in any one of claims 1-7, characterized in that, include: Step 1: Construct a user-item bipartite graph, specifically by extracting user IDs, item IDs, and the interaction matrix from the dataset. The set of nodes consists of all users and items. The set of edges represents user interactions with items (e.g., clicks, purchases). Construct a bipartite graph ; Step 2, Degree-sensitive pruning and edge-level noise reduction: The edge set obtained in Step 1... Each edge (u, i) is assigned a weight. ,in Set the dropout hyperparameter for the degree of nodes u and i; Weighted random sampling (1-dropout) edges are used to retain the sampled points, resulting in a trimmed image after edge denoising. ; Step 3, Graph Convolution: In the graph The above is represented by nodes. As input to layer 0 Perform L layers of lightweight graph convolution (LightGCN), specifically: , ,in: Are nodes u and i in The neighbor set on the Lth layer, output at the Lth layer. Embedded representation of nodes ; Step 4: Noise perturbation and node-level denoising: Set the hyperparameter 'scale' to modify the node representation obtained in Step 3. Apply within the same superquadrant, with a modulus length not exceeding *scale random uniform noise The enhanced representation of the node is obtained. Without causing ambiguity, the following text... Enhanced characterization The user section and the item section; Step 5, Modal Feature Mapping and Aggregation: Read the preprocessed modal features, map the features of different dimensions under different modalities to the same low-dimensional feature space, and aggregate them to obtain the aggregated modal features of the item, specifically: ,in: Modal features for items in textual and visual modalities. All are learnable parameters; Step 6: Construct an item-item semantic graph ; Step 7: Jointly train the model and perform real-time multimodal recommendations using the trained model in the online phase.

9. The multimodal recommendation method according to claim 8, characterized in that, Step 6 specifically includes: 6.1 Using all items as the node set For each mode Calculate the feature cosine similarity matrix of the item in this modality. in: For items In modality Modal characteristics below; 6.2 Preserving the cosine similarity matrix Take the k largest numbers from each column, binarize them into a 0-1 matrix, and then normalize them using Laplace's algorithm: The matrix is ​​obtained by weighted averaging the normalized matrices under different modes. The normalized adjacency matrix; 6.3 Representing items on the semantic map Using this as input, perform the same L-layer graph convolution as in step 3 to obtain item representations with semantic information. .

10. The multimodal recommendation method according to claim 8, characterized in that, Step 7 specifically includes: 7.1 Enhanced characterization obtained in step 4 For users to ultimately embed representations, item representations The item representation with semantic information obtained in step 6 The sum is used as the final embedded representation of the item, along with the aggregated modal features obtained in step 5. As a modal representation of the item, calculate the Bayesian Personalized Ranking (BPR) loss: ,in: , That is, for users , These are positive samples that exhibit interaction. These are non-interactive negative samples obtained through random negative sampling. 7.2 Calculate the self-supervised contrast loss , : Use self-supervised contrastive learning methods to constrain step 4 and Consistency, respectively for the user side and item side The calculation is performed using embedding pairs of the same node before and after noise addition as positive samples, and embedding pairs between different nodes before and after noise addition as negative samples. The corresponding information-noise contrastive estimates (InfoNCE) are calculated for each, resulting in the self-supervised loss. in: It is a temperature hyperparameter; 7.3 Calculating Interest-Mode Alignment Loss In the same training batch of data, user-item pairs with interaction are considered positive samples, and all other user-item pairs without interaction are considered negative samples. The user augmentation representation obtained in step 4 is then calculated. The aggregated modal features obtained in step 5 Information noise contrast estimation (InfoNCE) between the two is used to obtain the alignment loss. ,in: It is a temperature hyperparameter; 7.4 Calculate the total loss function and optimize the model parameters accordingly, specifically as follows: This is a hyperparameter.

Images