Multimodal data denoising framework, model training method, recommendation method and system

Abstract

The application discloses a kind of multimodal data denoising framework, model training method, recommended method and system, the multimodal data denoising framework in it includes modal perception re-labeling module, sorting loss construction and optimization module and knowledge distillation guide alignment module, through modal perception re-labeling module, user behavior log and multimodal article content are carried out interaction quality evaluation, identify and mark " clean interaction " and " suspected noise interaction ";Second, in sorting loss construction and optimization module, construct weighted DBPR loss function, dynamically adjust sample weight, improve the attention of model to high-quality interaction;Finally, in knowledge distillation guide alignment module, modal feature is mapped to recommended semantic space, utilizes Sinkhorn distance to construct distillation loss, realizes the effective alignment of modal feature and recommended semantics.The application significantly improves the accuracy, robustness and generalization ability of recommended system, is applicable to picture-text recommendation, audio-video content recommendation, advertisement click prediction and the like scene.

Description

Technical Field

[0001] This invention relates to the field of multimodal data denoising technology, and more specifically, to a multimodal data denoising framework, model training method, recommendation method, and system. Background Technology

[0002] Currently, multimodal machine learning systems demonstrate significant advantages in cross-modal feature fusion and complex behavior modeling. However, these systems face severe semantic consistency issues when processing heterogeneous modal data: firstly, structural differences exist in the underlying feature spaces of different modalities, making cross-modal semantic alignment difficult; secondly, distribution shifts easily occur between the learned modal representations and downstream task objectives, affecting model performance. These challenges make the development of techniques that can effectively address the problem of multimodal semantic heterogeneity particularly important.

[0003] In the field of multimodal learning, cross-modal alignment methods based on contrastive learning have become the mainstream technical approach. This method constructs a contrastive loss function between modalities and utilizes deep neural networks to jointly embed heterogeneous data such as text and images, enabling the semantic representations of different modalities to be aligned in a shared latent space. Modern feature extractors (such as ResNet and BERT) can capture high-order semantic features of each modality. Combined with task-related supervision signals, these methods achieve effective semantic fusion while preserving modal characteristics.

[0004] However, through extensive research and practice, the inventors of this application have discovered significant technical limitations in existing methods: their effectiveness heavily relies on the core assumption that "transferable underlying semantic structures exist between modalities." This assumption holds true when processing homogeneous multimodal data (such as text content with images); however, when facing complex scenarios like short videos—which require processing multi-source heterogeneous information such as visual scenes, audio, and bullet screen text simultaneously—different modalities may carry complementary or even conflicting semantics, making it difficult for the model to construct a unified representation space. More importantly, domain-adaptive training for specific tasks may distort the original semantic relationships between modalities, leading to pseudo-alignment and ultimately impairing the model's generalization ability and interpretability. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention proposes a universal Guided Simultaneous Distillation Denoising Framework (GUIDER). This framework can be integrated as a plug-and-play module into existing multimodal recommendation models, significantly improving recommendation accuracy and system robustness.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a multimodal data denoising framework is provided, comprising: The modality-aware recalibration module is used to extract features from the input multimodal item content, obtain the multimodal features of the items, calculate the similarity between user behavior features and the modal features of the items as a quality score of user-item interaction, and recalibrate the original user-item interaction data based on the relationship between the quality score and the preset quality score threshold. The ranking loss construction and optimization module is used to dynamically estimate the denoising weight of each positive sample based on the user-item interaction quality score, and introduce the denoising weight into the BPR loss function to form a weighted DBPR loss function. Here, the positive sample is the item that has interacted with the user, the BPR loss function is the Bayesian personalized ranking loss function, and the DBPR loss function is the dynamic Bayesian personalized ranking loss function. The knowledge distillation guided alignment module is used to map multimodal features to the recommendation semantic space to obtain modal semantic representations. The recommendation semantic representations are used as the target distribution and the modal features are used as the source distribution to construct a distillation loss function based on the optimal transmission distance.

[0007] Based on the same inventive concept, and according to a second aspect of the present invention, a model training method based on the multimodal data denoising framework described in the first aspect is provided, comprising: Collect user behavior data and multimodal item data, construct a user-item interaction matrix, standardize the multimodal features, and construct training, validation, and test sets; The modality-aware recalibration module is used to perform quality assessment and recalibration on the original user-item interaction data; The ranking loss construction and optimization module dynamically calculates the denoising weight of each positive sample based on the interactive recalibration results, and constructs a weighted DBPR loss function. The knowledge distillation-guided alignment module maps multimodal features to the recommendation semantic space, with the recommendation semantic representation as the target distribution and the modal features as the source distribution, and constructs a distillation loss function based on the optimal transmission distance; The total loss function is constructed based on the weighted DBPR loss function and the distillation loss function based on the optimal transmission distance, and the loss weights are dynamically adjusted using a multi-task optimization method. Using the training set, the model is trained through the Adam optimizer. In each training cycle, the total loss function is calculated and backpropagation is performed to update the model parameters. Use the validation set to perform hyperparameter tuning and select the optimal model parameters.

[0008] In one implementation, user behavior data and multimodal item data are collected to construct a user-item interaction matrix. The multimodal features are then standardized to construct training, validation, and test sets, including: Collect user behavior data from recommendation system logs and perform data cleaning on the collected user behavior data; The cleaned user behavior data is converted into a user-item interaction matrix, where rows represent users, columns represent items, and matrix elements represent the intensity of user behavior towards items. For user-item pairs that have not interacted, the matrix element value is 0. Collect multimodal data of items, including text descriptions, cover images, and video clips, and standardize the data format; The text description is preprocessed, and a pre-trained language model is used to extract text feature vectors. The cover image is preprocessed, and a pre-trained convolutional neural network is used to extract image feature vectors. Keyframes are extracted from the video clips, and each frame is preprocessed. Convolutional neural networks are used to extract image features from each frame. The features of all frames are aggregated to obtain the video feature vector. The extracted multimodal feature vectors are standardized. Align the standardized multimodal feature vectors with the user-item interaction matrix; The aligned user-item interaction matrix and multimodal feature vectors are stored in a standard format and divided into training, validation and test sets.

[0009] In one implementation, a modality-aware recalibration module is used to perform quality assessment and recalibration on the original user-item interaction data, including: Extract the user ID, item ID, and user behavior features corresponding to each interaction record from the aligned user-item interaction matrix; For each user action, its type and intensity are mapped into a continuous vector through an embedding layer or feature encoder; For each item, the corresponding text feature vector, image feature vector and video feature vector are extracted from the standardized multimodal feature library, and then concatenated or weighted and fused to form a unified item modal feature representation. A cross-modal matching model is used to calculate the similarity score between the user behavior feature vector and the item modal feature vector, which serves as the quality score for each interaction record. Statistical analysis was performed on the quality scores of all interaction records, and the interactions were recalibrated as clean interactions and suspected noisy interactions based on preset thresholds. The tagged interaction data is merged with the standardized stored interaction matrix to form a new interaction matrix that includes interaction quality tags.

[0010] In one implementation, a ranking loss construction and optimization module dynamically calculates the denoising weights for each positive sample based on the interactive recalibration results, constructing a weighted DBPR loss function, including: Extract the interaction quality label corresponding to each positive sample from the new interaction matrix, and obtain the corresponding quality score; For each positive sample, the denoising weight of the sample is dynamically calculated based on the interaction quality label and quality score. Clean interaction samples are given higher weights, while suspected noisy interaction samples are given lower weights. The calculated denoising weights are associated with and stored with the corresponding positive samples; The calculated denoising weights are incorporated into the BPR loss function to construct the DBPR loss function.

[0011] In one implementation, the calculated denoising weights are incorporated into the BPR loss function to construct the DBPR loss function, including: Extract all positive samples and their corresponding negative samples to form triples for DBPR loss calculation, including users, positive items, and negative items; For each triple, obtain the user embedding vector, positive item embedding vector, and negative item embedding vector from the recommendation model, and ensure that the dimensions of the embedding vectors are consistent. Calculate the user's prediction score for positive items and the user's prediction score for negative items in each triplet, and use dot product or cosine similarity to obtain two scalar values, which represent the user's preference for positive items and negative items, respectively. The difference between the predicted scores of each triplet is calculated to obtain the difference between the scores of positive and negative samples. The difference is then input into the Sigmoid function to obtain the probability value of each triplet, which represents the probability that the score of the positive sample is higher than the score of the negative sample. Take the logarithm of the probability value of each triplet and multiply it by the denoising weight of the corresponding positive sample to obtain the weighted log-likelihood loss of each triplet. The weighted log-likelihood loss of all triples is summed and negative to obtain the final DBPR loss function value.

[0012] In one implementation, a knowledge distillation-guided alignment module is used to map multimodal features to a recommendation semantic space, with the recommendation semantic representation as the target distribution and the modal features as the source distribution. A distillation loss function based on the optimal transmission distance is constructed, including: Obtain the embedded representations of users and items in the current batch, as the basis for the target distribution of the recommendation semantic space; From the standardized multimodal feature library, extract the text feature vector, image feature vector and video feature vector corresponding to the current batch of items, and concatenate or weightedly fuse them to form a unified modal feature representation of the items; A modal feature mapping network is constructed to map high-dimensional modal feature vectors to a low-dimensional space that is the same as the recommendation semantic space. The weight parameters of the mapping network are initialized by Xavier and then optimized. Using the mapped modal feature vector as the source distribution and the recommended semantic embedding vector as the target distribution, the SamplesLoss function is used to construct the Sinkhorn distance calculation object. Call the Sinkhorn distance calculation object, input the source distribution and target distribution, and calculate the optimal transmission distance loss value for the current batch; The calculated optimal transmission distance loss value is combined with the weighted DBPR loss function to form the total loss function of the model. Backpropagation is performed on the total loss function to calculate the gradients of the mapping network and the recommendation model. The Adam optimizer is then used to update the parameters of both the mapping network and the recommendation model simultaneously.

[0013] In one implementation, the model is trained using a training set via the Adam optimizer. At each training epoch, the total loss function is calculated and backpropagation is performed to update the model parameters, including: Load the training set data, make a one-to-one correspondence between the user-item interaction records in the training set and the multimodal feature vectors, perform data transformation, and move the data to the GPU device; The recommendation model parameters, including the user embedding matrix, item embedding matrix, and modal feature mapping network, are initialized using the Xavier uniform initialization method. Define the Adam optimizer, set the learning rate, weight decay coefficient and gradient clipping threshold, and pass the parameters of the recommendation model and the mapping network into the optimizer at the same time; At the beginning of each training cycle, the training set data is divided into multiple batches, each batch containing a fixed number of user-item interaction records and corresponding multimodal feature vectors; For each batch of data, user embedding vectors, positive item embedding vectors, and negative item embedding vectors are calculated using a recommendation model, and mapped modal feature vectors are calculated using a modal feature mapping network, where all vectors have the same dimension. Based on the constructed weighted DBPR loss function and the optimal transport distillation loss function, calculate the total loss function value for the current batch and record the value of each loss term; Backpropagation is performed on the total loss function to calculate the gradients of the recommendation model and the mapping network. The optimizer is used to update the model parameters. The gradient pruning operation is performed after backpropagation. After each training cycle, evaluate the model performance on the validation set and adjust the hyperparameters based on the validation set performance. Repeat the above training steps until the model's performance on the validation set no longer improves or reaches the preset maximum number of training epochs, then save the trained model parameters.

[0014] Based on the same inventive concept, according to the third aspect of the present invention, a recommended method is provided, which implements the model obtained by the method of the second aspect, including: The embedding representations of users and items are obtained from the trained model. By calculating the similarity between user embeddings and item embeddings, the predicted rating of each candidate item by the user is obtained. Candidate items are sorted according to their predicted scores, and the top K items with the highest scores are selected to generate a recommendation list as the personalized recommendation result.

[0015] Based on the same inventive concept, according to a fourth aspect of the present invention, a recommendation system is provided, comprising the multimodal data denoising framework described in the first aspect and a recommendation prediction output module, wherein the recommendation prediction output module is used for: The embedding representations of users and items are obtained from the trained model. By calculating the similarity between user embeddings and item embeddings, the predicted rating of each candidate item by the user is obtained. Candidate items are sorted according to their predicted scores, and the top K items with the highest scores are selected to generate a recommendation list, which serves as the personalized recommendation result. Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: This invention discloses a multimodal data denoising framework, comprising three core modules: a modality-aware recalibration module, a de-ranking loss construction and optimization module, and a knowledge distillation-guided alignment module. The modality-aware recalibration module assesses the interaction quality of user behavior logs and multimodal item content, identifying and labeling "clean interactions" and "suspected noisy interactions." In the de-ranking loss construction and optimization module, a confidence-based weighted DBPR loss function is constructed, dynamically adjusting sample weights to enhance the model's focus on high-quality interactions. In the optimal transport-guided knowledge distillation module, modal features are mapped to the recommendation semantic space, and a distillation loss is constructed using Sinkhorn distance to achieve effective alignment between modal features and recommendation semantics.

[0016] Furthermore, this invention provides a method for training a recommendation model using a multimodal data denoising framework, resulting in a well-trained recommendation model that can significantly improve the accuracy, robustness, and generalization ability of the recommendation system. This method is applicable to scenarios such as image and text recommendation, audio and video content recommendation, and ad click prediction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a block diagram of the multimodal data denoising framework in an embodiment of the present invention; Figure 2 This is a flowchart of the training method using a multimodal data denoising framework in an embodiment of the present invention; Figure 3 This is a schematic diagram of the recommendation system in an embodiment of the present invention. Detailed Implementation

[0019] Example 1 like Figure 1 As shown, the present invention provides a multimodal data denoising framework, comprising: The modality-aware recalibration module 101 is used to extract features from the input multimodal item content, obtain the multimodal features of the items, calculate the similarity between user behavior features and the modal features of the items as a quality score of user-item interaction, and recalibrate the original user-item interaction data according to the relationship between the quality score and the preset quality score threshold. The ranking loss construction and optimization module 102 is used to dynamically estimate the denoising weight of each positive sample based on the user-item interaction quality score, and introduce the denoising weight into the BPR loss function to form a weighted DBPR loss function. Here, the positive sample is the item that has interacted with the user, the BPR loss function is the Bayesian personalized ranking loss function, and the DBPR loss function is the dynamic Bayesian personalized ranking loss function. The knowledge distillation guided alignment module 103 is used to map multimodal features to the recommendation semantic space to obtain modal semantic representations. The recommendation semantic representations are used as the target distribution and the modal features are used as the source distribution to construct a distillation loss function based on the optimal transmission distance.

[0020] Specifically, the modality-aware recalibration module performs quality assessment and recalibration on the original user-item interaction data to identify and distinguish between high-quality interactions and noisy interactions. This module first extracts features from the input multimodal item content (such as text descriptions, image covers, video clips, etc.), obtaining text feature vectors, image feature vectors, and video feature vectors respectively. Then, based on a cross-modal matching model, it calculates the similarity score between user behavior features and item modal features to obtain a quality score for each interaction. According to a preset quality score threshold, the interaction data is reordered and labeled as "clean interactions" and "suspected noisy interactions," thereby effectively improving the quality of the input data and providing a high-quality data foundation for subsequent ranking optimization and knowledge distillation.

[0021] The ranking loss construction and optimization module, based on the traditional Bayesian Personalized Ranking (BPR) loss, introduces sample denoising weights calculated based on modal content relevance to construct a denoised ranking loss function suitable for implicit feedback scenarios. Specifically, this module dynamically estimates the denoising weight of each positive sample based on the interaction quality score output by the modality-aware recalibration module and incorporates it into the BPR loss function to form a weighted DBPR loss function. By optimizing this loss function, the model can pay more attention to high-quality interaction samples and reduce the negative impact of noisy samples on model training, thereby significantly improving the ranking performance and robustness of the recommendation system.

[0022] The knowledge distillation-guided alignment module aims to achieve effective alignment between modal features and the recommendation semantic space, thereby improving the generalization ability and robustness of recommendation models. This module first maps multimodal features (text, images, videos, etc.) to the recommendation semantic space to obtain modal semantic representations. Then, using the recommendation semantic representations as the target distribution and the modal features as the source distribution, an optimal transmission loss function based on optimal transmission distance (such as Sinkhorn distance or Entropy-regularized OT) is constructed. By jointly optimizing this loss function with the ranking loss, the model can achieve effective alignment between modal features and recommendation semantics, thus fully utilizing multimodal information and improving the overall performance of the recommendation system.

[0023] Example 2 Based on the same inventive concept, this embodiment provides a model training method based on the multimodal data denoising framework of Embodiment 1. Please refer to [link to embodiment]. Figure 2 ,include: S1: Collect user behavior data and multimodal item data, construct a user-item interaction matrix, standardize the multimodal features, and construct training, validation, and test sets; S2: Use the modality-aware recalibration module to perform quality assessment and recalibration on the original user-item interaction data; S3: Using the ranking loss construction and optimization module, based on the interactive recalibration results, the denoising weight of each positive sample is dynamically calculated to construct a weighted DBPR loss function; S4: Use the knowledge distillation-guided alignment module to map multimodal features to the recommendation semantic space, with the recommendation semantic representation as the target distribution and the modal features as the source distribution, and construct a distillation loss function based on the optimal transmission distance; S5: Construct the total loss function based on the weighted DBPR loss function and the distillation loss function based on the optimal transmission distance, and dynamically adjust the loss weights using a multi-task optimization method; S6: Use the training set to train the model through the Adam optimizer. Calculate the total loss function and perform backpropagation in each training cycle to update the model parameters. S7: Use the validation set to perform hyperparameter tuning and select the optimal model parameters.

[0024] In practice, user behavior data in S1 includes clicks, browsing, and ratings.

[0025] S2 utilizes a modality-aware recalibration module to calculate a similarity score between user behavior and item modal features (as an interaction quality score). Based on a preset threshold, it labels interaction data as "clean interaction" or "suspected noisy interaction," thereby improving the quality of input data. In practice, the similarity score ranges from [0,1], with a higher similarity score indicating better interaction quality.

[0026] S3 allows the model to focus more on high-quality interactive samples and reduce the impact of noisy samples. S5 uses a multi-task optimization method (such as MinNormSolver) to dynamically adjust the loss weights to achieve optimal overall performance. S6 uses the training set from the dataset obtained in step S1 to train the model through the Adam optimizer. In each training epoch, the total loss function is calculated and backpropagation is performed to update the model parameters and ensure model convergence. S7 also includes evaluating model performance on the test set (such as NDCG, Recall, etc.) to ensure the model's generalization ability and practical application effectiveness.

[0027] In one implementation, S1 includes: S1.1: Collect user behavior data from the recommendation system logs and perform data cleaning on the collected user behavior data; S1.2: Convert the cleaned user behavior data into a user-item interaction matrix, where rows represent users, columns represent items, and matrix elements represent the intensity of user behavior towards items. For user-item pairs that have not interacted, the matrix element value is 0. S1.3: Collect multimodal data of items, including text descriptions, cover images and video clips, and standardize the data format; S1.4: Preprocess the text description, extract text feature vectors using a pre-trained language model, preprocess the cover image, extract image feature vectors using a pre-trained convolutional neural network, extract keyframes from the video clips, preprocess the image of each frame, extract the image features of each frame using a convolutional neural network, aggregate the features of all frames, and obtain the video feature vector. S1.5: Standardize the extracted multimodal feature vectors; S1.6: Align the standardized multimodal feature vectors with the user-item interaction matrix; S1.7: Store the aligned user-item interaction matrix and multimodal feature vectors in a standard format and divide them into training, validation and test sets.

[0028] Specifically, user behavior data is collected from recommendation system logs, including user ID, item ID, behavior type (such as click, browse, rating, etc.), behavior timestamp, and behavior intensity (such as rating score or number of clicks). After collecting the data, these data are stored in a structured format (such as CSV or database table) to ensure that each record is complete and without omissions.

[0029] Data cleaning is performed on the collected user behavior data, including deduplication to remove duplicate and invalid records (such as records with empty user IDs or item IDs), and filtering of abnormal behavior data (such as records with behavior intensity exceeding a reasonable range).

[0030] The text description in S1.3 includes the item title, descriptive text, etc. The cover image can be an item image URL or image file, and the video clip can be a video URL or video file.

[0031] In S1.4, text preprocessing refers to performing natural language processing operations, including word segmentation, stop word removal, and stemming. The pre-trained language model can be BERT. Cover image preprocessing involves image preprocessing operations such as size normalization and color space conversion (e.g., RGB to grayscale). The pre-trained convolutional neural network can be ResNet or other networks. Feature aggregation across all frames includes average pooling or max pooling.

[0032] S1.5's standardization process includes L2 norm normalization or Z-score normalization to ensure that different modal features have the same dimensions and distribution.

[0033] The data alignment operation in S1.6 ensures that the multimodal feature vector corresponding to each item corresponds one-to-one with the item column in the user-item interaction matrix, thus avoiding data misalignment.

[0034] In S1.7, the standard format can be either a NumPy array or a PyTorch tensor.

[0035] In one implementation, S2 includes: S2.1: Extract the user ID, item ID, and user behavior features corresponding to each interaction record from the aligned user-item interaction matrix; S2.2: For each user action, its type and intensity are mapped to a continuous vector through an embedding layer or feature encoder; S2.3: For each item, extract the corresponding text feature vector, image feature vector and video feature vector from the standardized multimodal feature library, and concatenate or weightedly fuse them to form a unified item modal feature representation; S2.4: Use a cross-modal matching model to calculate the similarity score between the user behavior feature vector and the item modal feature vector, as the quality score for each interaction record; S2.5: Perform statistical analysis on the quality scores of all interaction records, and recalibrate them as clean interactions and suspected noisy interactions based on preset thresholds; S2.6: Merge the tagged interaction data with the standardized stored interaction matrix to form a new interaction matrix that includes interaction quality tags.

[0036] Specifically, the aligned user-item interaction matrix in S2.1 is obtained through the operation in S1.6. The operation in S2.2 facilitates the similarity calculation between user behavior and item modal features. The cross-modal matching model in S2.4 can be a cosine similarity function or a dot product similarity function. The preset threshold can be set according to the actual situation, for example, using the mean plus standard deviation or a fixed threshold such as 0.7. S2.6 ensures that the subsequent ranking loss construction and optimization modules can directly use this labeled information.

[0037] In one implementation, S3 includes: S3.1: Extract the interaction quality label corresponding to each positive sample from the new interaction matrix and obtain the corresponding quality score; S3.2: For each positive sample, dynamically calculate the denoising weight of the sample based on the interaction quality label and quality score, where clean interaction samples are assigned higher weights and suspected noisy interaction samples are assigned lower weights. S3.3: Associate and store the calculated denoising weights with the corresponding positive samples; S3.4: Introduce the calculated denoising weights into the BPR loss function to construct the DBPR loss function.

[0038] Specifically, positive samples refer to user-positive item pairs, representing items that have interacted with the user, while negative samples refer to user-negative item pairs, representing items that have not interacted with the user. Interaction quality labels refer to the calibration results of the modality-aware recalibration module, such as "clean" or "suspected noise."

[0039] In S3.2, clean interaction samples are assigned a higher weight, such as close to 1.0, while suspected noisy interaction samples are assigned a lower weight, such as close to 0.1. The weights can be calculated using a linear mapping or a nonlinear mapping based on a Gaussian function.

[0040] By using the associated storage in S3.3, it is ensured that each positive sample can correctly call its corresponding weight value during subsequent loss calculations, thus avoiding mismatch between weights and samples.

[0041] In one implementation, S3.4 includes: S3.4.1: Extract all positive samples and their corresponding negative samples to form triples for DBPR loss calculation, including users, positive items, and negative items; S3.4.2: For each triple, obtain the user embedding vector, positive item embedding vector, and negative item embedding vector from the recommendation model, and ensure that the dimensions of the embedding vectors are consistent. S3.4.3: Calculate the user's prediction score for positive items and the user's prediction score for negative items in each triplet. Use dot product or cosine similarity to calculate two scalar values, which represent the user's preference for positive items and negative items, respectively. S3.4.4: Calculate the difference between the predicted scores of each triplet to obtain the difference between the positive sample score and the negative sample score, and input the difference into the Sigmoid function to obtain the probability value of each triplet, which represents the probability that the positive sample score is higher than the negative sample score; S3.4.5: Take the logarithm of the probability value of each triplet and multiply it by the denoising weight of the corresponding positive sample to obtain the weighted log-likelihood loss of each triplet; S3.4.6: Sum the weighted log-likelihood losses of all triples and take the negative value to obtain the final DBPR loss function value.

[0042] Specifically, S3.4.2 also includes checking for gradients in the embedding vectors to ensure the correctness of subsequent backpropagation. S3.4.5 ensures that high-quality interaction samples contribute more to the loss function, while noisy samples contribute less. S3.4.6 sums the weighted log-likelihood losses of all triples and takes the negative value to ensure that a smaller loss function value indicates better model ranking performance. The DBPR loss function is calculated as follows:

[0043] in, For the set of triples in the training set, These represent users, positive items, and negative items, respectively. The denoising weights for positive samples (u,i) are... , This is a balancing coefficient used to control the weighting of similarity scores and quality pricing. For similarity scoring, normalize to [0,1]. As an indicator function, when user u interacts with item i, the tag... The value is 1 for clean interaction and 0 for suspected noise. , representing the prediction score of user u for item i. This represents the prediction score for user u regarding item j. The regularization coefficient is . These are the model parameters.

[0044] In one implementation, S4 includes: S4.1: Obtain the embedded representations of users and items in the current batch as the basis for the target distribution of the recommendation semantic space; S4.2: Extract the text feature vector, image feature vector and video feature vector corresponding to the current batch of items from the standardized multimodal feature library, and concatenate or weightedly fuse them to form a unified modal feature representation of the items; S4.3: Construct a modal feature mapping network to map high-dimensional modal feature vectors to a low-dimensional space that is the same as the recommendation semantic space. The weight parameters of the mapping network are initialized by Xavier and then optimized. S4.4: Using the mapped modal feature vector as the source distribution and the recommended semantic embedding vector as the target distribution, the SamplesLoss function is used to construct the Sinkhorn distance calculation object; S4.5: Call the Sinkhorn distance calculation object, input the source distribution and target distribution, and calculate the optimal transmission distance loss value for the current batch; S4.6: Combine the calculated optimal transmission distance loss value with the weighted DBPR loss function to form the model's total loss function; S4.7: Perform backpropagation on the total loss function, calculate the gradients of the mapping network and the recommendation model, and use the Adam optimizer to update the parameters of both the mapping network and the recommendation model simultaneously.

[0045] Specifically, S4.1 ensures that the embedding vector dimension is compatible with subsequent mapping operations. The modal feature mapping network in S4.3 typically adopts a multilayer perceptron (MLP) structure.

[0046] In S4.4, the SamplesLoss function from the geomloss library is used, with the distance type set to "sinkhorn", parameters p=1, and blur=0.1. This constructs a Sinkhorn distance calculation object. Here, loss is the distance type, set to "sinkhorn", indicating the optimal transmission distance using entropy regularization; p is the order of the Wasserstein distance, set to p=1, indicating the use of 1-Wasserstein distance to measure the minimum transmission cost between two distributions; blur is the entropy regularization coefficient, set to blur=0.1, used to smooth the optimal transmission plan. The smaller the value, the more accurate the calculation but the slower the convergence; the larger the value, the faster the calculation but the larger the approximation error; backend is the computation backend, using "tensorized" by default to support GPU acceleration.

[0047] In S4.5, the source distribution refers to the mapped modal features, the target distribution refers to the recommended semantic embedding, the optimal transmission distance loss value for the current batch is calculated, and the calculation process is ensured to be performed on the GPU to accelerate the computation.

[0048] S4.7 ensures effective alignment between modal features and the recommendation semantic space.

[0049] In one implementation, S6 includes: S6.1: Load training set data, make the user-item interaction records in the training set correspond one-to-one with the multimodal feature vectors, perform data transformation, and move the data to the GPU device; S6.2: Initialize the recommendation model parameters, including the user embedding matrix, item embedding matrix, and modal feature mapping network, using the Xavier uniform initialization method; S6.3: Define the Adam optimizer, set the learning rate, weight decay coefficient and gradient clipping threshold, and pass the parameters of the recommendation model and the mapping network into the optimizer at the same time; S6.4: At the beginning of each training cycle, the training set data is divided into multiple batches, each batch containing a fixed number of user-item interaction records and corresponding multimodal feature vectors; S6.5: For each batch of data, calculate the user embedding vector, positive item embedding vector, and negative item embedding vector through the recommendation model, and calculate the mapped modal feature vector through the modal feature mapping network, wherein all vectors have the same dimension; S6.6: Based on the constructed weighted DBPR loss function and the optimal transport distillation loss function, calculate the total loss function value for the current batch and record the value of each loss term; S6.7: Perform backpropagation on the total loss function, calculate the gradients of the recommendation model and the mapping network, update the model parameters using the optimizer, and perform gradient pruning after backpropagation; S6.8: After each training cycle, evaluate the model performance on the validation set and adjust the hyperparameters based on the validation set performance; S6.9: Repeat the above training steps until the model's performance on the validation set no longer improves or reaches the preset maximum number of training cycles, then save the trained model parameters.

[0050] Specifically, S6.1 converts the data into PyTorch tensor format and moves it to the GPU device to accelerate computation.

[0051] S6.2 employs the Xavier uniform initialization method to ensure that the initial values ​​of the parameters have an appropriate scale, thus avoiding gradient vanishing or exploding.

[0052] S6.3 simultaneously feeds the parameters of the recommendation model and the mapping network into the optimizer, thereby ensuring that all parameters can be updated synchronously.

[0053] S6.6 ensures that the loss calculation process is performed on the GPU and records the value of each loss term to monitor the training process.

[0054] The S6.7 gradient clipping operation is performed after backpropagation to prevent gradient explosion. Specifically, S6.7 includes: S6.7.1: After calculating the total loss function value in each forward propagation, the backward() method of the total loss tensor is called first to start the automatic differentiation mechanism and calculate the gradient of all tensors involved in the operation in the graph, including the user embedding, item embedding of the recommendation model and the weight parameters of the modality mapping network.

[0055] S6.7.2: During backpropagation, PyTorch automatically constructs a computation graph and applies the chain rule, starting from the loss function and calculating the gradient of each parameter in turn. The gradient values ​​are accumulated and stored in the .grad attribute of each tensor.

[0056] S6.7.3: Before calling optimizer.step() to update the parameters, use the torch.nn.utils.clip_grad_norm_() function to clip the gradients of all parameters and set the maximum norm to 1.0 to prevent gradient explosion.

[0057] S6.7.4: The gradient clipping operation only scales the gradient and does not change the gradient direction. After clipping, the optimizer.step() method is called. The optimizer adjusts the model parameters according to the update rules (such as Adam's first and second momentum) to complete one parameter update.

[0058] S6.7.5: After the parameters are updated, call the optimizer.zero_grad() method to clear the gradients of all parameters to avoid gradient accumulation affecting the next iteration and ensure that the gradient calculation of each batch is performed independently.

[0059] S6.7.6: Throughout the process, ensure that all parameters of the model and mapping network participate in backpropagation and updates, avoid missing any trainable parameters, and monitor gradient norm changes to promptly detect and handle abnormal gradient situations.

[0060] The `backward()` method is the core method of PyTorch's automatic differentiation engine, typically called as `loss.backward()`. Its core function is to execute the backpropagation algorithm, calculating the gradient of the loss function relative to all trainable parameters in the model. `optimizer.step()` performs a single parameter update operation, updating the model parameters according to a specific optimization algorithm based on the gradient calculated by `backward()`. The `optimizer.zero_grad()` method is used to clear the gradients of the model parameters to zero.

[0061] S6.8 evaluates model performance on the validation set after each training cycle, calculates metrics such as NDCG and Recall, and adjusts hyperparameters (such as learning rate and loss weights) based on the validation set performance to ensure the model has good generalization ability.

[0062] S6.9 Repeat the above training steps (S6.1~S6.8) until the model's performance on the validation set no longer improves or reaches the preset maximum number of training cycles. Save the trained model parameters to ensure that the model converges and its performance is stable.

[0063] Example 3 Based on the same inventive concept, this embodiment provides a recommended method, which implements the model obtained by the method described in Embodiment 2, including: The embedding representations of users and items are obtained from the trained model. By calculating the similarity between user embeddings and item embeddings, the predicted rating of each candidate item by the user is obtained. Candidate items are sorted according to their predicted scores, and the top K items with the highest scores are selected to generate a recommendation list as the personalized recommendation result.

[0064] Example 4 Based on the same inventive concept, this embodiment provides a recommendation system, please refer to [link / reference]. Figure 3 This includes the multimodal data denoising framework in Example 1 and the recommendation prediction output module, which is used for: The embedding representations of users and items are obtained from the trained model. By calculating the similarity between user embeddings and item embeddings, the predicted rating of each candidate item by the user is obtained. Candidate items are sorted according to their predicted scores, and the top K items with the highest scores are selected to generate a recommendation list as the personalized recommendation result.

[0065] Specifically, the modality-aware recalibration module performs quality assessment and recalibration on the original user-item interaction data to identify and distinguish between high-quality interactions and noisy interactions. This module first extracts features from the input multimodal item content (such as text descriptions, image covers, video clips, etc.), obtaining text feature vectors, image feature vectors, and video feature vectors respectively. Then, based on a cross-modal matching model, it calculates the similarity score between user behavior features and item modal features to obtain a quality score for each interaction. According to a preset quality score threshold, the interaction data is reordered and labeled as "clean interactions" and "suspected noisy interactions," thereby effectively improving the quality of the input data and providing a high-quality data foundation for subsequent ranking optimization and knowledge distillation.

[0066] The ranking loss construction and optimization module, based on the traditional Bayesian Personalized Ranking (BPR) loss, introduces sample denoising weights calculated based on modal content relevance to construct a denoised ranking loss function suitable for implicit feedback scenarios. Specifically, this module dynamically estimates the denoising weight of each positive sample based on the interaction quality score output by the modality-aware recalibration module and incorporates it into the BPR loss function to form a weighted DBPR loss function. By optimizing this loss function, the model can pay more attention to high-quality interaction samples and reduce the negative impact of noisy samples on model training, thereby significantly improving the ranking performance and robustness of the recommendation system.

[0067] The knowledge distillation-guided alignment module aims to achieve effective alignment between modal features and the recommendation semantic space, thereby improving the generalization ability and robustness of recommendation models. This module first maps multimodal features (text, images, videos, etc.) to the recommendation semantic space to obtain modal semantic representations. Then, using the recommendation semantic representations as the target distribution and the modal features as the source distribution, an optimal transmission loss function based on optimal transmission distance (such as Sinkhorn distance or Entropy-regularized OT) is constructed. By jointly optimizing this loss function with the ranking loss, the model can achieve effective alignment between modal features and recommendation semantics, thus fully utilizing multimodal information and improving the overall performance of the recommendation system.

[0068] The recommendation prediction output module is responsible for generating the final personalized recommendation results. This module first obtains the embedding representations of users and items from the trained model. Then, it calculates the similarity between user embeddings and item embeddings (such as dot product or cosine similarity) to obtain the user's predicted rating for each candidate item. The candidate items are then sorted according to the predicted ratings, and the top K items with the highest ratings are selected to generate a recommendation list, which is then output to the user. Through this module, the system can provide users with high-quality personalized recommendation services to meet their individual needs.

[0069] The modal content-based user-item interaction recalibration process is the foundation of the entire method. In this process, the system first extracts features from the input multimodal item content (such as text descriptions, image covers, video clips, etc.), obtaining text feature vectors, image feature vectors, and video feature vectors respectively. Then, a cross-modal matching model is used to calculate the similarity score between user behavior features and item modal features, obtaining a quality score for each interaction. Based on a preset quality score threshold, the interaction data is reordered and labeled as "clean interactions" and "potentially noisy interactions," thereby effectively improving the quality of the input data and providing a high-quality data foundation for subsequent ranking optimization and knowledge distillation.

[0070] Constructing a confidence-based Bayesian ranking denoising loss function is a key step in improving the ranking performance of recommender systems. This loss function, based on the traditional Bayesian Personalized Ranking (BPR) loss, introduces sample denoising weights calculated based on modal content relevance, forming a weighted DBPR loss function. Specifically, based on the interaction quality score output from the modality-aware recalibration process, the denoising weight of each positive sample is dynamically estimated and incorporated into the loss function. By optimizing this loss function, the model can focus more on high-quality interaction samples, reducing the negative impact of noisy samples on model training, thereby significantly improving the ranking performance and robustness of the recommender system.

[0071] A knowledge distillation mechanism based on optimal transmission distance aligns modal features with the recommendation semantic space, further enhancing the generalization ability and robustness of the recommendation model. This mechanism first maps multimodal features to the recommendation semantic space, obtaining modal semantic representations. Then, using the recommendation semantic representations as the target distribution and the modal features as the source distribution, an optimal transmission loss function based on the optimal transmission distance (such as Sinkhorn distance or Entropy-regularized OT) is constructed. By jointly optimizing this loss function with the ranking loss, the model can effectively align modal features with recommendation semantics, thereby fully utilizing multimodal information and improving the overall performance of the recommendation system.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A multimodal data denoising framework, characterized in that, include: The modality-aware recalibration module is used to extract features from the input multimodal item content, obtain the multimodal features of the items, calculate the similarity between user behavior features and the modal features of the items as a quality score of user-item interaction, and recalibrate the original user-item interaction data based on the relationship between the quality score and the preset quality score threshold. The ranking loss construction and optimization module is used to dynamically estimate the denoising weight of each positive sample based on the user-item interaction quality score, and introduce the denoising weight into the BPR loss function to form a weighted DBPR loss function. Here, the positive sample is the item that has interacted with the user, the BPR loss function is the Bayesian personalized ranking loss function, and the DBPR loss function is the dynamic Bayesian personalized ranking loss function. The knowledge distillation guided alignment module is used to map multimodal features to the recommendation semantic space to obtain modal semantic representations. The recommendation semantic representations are used as the target distribution and the modal features are used as the source distribution to construct a distillation loss function based on the optimal transmission distance.

2. A model training method based on the multimodal data denoising framework of claim 1, characterized in that, include: Collect user behavior data and multimodal item data, construct a user-item interaction matrix, standardize the multimodal features, and construct training, validation, and test sets; The modality-aware recalibration module is used to perform quality assessment and recalibration on the original user-item interaction data; The ranking loss construction and optimization module dynamically calculates the denoising weight of each positive sample based on the interactive recalibration results, and constructs a weighted DBPR loss function. The knowledge distillation-guided alignment module maps multimodal features to the recommendation semantic space, with the recommendation semantic representation as the target distribution and the modal features as the source distribution, and constructs a distillation loss function based on the optimal transmission distance; The total loss function is constructed based on the weighted DBPR loss function and the distillation loss function based on the optimal transmission distance, and the loss weights are dynamically adjusted using a multi-task optimization method. Using the training set, the model is trained through the Adam optimizer. In each training cycle, the total loss function is calculated and backpropagation is performed to update the model parameters. Use the validation set to perform hyperparameter tuning and select the optimal model parameters.

3. The model training method as described in claim 2, characterized in that, Collect user behavior data and multimodal item data, construct a user-item interaction matrix, standardize the multimodal features, and construct training, validation, and test sets, including: Collect user behavior data from recommendation system logs and perform data cleaning on the collected user behavior data; The cleaned user behavior data is converted into a user-item interaction matrix, where rows represent users, columns represent items, and matrix elements represent the intensity of user behavior towards items. For user-item pairs that have not interacted, the matrix element value is 0. Collect multimodal data of items, including text descriptions, cover images, and video clips, and standardize the data format; The text description is preprocessed, and a pre-trained language model is used to extract text feature vectors. The cover image is preprocessed, and a pre-trained convolutional neural network is used to extract image feature vectors. Keyframes are extracted from the video clips, and each frame is preprocessed. Convolutional neural networks are used to extract image features from each frame. The features of all frames are aggregated to obtain the video feature vector. The extracted multimodal feature vectors are standardized. Align the standardized multimodal feature vectors with the user-item interaction matrix; The aligned user-item interaction matrix and multimodal feature vectors are stored in a standard format and divided into training, validation and test sets.

4. The model training method as described in claim 3, characterized in that, Using a modality-aware recalibration module, the original user-item interaction data is subjected to quality assessment and recalibration, including: Extract the user ID, item ID, and user behavior features corresponding to each interaction record from the aligned user-item interaction matrix; For each user action, its type and intensity are mapped into a continuous vector through an embedding layer or feature encoder; For each item, the corresponding text feature vector, image feature vector and video feature vector are extracted from the standardized multimodal feature library, and then concatenated or weighted and fused to form a unified item modal feature representation. A cross-modal matching model is used to calculate the similarity score between the user behavior feature vector and the item modal feature vector, which serves as the quality score for each interaction record. Statistical analysis was performed on the quality scores of all interaction records, and the interactions were recalibrated as clean interactions and suspected noisy interactions based on preset thresholds. The tagged interaction data is merged with the standardized stored interaction matrix to form a new interaction matrix that includes interaction quality tags.

5. The model training method as described in claim 4, characterized in that, The ranking loss construction and optimization module dynamically calculates the denoising weights for each positive sample based on the interactive recalibration results, constructing a weighted DBPR loss function, including: Extract the interaction quality label corresponding to each positive sample from the new interaction matrix, and obtain the corresponding quality score; For each positive sample, the denoising weight of the sample is dynamically calculated based on the interaction quality label and quality score. Clean interaction samples are given higher weights, while suspected noisy interaction samples are given lower weights. The calculated denoising weights are associated with and stored with the corresponding positive samples; The calculated denoising weights are incorporated into the BPR loss function to construct the DBPR loss function.

6. The model training method as described in claim 5, characterized in that, The calculated denoising weights are incorporated into the BPR loss function to construct the DBPR loss function, which includes: Extract all positive samples and their corresponding negative samples to form triples for DBPR loss calculation, including users, positive items, and negative items; For each triple, obtain the user embedding vector, positive item embedding vector, and negative item embedding vector from the recommendation model, and ensure that the dimensions of the embedding vectors are consistent. Calculate the user's prediction score for positive items and the user's prediction score for negative items in each triplet, and use dot product or cosine similarity to obtain two scalar values, which represent the user's preference for positive items and negative items, respectively. The difference between the predicted scores of each triplet is calculated to obtain the difference between the scores of positive and negative samples. The difference is then input into the Sigmoid function to obtain the probability value of each triplet, which represents the probability that the score of the positive sample is higher than the score of the negative sample. Take the logarithm of the probability value of each triplet and multiply it by the denoising weight of the corresponding positive sample to obtain the weighted log-likelihood loss of each triplet. The weighted log-likelihood loss of all triples is summed and negative to obtain the final DBPR loss function value.

7. The model training method as described in claim 3, characterized in that, A knowledge distillation-guided alignment module maps multimodal features to the recommendation semantic space, using the recommendation semantic representation as the target distribution and the modal features as the source distribution. A distillation loss function based on the optimal transmission distance is constructed, including: Obtain the embedded representations of users and items in the current batch, as the basis for the target distribution of the recommendation semantic space; From the standardized multimodal feature library, extract the text feature vector, image feature vector and video feature vector corresponding to the current batch of items, and concatenate or weightedly fuse them to form a unified modal feature representation of the items; A modal feature mapping network is constructed to map high-dimensional modal feature vectors to a low-dimensional space that is the same as the recommendation semantic space. The weight parameters of the mapping network are initialized by Xavier and then optimized. Using the mapped modal feature vector as the source distribution and the recommended semantic embedding vector as the target distribution, the SamplesLoss function is used to construct the Sinkhorn distance calculation object. Call the Sinkhorn distance calculation object, input the source distribution and target distribution, and calculate the optimal transmission distance loss value for the current batch; The calculated optimal transmission distance loss value is combined with the weighted DBPR loss function to form the total loss function of the model. Backpropagation is performed on the total loss function to calculate the gradients of the mapping network and the recommendation model. The Adam optimizer is then used to update the parameters of both the mapping network and the recommendation model simultaneously.

8. The model training method as described in claim 3, characterized in that, Using the training set, the model is trained using the Adam optimizer. At each training epoch, the total loss function is calculated and backpropagation is performed to update the model parameters, including: Load the training set data, make a one-to-one correspondence between the user-item interaction records in the training set and the multimodal feature vectors, perform data transformation, and move the data to the GPU device; The recommendation model parameters, including the user embedding matrix, item embedding matrix, and modal feature mapping network, are initialized using the Xavier uniform initialization method. Define the Adam optimizer, set the learning rate, weight decay coefficient and gradient clipping threshold, and pass the parameters of the recommendation model and the mapping network into the optimizer at the same time; At the beginning of each training cycle, the training set data is divided into multiple batches, each batch containing a fixed number of user-item interaction records and corresponding multimodal feature vectors; For each batch of data, user embedding vectors, positive item embedding vectors, and negative item embedding vectors are calculated using a recommendation model, and mapped modal feature vectors are calculated using a modal feature mapping network, where all vectors have the same dimension. Based on the constructed weighted DBPR loss function and the optimal transport distillation loss function, calculate the total loss function value for the current batch and record the value of each loss term; Backpropagation is performed on the total loss function to calculate the gradients of the recommendation model and the mapping network. The optimizer is used to update the model parameters. The gradient pruning operation is performed after backpropagation. After each training cycle, evaluate the model performance on the validation set and adjust the hyperparameters based on the validation set performance. Repeat the above training steps until the model's performance on the validation set no longer improves or reaches the preset maximum number of training epochs, then save the trained model parameters.

9. A recommended method, characterized in that, The model implementation using the method described in any one of claims 2-8 includes: The embedding representations of users and items are obtained from the trained model. By calculating the similarity between user embeddings and item embeddings, the predicted rating of each candidate item by the user is obtained. Candidate items are sorted according to their predicted scores, and the top K items with the highest scores are selected to generate a recommendation list as the personalized recommendation result.

10. A recommendation system, characterized in that, Including the multimodal data denoising framework as described in claim 1 and the recommendation prediction output module, the recommendation prediction output module is used for: The embedding representations of users and items are obtained from the trained model. By calculating the similarity between user embeddings and item embeddings, the predicted rating of each candidate item by the user is obtained. Candidate items are sorted according to their predicted scores, and the top K items with the highest scores are selected to generate a recommendation list as the personalized recommendation result.

Images