A multi-modal semi-supervised learning method and device based on cyclic data distillation

Abstract

The present application relates to the technical field of multi-modal learning, semi-supervised learning and sentiment computing, in particular to a multi-modal semi-supervised learning method and device based on cyclic data distillation. The method and device utilize multiple single-modal pre-training models, distill reliable and class-representative data from a large amount of unlabeled data through a confidence-enhanced pseudo-label module to train a multi-modal sentiment recognition model, and provide performance feedback to update all single-modal models. It jointly trains the multi-modal sentiment recognition model and the single-modal model in a cyclic and mutually beneficial manner, and finally obtains an effective multi-modal sentiment recognition model.

Description

Technical Field

[0001] This invention relates to the fields of multimodal learning, semi-supervised learning, and affective computing, specifically a multimodal semi-supervised learning method and apparatus based on cyclic data distillation. Background Technology

[0002] (1) Multimodal emotion recognition algorithm

[0003] Multimodal emotion recognition technology integrates different types of human data, such as facial expressions, voice tone, and text content, to capture and analyze emotional information from multiple perspectives. This effectively helps understand users' true emotions, making human-computer interaction more natural and human-like. Multimodal emotion recognition technology is widely used in social media, e-commerce, advertising, customer service, healthcare, and other fields, enabling businesses and individuals to better understand users' emotional states and make corresponding decisions or adjustments.

[0004] Currently, the training of multimodal sentiment recognition models heavily relies on labeled datasets. Labeled datasets provide ample information for model training and validation, helping models better understand data characteristics and patterns. However, the size of current multimodal sentiment datasets is generally small. The largest multimodal sentiment dataset, CMU-MOSEI, has only about 23,000 samples, while the MELD and IEMOCAP datasets have 13,708 and 10,039 samples respectively, and other datasets have fewer than 3,000 samples. Therefore, the limited size of existing multimodal sentiment datasets severely restricts the development of multimodal sentiment recognition models. Furthermore, due to the complexity of multimodal sentiment annotation, constructing large-scale labeled datasets requires significant time and manpower. Therefore, researchers are exploring multimodal semi-supervised learning algorithms to train multimodal sentiment recognition models using large amounts of unlabeled data, reducing the model's dependence on large labeled datasets.

[0005] Secondly, most multimodal emotion recognition studies utilize comprehensive and complementary multimodal features, which often contain a large amount of redundant information and interfere with the model's discriminative performance. This negative impact of redundant information on model performance is particularly pronounced in noisy semi-supervised learning environments. Some studies introduce gating mechanisms to reduce redundant information; for example, multi-level fusion networks control the redundant information of long multimodal sequences by incorporating forget gate modules. Other methods employ attention mechanisms to weaken redundant features; for instance, the AtM-DNN model introduces an attention mechanism to control the weights of each modality's information flowing into the multimodal fusion feature, thereby reducing redundant information from weaker modalities. However, typical attention weight matrices are large and dense, and their element values ​​may generally be too small to effectively highlight important features. Furthermore, sparse representation algorithms are also an effective redundancy reduction technique; for example, multimodal sparse Transformer networks use traditional top-q sparse methods to retain the most important feature fragments. However, the complex optimization methods of sparse representation algorithms are difficult to adapt to gradient backpropagation optimization methods, making most sparse representation algorithms unsuitable for deep learning models. Therefore, reducing redundant features in multimodal models and improving model robustness are among the research problems in multimodal feature fusion and multimodal semi-supervised learning.

[0006] (2) Semi-supervised learning algorithm

[0007] Semi-supervised learning is a learning framework that uses a small number of labeled samples and a large number of unlabeled samples to train a model. It can effectively reduce the dependence on large labeled datasets, reduce labeling costs and model training costs, and improve the generalization performance of the model.

[0008] In the field of multimodal emotion recognition, there is relatively little research on semi-supervised learning, which is mainly divided into pseudo-labeling methods, consistency regularization methods, graph-based methods, and generative model-based methods.

[0009] The pseudo-labeling method refers to a cyclical training process where a model predicts unlabeled data, and the resulting pseudo-labeled data is used to train the model along with a small amount of labeled data. For example, Fixmatch uses high-confidence predictions exceeding a fixed threshold as pseudo-labels for unlabeled data. However, the pseudo-labeling method is prone to confirmation bias, where the model that generates incorrect predictions reuses erroneous information during training, leading to error accumulation that can negatively impact the quality of pseudo-labels and the model's recognition performance. Some studies employ preheating or high-threshold operations to generate reliable pseudo-labels. However, setting high thresholds can cause the model to favor learning strong-class samples while neglecting weak-class samples, resulting in the pseudo-labeled dataset deviating from the original data distribution.

[0010] Consistency regularization is a regularization constraint method that constrains a model to produce consistent predictions under different input disturbances, thereby uncovering the model's generalization and versatility. For example, temporal ensemble models use the squared difference loss function to constrain the output distribution to remain consistent across multiple epochs, thus uncovering potential information from unlabeled data. However, consistency regularization is not suitable for multimodal data with significant modal differences.

[0011] Graph-based methods propagate labels from labeled nodes (labeled data points) to unlabeled nodes (unlabeled data points) based on node similarity, such as dynamic graph fusion label propagation networks.

[0012] Generative model-based methods learn general representations from large amounts of unlabeled data through a data reconstruction process. Common reconstruction methods include autoencoders and variational autoencoders. For example, cross-modal distribution matching models learn general features from a wealth of multimodal unlabeled data through the reconstruction mechanism of autoencoders, effectively improving model performance.

[0013] The current scale of multimodal sentiment datasets severely limits the training and performance of multimodal sentiment recognition models. Furthermore, constructing large-scale labeled datasets requires significant time and manpower. Therefore, researchers have explored multimodal semi-supervised learning algorithms to train multimodal sentiment recognition models using large amounts of unlabeled data. However, in multimodal semi-supervised learning algorithms, consistency regularization is not suitable for multimodal data with significant modal differences, while pseudo-labeling suffers from confirmation bias and pseudo-label data distribution bias, greatly reducing model performance and reliability. Secondly, multimodal sentiment features contain a large amount of redundant information, which can easily interfere with the model's sentiment discrimination performance under high-noise semi-supervised learning framework training. Summary of the Invention

[0014] To address at least one of the shortcomings of existing technologies, this invention provides a multimodal semi-supervised learning method and apparatus based on cyclic data distillation, which can effectively mitigate the problems of false label confirmation bias and false label data distribution bias, and maintain excellent robustness in high-noise semi-supervised training.

[0015] To achieve the above objectives, the present invention provides the following technical solution:

[0016] In a first aspect, the present invention provides a multimodal semi-supervised learning method based on cyclic data distillation, comprising:

[0017] Obtain an unlabeled dataset and a labeled dataset, wherein the labeled dataset is divided into a training set and a validation set;

[0018] During Epochk∈[1,kini], the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset;

[0019] During Epochk > Kini, the unimodal teacher model is used to predict the unlabeled dataset to obtain prediction results. The prediction results are then processed using a pre-built confidence-enhancing pseudo-label module to obtain a pseudo-labeled dataset. The multimodal student model is iteratively trained using the training sets of the pseudo-labeled dataset and the labeled dataset, and multiple unimodal teacher models are iteratively updated. Simultaneously, the pseudo-labeled dataset is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled dataset reaches the preset iteration condition.

[0020] Output the trained multimodal student model.

[0021] Further, obtain an unlabeled dataset and a labeled dataset. The labeled dataset is divided into a training set and a validation set. Specifically, the training set of the labeled dataset is... The validation set of the labeled dataset is Unlabeled dataset is

[0022] The steps following the acquisition also include: initializing the unimodal teacher model f. T1 ,…,f TJ Model parameters And multimodal student model f S Model parameters wf S , and initialize the parameter Epochk = 1 and the category parameter c ∈ (1,…,C).

[0023] Furthermore, the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset, specifically including:

[0024] w fS ←w fS -η(▽L(w fS ,X))

[0025] w fTj ←w fTj -η(▽L(w fTj ,X))

[0026] Furthermore, the prediction results are processed using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label dataset, specifically including:

[0027] The prediction results are integrated using a pre-built modal adaptation module to obtain a multimodal ensemble result:

[0028] Among them, each single-modal teacher model f Tj From my own perspective, on all unlabeled datasets Make predictions, and then pass them through the modal adaptation module f. MAE Integrating predictions from all unimodal teacher models improves the P-value of the multimodal ensemble result. U Confidence level:

[0029]

[0030] The multimodal ensemble result is distilled using a pre-built category-aware pseudo-label module to obtain a pseudo-label dataset:

[0031] Among them, according to the multimodal integration result P U By using a category-aware pseudo-label module, a pseudo-label dataset φ is distilled from the unlabeled dataset U. k ;

[0032]

[0033] Category-aware pseudo-label module f CPL Based on the new pseudo-label dataset φ k The data distribution is used to update the thresholds for each category in stages for Epochk+1:

[0034]

[0035] Furthermore, the training sets of the pseudo-label dataset and the labeled dataset are used to iteratively train the multimodal student model and iteratively update multiple unimodal teacher models, specifically including:

[0036] Pseudo-label dataset φ k The training set X of the labeled dataset is subjected to the total loss function L(w) fS ,X∪φ k Jointly train the multimodal student model f S And update the single-modal teacher model f T1 ,…,f TJ Classifier C j ,j∈{1,…,J}.

[0037]

[0038] w fS ←w fS -η(▽L(w fS ,X∪φ k ))

[0039]

[0040] Furthermore, the loss function value of the multimodal student model on the validation set of the labeled dataset reaches a preset iteration condition, specifically including:

[0041] Loss function value of the multimodal student model on the validation set of the labeled dataset This will serve as a condition for terminating network training, when If the network does not decrease within 10 consecutive epochs, it will stop training and output the multimodal student model f. S and its parameters Otherwise, continue the loop iteration.

[0042] Secondly, the present invention provides a multimodal semi-supervised learning device based on cyclic data distillation, characterized in that it comprises:

[0043] A data acquisition unit is used to acquire unlabeled datasets and labeled datasets, wherein the labeled datasets are divided into training sets and validation sets.

[0044] Fine-tuning unit, which is used in Epochk∈[1,k] ini During this period, the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset;

[0045] The pseudo-labeled dataset and the student-teacher model iterative update unit are used for Epoch k > k. ini During this process, the unimodal teacher model is used to predict the unlabeled dataset to obtain prediction results. The prediction results are then processed using a pre-built confidence-enhancing pseudo-label module to obtain a pseudo-label dataset. The multimodal student model is iteratively trained using the training sets of the pseudo-label dataset and the labeled dataset, and multiple unimodal teacher models are iteratively updated. Simultaneously, the pseudo-label dataset is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled dataset reaches the preset iteration condition.

[0046] The model output unit is used to output the trained multimodal student model.

[0047] Thirdly, the present invention also provides an electronic device, including a processor and a memory;

[0048] The memory is used to store programs;

[0049] The processor executes the program to implement the method described above.

[0050] Fourthly, the present invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0051] Fifthly, the present invention also provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the preceding method.

[0052] Compared with the prior art, the advantages of this invention are as follows:

[0053] (1) Current research on multimodal semi-supervised learning involves pseudo-label generation and training operating on the same multimodal structure, meaning that both the pseudo-label generator and the operator are part of the same structure. This can easily lead to confirmation bias and vicious training loops. The multimodal semi-supervised learning framework based on cyclic data distillation proposed in this invention fully utilizes the performance advantages of mature unimodal pre-trained models. It uses multiple unimodal pre-trained models as teacher models and a multimodal emotion recognition model as student models. Through a confidence-enhancing pseudo-label module, reliable pseudo-label data is distilled from a large amount of unlabeled data, and then the multimodal student model is trained cyclically. This framework effectively utilizes the performance and stability advantages of the unimodal teacher model, ensuring the stability and reliability of the multimodal semi-supervised learning framework in cyclic generation and training.

[0054] (2) Pseudo-labeling algorithms are prone to confirmation bias, which reduces the reliability of pseudo-labels. Some methods set high thresholds to improve the reliability of pseudo-labels, but this can easily cause the model to favor learning strong class samples while ignoring weak class samples, resulting in the pseudo-label dataset deviating from the original data distribution and causing data imbalance. The confidence enhancement pseudo-labeling module proposed in this invention improves the confidence of single-modal prediction and ensemble output through a modality adaptive ensemble module, and provides appropriate thresholds for different categories at different stages through a category-aware threshold module. This controls the number of strong class samples and accommodates more weak class samples, ensuring the consistency of data distribution between the pseudo-label dataset and the original dataset, improving the reliability of the pseudo-label dataset, and effectively mitigating confirmation bias.

[0055] (3) High-dimensional multimodal features tend to contain a lot of redundant information. In high-noise semi-supervised training, redundant information will seriously interfere with the model's discriminative performance. The sparse interaction attention module proposed in this invention can effectively encode multimodal interaction relationships into sparse interaction attention weights without complex optimization methods through deep sparse representation algorithms within and between modalities. It highlights a few but important emotional features with a small number of non-zero weights, suppresses a large number of redundant features, and effectively resists the interference of high-noise sample sets. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of multimodal semi-supervised learning based on cyclic data distillation in an embodiment of the present invention;

[0058] Figure 2 This is a diagram of a multimodal semi-supervised learning framework based on cyclic data distillation in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] Example:

[0061] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0062] See Figure 1 This embodiment provides a multimodal semi-supervised learning method based on cyclic data distillation, which may specifically include the following process:

[0063] Step 1: Given the training set of the labeled dataset and verification set

[0064] and unlabeled datasets Initialize the single-modal teacher model f T1 ,…,f TJ Model parameters And multimodal student model f S Model parameters

[0065] And initialize the parameter Epochk = 1 and the category parameter c ∈ (1, ..., C).

[0066] Step 2: In Epochk∈[1,k] ini During this period, all unimodal teacher models and multimodal student models were fine-tuned using the training set X of the labeled dataset:

[0067]

[0068] Step 3: In Epochk > k ini During this period, each unimodal teacher model f Tj From my own perspective, on all unlabeled datasets Make predictions, and then pass them through the modal adaptation module f. MAE Integrating predictions from all unimodal teacher models improves the P-value of the multimodal ensemble result. U Confidence level:

[0069]

[0070] Step 4: Based on the multimodal integration result P U By using a category-aware pseudo-label module, a reliable and category-representative pseudo-label dataset φ is distilled from the unlabeled dataset U. k .

[0071]

[0072]

[0073] Step 5: Category-aware pseudo-label module f CPL Based on the new pseudo-label dataset φ k The data distribution is used to update the thresholds for each category in Epochk+1.

[0074]

[0075] Step 6: New pseudo-label dataset φ k The training set X of the labeled dataset is subjected to the total loss function.

[0076] Jointly train multimodal student model f S And update the single-modal teacher model f T1 ,…,f TJ Classifier C j ,j∈{1,…,J}.

[0077]

[0078] Step 7: Loss function values ​​of the multimodal student model on the validation set of the labeled dataset.

[0079] This will serve as a condition for terminating network training. When If the network does not decrease within 10 consecutive cycles, it will stop training and output the multimodal student model f. S and its parameters Otherwise, continue the cycle from step 3 to step 7.

[0080] like Figure 2 As shown, the framework consists of a unimodal teacher model, a confidence-enhancing pseudo-label module, and a multimodal student model. Each unimodal teacher model predicts unlabeled samples from its own perspective. Then, the confidence-enhancing pseudo-label module integrates the predictions of all unimodal teacher models in a sharpening and uncertainty-based weighted manner, reducing interference from suboptimal categories and modal uncertainty, thus improving the confidence of the integrated output. Next, based on the integrated output, the confidence-enhancing pseudo-label module filters a large amount of unlabeled data using a stage-adaptive category-aware threshold, distilling high-confidence, category-representative pseudo-label data. The multimodal student model, based on sparse interactive attention, progressively learns rich discriminative information from reliable pseudo-label and labeled datasets. Finally, the performance of the multimodal student model on labeled data is used to further update all unimodal teacher models. This multimodal semi-supervised learning framework based on cyclic data distillation trains the multimodal student model and the unimodal teacher model in a mutually beneficial manner, iteratively training an effective multimodal student model (multimodal sentiment recognition model).

[0081] In practical implementation, regarding the single-modal teacher model:

[0082] Each unimodal model is relatively mature and possesses pre-trained models with strong generalization and good predictive performance. Therefore, this invention utilizes multiple unimodal pre-trained models as teacher models to jointly label unlabeled data, construct a pseudo-labeled dataset, and then train a multimodal sentiment recognition model.

[0083] Each unimodal teacher model consists of a pre-trained model and a classifier. For example, the text teacher model comprises a text pre-trained model and a text classifier. The text pre-trained model is a RoBERTa pre-trained model containing 24 Transformer layers and 1 fully connected layer. The text classifier contains 1 fully connected layer and 1 softmax function. The speech teacher model consists of a speech pre-trained model and a speech classifier. The speech pre-trained model is a RoBERTa pre-trained model, but this model only contains 12 Transformer layers and 1 fully connected layer and is re-pre-trained on the Librisspeech-960 dataset. The speech classifier contains 1 fully connected layer and 1 softmax function. The video teacher model consists of a video pre-trained model and a video classifier. The video pre-trained model is the Fabnet facial attribute network, and the video classifier contains 1 fully connected layer and 1 softmax function. Similarly, other unimodal teacher models can consist of pre-trained models and their classifiers from current unimodal research areas.

[0084] Understandably, the pre-trained model for a unimodal teacher model can be any other effective pre-trained model in that unimodal domain. For example, the pre-trained model for a text teacher model can be Transformer, BERT, etc., the pre-trained model for a speech teacher model can be Transformer, HuBERT, etc., and the pre-trained model for a video teacher model can be 3D ResNet, CLIP, etc.

[0085] In practical implementation, regarding the confidence enhancement pseudo-label module:

[0086] The confidence-enhancing pseudo-label module aims to distill reliable and class-representative pseudo-label data from large amounts of unlabeled data. It consists of a modality-adaptive ensemble module and a class-aware pseudo-label module.

[0087] A. Modal Adaptive Integration Module

[0088] Different unimodal teacher models predict unlabeled samples from their respective perspectives and provide corresponding prediction confidence scores. These prediction confidence scores represent probability distributions across different classes, thus exhibiting categories such as the highest probability class (optimal class), the second highest probability class (suboptimal class), and the third highest probability class. However, different modalities exhibit varying prediction confidence scores for the same sample. Therefore, integrating the prediction confidence scores of multiple unimodal teacher models can easily highlight the suboptimal class, thereby threatening the optimal class and reducing the reliability of the multimodal ensemble output. To address these issues, the modality adaptive ensemble module effectively improves the confidence scores of both unimodal predictions and the ensemble output.

[0089] First, the modal adaptive ensemble module utilizes an orthogonal diversity loss function to reduce the interference of suboptimal classes. Given the predictions of the j-th unimodal teacher model... Where C represents the number of categories. Assuming c represents the category corresponding to the actual label, then... The predicted probability of the true label category. Probability combinations for non-true label categories:

[0090]

[0091] The orthogonal diversity loss function is:

[0092]

[0093] Where J represents the number of modalities. The orthogonal diversity loss function encourages the probability combinations of non-true label categories among each unimodal teacher model to be orthogonal to each other, that is, to output different suboptimal category probability combinations, effectively reducing the suboptimal category interference of subsequent multimodal ensemble outputs and improving the confidence of unimodal predictions and multimodal ensemble outputs.

[0094] Next, the modality adaptive ensemble module evaluates the prediction uncertainty of each unimodal teacher model and adaptively integrates the prediction results of all unimodal teacher models based on the prediction uncertainty, further improving the confidence of the multimodal ensemble output. The process is as follows:

[0095]

[0096] Where E j This refers to the prediction uncertainty of the j-th unimodal teacher model, α j The weights for participating in multimodal integration are calculated based on the prediction uncertainty of the j-th unimodal teacher model, which can reduce the impact of uncertain predictions on the multimodal integration output. U It is the ensemble prediction result of all unimodal teacher models.

[0097] B. Category-Aware Pseudo-Label Module

[0098] Pseudo-labeling methods typically employ a fixed, uniform high threshold to filter unlabeled data. However, this method is prone to selecting a large number of strong-class samples while blocking most weak-class samples, resulting in an imbalanced pseudo-label dataset or a deviation from the original data distribution due to an excessive number of strong-class samples and an insufficient number of weak-class samples. To address this issue, a category-aware pseudo-labeling module provides appropriate thresholds for different categories at different stages, effectively distilling reliable and class-representative pseudo-label datasets from a large amount of unlabeled data, thus alleviating the data imbalance problem.

[0099] The category-aware pseudo-label module is divided into an initial stage, a learning stage, and a stable stage.

[0100] The initial phase requires ensuring the task adaptability and stability of the entire framework. In the first k steps of the initial phase...ini For each epoch, the classifiers for all unimodal teacher models are fine-tuned using a labeled dataset, while a uniform preset threshold is applied to different categories. Filter out the multimodal integration result P U More than Θ k From unlabeled samples, a relatively stable initial pseudo-label dataset Φ is generated. k It is used for training during the learning phase.

[0101] During the learning phase, it is expected that different categories will have different thresholds. These thresholds will gradually control the number of samples from the strong category and allow for more samples from the weak category, thus providing a sufficient number of representative samples for each category. Therefore, the learning phase uses the previous pseudo-label dataset Φ... k-1 Based on the data distribution, adaptively adjust the thresholds for different categories. The threshold adjustments during the learning phase are as follows:

[0102]

[0103] Where I(·) is an indicator function that returns 1 when the expression within the parentheses is true. s For the pseudo-label dataset Φ k-1 The pseudo-label of the s-th sample. M is the total number of unlabeled samples, Ω c For the pseudo-label dataset Φ k-1 The total number of class c samples, Let be the confidence threshold for class c in the k-th epoch. From the above formula, we can observe that for the current pseudo-label dataset Φ... k-1 There are fewer class c samples. The size will decrease to accommodate more class c samples in the current epoch, providing more representative samples for the weak classes. Simultaneously, as the size of the pseudo-label dataset increases, the above formula can achieve the class threshold Θ. k The gradual growth model.

[0104] The stabilization phase aims to ensure that the thresholds for all categories reach a dynamically stable state as the pseudo-labeled dataset stabilizes. Therefore, when the size of the pseudo-labeled dataset exceeds the amount of remaining unlabeled data, the threshold adjustment stabilization phase begins.

[0105]

[0106] During the stable phase, if the number of pseudo-labeled samples of class c is the largest, then The number of samples in this category will be strictly controlled at a high preset threshold, while the thresholds for other categories will be aligned and adjusted according to the threshold for category c, ensuring that the distribution of pseudo-label data remains relatively stable.

[0107] In practical implementation, regarding the multimodal student model:

[0108] The pseudo-labeled dataset and the labeled dataset will be used together to train the multimodal student model. However, high-dimensional multimodal features contain a large amount of redundant information, which seriously interferes with the model's sentiment discrimination performance, especially in noisy semi-supervised training scenarios. To address this issue, the multimodal student model is designed as a sparse interactive attention network. The sparse interactive attention network encodes multimodal interaction features into sparse interactive attention weights through intra-modal sparse modules and inter-modal sparse modules. It highlights sentiment features with a small number of non-zero weights, ignores a large number of redundant features, and effectively prevents overfitting.

[0109] The sparse interactive attention network consists of a pre-trained feature extractor, a sparse interactive attention module, and a sentiment classifier. The pre-trained feature extractor is consistent with the unimodal pre-trained model of the unimodal teacher model.

[0110] A. Sparse Interactive Attention

[0111] The sparse interactive attention module is the core component of the sparse interactive attention network, mainly composed of intra-modal sparse modules and inter-modal sparse modules.

[0112] Intramodal sparse modules preserve unimodal features effective for sentiment representation and suppress redundant features, providing sparse yet effective unimodal features for subsequent multimodal interactions. It cleverly utilizes the scaling factor of batch normalization to achieve sparsity constraints, sparsifying unimodal features with low computational cost. The following is the expression for batch normalization:

[0113]

[0114] in Features within the initial mode elements, For the output features of the j-th unimodal pre-trained feature extractor, This is for splicing operations. and These are the mean and standard deviation of the current mini-batch of data, respectively. and These are the trainable scaling factor and bias, respectively. When hour, It will become inactive, then These may be redundant feature points. Therefore, intramodal sparse modules will... Considered as a sparsity factor, further sparse single-modal features are defined as follows:

[0115]

[0116] Where τ is the sparsity threshold of the intramodal sparse modules, and For sparse mode features X s Element.

[0117] Intermodal sparse modules capture vast and complex multimodal interactions and encode them as sparse interaction attention weights. These weights concentrate a small number of important features with a few non-zero weights, while suppressing a large number of redundant features. Given sparse intramodal features X... s Calculate the interaction relationships between multimodalities:

[0118]

[0119] Where I is the similarity matrix. The similarity matrix I is a weight matrix that adjusts the similarity between features across all modalities as the network trains. Therefore, the similarity matrix I contains comprehensive multimodal interaction information. However, this matrix is ​​high-dimensional and its elements are too dense, which may contain a lot of redundant and useless information, interfering with sentiment representation.

[0120] To address the aforementioned issues, we utilize the Euclidean projection theory of the probabilistic simplex to design an intermodal sparsity method based on the probabilistic simplex. This method encodes the dense similarity matrix I into effective sparse interactive attention weights, highlighting a few but effective features while suppressing a large amount of redundant information.

[0121] The inter-modal sparsity method based on probabilistic simplex is used to process the elements v of the similarity matrix I. (j) Sort in descending order v (1) ≥v (2) ≥…≥v (μ) , where μ is the total number of elements. Next, the adaptive sparsity threshold θ is calculated:

[0122]

[0123] Sparse interaction attention weights are calculated based on the sparse threshold:

[0124]

[0125] It is worth noting that the inter-modal sparsity algorithm based on the probabilistic simplex is a sample-adaptive deep sparse representation algorithm because different samples have different values ​​of ρ, and thus different sparsity thresholds θ. Secondly, the dense similarity matrix I can be cleverly used to calculate the sparse interaction attention weights Z with a probability distribution using the inter-modal sparsity algorithm based on the probabilistic simplex. At the same time, it is a fast deep sparse representation algorithm that does not require additional optimization and is suitable for deep learning networks.

[0126] Given sparse interactive attention weights Z, and applied to features X within the sparse modality. s The multimodal sparse features F are obtained. s :

[0127] F s =X s Z

[0128] Finally, to further enhance the representational power of features and prevent the gradient vanishing problem during training, a double residual connection structure is added to the sparse interactive attention structure, as shown below:

[0129] F res =X+F s =X+X s Z = X + f II (X)

[0130]

[0131] Where f II (·) represents the correlation function between intramodal sparse modules and intermodal sparse modules, f FFN (·) represents the feedforward neural network FFN, F res For residual characteristics, This represents the output feature of the sparse interactive attention module.

[0132] B. Sentiment Classifier

[0133] The sentiment classifier of the multimodal student model consists of a single fully connected layer, a softmax function, and a cross-entropy loss function, as shown below:

[0134]

[0135]

[0136] Where w1 and b1 are the weights and biases of the fully connected layer. The number of samples in a small batch of data. and The first The predicted value and label of each sample.

[0137] In addition to the sentiment classifier for the multimodal student model, a unimodal classifier C needs to be loaded and trained during training of the multimodal student model. j ,j∈{1,…,J}. The structure of these unimodal classifiers is similar to that of the classifier C in the unimodal teacher model. j The classes, j∈{1,…,J}, maintain consistency and share parameters. Single-modal classifier C j ,j∈{1,…,J} represents the following:

[0138]

[0139]

[0140] in For the output features of a single-modal pre-trained model j∈{1,…,J}, w j and b j These are single-modal classifiers C j The weights and biases, and For a single-modal classifier C j For the The predicted value for each sample.

[0141] Understandably, the multimodal semi-supervised learning framework based on cyclic data distillation is applicable not only to multimodal emotion recognition tasks but also to other multimodal classification tasks.

[0142] Optionally, in the multimodal semi-supervised learning framework based on cyclic data distillation, the multimodal student model can be the sparse interactive attention network proposed in this invention, or other multimodal feature fusion networks, such as multimodal Transformer networks, multimodal interactive attention networks, etc.

[0143] This invention proposes a multimodal semi-supervised learning method and apparatus based on cyclic data distillation. The framework utilizes multiple unimodal pre-trained models and, through a confidence-enhanced pseudo-label module, distills reliable and class-representative data from a large amount of unlabeled data to train a multimodal sentiment recognition model, providing performance feedback to update all unimodal models. It jointly trains the multimodal sentiment recognition model and unimodal models in a cyclically mutually beneficial manner, ultimately obtaining an effective multimodal sentiment recognition model. Simultaneously, based on sparse interactive attention, the multimodal sentiment recognition model adaptively sparses multimodal interaction relationships and encodes them as sparse interactive attention weights. A small number of non-zero weights are used to highlight a few but important sentiment features while suppressing a large number of redundant features, enhancing the model's robustness to noisy pseudo-labeled data.

[0144] In summary, the multimodal semi-supervised learning framework based on cyclic data distillation proposed in this invention progressively improves the confidence of unimodal prediction, ensemble output, and pseudo-label datasets, effectively mitigating the pseudo-label confirmation bias problem. Simultaneously, this framework distills reliable, class-representative pseudo-label datasets that maintain the original data distribution from a large amount of unlabeled data, providing reliable data support for the training of the multimodal sentiment recognition model. Furthermore, its cyclic data distillation mechanism jointly trains the multimodal and unimodal models in a mutually beneficial manner, iteratively training an effective multimodal sentiment recognition model. Secondly, the sparse interaction attention algorithm of the multimodal sentiment recognition model is an attention mechanism based on deep sparse representations. It encodes multimodal interactions into sparse interaction attention weights with a small number of non-zero values, rather than a dense weight matrix, highlighting effective sentiment features while suppressing a large number of redundant features, and maintaining excellent robustness in noisy semi-supervised training.

[0145] Example 2:

[0146] Based on the same inventive concept, embodiments of the present invention also provide a multimodal semi-supervised learning device based on cyclic data distillation, comprising:

[0147] A data acquisition unit is used to acquire unlabeled datasets and labeled datasets, wherein the labeled datasets are divided into training sets and validation sets.

[0148] Fine-tuning unit, which is used in Epochk∈[1,k] ini During this period, the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset;

[0149] The pseudo-labeled dataset and the student-teacher model iterative update unit are used for Epoch k > k. ini During this process, the unimodal teacher model is used to predict the unlabeled dataset to obtain prediction results. The prediction results are then processed using a pre-built confidence-enhancing pseudo-label module to obtain a pseudo-label dataset. The multimodal student model is iteratively trained using the training sets of the pseudo-label dataset and the labeled dataset, and multiple unimodal teacher models are iteratively updated. Simultaneously, the pseudo-label dataset is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled dataset reaches the preset iteration condition.

[0150] The model output unit is used to output the trained multimodal student model.

[0151] Since this device corresponds to the multimodal semi-supervised learning method based on cyclic data distillation in this embodiment of the invention, and the principle of this device in solving the problem is similar to that of this method, the implementation of this device can refer to the implementation process of the above method embodiment, and repeated parts will not be described again.

[0152] Example 3:

[0153] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to realize the multimodal semi-supervised learning method based on cyclic data distillation as described above.

[0154] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0155] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, performing various server functions and processing data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by accessing data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or more of a Central Processing Unit (CPU) and a modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0156] Since this electronic device is the electronic device corresponding to the multimodal semi-supervised learning method based on cyclic data distillation in the embodiments of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0157] Example 4

[0158] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described above.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0160] Since the storage medium is the storage medium corresponding to the multimodal semi-supervised learning method based on cyclic data distillation in the embodiments of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0161] Example 5:

[0162] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the multimodal semi-supervised learning method based on cyclic data distillation according to various exemplary embodiments of the present application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0163] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0164] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0165] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal semi-supervised learning method based on cyclic data distillation, characterized in that, include: Obtain an unlabeled dataset and a labeled dataset, wherein the labeled dataset is divided into a training set and a validation set; Initialize the single-modal teacher model Model parameters and multimodal student models Model parameters and initialize the parameter Epoch and category parameters The single-modal teacher model includes a text teacher model, a voice teacher model, and a video teacher model. In Epoch During this period, the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset; In Epoch During this process, the unimodal teacher model is used to predict the unlabeled dataset to obtain prediction results. The prediction results are then processed using a pre-built confidence-enhancing pseudo-label module to obtain a pseudo-label dataset. The multimodal student model is iteratively trained using the training sets of the pseudo-label dataset and the labeled dataset, and multiple unimodal teacher models are iteratively updated. Simultaneously, the pseudo-label dataset is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled dataset reaches the preset iteration condition. Output the trained multimodal student model, which is used for multimodal emotion recognition of text, speech and video.

2. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 1, characterized in that, Obtain both an unlabeled dataset and a labeled dataset. The labeled dataset is divided into a training set and a validation set. Specifically, the training set of the labeled dataset is... The validation set for the labeled dataset is The unlabeled dataset is .

3. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 2, characterized in that, The prediction results are processed using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label dataset, specifically including: The prediction results are integrated using a pre-built modal adaptation module to obtain a multimodal integration result; Each single-modal teacher model From my own perspective, on all unlabeled datasets Make predictions, and then use the modal adaptation module. Integrating predictions from all unimodal teacher models improves the multimodal ensemble results. Confidence level; The multimodal ensemble result is distilled using a pre-built category-aware pseudo-label module to obtain a pseudo-label dataset: Among them, based on the multimodal integration results By using a category-aware pseudo-label module, from an unlabeled dataset Distillation of pseudo-label dataset ; Category-aware pseudo-label module Based on the new pseudo-label dataset Data distribution is updated in stages (Epoch) Thresholds for each category.

4. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 3, characterized in that, The multimodal student model is iteratively trained using the training sets of the pseudo-labeled dataset and the labeled dataset, and multiple unimodal teacher models are iteratively updated, specifically including: Pseudo-label dataset Training set of the label dataset Through the total loss function Training multimodal student models And update the single-modal teacher model. Classifier .

5. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 4, characterized in that, The loss function value of the multimodal student model on the validation set of the labeled dataset reaches a preset iteration condition, specifically including: Loss function value of the multimodal student model on the validation set of the labeled dataset This will serve as a condition for terminating network training, when If the network does not decline within 10 consecutive epochs, it will stop training and output a multimodal student model. and its parameters Otherwise, continue the loop iteration.

6. A multimodal semi-supervised learning device based on cyclic data distillation, characterized in that, include: The data acquisition unit is used to acquire unlabeled and labeled datasets, wherein the labeled dataset is divided into training and validation sets; and to initialize the unimodal teacher model. Model parameters and multimodal student models Model parameters and initialize the parameter Epoch and category parameters The single-modal teacher model includes a text teacher model, a voice teacher model, and a video teacher model. Fine-tuning unit, which is used in Epoch During this period, the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset; The pseudo-labeled dataset and student-teacher model iterative update unit are used in Epoch. During this process, the unimodal teacher model is used to predict the unlabeled dataset to obtain prediction results. The prediction results are then processed using a pre-built confidence-enhancing pseudo-label module to obtain a pseudo-label dataset. The multimodal student model is iteratively trained using the training sets of the pseudo-label dataset and the labeled dataset, and multiple unimodal teacher models are iteratively updated. Simultaneously, the pseudo-label dataset is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled dataset reaches the preset iteration condition. The model output unit is used to output the trained multimodal student model, which is used for multimodal emotion recognition of text, speech and video.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal semi-supervised learning method based on cyclic data distillation as described in any one of claims 1 to 5.

Images