Multi-mode semi-supervised learning method and device based on cyclic data distillation

Abstract

The invention relates to the technical field of multi-modal learning, semi-supervised learning and emotion calculation, in particular to a multi-modal semi-supervised learning method and device based on cyclic data distillation. According to the method and the device, a plurality of single-mode pre-training models are utilized, a false label module is enhanced through confidence, reliable data with category representativeness are distilled from a large amount of label-free data to train a multi-mode emotion recognition model, and performance feedback is provided to update all the single-mode models. According to the method, a multi-modal emotion recognition model and a single-modal model are jointly trained in a cyclic mutual benefit mode, and finally an effective multi-modal emotion recognition model is obtained.

Description

Technical Field

[0001] The present invention relates to the technical fields of multimodal learning, semi-supervised learning and sentiment computing, and in particular to a multimodal semi-supervised learning method and device based on cyclic data distillation. Background Art

[0002] (1) Multimodal emotion recognition algorithm

[0003] Multimodal emotion recognition technology integrates different types of human data such as facial expressions, voice intonation, and text content to capture and analyze emotional information from multiple angles, effectively understand the user's true emotions, and make human-computer interaction more natural and humane. Multimodal emotion recognition technology is widely used in social media, e-commerce, advertising, customer service, healthcare and other fields, helping companies or individuals better understand the user's emotional state and make corresponding decisions or adjustments.

[0004] At present, the training of multimodal emotion recognition models is highly dependent on labeled datasets. Labeled datasets can provide sufficient information for model training and verification, and help the model better understand the characteristics and laws of the data. However, the data size of current multimodal emotion datasets is generally small. The largest multimodal emotion dataset, the CMU-MOSEI dataset, has only about 23,000 samples, the MELD dataset and the IEMOCAP dataset have 13,708 and 10,039 samples respectively, and the sample sizes of other datasets are less than 3,000. Therefore, the data size of existing multimodal emotion datasets has severely restricted the development of multimodal emotion recognition models. At the same time, due to the complexity of the multimodal emotion annotation process, it takes a lot of time and manpower to build a large-scale annotated dataset. Therefore, scholars study multimodal semi-supervised learning algorithms, train multimodal emotion recognition models through a large amount of unlabeled data, and reduce the model's dependence on large labeled datasets.

[0005] Secondly, most multimodal emotion recognition works study comprehensive and complementary multimodal features, which often contain a lot of redundant information and interfere with the discriminative performance of the model. Especially in the semi-supervised learning environment with high noise, the negative impact of redundant information on model performance is more significant and prominent. Some research works introduce gating mechanisms to reduce redundant information. For example, multi-level fusion networks control the redundant information of multimodal long sequences by fusing forget gate modules. Other methods use attention mechanisms to weaken redundant features. For example, the AtM-DNN model introduces an attention mechanism to control the weight of each modal information flowing into the multimodal fusion feature, thereby reducing the redundant information of the weak modality. However, the general attention weight matrix is ​​large and dense, and its element values ​​may generally be too small to effectively highlight important features. Secondly, sparse representation algorithms are also an effective redundancy reduction technology. For example, the multimodal sparse Transformer network uses the traditional top-q sparse method to retain the most important feature fragments. However, the complex optimization method of sparse representation algorithms is difficult to adapt to the gradient backpropagation optimization method, which makes most sparse representation algorithms difficult to adapt to deep learning models. Therefore, reducing multimodal redundant features and improving the robustness of the model is one of the research issues in multimodal feature fusion and multimodal semi-supervised learning.

[0006] (2) Semi-supervised learning algorithm

[0007] Semi-supervised learning refers to a learning framework that uses a small number of labeled samples and a large number of unlabeled samples to train models. It can effectively reduce the dependence on large labeled datasets, reduce annotation costs and model training costs, and improve model generalization performance.

[0008] In the field of multimodal emotion recognition, there are relatively few studies on semi-supervised learning, which are mainly divided into pseudo-labeling methods, consistency regularization methods, graph-based methods, and generative model-based methods.

[0009] The pseudo-labeling method refers to a cyclic training process in which a model is used to predict unlabeled data, obtain pseudo-labeled data, and train the model together with a small amount of labeled data. For example, Fixmatch uses high-confidence predictions that exceed a fixed threshold as pseudo-labels for unlabeled data. However, the pseudo-labeling method is prone to confirmation bias, that is, the model that produces incorrect predictions reuses incorrect information during training, resulting in error accumulation, which can easily affect the quality of the pseudo-label and the recognition performance of the model. Some research works use preheating or high-threshold operations to enable the model to generate reliable pseudo-labels. However, setting a high threshold can easily cause the model to tend to learn strong category samples and ignore weak category samples, resulting in the pseudo-label data set deviating from the original data distribution.

[0010] The consistency regularization method is a regularization constraint method, which constrains the model to produce consistent prediction results under different input interferences, thereby mining the general ability and generalization ability of the model. For example, the time integration model constrains the output distribution to remain consistent over multiple epochs through the squared difference loss function, thereby mining the potential information of unlabeled data. However, the consistency regularization method is not suitable for multimodal data with large modal differences.

[0011] Graph-based methods propagate labels from labeled nodes (labeled data points) to unlabeled nodes (unlabeled data points) according to node similarity, such as dynamic graph fusion label propagation networks.

[0012] Generative model-based methods learn universal representations from a large amount of unlabeled data through a data reconstruction process. General reconstruction methods include autoencoders and variational autoencoders. For example, the cross-modal distribution matching model learns universal features from richer multimodal unlabeled data through the reconstruction mechanism of the autoencoder, effectively improving model performance.

[0013] The current scale of multimodal emotion datasets severely limits the training and performance of multimodal emotion recognition models, and building large-scale annotated datasets requires a lot of time and manpower. Therefore, scholars have studied multimodal semi-supervised learning algorithms and used a large amount of unlabeled data to train multimodal emotion recognition models. However, in multimodal semi-supervised learning algorithms, the consistency regularization method is not suitable for multimodal data with large modal differences, and the pseudo-label method has confirmation bias problems and pseudo-label data distribution bias problems, which greatly reduce the performance and reliability of the model. Secondly, multimodal emotion features contain a lot of redundant information, which is easy to interfere with the emotion discrimination performance of the model under the training of a high-noise semi-supervised learning framework. Summary of the invention

[0014] In response to at least one deficiency in the prior art, the present invention provides a multimodal semi-supervised learning method and device based on cyclic data distillation, which can effectively alleviate the pseudo-label confirmation bias problem and the pseudo-label data distribution bias problem, and maintain excellent robustness in high-noise semi-supervised training.

[0015] To achieve the above object, the present invention provides the following technical solutions:

[0016] In a first aspect, the present invention provides a multimodal semi-supervised learning method based on cyclic data distillation, comprising:

[0017] Obtain an unlabeled data set and a labeled data set, wherein the labeled data set is divided into a training set and a validation set;

[0018] During Epochk∈[1,kini], the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset;

[0019] During Epochk>kini, the unlabeled data set is predicted by the unimodal teacher model to obtain a prediction result, and the prediction result is processed by a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label data set; the multimodal student model is iteratively trained and multiple unimodal teacher models are iteratively updated by using the training set of the pseudo-label data set and the labeled data set, and at the same time, the pseudo-label data set is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled data set reaches a preset iteration condition;

[0020] Output the trained multimodal student model.

[0021] Furthermore, an unlabeled data set and a labeled data set are obtained, and the labeled data set is divided into a training set and a validation set. Specifically, the training set of the labeled data set is The validation set of the labeled dataset is The unlabeled dataset is

[0022] After the acquisition step, it also includes: initializing the single-modal teacher model f T1 ,…,f TJ Model parameters and the multimodal student model f S The model parameters wf S , and initialize the parameter Epochk=1 and the category parameter c∈(1,…,C).

[0023] Further, the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled data set, specifically including:

[0024] w fS ←w fS -η(▽L(w fS ,X))

[0025] w fTj ←w fTj -η(▽L(w fTj ,X))

[0026] Furthermore, the prediction results are processed using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label dataset, specifically including:

[0027] The prediction results are integrated using the pre-built modal adaptation module to obtain a multi-modal integration result:

[0028] Among them, each single-modal teacher model f Tj From your own perspective, all unlabeled datasets Make predictions and then pass the modal adaptation module f MAE Integrate all single-modal teacher model prediction results to improve the multimodal integration result P U Confidence of:

[0029]

[0030] The multimodal integration results are distilled using the pre-built category-aware pseudo-label module to obtain a pseudo-label dataset:

[0031] Among them, according to the multimodal integration result P U , through the category-aware pseudo-label module, a pseudo-label dataset φ is distilled from the unlabeled dataset U k ;

[0032]

[0033] Category-aware pseudo-label module f CPL According to the new pseudo-label dataset φ k Update the thresholds of each category of Epochk+1 in stages according to the data distribution:

[0034]

[0035] Further, the pseudo-label dataset and the training set of the label dataset are used to iteratively train the multimodal student model and iteratively update the plurality of single-modal teacher models, specifically including:

[0036] Pseudo-labeled dataset φ k The training set X of the labeled dataset is passed through the total loss function L(w fS ,X∪φ k ) Jointly train the multimodal student model f S , and update the unimodal teacher model f T1 ,…,f TJ The classifier C j ,j∈{1,…,J}.

[0037]

[0038] w fS ←w fS -η(▽L(w fS ,X∪φ k ))

[0039]

[0040] Furthermore, the loss function value of the multimodal student model on the validation set of the label data set reaches a preset iteration condition, specifically including:

[0041] The loss function value of the multimodal student model on the validation set of the labeled dataset It will be used as the condition for terminating network training when If the value does not decrease within 10 consecutive epochs, the network will stop training and output the multimodal student model f S and its parameters Otherwise continue looping.

[0042] In a second aspect, the present invention provides a multimodal semi-supervised learning device based on cyclic data distillation, characterized in that it includes:

[0043] A data acquisition unit, which is used to acquire an unlabeled data set and a labeled data set, wherein the labeled data set is divided into a training set and a validation set;

[0044] The fine-tuning unit is used to adjust the ini ] during which the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset;

[0045] Pseudo-labeled dataset and student-teacher model iterative update unit, which is used to update the model in Epochk>k ini During this period, the unlabeled data set is predicted by using the unimodal teacher model to obtain a prediction result, and the prediction result is processed by using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label data set; the multimodal student model is iteratively trained using the pseudo-label data set and the training set of the labeled data set, and multiple unimodal teacher models are iteratively updated. At the same time, the pseudo-label data set is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled data set reaches a preset iteration condition;

[0046] The model output unit is used to output the trained multimodal student model.

[0047] In a third aspect, the present invention further provides an electronic device, including a processor and a memory;

[0048] The memory is used to store programs;

[0049] The processor executes the program to implement the method described above.

[0050] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0051] In a fifth aspect, the present invention further provides a computer program product or a computer program, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) In current multimodal semi-supervised learning research, pseudo-label generation and training both act on the same multimodal structure, that is, the pseudo-label generator and the actor are the same multimodal structure, which is prone to confirmation bias and vicious cycle training. The multimodal semi-supervised learning framework based on cyclic data distillation proposed in the present invention fully utilizes the performance advantages of mature unimodal pre-trained models, uses multiple unimodal pre-trained models as teacher models, and multimodal emotion recognition models as student models. Through the confidence enhancement pseudo-label module, reliable pseudo-label data is distilled from a large amount of unlabeled data, and then the multimodal student model is cyclically trained. This framework effectively utilizes the performance and stability advantages of the unimodal teacher model to ensure the stability and reliability of the multimodal semi-supervised learning framework in cyclic generation and training.

[0054] (2) Pseudo-labeling algorithms are prone to confirmation bias, which reduces the reliability of pseudo-labeling. Some methods set high thresholds to improve the reliability of pseudo-labeling, but it is easy to make the model tend to learn strong class samples and ignore weak class samples, which causes the pseudo-labeling data set to deviate from the original data distribution and cause data imbalance. The confidence-enhanced pseudo-labeling module proposed in the present invention improves the confidence of single-modal prediction and integrated output through a modality adaptive integration module, and provides appropriate thresholds for different categories at different stages through a category-aware threshold module, thereby controlling the number of strong category samples and accommodating more weak category samples, ensuring the consistency of data distribution between the pseudo-labeling data set and the original data set, and improving the reliability of the pseudo-labeling data set, effectively reducing confirmation bias.

[0055] (3) High-dimensional multimodal features tend to contain a large amount of redundant information. In semi-supervised training with high noise, redundant information will seriously interfere with the discriminative performance of the model. The sparse interactive attention module proposed in the present invention can effectively encode multimodal interactive relationships into sparse interactive attention weights through deep sparse representation algorithms within and between modalities without complex optimization methods, and can highlight a small number of important emotional features with a small number of non-zero weights, suppress a large number of redundant features, and effectively resist the interference of high-noise sample sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 Flow chart of multimodal semi-supervised learning based on cyclic data distillation in an embodiment of the present invention;

[0058] Figure 2 This is a diagram of a multimodal semi-supervised learning framework based on cyclic data distillation in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] Example:

[0061] It should be noted that the terms "including" and "having" and any variations of the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] See also Figure 1 , a multimodal semi-supervised learning method based on cyclic data distillation provided in this embodiment may specifically include the following process:

[0063] Step 1: Given a training set of labeled datasets and validation set

[0064] and unlabeled datasets Initialize the unimodal teacher model f T1 ,…,f TJ Model parameters and the multimodal student model f S Model parameters

[0065] And initialize the parameter Epochk=1 and the category parameter c∈(1,…,C).

[0066] Step 2: In Epochk∈[1,k ini ] During this period, all unimodal teacher models and multimodal student models are fine-tuned using the training set X of the labeled dataset:

[0067]

[0068] Step 3: At Epoch k > k ini During this period, each unimodal teacher model f Tj From your own perspective, all unlabeled datasets Make predictions and then pass the modal adaptation module f MAE Integrate all single-modal teacher model prediction results to improve the multimodal integration result P U Confidence of:

[0069]

[0070] Step 4: Based on the multimodal integration result P U , through the category-aware pseudo-label module, a reliable and category-representative pseudo-label dataset φ is distilled from the unlabeled dataset U k .

[0071]

[0072]

[0073] Step 5: Category-aware pseudo-label module f CPL According to the new pseudo-label dataset φ k Update the thresholds of each category in Epochk+1 based on the data distribution.

[0074]

[0075] Step 6: New pseudo-labeled dataset φ k The training set X of the labeled dataset is passed through the total loss function

[0076] Jointly train a multimodal student model f S , and update the unimodal teacher model f T1 ,…,f TJ The classifier C j ,j∈{1,…,J}.

[0077]

[0078] Step 7: Loss function value of the multimodal student model on the validation set of the labeled dataset

[0079] It will be used as the condition for terminating network training. If it does not decrease within 10 consecutive times, the network will stop training and output the multimodal student model f S and its parameters Otherwise, continue to loop steps 3 to 7.

[0080] like Figure 2 As shown in the figure. The framework consists of a unimodal teacher model, a confidence-enhanced pseudo-label module, and a multimodal student model. Each unimodal teacher model predicts unlabeled samples from its own perspective. Next, the confidence-enhanced pseudo-label module integrates the prediction results of all unimodal teacher models in a sharpened and uncertainty-based weighted manner to reduce the interference from suboptimal categories and modality uncertainty, and improve the confidence of the integrated output. Then, the confidence-enhanced pseudo-label module screens a large amount of unlabeled data based on the integrated output results through a stage-adaptive category-aware threshold, and distills pseudo-label data with high confidence and category representativeness. The multimodal student model based on sparse interactive attention gradually learns rich discriminative information from reliable pseudo-label and label data sets. Finally, all unimodal teacher models are further updated using the performance of the multimodal student model on labeled data. The multimodal semi-supervised learning framework based on cyclic data distillation jointly trains multimodal student models and unimodal teacher models in a cyclic and mutually beneficial manner, and iteratively trains an effective multimodal student model (multimodal emotion recognition model).

[0081] In the specific implementation, regarding the unimodal teacher model:

[0082] Each unimodal model is relatively mature and has a pre-trained model with strong generalization and good prediction performance. Therefore, the present invention uses multiple unimodal pre-trained models as teacher models, jointly annotates unlabeled data, constructs a pseudo-label data set, and then trains a multimodal emotion recognition model.

[0083] Each unimodal teacher model consists of a pre-trained model and a classifier. For example, the text teacher model consists of a text pre-trained model and a text classifier. The text pre-trained model is a RoBERTa pre-trained model, which contains 24 layers of Transformer and 1 fully connected layer. The text classifier contains 1 fully connected layer and 1 softmax function. The speech teacher model consists of a speech pre-trained model and a speech classifier. The speech pre-trained model is a RoBERTa pre-trained model, but the model only contains 12 layers of Transformer and 1 fully connected layer and is re-pretrained on the Librisspeech-960 dataset. The speech classifier contains 1 fully connected layer and 1 softmax function. The video teacher model consists of a video pre-trained model and a video classifier. The video pre-trained model is the facial attribute network Fabnet, and the video classifier contains 1 fully connected layer and 1 softmax function. Similarly, other unimodal teacher models can be composed of pre-trained models and their classifiers in the current unimodal research field.

[0084] It is understandable that the pre-trained model of the unimodal teacher model can be other effective pre-trained models in the unimodal field. For example, the pre-trained model of the text teacher model can be Transformer, BERT, etc., the pre-trained model of the voice teacher model can be Transformer, HuBERT, etc., and the pre-trained model of the video teacher model can be 3D ResNet, CLIP, etc.

[0085] In the specific implementation, regarding the confidence enhanced pseudo label module:

[0086] The confidence-enhanced pseudo-label module aims to distill reliable and category-representative pseudo-label data from a large amount of unlabeled data. It consists of a modality-adaptive integration module and a category-aware pseudo-label module.

[0087] A. Modal Adaptive Integration Module

[0088] Different unimodal teacher models predict unlabeled samples from their own perspectives and provide corresponding prediction confidence. The prediction confidence is a probability distribution over different categories, so there is the highest probability category (optimal category), the second highest probability category (suboptimal category), the third highest probability category, etc. However, different modalities have different prediction confidences on the same sample, so the prediction confidence of integrating multiple unimodal teacher models is likely to highlight the suboptimal category, thereby threatening the optimal category and reducing the reliability of the multimodal integrated output. To address the above issues, the modal adaptive integration module effectively improves the confidence of unimodal predictions and integrated outputs.

[0089] First, the modality adaptation integration module uses the orthogonal diversity loss function to reduce the interference of suboptimal categories. Given the prediction of the j-th unimodal teacher model Where C is the number of categories. Assuming c is the category corresponding to the true label, then is the predicted probability of the true label category, is the probability combination of non-real label categories:

[0090]

[0091] The orthogonal diversity loss function is:

[0092]

[0093] Where J is the number of modalities. The orthogonal diversity loss function encourages the probability combinations of non-real label categories between each unimodal teacher model to be orthogonal to each other, that is, output different suboptimal category probability combinations, effectively reducing the suboptimal category interference of subsequent multimodal integration output, and improving the confidence of unimodal prediction and multimodal integration output.

[0094] Next, the modality adaptive integration module evaluates the prediction uncertainty of each single-modality teacher model and adaptively integrates the prediction results of all single-modality teacher models according to the prediction uncertainty to further improve the confidence of the multimodal integration output. The process is expressed as follows:

[0095]

[0096] Where E j is the prediction uncertainty of the j-th unimodal teacher model, α j It is the weight of participating in multimodal integration calculated according to the prediction uncertainty of the j-th single-modal teacher model, which can reduce the impact of uncertain prediction on the multimodal integration output. U is the ensemble prediction result of all unimodal teacher models.

[0097] B. Category-aware pseudo-label module

[0098] The pseudo-labeling method generally uses a fixed and uniform high threshold to filter unlabeled data. However, this method tends to select a large number of strong category samples and intercept most of the weak category samples, resulting in too many strong category samples and too few weak category samples, which leads to an imbalanced pseudo-label dataset or deviation from the original data distribution. To address the above problems, the category-aware pseudo-labeling module provides appropriate thresholds for different categories at different stages, effectively distilling a reliable and category-representative pseudo-label dataset from a large amount of unlabeled data, alleviating the problem of data imbalance.

[0099] The category-aware pseudo-label module is divided into the initial stage, the learning stage, and the stabilization stage.

[0100] In the initial stage, it is necessary to ensure the task adaptability and stability of the entire framework.ini epochs, fine-tuning the classifiers of all unimodal teacher models through the labeled dataset, while using a unified preset threshold for different categories Filter out the multimodal integration results P U Exceed Θ k Unlabeled samples generate a relatively stable initial pseudo-label dataset Φ k , used for training in the learning phase.

[0101] The learning phase expects different categories to have different thresholds. At the same time, the thresholds of different categories can gradually control the number of strong category samples and allow more weak category samples to pass through, providing a sufficient number of representative samples for different categories. Therefore, the learning phase is based on the previous pseudo-label dataset Φ k-1 According to the data distribution, the thresholds of different categories are adjusted adaptively. The threshold adjustment during the learning phase is as follows:

[0102]

[0103] Where I(·) is an indicator function that returns 1 when the expression in the brackets is true. s is a pseudo-label dataset Φ k-1 The pseudo label of the sth sample. M is the total number of unlabeled samples, Ω c is a pseudo-label dataset Φ k-1 The total number of class c samples, is the confidence threshold of class c in the kth epoch. From the above formula, we can observe that the current pseudo-label dataset Φ k-1 There are fewer samples of type C. will become smaller, so as to accommodate more c-class samples in the current epoch and output more representative samples for weak categories. At the same time, as the size of the pseudo-label dataset increases, the above formula can achieve the category threshold Θ k gradual growth pattern.

[0104] In the stable stage, it is expected that all category thresholds can tend to a dynamic stable state as the pseudo-label dataset stabilizes. Therefore, when the data size of the pseudo-label dataset is larger than the amount of remaining unlabeled data, the threshold adjustment stable stage is entered:

[0105]

[0106] In the stable stage, if the number of pseudo-label samples of category c is the largest, then It will be strictly controlled at a higher preset threshold to control the number of samples in this category, while other category thresholds are aligned and adjusted according to the category C threshold to ensure that the distribution of pseudo-label data remains relatively stable.

[0107] In specific implementation, regarding the multimodal student model:

[0108] The pseudo-labeled dataset and the labeled dataset will jointly train the multimodal student model. However, high-dimensional multimodal features contain a lot of redundant information, which seriously interferes with the model's sentiment discrimination performance, especially in high-noise semi-supervised training scenarios. To address the above problems, the multimodal student model is designed as a sparse interactive attention network. The sparse interactive attention network encodes multimodal interaction features into sparse interactive attention weights through intra-modal sparse modules and inter-modal sparse modules, focusing on highlighting emotional features with fewer non-zero weights, ignoring a large number of redundant features, and effectively preventing overfitting.

[0109] The sparse interactive attention network consists of a pre-trained feature extractor, a sparse interactive attention module, and a sentiment classifier. The pre-trained feature extractor is consistent with the unimodal pre-trained model of the unimodal teacher model.

[0110] A. Sparse Interaction Attention

[0111] The sparse interactive attention module is the core component of the sparse interactive attention network, which mainly consists of an intra-modality sparse module and an inter-modality sparse module.

[0112] The intra-modal sparse module retains unimodal features that are effective for sentiment representation and suppresses redundant features, providing sparse but effective unimodal features for subsequent multimodal interactions. It cleverly uses the scaling factor of batch normalization to implement sparse constraints and sparse unimodal features at a lower computational cost. The following is the expression of batch normalization:

[0113]

[0114] in is the initial modal feature Elements of is the output feature of the j-th unimodal pre-trained feature extractor, For splicing operation. and are the mean and standard deviation of the current mini-batch data, and and are the trainable scaling factor and bias, respectively. hour, will be inactivated, then may be redundant feature points. Therefore, the intra-modal sparse module will Considered as a sparse factor, further sparse unimodal features:

[0115]

[0116] Among them, τ is the sparse threshold of the intra-modal sparse module, and is the sparse modal feature X s elements.

[0117] The inter-modal sparse module captures the large and complex multi-modal interaction relationship and encodes it into sparse interaction attention weights, which focuses on a few but important features with a small number of non-zero weights and suppresses a large number of redundant features. s , calculate the interaction relationship between multi-modal:

[0118]

[0119] Where I is the similarity matrix, is a weight matrix, which can adjust the similarity between all modal features as the network is trained. Therefore, the similarity matrix I contains comprehensive multimodal interaction information, but the matrix has high dimensions and too dense elements, which may contain a lot of redundant and useless information, interfering with the emotional representation.

[0120] To solve the above problems, a probability simplex-based inter-modal sparse method is designed by utilizing the Euclidean projection theory of probability simplex. The dense similarity matrix I is encoded into effective sparse interactive attention weights, which can focus on highlighting a few but effective features and suppress a large amount of redundant information.

[0121] The inter-modal sparse method based on probability simplex is used to calculate the element v of the similarity matrix I. (j) Sort in descending order v (1) ≥v (2) ≥…≥v (μ) , where μ is the total number of elements. Next, the adaptive sparse threshold θ is calculated:

[0122]

[0123] The sparse interaction attention weights are calculated based on the sparse threshold:

[0124]

[0125] It is worth noting that the inter-modal sparse algorithm based on probability simplex is a sample-adaptive deep sparse representation algorithm, because different samples have different ρ and thus different sparse thresholds θ. Secondly, the dense similarity matrix I can cleverly calculate the sparse interactive attention weight Z with probability distribution through the inter-modal sparse algorithm based on probability simplex. At the same time, it is a fast deep sparse representation algorithm that does not require additional optimization and is suitable for deep learning networks.

[0126] Given the sparse interaction attention weight Z, and acting on the sparse intra-modal features X s , get the multimodal sparse feature F s :

[0127] F s =X s Z

[0128] Finally, in order to further enhance the representation ability of features and prevent the gradient vanishing problem in the training process, a double residual connection structure is added to the sparse interactive attention structure, and its structure is as follows:

[0129] F res =X+F s =X+X s Z=X+f II (X)

[0130]

[0131] where f II (·) is the correlation function between the intra-modal sparse module and the inter-modal sparse module, f FFN (·) is the feed-forward neural network FFN, F res is the residual feature, is the output feature of the sparse interaction attention module.

[0132] B. Sentiment Classifier

[0133] The sentiment classifier of the multimodal student model consists of 1 fully connected layer, a softmax function, and a cross entropy loss function, which is expressed as follows:

[0134]

[0135]

[0136] Where w1 and b1 are the weight and bias of the fully connected layer, is the number of samples in a mini-batch of data, and Respectively The predicted values ​​and labels of samples.

[0137] In addition to the sentiment classifier of the multimodal student model, the unimodal classifier C that is used to train the multimodal student model needs to be loaded during training. j , j∈{1,…,J}. The structures of these unimodal classifiers are similar to the classifier C of the unimodal teacher model. j ,j∈{1,…,J} remain consistent and share parameters with each other. Unimodal classifier C j ,j∈{1,…,J} is represented as follows:

[0138]

[0139]

[0140] in is the output feature of the unimodal pre-trained model j∈{1,…,J}, w j and b j They are unimodal classifiers C j The weights and biases of C is a unimodal classifier j For The predicted value of a sample.

[0141] It can be understood that the multimodal semi-supervised learning framework based on cyclic data distillation is applicable not only to multimodal emotion recognition tasks, but also to other multimodal classification tasks.

[0142] Optionally, in a multimodal semi-supervised learning framework based on cyclic data distillation, the multimodal student model can be the sparse interactive attention network proposed in the present invention, or it can be other multimodal feature fusion networks, such as a multimodal Transformer network, a multimodal interactive attention network, etc.

[0143] The present invention proposes a multimodal semi-supervised learning method and device based on cyclic data distillation. The framework uses multiple unimodal pre-trained models, and through a confidence-enhanced pseudo-label module, distills reliable and category-representative data from a large amount of unlabeled data to train a multimodal emotion recognition model, and provides performance feedback to update all unimodal models. It jointly trains multimodal emotion recognition models and unimodal models in a cyclic and mutually beneficial manner, and finally obtains an effective multimodal emotion recognition model. At the same time, the multimodal emotion recognition model based on sparse interactive attention adaptively sparses multimodal interaction relationships and encodes them into sparse interactive attention weights, using a small number of non-zero weights to focus on highlighting a few but important emotional features and suppress a large number of redundant features, thereby enhancing the model's robustness to high-noise pseudo-label data.

[0144] In summary, the multimodal semi-supervised learning framework based on cyclic data distillation proposed in the present invention gradually improves the confidence of unimodal prediction, integrated output, and pseudo-label data sets, and effectively alleviates the problem of pseudo-label confirmation bias; at the same time, the framework distills reliable, category-representative, and original data distribution-maintaining pseudo-label data sets from a large amount of unlabeled data, providing reliable data support for the training of multimodal emotion recognition models. Moreover, its cyclic data distillation mechanism jointly trains multimodal models and unimodal models in a cyclically mutually beneficial manner, and iteratively trains an effective multimodal emotion recognition model. Secondly, the sparse interactive attention algorithm of the multimodal emotion recognition model is an attention mechanism based on deep sparse representation, which can encode multimodal interactions into sparse interactive attention weights with a small number of non-zero values ​​instead of dense weight matrices, focusing on highlighting effective emotional features and suppressing a large number of redundant features, and maintaining excellent robustness in high-noise semi-supervised training.

[0145] Embodiment 2:

[0146] Based on the same inventive concept, an embodiment of the present invention further provides a multimodal semi-supervised learning device based on cyclic data distillation, comprising:

[0147] A data acquisition unit, which is used to acquire an unlabeled data set and a labeled data set, wherein the labeled data set is divided into a training set and a validation set;

[0148] The fine-tuning unit is used to adjust the ini ] during which the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset;

[0149] Pseudo-labeled dataset and student-teacher model iterative update unit, which is used to update the model in Epochk>k ini During this period, the unlabeled data set is predicted by using the unimodal teacher model to obtain a prediction result, and the prediction result is processed by using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label data set; the multimodal student model is iteratively trained using the pseudo-label data set and the training set of the labeled data set, and multiple unimodal teacher models are iteratively updated. At the same time, the pseudo-label data set is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled data set reaches a preset iteration condition;

[0150] The model output unit is used to output the trained multimodal student model.

[0151] Since the device is a device corresponding to the multimodal semi-supervised learning method based on cyclic data distillation of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0152] Embodiment 3:

[0153] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described above.

[0154] It is understandable that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0155] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a chip.

[0156] Since the electronic device is the electronic device corresponding to the multimodal semi-supervised learning method based on cyclic data distillation of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0157] Example 4

[0158] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described above.

[0159] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0160] Since the storage medium is the storage medium corresponding to the multimodal semi-supervised learning method based on cyclic data distillation in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0161] Embodiment 5:

[0162] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a computer device, the program code is used to enable the computer device to perform the steps of the multimodal semi-supervised learning method based on cyclic data distillation according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0163] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0164] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0165] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal semi-supervised learning method based on cyclic data distillation, characterized in that: include: Obtain an unlabeled data set and a labeled data set, wherein the labeled data set is divided into a training set and a validation set; In Epochk∈[1,k ini ] during which the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset; In Epoch k>k ini During this period, the unlabeled data set is predicted by using the unimodal teacher model to obtain a prediction result, and the prediction result is processed by using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label data set; the multimodal student model is iteratively trained using the pseudo-label data set and the training set of the labeled data set, and multiple unimodal teacher models are iteratively updated. At the same time, the pseudo-label data set is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled data set reaches a preset iteration condition; Output the trained multimodal student model.

2. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 1, characterized in that: Obtain an unlabeled dataset and a labeled dataset, wherein the labeled dataset is divided into a training set and a validation set. Specifically, the training set of the labeled dataset is The validation set of the labeled dataset is The unlabeled dataset is After the acquisition step, it also includes: initializing the single-modal teacher model f T1 ,…,f TJ The model parameters w fTj and the multimodal student model f S The model parameters w fS , and initialize the parameter Epochk=1 and the category parameter c∈(1,…,C).

3. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 2, characterized in that: The pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset, specifically including:

4. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 3, characterized in that: The prediction results are processed using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label dataset, specifically including: The prediction results are integrated using the pre-built modal adaptation module to obtain a multi-modal integration result: Among them, each single-modal teacher model f Tj From your own perspective, all unlabeled datasets Make predictions and then pass the modal adaptation module f MAE Integrate all single-modal teacher model prediction results to improve the multimodal integration result P U Confidence of: The multimodal integration results are distilled using the pre-built category-aware pseudo-label module to obtain a pseudo-label dataset: Among them, according to the multimodal integration result P U , through the category-aware pseudo-label module, a pseudo-label dataset φ is distilled from the unlabeled dataset U k ; Category-aware pseudo-label module f CPL According to the new pseudo-label dataset φ k Update the thresholds of each category of Epochk+1 in stages according to the data distribution; 5. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 4, characterized in that: Iteratively training the multimodal student model and iteratively updating the plurality of single-modal teacher models using the training set of the pseudo-labeled dataset and the labeled dataset specifically includes: Pseudo-labeled dataset φ k The training set X of the labeled dataset is passed through the total loss function Train the multimodal student model f S , and update the unimodal teacher model f T1 ,…,f TJ The classifier C j ,j∈{1,…,J}.

6. The multimodal semi-supervised learning method based on cyclic data distillation according to claim 5, characterized in that: The loss function value of the multimodal student model on the validation set of the label data set reaches a preset iteration condition, specifically including: The loss function value of the multimodal student model on the validation set of the labeled dataset It will be used as the condition for terminating network training when If the value does not decrease within 10 consecutive epochs, the network will stop training and output the multimodal student model f S and its parameters Otherwise continue looping.

7. A multimodal semi-supervised learning device based on cyclic data distillation, characterized in that: include: A data acquisition unit, which is used to acquire an unlabeled data set and a labeled data set, wherein the labeled data set is divided into a training set and a validation set; The fine-tuning unit is used to adjust the ini ] during which the pre-built unimodal teacher model and multimodal student model are adjusted using the training set of the labeled dataset; Pseudo-labeled dataset and student-teacher model iterative update unit, which is used to update the model in Epochk>k ini During this period, the unlabeled data set is predicted by using the unimodal teacher model to obtain a prediction result, and the prediction result is processed by using a pre-built confidence-enhanced pseudo-label module to obtain a pseudo-label data set; the multimodal student model is iteratively trained using the pseudo-label data set and the training set of the labeled data set, and multiple unimodal teacher models are iteratively updated. At the same time, the pseudo-label data set is updated by dynamically adjusting the threshold until the loss function value of the multimodal student model on the validation set of the labeled data set reaches a preset iteration condition; The model output unit is used to output the trained multimodal student model.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the multimodal semi-supervised learning method based on cyclic data distillation as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal semi-supervised learning method based on cyclic data distillation as described in any one of claims 1 to 6 is implemented.

Images