Visual emotion model pre-training method based on semi-supervised learning
By combining semi-supervised learning with deep clustering and pseudo-labeling mechanisms to pre-train visual emotion models, the problem of poor generalization performance of visual emotion recognition models under labeling difficulties and imbalanced data is solved, and more accurate emotion recognition is achieved.
Patent Information
- Application Number
- CN202511184863.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-22
- Publication Date
- 2025-12-26
AI Technical Summary
Existing visual emotion recognition models suffer from poor generalization performance and struggle to achieve efficient recognition due to challenges such as difficulty in labeling emotion data, limited data volume, uneven data distribution, and the subjectivity of labels.
We employ a semi-supervised learning-based pre-training method for visual emotion models. By combining labeled and unlabeled datasets, we utilize deep clustering and pseudo-labeling mechanisms for feature extraction and model updating, and construct a joint total loss function to improve the model's generalization ability.
It significantly improves the model's adaptability to unknown data and its recognition accuracy, enhances its ability to recognize emotional states, and is suitable for a variety of practical application scenarios.
Smart Images
Figure CN121213993A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of deep learning and the field of visual emotion recognition, and in particular to a visual emotion model pre-training method based on semi-supervised learning. BACKGROUND
[0002] Visual emotion recognition technology, as an important branch of emotion recognition research, mainly focuses on how to accurately identify human emotional states by analyzing visual information such as facial expressions and body movements contained in images or videos. In recent years, researchers have mainly focused on static facial expression recognition methods in facial expression recognition. However, the capture and analysis of dynamic facial expressions provide more rich and accurate emotional data. Therefore, research has gradually shifted to more challenging dynamic facial video emotion recognition.
[0003] In the field of video, researchers have made significant progress in exploring and utilizing video temporal information. Some methods utilize the continuity between video frames to learn the features of time series. In addition, some studies have effectively improved the model's ability to capture complex activity sequences in videos by randomly sorting video frames and having the model predict the correct frame order. Pre-training techniques have become a common strategy in the field of deep learning. Pre-trained models based on large data have become a popular research direction due to their strong versatility and transfer learning capabilities. These large general-purpose pre-trained models are usually pre-trained on well-designed tasks, and the learned general-purpose features have good versatility and transferability, which can effectively be applied to various downstream emotion recognition tasks, demonstrating superior performance.
[0004] However, the training process of traditional pre-trained models often relies on large-scale labeled data, which is particularly problematic in the field of emotional data, as the labeling of emotional data is not only time-consuming and labor-intensive, but also subjective, resulting in some inconsistencies in the labeling results.
[0005] Current researchers have made some achievements in the field of visual emotion, but still face the following challenges: 1) Emotion is subjective, and different people may have different emotional understanding of the same video, so labeling emotional data needs to consider subjectivity and diversity, making labeling work relatively difficult and time-consuming. And although it is relatively easy to collect emotional video data, due to the high cost of labeling, the existing emotional data set is limited in size, making it difficult to meet the needs of deep learning and other models that require large-scale data for training.
[0006] 2) The number of samples of various emotion categories in the existing emotional data set is quite different, which may cause the model to be biased towards the category with more quantity during the training process, and ignore the category with less quantity, thereby affecting the generalization ability of the model to all emotion categories.
[0007] In summary, the current emotional data set is faced with the problems and challenges of labeling difficulty, limited data volume, uneven data distribution, complex emotional context, and label objectivity. How to improve the generalization of the model under such conditions is the purpose of the present application. SUMMARY
[0008] The purpose of the present application is to provide a visual emotion model pre-training method based on semi-supervised learning to overcome the problems of poor generalization performance and unsatisfactory recognition effect of existing methods.
[0009] To achieve the above task, the present application adopts the following technical solutions: The visual emotion model pre-training method based on semi-supervised learning comprises: Obtaining a video segment of a to-be-recognized emotion category, inputting it into a trained visual emotion model after preprocessing to obtain a prediction result of the emotion category corresponding to the video segment; wherein the training process of the visual emotion model comprises: First, initialize the model, use the labeled data set to preliminarily train the video encoder and the classifier; then enter the iterative training stage of semi-supervised continuous learning, in each training round, block and combine the unlabeled data set to construct the training data of the current round; for each video segment sample in the training data, use the encoder to extract features and enhance data to obtain the feature set of the current round; perform deep clustering and cluster label generation on the feature set, and finally perform purity screening of the clusters and label consistency verification to form a high-purity cluster subset; construct a joint total loss function based on the high-purity cluster subset to update the visual emotion model.
[0010] Further, the data set construction process of the visual emotion model is: The data set comprises a labeled data set and an unlabeled data set . The labeled data set contains a series of video samples , wherein is a labeled video segment of facial expression, is the corresponding artificial labeled true emotion label; The unlabeled data set contains only facial expression unlabeled video segments without corresponding emotion labels; The labeled video segments in the labeled data set and the unlabeled video segments in the unlabeled data set Preprocessing is performed, including: video frame extraction, face detection and alignment, and image cropping and normalization.
[0011] Further, model initialization is performed, and the video encoder and classifier are initially trained using a labeled dataset, including: labeled dataset The labeled video clips The input video encoder yields the feature representation. Then, the predicted label is obtained through a classifier. ; predict labels by calculation and video clips Corresponding real emotional tags The cross-entropy loss between the two is used to perform preliminary optimization of the parameters of the video encoder and classifier.
[0012] Furthermore, the unlabeled dataset is divided into blocks and combined to construct the training data for the current round, including: Unlabeled dataset Divided into Disjoint unlabeled data sub-blocks ; in the current number In each round, select an unlabeled data sub-block. and labeled datasets Combined to form the training data for the current round. .
[0013] Furthermore, for each video segment sample in the training data, feature extraction and data augmentation are performed using the encoder to obtain the feature set for the current round, including: For training data The labeled video clips Through the encoder of the current round Directly extract its features For training data Unlabeled video clips Two different random data augmentations were performed to obtain two augmented views. and And respectively through the encoder of the current round Extract its features and ; All for labeled video clips Unlabeled video clips Extracted features , and Together they constitute the feature set of the current round. .
[0014] Furthermore, deep clustering and cluster label generation are performed on the feature set, including: feature set Input the data into the K-means clustering module and set the number of clusters. , feature set All features are clustered into Cluster Then, for each cluster Assign a cluster label The allocation method adopts a voting mechanism: statistical clusters The distribution of pseudo-labels output by the current classifier for all features is used, and the pseudo-label with the highest frequency is taken as the cluster label for that cluster. .
[0015] Furthermore, cluster purity screening and label consistency verification are performed to form a high-purity cluster subset, including: Purity screening: Calculate purity for each cluster internal purity ;purity The pseudo-label defined as the feature within a cluster and the cluster label of that cluster. Matching proportions:
[0016] in, It is a cluster The number of features in It is a feature pseudo-tags It is an indicator function; only purity is preserved. Higher than the preset threshold The clusters form a filtered cluster set. ; Tag consistency check: for the filtered cluster set The reliability of each cluster is verified; the distribution of real sentiment labels corresponding to the labeled video segments within the cluster is statistically analyzed, and the most frequently occurring real sentiment label is found. Only when Cluster label of the cluster Only when there is consistency will the cluster be ultimately preserved, forming a high-purity cluster subset. .
[0017] Furthermore, a joint total loss function is constructed based on a subset of high-purity clusters, including: Joint total loss function It consists of three parts: Monitoring losses For high-purity cluster subsets The features contained correspond to labeled video clips. Calculate the cross-entropy loss between the true sentiment label and the cluster label; Comparative loss For high-purity cluster subsets The features contained in the unlabeled video clips Calculate the two features extracted by the encoder in the current round. and Cosine similarity loss between them; Continuous learning loss Introduce a projection network The projection network The current number Round encoder Extracted features Project back into the feature space of the previous round; calculate the projected features. Compared with the previous encoder Extracted features The mean square error between them; Joint total loss function Represented as:
[0018] in To balance the hyperparameters of different loss terms.
[0019] A terminal device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, it implements the visual emotion model pre-training method based on semi-supervised learning.
[0020] A computer-readable storage medium storing a computer program; when executed by a processor, the computer program implements the pre-training method for a visual emotion model based on semi-supervised learning.
[0021] Compared with the prior art, the present invention has the following technical features: This invention primarily designs a pseudo-labeling mechanism based on deep clustering and a block-based training mechanism that integrates continuous learning, proposing a visual sentiment pre-training model based on semi-supervised learning. The pseudo-labeling mechanism based on deep clustering reduces the impact of noisy data, making it easier for the model to learn key sentiment features and improving the accuracy of cluster labels. The block-based training mechanism that integrates continuous learning provides a progressively deeper process for feature learning and semantic understanding of unlabeled data, helping the model refine the features it has acquired and gradually expand its feature representation range to include more diverse and comprehensive visual sentiment features. Attached Figure Description
[0022] Figure 1 Structure diagram of visual emotion model in the method of the application. DETAILED DESCRIPTION
[0023] The semi-supervised learning technology provides an innovative perspective. The semi-supervised learning has obvious advantages in the case of insufficient labeled data. By combining a large amount of unlabeled data and a small amount of labeled data, the semi-supervised learning method widens the vision of model training, enhances the adaptability of the model to unknown data distribution, can fully utilize the implicit knowledge in the unlabeled data, significantly improves the generalization ability of the model, and realizes more accurate and more detailed recognition of the emotional state.
[0024] The application provides a visual emotion model pre-training method based on semi-supervised learning, which comprises the following steps: A video segment to be recognized in an emotional category is obtained, preprocessed and input into a trained visual emotion model to obtain a prediction result of the emotional category corresponding to the video segment; wherein the training process of the visual emotion model comprises the following steps: First, the model is initialized, and the video encoder and the classifier are preliminarily trained using a labeled data set; then, an iterative training phase of semi-supervised continuous learning is entered, and in each training round, the unlabeled data set is divided and combined to construct the training data of the current round; for each video segment sample in the training data, the encoder is used for feature extraction and data enhancement to obtain a feature set of the current round; depth clustering and cluster label generation are performed on the feature set, and finally, purity screening of the cluster and label consistency checking are performed to form a high-purity cluster subset; a joint total loss function is constructed based on the high-purity cluster subset to update the visual emotion model.
[0025] The specific implementation process of the application will be further described in detail below with reference to the accompanying drawings.
[0026] 1. Data set construction and preprocessing
[0027] Before the visual emotion model training starts, the data set required for training needs to be constructed first; the data set comprises a labeled data set and an unlabeled data set .
[0028] The labeled data set comprises a series of video samples , wherein is a labeled video segment of a facial expression, is a real emotional label (for example, one of the emotional categories of happiness, sadness, anger, etc.) artificially labeled corresponding thereto.
[0029] The unlabeled data set : Contains a large number of unlabeled video clips with only facial expressions However, there is no corresponding emotional label; usually, The data size is much larger than .
[0030] Before inputting data into a visual sentiment model, labeled datasets can be used. Unlabeled datasets The labeled video clips Unlabeled video clips Preprocessing includes: video frame extraction, which breaks down the video into a series of static image frames; face detection and alignment, which locates the face region in each frame and performs standardized alignment; and image cropping and normalization, which crops out the face region and scales it to a uniform size (e.g., 224x224 pixels).
[0031] The dataset is divided into a training set and a validation set.
[0032] 2. Model initialization.
[0033] Model training is an iterative process designed to teach the model the mapping relationship between videos and emotions.
[0034] Before starting semi-supervised training, use a labeled dataset. For video encoders The video encoder is initially trained with an initial classifier; the video encoder is a Transformer encoder.
[0035] Specifically, there will be labeled datasets The labeled video clips The input video encoder yields the feature representation. Then, the predicted label is obtained through a classifier. ; predict labels by calculation and video clips Corresponding real emotional tags The cross-entropy loss between the two is used to initially optimize the parameters of the video encoder and classifier to obtain a reasonable set of initial parameters, providing a good starting point for subsequent semi-supervised learning.
[0036] 3. Semi-supervised continuous learning.
[0037] After model initialization, the model enters the iterative training phase of semi-supervised continuous learning; this phase is divided into multiple training epochs, with each training epoch... In the middle, perform the following operations: (1) Data segmentation and combination.
[0038] Unlabeled dataset partitioned into non-intersecting unlabeled data sub-blocks ; in the current round, one unlabeled data sub-block is selected (where is a modulo operation), and combined with the labeled dataset to form the training data for the current round ; thus the training data contains both labeled video segments and unlabeled video segments ; this sub-blocking strategy enables the model to access new unlabeled data in each round, achieving progressive continual learning.
[0039] (2) Feature extraction and data augmentation.
[0040] For the labeled video segments in the training data , their features are directly extracted by the current round encoder ; For the unlabeled video segments in the training data , two different random data augmentations (e.g., random cropping, horizontal flipping, etc.) are performed on them to obtain two augmented views and , and their features and are extracted by the current round encoder , respectively.
[0041] All the extracted features , and for the labeled video segments , the unlabeled video segments form the feature set for the current round.
[0042] (3) Deep clustering and cluster label generation.
[0043] The feature set is input into the K-means clustering module, and the number of clusters is usually set to be much larger than the total number of emotion categories), and the feature set is clustered into clusters ( ); then, each cluster is assigned a cluster label The allocation method adopts a voting mechanism: statistical clusters The distribution of pseudo-labels output by the current classifier for all features is used, and the pseudo-label with the highest frequency is taken as the cluster label for that cluster. .
[0044] (4) Screening of high-purity clusters.
[0045] To ensure the quality of pseudo-tags, the generated... Each cluster undergoes a two-stage screening: Purity screening: First, calculate the purity of each cluster. internal purity ;purity The pseudo-label defined as the feature within a cluster and the cluster label of that cluster. The formula for calculating the matching ratio is:
[0046] in, It is a cluster The number of features in It is a feature pseudo-tags It is an indicator function (taking 1 if the condition is true, otherwise taking 0); only purity is preserved. Higher than the preset threshold The clusters form a filtered cluster set. .
[0047] Tag consistency check: for the filtered cluster set For each cluster, its reliability is further verified; the distribution of real sentiment labels corresponding to the labeled video clips within that cluster (i.e., the real sentiment labels of the labeled video clips) is statistically analyzed, and the real sentiment label with the highest frequency is found. Only when Cluster label of the cluster When consistent ( Only after this process is the cluster ultimately preserved, forming a high-purity cluster subset. .
[0048] This rigorous screening mechanism effectively filters out low-quality, noisy pseudo-labeled data.
[0049] 4. Joint total loss function and model update.
[0050] Based on high-purity cluster subsets Calculate a joint total loss function To update the parameters of the visual emotion model; joint total loss function It consists of three parts: Monitoring losses For high-purity cluster subsets The features contained correspond to labeled video clips. (Its true emotional label is) The cluster label of the cluster is ), calculate the cross-entropy loss between its true sentiment label and cluster label to utilize high-quality supervision signals.
[0051]
[0052] Comparative loss For high-purity cluster subsets The features contained in the unlabeled video clips Calculate its two features and The cosine similarity loss between the samples is used to bring the feature representations of different views of the same source sample closer together, enabling the model to learn more robust features.
[0053]
[0054] Continuous learning loss To mitigate the catastrophic forgetting phenomenon (forgetting old knowledge) that occurs when the model learns new data blocks, a projection network is introduced. The network will currently be the [number]th [number]. Round encoder Extracted features Project back into the feature space of the previous round; calculate the projected features Compared with the previous encoder Extracted features The mean squared error (MSE) between features is used to maintain the continuity of the feature representation.
[0055] in, This represents the training data for the current round. The first in A sample of video clips, which may be labeled video clips. Or unlabeled video clips .
[0056] The total loss function is the weighted sum of the three:
[0057] in To balance the hyperparameters of different loss terms, the backpropagation algorithm is used, based on the total loss... Update encoder Classifiers and projection networks The parameters.
[0058] Each training round is iteratively performed until a preset number of training rounds is reached or the performance of the visual emotion model converges on the validation set, and finally a trained visual emotion model with strong generalization ability is obtained.
[0059] 5. Application inference of the visual emotion model.
[0060] When the model training is completed, it can be deployed in practical applications for emotion recognition of unknown videos. The specific steps are as follows: Obtain the data to be tested: obtain a video segment to be identified for emotion classification; Data preprocessing: the same preprocessing operation as in the training phase is performed on the video segment; Emotion recognition: input the processed video segment into the trained video encoder and classifier of the visual emotion model; Output result: the classifier outputs the emotion classification prediction result (e.g. "happy") corresponding to the visual emotion model; this result can be used for intelligent feedback of human-computer interaction systems, driver state monitoring, student concentration analysis in online education, mental health assessment, and other application scenarios.
[0061] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A pre-training method for visual emotion models based on semi-supervised learning, characterized in that, include: The video clips to be identified in terms of emotion category are obtained, preprocessed, and then input into a trained visual emotion model to obtain the prediction results of the emotion category corresponding to the video clips. The training process of the visual emotion model includes: First, model initialization is performed using a labeled dataset to initially train the video encoder and classifier. Then, a semi-supervised continuous learning iterative training phase begins. In each training epoch, the unlabeled dataset is divided and combined to construct the training data for that epoch. For each video segment sample in the training data, the encoder is used for feature extraction and data augmentation to obtain the feature set for that epoch. Deep clustering and cluster label generation are then performed on the feature set. Finally, cluster purity is selected and label consistency is verified to form a high-purity cluster subset. Based on the high-purity cluster subset, a joint total loss function is constructed to update the visual emotion model.
2. The visual emotion model pre-training method based on semi-supervised learning according to claim 1, characterized in that, The data set construction process for the visual emotion model is as follows: The dataset includes labeled datasets. Unlabeled datasets Two parts; Labeled datasets : Contains a series of video samples ,in It is a tagged video clip with facial expressions. These are the corresponding manually labeled, genuine sentiment tags; Unlabeled datasets : Unlabeled video clips containing only facial expressions However, there is no corresponding emotional tag; For labeled datasets Unlabeled datasets The labeled video clips Unlabeled video clips Preprocessing is performed, including: video frame extraction, face detection and alignment, and image cropping and normalization.
3. The visual emotion model pre-training method based on semi-supervised learning according to claim 1, characterized in that, Model initialization is performed, using a labeled dataset to initially train the video encoder and classifier, including: labeled dataset The labeled video clips The input video encoder yields the feature representation. Then, the predicted label is obtained through a classifier. ; predict labels by calculation and video clips Corresponding real emotional tags The cross-entropy loss between the two is used to perform preliminary optimization of the parameters of the video encoder and classifier.
4. The visual emotion model pre-training method based on semi-supervised learning according to claim 1, characterized in that, The unlabeled dataset is divided into blocks and combined to construct the training data for the current round, including: Unlabeled dataset Divided into Disjoint unlabeled data sub-blocks ; in the current number In each round, select an unlabeled data sub-block. and labeled datasets Combined to form the training data for the current round. .
5. The visual emotion model pre-training method based on semi-supervised learning according to claim 1, characterized in that, For each video segment sample in the training data, feature extraction and data augmentation are performed using the encoder to obtain the feature set for the current round, including: For training data The labeled video clips Through the encoder of the current round Directly extract its features For training data Unlabeled video clips Two different random data augmentations were performed to obtain two augmented views. and And respectively through the encoder of the current round Extract its features and ; All for labeled video clips Unlabeled video clips Extracted features , and Together they constitute the feature set of the current round. .
6. The visual emotion model pre-training method based on semi-supervised learning according to claim 1, characterized in that, Deep clustering and cluster label generation are performed on the feature set, including: feature set Input the data into the K-means clustering module and set the number of clusters. , feature set All features are clustered into Cluster Then, for each cluster Assign a cluster label The allocation method adopts a voting mechanism: statistical clusters The distribution of pseudo-labels output by the current classifier for all features is used, and the pseudo-label with the highest frequency is taken as the cluster label for that cluster. .
7. The visual emotion model pre-training method based on semi-supervised learning according to claim 1, characterized in that, Cluster purity screening and label consistency verification are performed to form a high-purity cluster subset, including: Purity screening: Calculate purity for each cluster internal purity ;purity The pseudo-label defined as the feature within a cluster and the cluster label of that cluster. Matching proportions: in, It is a cluster The number of features in It is a feature pseudo-tags It is an indicator function; only purity is preserved. Higher than the preset threshold The clusters form a filtered cluster set. ; Tag consistency check: for the filtered cluster set The reliability of each cluster is verified; the distribution of real sentiment labels corresponding to the labeled video segments within the cluster is statistically analyzed, and the most frequently occurring real sentiment label is found. Only when Cluster label of the cluster Only when there is consistency will the cluster be ultimately preserved, forming a high-purity cluster subset. .
8. The method for pre-training a visual emotion model based on semi-supervised learning according to claim 1, characterized in that, The joint total loss function is constructed based on high-purity cluster subsets, including: Joint total loss function It consists of three parts: Monitoring losses For high-purity cluster subsets The features contained correspond to labeled video clips. Calculate the cross-entropy loss between the true sentiment label and the cluster label; Comparative loss For high-purity cluster subsets The features contained in the unlabeled video clips Calculate the two features extracted by the encoder in the current round. and Cosine similarity loss between them; Continuous learning loss Introduce a projection network The projection network The current number Round encoder Extracted features Project back into the feature space of the previous round; calculate the projected features. Compared with the previous encoder Extracted features The mean square error between them; Joint total loss function Represented as: in To balance the hyperparameters of different loss terms.
9. A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, When the processor executes a computer program, it implements the visual emotion model pre-training method based on semi-supervised learning as described in any one of claims 1-8.
10. A computer-readable storage medium storing a computer program; characterized in that, When the computer program is executed by the processor, it implements the visual emotion model pre-training method based on semi-supervised learning as described in any one of claims 1-8.