Student expression data semi-automatic labeling method and system based on multi-modal fusion

Abstract

This invention discloses a semi-automatic annotation method and system for student facial expression data based on multimodal fusion, comprising: S1 generating multidimensional expressions based on a semantic conditional diffusion model, and simultaneously outputting discrete expression classifications and continuous VAD emotion dimension labels through text prompts in an educational setting; S2 obtaining synthetic expression images and their corresponding 68 facial key point coordinates based on S1, performing classroom scene-adaptive style transfer, and using a key point-constrained adversarial generative network to preserve expression features and adapt to classroom lighting and viewing angle; S3 identifying the classification probability of discrete expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension, using hybrid uncertainty-driven active learning, and combining classification entropy and regression variance to select high-value samples; S4 performing identity decoupling and privacy anonymization on the selected high-value samples, reconstructing the face through a 3D deformation model and specifically blurring the eyebrow region to achieve anonymity protection. This invention can ensure privacy compliance, reduce annotation costs, and improve model accuracy and robustness.

Description

Technical Field

[0001] The invention relates to the fields of computer vision and intelligent education technology, specifically to a semi-automatic annotation method and system for student facial expression data based on multimodal fusion. Background Technology

[0002] With the accelerated development of smart classrooms, teachers urgently need to obtain classroom facial expression data in real time to understand students' emotional states and accurately adjust teaching strategies. However, existing methods for obtaining classroom facial expression data involve manually annotating discrete facial expressions and continuous VAD emotions frame by frame, which has the following problems:

[0003] 1. The student facial expression dataset is small in size and has an imbalanced class structure. In addition, the lighting, viewing angle and occlusion of real classrooms vary greatly, resulting in poor cross-domain performance of the model.

[0004] 2. Not only is manual operation time-consuming and labor-intensive, but it is also expensive;

[0005] 3. Overall blurring or DeepPrivacy anonymization can damage the texture of eyebrows and corners of the mouth. Public experiments show that the recognition rate of AU12 / AU6 decreases by 5-12%, and there is a risk of re-recognition.

[0006] 4. Difficult to meet classroom privacy compliance requirements.

[0007] In conclusion, a low-cost, high-quality, and cross-domain-usable semi-automatic annotation method for student facial expression data is needed to obtain facial expression data for classroom scenarios. Summary of the Invention

[0008] This invention proposes a semi-automatic annotation method and system for student facial expression data based on multimodal fusion, which solves the four major technical pain points of "data scarcity, expensive annotation, cross-domain drift, and privacy violation" faced by student facial expression data collection and annotation in classroom scenarios.

[0009] To achieve one of the above objectives, the present invention adopts the following technical solution:

[0010] A semi-automatic annotation method for student facial expression data based on multimodal fusion includes the following steps:

[0011] S1: Generate multi-dimensional expressions based on the semantic conditional diffusion model. Simultaneously output discrete expression classification and continuous VAD emotion dimension labels through text prompts in educational scenarios, and divide them into training set, validation set and test set;

[0012] S2: Based on the discrete expression classification and continuous VAD emotion dimension labels, obtain the synthesized expression image and its corresponding 68 facial key point coordinates, perform classroom scene adaptive style transfer, and use key point-constrained adversarial generative network to preserve expression features and adapt to classroom lighting and viewing angle.

[0013] S3: Utilize a classroom facial expression recognition model to identify the classification probability of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension, hybrid uncertainty-driven active learning, and combine classification entropy and regression variance to screen high-value samples;

[0014] S4: The high-value samples selected in S3 are subjected to privacy desensitization by decoupling their identities. The faces are reconstructed using a 3D deformation model, and the eyebrow area is specifically blurred to achieve anonymity protection.

[0015] Furthermore, S1: generating multi-dimensional expressions based on a semantic conditional diffusion model, and simultaneously outputting discrete expression classifications and continuous VAD emotion dimension labels through text prompts in educational scenarios; specifically including:

[0016] S101: The discrete facial expression classification labels, which include six categories: focused, confused, distracted, interactive questioning, fatigued, and neutral, are converted into 128-dimensional vectors through an embedding layer;

[0017] S102: Map the continuous VAD emotion vector containing the three dimensions of valence, arousal and dominance to a 128-dimensional vector through a fully connected layer;

[0018] S103: The 128-dimensional vectors of S101 and S102, together with the 512-dimensional field educational scene text prompts containing school age parameters and classroom environment parameters extracted by the multimodal pre-trained model CLIP text encoder, are input into the U-Net network and cross-attention fusion is performed in its 4th to 16th layers to construct an educational semantic conditional diffusion model.

[0019] S104: The probability distribution of discrete facial expression classification and the three-dimensional numerical values ​​of continuous VAD emotion vectors are output synchronously through a single forward propagation.

[0020] Furthermore, the educational scenario text prompts must include at least the following three elements:

[0021] Age group label: one of the following four categories: lower elementary school, upper elementary school, junior high school, and senior high school;

[0022] Classroom environmental parameters: a description of the combination of lighting type and obstructions;

[0023] Facial expression keywords: focus, confusion, distraction, interactive questioning, fatigue, neutral, one of the following six categories.

[0024] Furthermore, the formula for the cross-attention fusion is as follows:

[0025]

[0026] Where || denotes vector concatenation, and d=128 is the feature dimension. , , These are the key matrices corresponding to discrete labels, VAD vectors, and text prompts, respectively.

[0027] Further, step S2: Based on the discrete expression classification and continuous VAD emotion dimension labels, obtain the synthesized expression image and its corresponding 68 facial keypoint coordinates, perform classroom scene-adaptive style transfer, and use a keypoint-constrained adversarial generative network to preserve expression features and adapt to classroom lighting and viewing angle; specifically including:

[0028] S201: Obtain the synthesized facial expression image and its corresponding coordinates of 68 facial key points, wherein the key point coordinates include the muscle movement features of eyebrows, eyes, and mouth;

[0029] S202: Input the synthesized facial expression image into the generator and output a style-transferred image with classroom scene lighting, background and perspective features;

[0030] S203: The style-transferred image and the real classroom scene image are adversarially discriminated against by a discriminator to generate an adversarial loss;

[0031] S204: Calculate the Euclidean distance L2 of the 68 facial key points before and after migration to form the key point consistency loss, and calculate the mean-variance distance of the images before and after migration in the Lab color model to form the illumination consistency loss.

[0032] S205: By jointly optimizing adversarial loss, keypoint consistency loss, and illumination consistency loss, and automatically updating the weight ratio of the three based on the validation set and keypoint error at fixed iterations during the training process, facial muscle movements are kept true under classroom perspective and lighting conditions.

[0033] Furthermore, the joint optimization includes an adaptive weight adjustment module. Based on the Fraser initial distance (FID) and keypoint error of the validation set, the adaptive weight adjustment module dynamically adjusts the weight allocation of adversarial loss, keypoint consistency loss, and illumination consistency loss until the difference in the rate of decrease of the three types of losses remains within a preset convergence threshold range, thus ensuring the fidelity of facial muscles while adapting to different classroom lighting conditions.

[0034] Furthermore, S3: Utilizing a classroom facial expression recognition model to identify the classification probabilities of discrete facial expression categories and the three-dimensional predicted values ​​of the continuous VAD emotion dimension, hybrid uncertainty-driven active learning is employed, combining classification entropy and regression variance to screen high-value samples; specifically including:

[0035] S301: Utilize the trained classroom facial expression recognition model to perform forward inference on style-aligned images, and simultaneously output the classification probability distribution of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension.

[0036] S302: Calculate the classification entropy H of the classification probability distribution. This classification entropy H is an indicator that measures the uncertainty of the model's prediction of discrete facial expression categories. Calculate the variance of the three-dimensional predicted values ​​based on Monte Carlo sampling. The variance This is used to evaluate the stability of model predictions, and then to construct a joint uncertainty index. ,in, and Dynamic updates are performed using Bayesian optimization to adjust the classification entropy H and variance, respectively. The contribution ratio in the joint uncertainty index U, and satisfying =1, ;

[0037] S303: Sort by joint uncertainty index U in descending order and select the top... The samples were used as samples to be manually reviewed;

[0038] The former After manual verification, the samples are immediately added to the training set, triggering the next round of Bayesian optimization to re-estimate. , This forms a continuous feedback loop.

[0039] Furthermore, the Monte Carlo sampling is achieved by enabling the model's Dropout layer, requiring no additional network parameters.

[0040] Furthermore, step S4 involves: decoupling the identity of the high-value samples selected in step S3 to achieve privacy anonymity protection by reconstructing the face using a 3D deformation model and specifically blurring the eyebrow area; specifically including:

[0041] S401: Parametric decomposition of the input face image is performed using the 3D deformation model 3DMM to generate identity coefficients, expression coefficients, and texture coefficients;

[0042] S402: After setting the identity coefficient to zero, reconstruct the face to obtain an anonymized base image with neutral geometric shape;

[0043] S403: Apply Gaussian blur to the eyebrow region of the anonymized base image to generate the final anonymized face image;

[0044] S404: Perform privacy compliance verification on the final anonymized face image to ensure that it meets the preset cross-identity similarity threshold and intra-class similarity variance threshold.

[0045] To achieve the second objective mentioned above, the present invention adopts the following technical solution:

[0046] A semi-automatic annotation system for student facial expression data based on multimodal fusion includes a data acquisition module, a data generation and transfer module, and an anonymization and annotation module.

[0047] The data acquisition module is used to acquire student facial images;

[0048] The data generation and migration module includes a synthetic data generation unit and a domain adaptation unit.

[0049] The synthetic data generation unit is used to generate multi-dimensional expressions based on the semantic conditional diffusion model, and synchronously output discrete expression classification and continuous VAD emotion dimension labels through educational scenario text prompts, and batch generate labeled synthetic expression images and their corresponding 68 facial key point coordinates.

[0050] The domain adaptation unit is used to transfer the style of the synthesized facial expression image to the real classroom environment, perform classroom scene adaptive style transfer, and use a key point-constrained adversarial generative network to preserve facial expression features and adapt to classroom lighting and viewing angle to generate a classroom facial expression recognition model.

[0051] The anonymization and annotation module includes an active learning unit and a privacy desensitization unit.

[0052] The active learning unit is used to identify the classification probability of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension using the classroom facial expression recognition model. It combines uncertainty-driven active learning and uses classification entropy and regression variance to screen high-value samples.

[0053] The privacy desensitization unit is used to decouple the identity of the input face image using a 3D deformation model, set the identity features to zero, and retain only the expression and texture features. Then, it performs local blurring only in the eyebrow area to generate an anonymized face image, thereby achieving anonymity protection.

[0054] The beneficial effects of this invention are:

[0055] This invention proposes a semi-automatic annotation method for facial expression data that integrates multimodal generation, cross-domain transfer, and active learning. It generates synthetic facial expression images with multidimensional labels through a conditional diffusion model, then adapts them to the real classroom environment using keypoint-constrained style transfer, and combines hybrid uncertainty active learning and local anonymization to significantly reduce manual annotation and improve model generalization ability. This provides a safe and efficient data foundation for large-scale classroom emotion analysis. Through front-end multimodal fusion, it overcomes four major challenges: scarcity of classroom facial expression data, expensive annotation, cross-domain drift, and privacy violation. Overall, this invention uses four technologies—educational semantic-driven diffusion-based data augmentation, keypoint-constrained cross-domain style transfer, entropy-variance coupled active learning sampling, and 3DMM local anonymization—to collaboratively construct an end-to-end closed loop. This systematically alleviates the structural contradictions of classroom facial expression data in terms of scarcity, high annotation costs, inter-domain distribution drift, and privacy compliance, providing a feasible data-algorithm integrated paradigm for scalable emotion computing in smart education scenarios.

[0056] The system of this invention generates synthetic facial expression images with multi-dimensional labels (discrete classification + continuous VAD) based on an educational semantic conditional diffusion model through a synthetic data generation unit. A domain adaptation unit utilizes a keypoint-constrained adversarial generative network to transfer the style of the synthetic data to a real classroom scene. An active learning unit uses mixed uncertainty evaluation to screen high-value labeled samples. A privacy desensitization unit employs a 3D deformation model to achieve identity decoupling and local eyebrow blurring.

[0057] This invention systematically overcomes the four major bottlenecks in classroom facial expression analysis—namely, data scarcity, expensive annotation, significant domain differences, and difficulty in protecting privacy—from the data source, providing a reliable, compliant, and scalable data foundation solution for all upper-level smart education applications. For the first time, it systematically integrates generative artificial intelligence, active learning, and verifiable privacy computing to construct a dedicated data infrastructure for educational scenarios, fundamentally empowering the possibilities and reliability of all upper-level applications. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of a specific embodiment 1 of the present invention;

[0060] Figure 2 This is a principle block diagram of a specific embodiment 2 of the present invention. Detailed Implementation

[0061] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Specific Implementation Example 1:

[0063] See Figure 1 As shown, a semi-automatic annotation method for student facial expression data based on multimodal fusion is proposed. This invention is designed for classroom scenarios. Two high-definition network cameras, one visible light camera and one infrared camera, are installed at a 15° downward angle on the classroom ceiling. Frame-level alignment is achieved through the IEEE 1588 time synchronization protocol to ensure data accuracy regardless of day or night. The camera output is H.264 encoded and sent to an edge computing node. The node has a built-in lightweight face detection model and uses a single-stage detector to quickly locate face regions. The single-stage detector refers to a 2D face detector (such as lightweight SSD or YOLO). The acquired face information is used as input images for classroom facial expression recognition in step S3 for real-world inference, and as privacy-desensitized input images for identity decoupling processing in step S4.

[0064] Includes the following steps:

[0065] Step S1: Generate multi-dimensional expressions based on the semantic conditional diffusion model, construct an educational semantic conditional diffusion model, and output discrete expression classification and continuous VAD emotion dimension labels simultaneously through educational scenario text prompts, and divide them into training set, validation set and test set;

[0066] This educational semantic conditional diffusion model is a customized improvement on existing semantic conditional diffusion models for educational scenarios, such as Stable Diffusion. Existing conditional diffusion models can only generate images based on text prompts, cannot output multi-dimensional labels simultaneously, and lack specific adaptation to educational scenarios (age group, classroom lighting, occlusion, etc.).

[0067] This educational semantic conditional diffusion model employs three key features: First, it uses a cross-attention fusion mechanism to simultaneously output discrete facial expression classification probability distributions and continuous three-dimensional emotional dimension values ​​(VAD) while generating images, eliminating the need for secondary annotation. Second, it designs text prompt templates containing three elements: school age group, classroom environment parameters, and facial expression keywords. These text prompts can flexibly embed scene information such as school age group, lighting type, and occlusions, ensuring high consistency between the generated data and real classroom settings. This significantly alleviates class imbalance and domain drift, reducing subsequent transfer and training costs. Third, the generated synthetic facial expression images include the coordinates of 68 facial key points, providing input for subsequent style transfer with key point constraints.

[0068] This educational semantic conditional diffusion model utilizes conditional diffusion to directly generate synthetic data with multi-dimensional labels (discrete category + continuous emotion dimension) based on educational semantic cues (such as "junior high school student, under natural light, confused expression"). This fundamentally solves the problems of data scarcity and category imbalance, transforming data acquisition from "passive collection" to "active generation".

[0069] Specifically, it includes:

[0070] Step S101: The discrete facial expression classification labels, including six categories—focused, confused, distracted, interactive questioning, fatigued, and neutral—are converted into 128-dimensional vectors through an embedding layer. In this specific embodiment: the six discrete labels of "focused, confused, distracted, interactive questioning, fatigued, and neutral" are first converted into 6-dimensional one-hot vectors, and then linearly mapped into 128-dimensional dense vectors through the embedding layer. This compresses the high-dimensional sparse category information into a semantically continuous space, reducing the number of parameters while preserving the similarity relationship between categories. This allows the subsequent diffusion model to process both discrete facial expressions and continuous emotion vectors within the same latent space, improving generation consistency and training efficiency.

[0071] Step S102: The continuous emotion three-dimensional model VAD, which includes valence, arousal, and dominance, is mapped to a 128-dimensional vector through a fully connected layer. In this specific embodiment, the original VAD emotion three-dimensional model, with values ​​ranging from [-1, 1], is mapped to a 128-dimensional dense space through a fully connected network. This mapping projects continuous scalars from a low-dimensional Euclidean space to a high-dimensional semantic space, amplifying the nonlinear coupling between dimensions and compressing the sensitivity of numerical fluctuations to the model. This allows the diffusion network to deeply interact with discrete expression embeddings within the same channel. The 128-dimensional vector has stronger expressive power, can capture subtle emotional differences, and maintains controllable parameter counts, providing a consistent and efficient input format for subsequent cross-attention fusion.

[0072] Step S103: The two types of 128-dimensional vectors from Step S101 and Step S102, along with the 512-dimensional educational scene text prompts containing age group parameters and classroom environment parameters extracted by the text encoder of the multimodal pre-trained model CLIP, are input into the U-Net network and cross-attention fusion is performed in its 4th to 16th layers to construct an educational semantic conditional diffusion model.

[0073] In this specific embodiment: Image generation based on the diffusion model and multimodal feature fusion are achieved by encoding text prompts containing semantic information about school age, classroom environment, and teaching context into a 512-dimensional feature vector through the text encoder of the multimodal pre-trained model CLIP. These three types of features are then input into the U-Net network and fused through cross-attention in specific layers 4 to 16. The U-Net, as a denoising network for the conditional diffusion model, is used for image generation. The cross-attention fusion mechanism involves interacting the text features extracted by the CLIP text encoder with the image features in the intermediate layers of the U-Net, thereby controlling the semantic content of the generated image, such as school age, classroom environment, and facial expression. The final output is a synthetic facial image with multi-dimensional labels and the coordinates of 68 facial key points. First, by using a query-key-value attention mechanism, dynamic calibration of facial expression semantics and emotional features under scene context conditions is achieved. Second, the phased progressive fusion strategy of layers 4-16 avoids information overload in the early layers while ensuring full interaction of deep features. Finally, by standardizing the feature dimensions and uniformly reducing them to 128 dimensions, the spatial compatibility of features of different modalities is significantly improved.

[0074] The educational scenario text prompts should include at least the following three elements:

[0075] Age group label: any one of the following four categories: lower elementary school, upper elementary school, junior high school, or senior high school;

[0076] Classroom environment parameters: a description of the combination of lighting type (natural light / fluorescent lamp / backlight) and obstructions (books / glasses / arms);

[0077] Facial expression keywords: any one of the following six categories: focused, confused, distracted, interactive questioning, tired, or neutral.

[0078] The formula for the cross-attention fusion is:

[0079]

[0080] Where || denotes vector concatenation, and d=128 is the feature dimension. , , These are the key matrices corresponding to discrete labels, VAD vectors, and text prompts, respectively.

[0081] Step S104: Simultaneously output the probability distribution of discrete facial expression classification and the three-dimensional values ​​of the continuous VAD emotion vector through a single forward propagation. In this specific embodiment: Utilizing the single forward propagation of the U-Net network, two key pieces of information can be obtained simultaneously: the probability distribution of discrete facial expression classification and the three-dimensional values ​​of the continuous VAD emotion vector. The probability distribution of discrete facial expression classification represents the network's predicted probability of facial expression categories in the image, including six categories: focused, confused, distracted, interactive questioning, fatigued, and neutral. The three-dimensional values ​​of the continuous VAD emotion vector cover three dimensions: valence, arousal, and dominance. This design ensures that the generated image not only visually matches the input educational scenario text prompts but also has accurate labels in the emotion dimension. Such output greatly facilitates subsequent facial expression analysis and emotion understanding tasks, directly providing the rich annotation information needed for training and evaluating classroom facial expression recognition models. Furthermore, this synchronous output method improves data processing efficiency and reduces the time and resources required for separately annotating various types of information, thereby significantly reducing costs while ensuring data quality.

[0082] In this specific embodiment, the following dataset partitioning and processing strategies were adopted:

[0083] Training set: Select most of the data from the dataset as the training set to train the conditional diffusion model and style transfer network. Ensure that the training set contains samples of various expression categories to cover six types of expressions: focused, confused, distracted, interactive questioning, fatigue, and neutral, as well as different classroom environmental conditions, such as different lighting and occlusion conditions. The training set accounts for 70% of the total data.

[0084] Validation set: A small portion of the dataset is further selected as the validation set to evaluate model performance and adjust model parameters, such as weight ratio and learning rate, during training. The validation set accounts for 20% of the dataset.

[0085] Test set: The remaining data will be used as the test set to evaluate the final performance of the model. The test set should include some expression types or special cases that have not appeared in the training and validation sets to test the model's generalization ability. The test set should account for 10% of the total data.

[0086] To improve the model's generalization performance, data augmentation strategies were introduced during the training phase, such as applying random cropping, rotation, and brightness adjustments to the training samples to increase data diversity. Model parameter optimization was achieved through backpropagation, using the training dataset for iterative learning. After training, an independent test set was used to evaluate the model's final performance. Key performance indicators such as accuracy and recall were calculated to objectively assess the model's overall effectiveness.

[0087] Step S2: Perform classroom scene-adaptive style transfer on the discrete facial expression classification and continuous VAD emotion dimension labels obtained in S1. Employ a keypoint-constrained adversarial generative network to preserve facial expression features and adapt to classroom lighting and viewing angle, generating a classroom facial expression recognition model. Specifically, this includes:

[0088] Step S201: Obtain the corresponding synthetic expression image and its corresponding 68 facial key point coordinates from the discrete expression classification and continuous VAD emotion dimension labels in step S1. The facial key point coordinates include the muscle movement features of eyebrows, eyes, and mouth.

[0089] In this specific embodiment: obtaining the synthesized facial expression image and its corresponding coordinates of 68 facial key points is a crucial step in achieving accurate facial expression analysis, such as... Figure 2 As shown, the 68 facial keypoints are distributed as follows: 17 points for the facial contour (from the forehead around the chin to the other side of the forehead); 5 points each for the left and right eyebrows, totaling 10 points (eyebrow initiation, brow peak, eyebrow tail); 6 points each for the left and right eyes, totaling 12 points (inner and outer corners of the eyes, upper and lower eyelid contours); 9 points for the nose (bridge of the nose, nasal alae, tip of the nose); and 20 points for the mouth (outer lip contour, inner lip contour, left and right corners of the mouth). These facial keypoints, including feature points around the eyebrows, eyes, and mouth, are crucial for capturing and quantifying facial expressions. By accurately locating these facial keypoints, the movements of facial muscles can be described in detail, such as eyebrow raising, eye opening and closing, and the upward or downward tilting of the corners of the mouth. This information not only helps in understanding subtle changes in expressions but is also essential for subsequent style transfer and expression recognition tasks. For example, keypoints for the eyebrows and corners of the mouth are particularly important for recognizing expressions such as confusion or happiness. Therefore, this step ensures that the generated dataset provides high-quality annotation information for the model, thereby improving the accuracy and reliability of expression recognition. A facial landmark detector is invoked to accurately locate key positions such as eyebrows, corners of the eyes, and corners of the mouth within a certain time frame; in this specific embodiment, this is accomplished within 5 milliseconds. The final output is a well-structured sequence of key points for a single person, which serves as the raw input data for subsequent processes.

[0090] Step S202: Input the synthesized facial expression image into the generator and output a style-transferred image with classroom scene lighting, background and perspective features;

[0091] In this specific embodiment, the process of inputting the synthesized facial expression image into the generator involves using a style transfer network designed to style-transfer the synthesized image to match the features of a real classroom scene. This process includes adjusting lighting conditions to match natural or artificial lighting within the classroom, modifying the background to reflect the classroom's interior decoration and layout, and applying perspective transformations to simulate facial expressions from different viewpoints. The goal of style transfer is to generate an image that visually matches the real classroom scene while preserving the facial expression features of the original synthesized image. This processed image will be used in subsequent annotation and analysis steps to ensure the usability and accuracy of the data.

[0092] Step S203: Perform adversarial discrimination between the style-transferred image and the real classroom scene image using a discriminator to generate adversarial loss;

[0093] Specifically, it includes:

[0094] The discriminator's role is to distinguish between style-transfer processed images and real classroom scene images, thereby generating adversarial loss. Specifically, the discriminator is a trained neural network that attempts to identify which images are real and which are generated through style transfer. Through this adversarial training, the style transfer network is forced to generate more realistic images that are difficult for the discriminator to recognize.

[0095] Adversarial loss is the loss function output by the discriminator, typically employing binary cross-entropy loss. This loss function measures the accuracy of the discriminator in distinguishing between real and generated images. The goal of style transfer networks is to minimize adversarial loss, i.e., to generate images that can "fool" the discriminator, making it unable to distinguish between generated and real images. In this way, style transfer networks can learn the complex lighting, background, and perspective features of real classroom scenes and apply these features to synthesized facial expression images.

[0096] Step S204: Calculate the Euclidean distance L2 of the 68 facial keypoints before and after style transfer. This distance measures the positional shift of the same facial keypoint before and after style transfer, forming a keypoint consistency loss. Also, calculate the mean-variance distance of the images in the Lab color model before and after style transfer to form an illumination consistency loss. In this specific embodiment: to ensure the consistency of facial keypoint positions during style transfer, the Euclidean distance L2 of the 68 facial keypoints in the images before and after style transfer is calculated to form a keypoint consistency loss. Let the coordinates of the keypoint before transfer be (x1, y1), and the corresponding coordinates after transfer be (x2, y2). Then, the formula for calculating the Euclidean distance L2 is:

[0097]

[0098] The summation or averaging of the Euclidean distances L2 of all 68 keypoints constitutes the keypoint consistency loss. This loss constrains the style transfer network to ensure that the positions of muscle motion features such as eyebrows, eyes, and mouth do not shift while changing the image style (lighting, background, viewpoint), thus preserving the semantic integrity of expressions. Because Euclidean distance is rotationally invariant to shifts in all directions, it is more suitable for measuring the actual displacement of keypoints in the two-dimensional image plane.

[0099] The introduction of this keypoint consistency loss aims to preserve key facial feature points, such as those around the eyebrows, eyes, and mouth, during image style transfer, preventing displacement or deformation and thus ensuring the naturalness and authenticity of the expression.

[0100] Furthermore, to maintain consistency in lighting conditions between the transferred image and the real classroom scene image, this method further calculates the mean-variance distance between the images before and after transfer using the Lab color model, forming the lighting consistency loss. Utilizing the Lab color model allows for more effective separation of color and brightness information, enabling the lighting consistency loss to be specifically optimized for lighting variations. By minimizing this mean-variance distance, it is ensured that the style-transferred image matches the lighting conditions of the real classroom scene, thereby improving the realism and usability of the final image.

[0101] By using a generative adversarial network constrained by keypoints, the style of synthetic data can be transferred to a real classroom scene. At the same time, the consistency constraint of facial keypoints ensures that the muscle movement features of facial expressions are not destroyed. This makes the generated data both realistic and usable, and retains accurate facial semantics, such as eyebrows and corners of the mouth.

[0102] Step S205: By jointly optimizing the adversarial loss, the keypoint consistency loss, and the illumination consistency loss, and during this training process, automatically updating the weight ratios of the three based on the Frechet Inception Distance (FID) and facial keypoint localization error every fixed number of iterations, the facial muscle movements remain faithful under classroom viewing angles and lighting conditions. The adversarial loss evaluates the difference between the generated image and the real image through a discriminator, the keypoint consistency loss ensures that the positions of facial keypoints in the images before and after transfer remain consistent, and the illumination consistency loss measures and adjusts the illumination difference between the transferred image and the real image in the Lab color model, thus generating a classroom facial expression recognition model.

[0103] In this specific embodiment: During training, after a set number of iterations, the system automatically adjusts the weight ratios of the three loss functions based on the FID of the style-transferred image and the facial landmark localization error calculated on the validation set. This dynamic weight adjustment mechanism ensures that the relative importance of different loss terms is optimized in real time during training, thereby effectively maintaining the detail fidelity of facial muscle movements under specific classroom viewing angles and lighting conditions.

[0104] The joint optimization includes an adaptive weight adjustment module, which achieves dynamic weight control by constructing a multi-objective optimization framework:

[0105] First, based on the style consistency index (FID) and expression fidelity index (keypoint error) calculated in real time on the validation set, the gradient descent method is used to dynamically adjust the weight distribution of illumination loss, keypoint loss, and style loss, so that the difference in the descent rate of the three types of losses is maintained at a preset convergence threshold. , ;

[0106] Simultaneously, a dynamic proportional model of illumination loss weight and keypoint loss weight is constructed, and a proportional constraint coefficient is introduced. , The forced weighting ratio of the two types of losses must satisfy When a weight ratio exceeding the limit is detected, a gradient pruning method is used to correct the weight ratio to the legal range.

[0107] Finally, set the global weight threshold. , The updated weight values ​​are validated in real time. When any weight... or When this occurs, a weight redistribution process is triggered, forcing all weights to meet the requirements. And the normalization condition.

[0108] Step S3: Use the classroom facial expression recognition model to identify the classification probability of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension. Hybrid uncertainty-driven active learning is used, and high-value samples are selected by combining classification entropy and regression variance, which reduces the amount of manual annotation.

[0109] Specifically, it includes:

[0110] Step S301: Use the trained classroom expression recognition model to perform forward inference on the style-aligned image, and simultaneously output the classification probability distribution of discrete expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension.

[0111] In this specific embodiment, two sets of output data are generated simultaneously: the first set is the classification probability distribution of discrete facial expression categories, covering the probability values ​​of six facial expression states such as focus, confusion, inattentiveness, interactive questioning, fatigue, and neutrality; the second set is the three-dimensional predicted values ​​of the continuous VAD emotion dimension, namely, the quantitative estimates of valence, arousal, and dominance. The key to this process is that the classroom facial expression recognition model can not only identify the facial expression categories in images, but also further quantify the emotional intensity and tendency contained in the expressions, providing richer and more detailed information for subsequent data analysis and behavioral understanding.

[0112] Step S302: Calculate the classification entropy H of the classification probability distribution. This classification entropy H is an indicator that measures the uncertainty of the model's prediction of discrete facial expression categories, and is based on Monte Carlo sampling. The regression variance of the three-dimensional predicted values ​​was calculated 0 times. The regression variance This is used to evaluate the stability of model predictions, and then to construct a joint uncertainty index. ,in, and Dynamic updates are performed using Bayesian optimization to adjust the classification entropy H and variance, respectively. The contribution ratio in the joint uncertainty index U is used to balance the impact of classification uncertainty and regression instability on sample selection. That is, α and β are the weighting coefficients of classification entropy H and regression variance σ² in the joint uncertainty U, respectively.

[0113] In this specific embodiment: First, the trained classroom expression recognition model is used to perform forward inference on the style-aligned image to obtain the predicted probability distribution of each expression category;

[0114] Then, the classification entropy H is calculated based on this probability distribution. Classification entropy H is a metric that measures the uncertainty of the model's prediction of discrete facial expression categories and is used to quantify the purity of the probability distribution output by the model. The higher the classification entropy, the more uncertain the model's prediction of a certain facial expression category.

[0115] The Monte Carlo sampling method is used, and each image is sampled T times ( 0) Forward inference to obtain a set of predicted values ​​for expression categories and continuous VAD sentiment dimensions; calculate the regression variance of these predicted values. Regression variance Variance is used to evaluate the stability of a model's predictions, i.e., the degree of fluctuation in the predicted values. Smaller variance indicates that the model's predictions are more stable, while larger variance indicates that the model's predictions of sentiment values ​​fluctuate more when predicting the same image multiple times, i.e., the prediction stability is lower.

[0116] The classification entropy H and regression variance Combined, construct a joint uncertainty index This metric combines classification uncertainty and prediction stability to measure the overall uncertainty of a sample, guiding the selection of high-value samples. It is dynamically updated through Bayesian optimization. and The value is used to find the optimal weight ratio, such that... =1, and This dynamic adjustment mechanism allows the model to automatically adjust its focus based on different expression categories and prediction stability, thereby more effectively guiding the selection of high-value samples. This design ensures that the active learning module can focus on both expression categories that the model cannot "distinguish" (high entropy) and emotional dimensions that the model cannot "measure" (high variance), thus filtering out the most informative manually reviewed samples.

[0117] Step S303: Sort by joint uncertainty index U in descending order and select the top... The samples were used as samples to be manually reviewed, among which ;

[0118] In this specific embodiment: all samples are sorted according to the joint uncertainty index U, and the top... ( The samples with the highest uncertainty are used as samples to be manually reviewed. These samples have the highest uncertainty and are therefore most likely to contain key information, requiring further manual verification. The results of the manually reviewed samples will be immediately added to the training set and trigger the next round of Bayesian optimization to re-estimate. This forms a continuous feedback loop, further improving the model's generalization ability and annotation efficiency.

[0119] For the transferred data, it is not necessary to manually label all of them. Design a joint uncertainty index that integrates classification entropy and regression variance, so that the classroom expression recognition model can automatically select the most "uncertain" and information-rich samples (about 5%-20%), and only these will be handed over to manual review.

[0120] Step S4: The high-value samples selected in S3 undergo identity decoupling for privacy anonymization. The face is reconstructed using a 3D deformation model, with specific blurring of the eyebrow region to achieve K-anonymity protection. The 3D deformation model (3DMM) identity decoupling technology is introduced into educational scenarios. By setting facial identity features to zero and specifically blurring only key identity regions such as the eyebrows, personal identity information is removed while maximizing the preservation of facial expression details. This ensures the generated data meets privacy compliance requirements such as K-anonymity, making it a preferred implementation method. Alternatively, specific blurring of key identity regions such as the eyes, nose, mouth contours, and facial contours can also be used. The benefits of identity decoupling for privacy anonymization are as follows:

[0121] 1. It directly addresses the two major pain points raised: "privacy violation" and "compliance difficulties";

[0122] 2. The proposed solution of "preserving facial expression details while protecting privacy" is a solution that has not been achieved in existing technologies, constituting an important technical feature that distinguishes this technology from existing technologies.

[0123] Specifically, it includes:

[0124] Step S401: Using the high-value sample images output after active learning in step S3, i.e. face images, the input face images are parametrically decomposed using a 3D deformation model to generate identity coefficients, expression coefficients, and texture coefficients.

[0125] In this specific embodiment: First, a 3DMM deformation model is selected or constructed. The 3DMM uses an existing 3D facial statistical model, which can describe the geometry and appearance of a face. Then, the face image to be processed is input into the 3DMM model. Through the model's parametric expression, three sets of coefficients are obtained: identity coefficient, expression coefficient, and texture coefficient. The identity coefficient captures stable and unchanging facial features between individuals; the expression coefficient reflects temporary changes caused by muscle movement; and the texture coefficient describes skin details such as spots and wrinkles.

[0126] Step S402: After setting the identity coefficient to zero, reconstruct the face to obtain an anonymized base image with neutral geometric shape;

[0127] In this specific embodiment: the identity coefficient is set to zero, thereby eliminating individual identity information contained in the image. Based on this, the face is reconstructed using the remaining expression coefficient and texture coefficient to generate an anonymized base image with a neutral geometric shape.

[0128] Step S403: Apply Gaussian blur to the eyebrow region of the anonymized base image to generate the final anonymized face image;

[0129] In this specific embodiment: First, the eyebrow region in the anonymized base image is identified and located. This region typically contains key facial feature points, such as the brow ridge, the inner corner of the eyebrow, and the outer corner of the eyebrow. Next, a Gaussian blur algorithm is applied to this region, where the standard deviation of the blur kernel is... The setting is within the range of [3,5] pixels to ensure that identity information is effectively concealed while minimizing the impact on the recognition of facial features.

[0130] Step S404: Perform privacy compliance verification on the final anonymized face image to ensure it meets the preset cross-identity similarity threshold and intra-class similarity variance threshold. "Intra-class" refers to the same student individual. Intra-class similarity variance measures the fluctuation in feature similarity between multiple anonymized face images of the same identity. An intra-class similarity variance greater than the preset threshold of 0.15 ensures sufficient feature differences between different images of the same person, preventing excessive stability in anonymization that could lead to traceability. This metric, together with the cross-identity similarity threshold, constitutes a dual verification standard for privacy compliance.

[0131] In this specific embodiment: First, the FaceNet deep learning-based face recognition model is used to extract image features, and the maximum similarity between the features and different identity sample libraries is verified to be less than a threshold of 0.35. This identity sample library is a dynamically constructed control sample library during the verification process. Second, the intra-class feature variance is calculated for 20 anonymized images generated for the same identity, and verified to be greater than a preset threshold of 0.15. In specific implementation, a control sample library of 200 identities is automatically constructed, containing 200 individual images that do not overlap with the identity of the image to be verified. The image sources can be existing anonymized images or publicly available face datasets, such as the authoritative publicly available evaluation dataset for face recognition in the unconstrained environment Labeled Faces in the Wild (LFW), established by the University of Massachusetts. The similarity matrix is ​​calculated using a batch processing method. When any verification fails, an enhanced anonymization module is triggered to add Gaussian noise with a standard deviation of 0.1 to the eyebrow region until both threshold requirements are met simultaneously. Images that pass verification are then digitally watermarked and their processing is logged, ensuring compliance with the anonymity standard of K≥20. Here, K represents that each individual is indistinguishable from at least K-1 other individuals. K≥20 means that after anonymization, each student is indistinguishable from at least 19 other students in the feature space, thus meeting privacy compliance requirements. This standard is achieved through dual verification using a cross-identity similarity threshold of 0.35 and an intra-class variance threshold of 0.15. The sample library size of 200 identities also supports the setting of the K value. If the intra-class feature variance is greater than the threshold of 0.15, the verification passes; if it is lower, there is a risk of feature solidification leading to traceable identities. If the cross-identity similarity is below the threshold of 0.35, the verification is successful; if it is above the threshold, there is a risk of re-identification. The cross-identity similarity threshold of 0.35 and the intra-class variance threshold of 0.15 are preferred values ​​in this embodiment, but they are not fixed. In actual applications, they can be flexibly adjusted according to privacy compliance requirements (such as the K value in K-anonymity and the security level of the application scenario).

[0132] This embodiment fully demonstrates that the present invention can achieve accurate labeling of facial expression data in educational scenarios through its core multimodal feature fusion and adaptive optimization mechanism, effectively solving the core pain point of traditional methods that rely on a single modality and are difficult to balance classification and regression tasks. Specific Implementation Example 2:

[0134] See Figure 3 As shown, a semi-automatic annotation system for student facial expression data based on multimodal fusion includes a data acquisition module 1, a data generation and transfer module 2, and an anonymization and annotation module 3.

[0135] The data acquisition module 1 is used to acquire student facial images in real time and continuously from classroom surveillance video. In implementation, two high-definition network cameras, one visible light camera and one infrared camera, are installed at a 15° downward angle on the classroom ceiling. Frame-level alignment is achieved through the IEEE 1588 time synchronization protocol to ensure data accuracy regardless of day or night. The camera outputs are H.264 encoded and sent to an edge computing node. The node has a built-in lightweight face detection model that uses a single-stage detector to quickly locate the face region and further calls a 68-point keypoint extractor to accurately locate key positions such as eyebrows, corners of the eyes, and corners of the mouth within 5ms. Finally, it outputs a facial image and corresponding keypoint sequence as the raw input for subsequent modules.

[0136] The data generation and migration module 2 includes a synthetic data generation unit and a domain adaptation unit.

[0137] The synthetic data generation unit is used to generate multi-dimensional expressions based on a semantic conditional diffusion model. It simultaneously outputs discrete expression classifications and continuous VAD (Voice over Expression) emotion dimension labels through educational scenario text prompts, batch-generating labeled synthetic expression images and their corresponding 68 facial key point coordinates. This enables batch generation of labeled expression images even in scenarios lacking real labels. Specifically, the educational semantic conditional diffusion model is first invoked. This model uses four types of prompts—age group, lighting type, occlusion, and expression keywords—as conditions to output synthetic expression images simultaneously possessing six discrete expression categories and a continuous emotion dimension.

[0138] The domain adaptation unit is used to transfer the style of the synthesized facial expression image to the real classroom environment, perform classroom scene adaptive style transfer, and use a keypoint-constrained adversarial generative network to preserve facial expression features and adapt to classroom lighting and viewing angle to generate a classroom facial expression recognition model. By performing style transfer on the synthesized facial expression image, it makes it consistent with the real scene in terms of classroom lighting, background texture and perspective angle. At the same time, the keypoint consistency constraint ensures that the key muscle textures such as eyebrows and corners of the mouth are not destroyed, thereby achieving cross-domain adaptation and fidelity.

[0139] The anonymization and annotation module 3 includes an active learning unit and a privacy desensitization unit.

[0140] The active learning unit utilizes the classroom facial expression recognition model to identify the classification probabilities of discrete facial expression categories and the three-dimensional predicted values ​​of the continuous VAD emotion dimension. It employs hybrid uncertainty-driven active learning, combining classification entropy and regression variance to select high-value samples. The system uses a hybrid uncertainty active learning strategy to evaluate style-aligned images, prioritizing high-uncertainty samples for manual review. The review results are immediately fed back into the training set, forming a continuous iterative closed loop, significantly reducing manual annotation and improving model generalization ability. This achieves efficient and low-cost data annotation while ensuring student privacy.

[0141] The privacy desensitization unit is used to decouple the identity of the input face image using a three-dimensional deformation model, setting the identity features to zero and retaining only the expression and texture features. Then, it performs local blurring only in the eyebrow area to generate an anonymized face image, which satisfies privacy compliance and avoids destroying expression details.

[0142] The functions and roles of the data acquisition module 1, the data generation and migration module 2, and the anonymization and labeling module 3 are the same as those in Specific Embodiment 1, so this Specific Embodiment is omitted here and will not be described again.

[0143] The technical solution provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A semi-automatic annotation method for student facial expression data based on multimodal fusion, characterized in that, Includes the following steps: S1: Generate multi-dimensional expressions based on the semantic conditional diffusion model, construct an educational semantic conditional diffusion model, and synchronously output discrete expression classification and continuous VAD emotion dimension labels through text prompts in educational scenarios, which are divided into training set, validation set and test set; S2: Based on the discrete expression classification and continuous VAD emotion dimension labels, obtain the synthesized expression image and its corresponding 68 facial key point coordinates, perform classroom scene adaptive style transfer, use key point-constrained adversarial generative network to preserve expression features and adapt to classroom lighting and viewing angle, and generate a classroom expression recognition model. S3: Utilize the classroom facial expression recognition model to identify the classification probability of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension, hybrid uncertainty-driven active learning, and combine classification entropy and regression variance to screen high-value samples; S4: The high-value samples selected in S3 are subjected to privacy desensitization by decoupling their identities. The faces are reconstructed using a 3D deformation model, and the eyebrow area is specifically blurred to achieve anonymity protection.

2. The semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 1, characterized in that: S1: Generates multi-dimensional expressions based on a semantic conditional diffusion model, and synchronously outputs discrete expression classifications and continuous VAD emotion dimension labels through text prompts in educational scenarios; specifically including: S101: The discrete facial expression classification labels, which include six categories: focused, confused, distracted, interactive questioning, fatigued, and neutral, are converted into 128-dimensional vectors through an embedding layer; S102: Map the continuous VAD emotion vector containing the three dimensions of valence, arousal and dominance to a 128-dimensional vector through a fully connected layer; S103: The 128-dimensional vectors of S101 and S102, together with the 512-dimensional field educational scene text prompts containing school age parameters and classroom environment parameters extracted by the multimodal pre-trained model CLIP text encoder, are input into the U-Net network and cross-attention fusion is performed in its 4th to 16th layers to construct an educational semantic conditional diffusion model. S104: The probability distribution of discrete facial expression classification and the three-dimensional numerical values ​​of continuous VAD emotion vectors are output synchronously through a single forward propagation.

3. The semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 2, characterized in that: The educational scenario text prompts must include at least the following three elements: Age group label: one of the following four categories: lower elementary school, upper elementary school, junior high school, and senior high school; Classroom environmental parameters: a description of the combination of lighting type and obstructions; Facial expression keywords: focus, confusion, distraction, interactive questioning, fatigue, neutral, one of the following six categories.

4. A semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 2, characterized in that, The formula for the cross-attention fusion is: ； in || indicates vector concatenation, where d=128 is the feature dimension. , , These are the key matrices corresponding to discrete labels, VAD vectors, and text prompts, respectively.

5. A semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 1 or 2, characterized in that, Step S2: Based on the discrete expression classification and continuous VAD emotion dimension labels, obtain the synthesized expression image and its corresponding 68 facial key point coordinates, perform classroom scene-adaptive style transfer, and use a key point-constrained adversarial generative network to preserve expression features and adapt to classroom lighting and viewing angle to generate a classroom expression recognition model; specifically including: S201: Obtain the synthesized facial expression image and its corresponding 68 facial key point coordinates from the discrete expression classification and continuous VAD emotion dimension labels. The key point coordinates include the muscle movement features of eyebrows, eyes, and mouth. S202: Input the synthesized facial expression image into the generator and output a style-transferred image with classroom scene lighting, background and perspective features; S203: The style-transferred image and the real classroom scene image are adversarially discriminated against by a discriminator to generate an adversarial loss; S204: Calculate the Euclidean distance L2 of the 68 facial key points before and after migration to form the key point consistency loss, and calculate the mean-variance distance of the images before and after migration in the Lab color model to form the illumination consistency loss. S205: By jointly optimizing adversarial loss, keypoint consistency loss, and illumination consistency loss, and automatically updating the weight ratio of the three based on the validation set and keypoint error at fixed iterations during the training process, the facial muscle movements are kept true under classroom perspective and lighting conditions, thus generating a classroom facial expression recognition model.

6. A semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 5, characterized in that, The joint optimization includes an adaptive weight adjustment module. Based on the Fraser initial distance (FID) and keypoint error of the validation set, the adaptive weight adjustment module dynamically adjusts the weight allocation of adversarial loss, keypoint consistency loss, and illumination consistency loss until the difference in the rate of decrease of the three types of losses remains within a preset convergence threshold range, thus ensuring the fidelity of facial muscles while adapting to different classroom lighting conditions.

7. A semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 1, characterized in that, S3: Utilizing the classroom facial expression recognition model to identify the classification probabilities of discrete facial expression categories and the three-dimensional predicted values ​​of the continuous VAD emotion dimension, hybrid uncertainty-driven active learning is employed, combining classification entropy and regression variance to screen high-value samples; specifically including: S301: Utilize the trained classroom facial expression recognition model to perform forward inference on style-aligned images, and simultaneously output the classification probability distribution of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension. S302: Calculate the classification entropy H of the classification probability distribution. This classification entropy H is an indicator that measures the uncertainty of the model's prediction of discrete facial expression categories. Calculate the regression variance of the three-dimensional predicted values ​​based on Monte Carlo sampling. The regression variance This is used to evaluate the stability of model predictions, and then to construct a joint uncertainty index. ,in, and Dynamic updates are performed using Bayesian optimization to adjust the classification entropy H and regression variance, respectively. The contribution ratio in the joint uncertainty index U, and satisfying =1, ; S303: Sort by joint uncertainty index U in descending order and select the top... The samples were used as samples to be manually reviewed; The former After manual verification, the samples are immediately added to the training set, triggering the next round of Bayesian optimization to re-estimate. , This forms a continuous feedback loop.

8. A semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 7, characterized in that, The Monte Carlo sampling is achieved by enabling the model's Dropout layer, requiring no additional network parameters.

9. A semi-automatic annotation method for student facial expression data based on multimodal fusion according to claim 1, characterized in that, Step S4: The high-value samples selected in S3 undergo identity decoupling and privacy anonymization. This is achieved by reconstructing the face using a 3D deformation model and specifically blurring the eyebrow area to protect anonymity. Specifically, this includes: S401: Parametric decomposition of the input face image is performed using the 3D deformation model 3DMM to generate identity coefficients, expression coefficients, and texture coefficients; S402: After setting the identity coefficient to zero, reconstruct the face to obtain an anonymized base image with neutral geometric shape; S403: Apply Gaussian blur to the eyebrow region of the anonymized base image to generate the final anonymized face image; S404: Perform privacy compliance verification on the final anonymized face image to ensure that it meets the preset cross-identity similarity threshold and intra-class similarity variance threshold.

10. A semi-automatic annotation system for student facial expression data based on multimodal fusion, characterized in that: It includes a data acquisition module, a data generation and migration module, and an anonymization and annotation module. The data acquisition module is used to acquire student facial images; The data generation and migration module includes a synthetic data generation unit and a domain adaptation unit. The synthetic data generation unit is used to generate multi-dimensional expressions based on the semantic conditional diffusion model, and synchronously output discrete expression classification and continuous VAD emotion dimension labels through educational scenario text prompts, and batch generate labeled synthetic expression images and their corresponding 68 facial key point coordinates. The domain adaptation unit is used to transfer the style of the synthesized facial expression image to the real classroom environment. It uses a key-point constrained adversarial generative network to preserve facial expression features and adapt to classroom lighting and viewing angle to generate a classroom facial expression recognition model. The anonymization and annotation module includes an active learning unit and a privacy desensitization unit. The active learning unit is used to identify the classification probability of discrete facial expression categories and the three-dimensional predicted value of the continuous VAD emotion dimension using the classroom facial expression recognition model. It combines uncertainty-driven active learning and uses classification entropy and regression variance to screen high-value samples. The privacy desensitization unit is used to decouple the identity of the input face image using a 3D deformation model, reconstruct the face, and specifically blur the eyebrow area to achieve anonymity protection.

Images