A video micro-expression recognition method based on supervised prototype self-adaptive contrast learning

Abstract

This invention belongs to the field of computer vision and artificial intelligence technology, and relates to a video micro-expression recognition method based on supervised prototype adaptive contrastive learning. The method includes calculating the dense optical flow field using the TV-L1 algorithm based on the start frame and vertex frame of the micro-expression video sequence to obtain the horizontal and vertical optical flow components, then obtaining the optical strain map, and normalizing and resizing the three components respectively. A motion-aware feature extractor (MSFE) is constructed, a supervised prototype adaptive contrastive learning module (SPACoL) is constructed, and a prototype contrastive loss function is established. L PCL and classification cross-entropy loss function L CE By jointly optimizing the above loss function and training the network end-to-end, the video to be identified is input into the trained network to obtain the micro-expression category recognition result. This invention solves the technical problems of difficulty in extracting subtle motion features and imbalanced dataset distribution in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and in particular relates to a video micro-expression recognition method based on supervised prototype adaptive contrastive learning. Background Technology

[0002] Microexpressions are facial expressions that last for extremely short periods and involve very little movement. They typically occur in high-risk situations where people are trying to suppress or conceal their true emotions. Due to their involuntary and difficult-to-fake nature, microexpressions have extremely high application value in fields such as national security, psychological clinical diagnosis, lie detection, and human-computer interaction.

[0003] However, in practical applications, micro-expression recognition faces two major technical bottlenecks. First, subtle motion features are difficult to extract. Micro-expressions often occur only in localized areas of the face, such as a slight raise of the eyebrows or a slight movement of the mouth, and are often accompanied by head posture and lighting changes. Traditional deep learning methods, such as C3D (Convolutional3D) and CNN-LSTM (Convolutional Neural Network and Long Short-Term Memory Network), tend to capture high-frequency texture information or large-amplitude movements, easily ignoring the low-intensity, non-rigid motion features unique to micro-expressions. Second, the imbalanced distribution and sample imbalance of the dataset. In existing micro-expression datasets with three classifications, the number of samples in the "negative" category far exceeds those in the "positive" and "surprised" categories. Traditional supervised learning methods typically use the cross-entropy loss function, which treats each sample equally, leading to the model easily overfitting to the majority class during training, while achieving extremely low accuracy in recognizing the minority class. Furthermore, while existing instance-level contrastive learning methods can enhance feature representation, they ignore the semantic category relationships between samples and cannot effectively construct clear inter-class boundaries. To address the aforementioned issues, there is an urgent need for a micro-expression recognition method that can explicitly perceive subtle facial movements and effectively overcome the long-tail distribution problem. Summary of the Invention

[0004] The purpose of this invention is to provide a video micro-expression recognition method based on supervised prototype adaptive contrastive learning, which solves the technical problems of existing methods, such as difficulty in extracting subtle motion features, imbalanced dataset distribution, and the fact that existing instance-level contrastive learning methods, while enhancing feature representation, ignore the semantic category relationship between samples and cannot effectively construct clear inter-class boundaries.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a video micro-expression recognition method based on supervised prototype adaptive contrastive learning, comprising the following steps: S1: Based on the start frame and vertex frame of the given micro-expression video sequence, the dense optical flow field is calculated using the Total Variation-L1 algorithm to obtain the horizontal optical flow component and the vertical optical flow component. Based on the two, the optical strain map is calculated, and the three components are normalized and size adjusted respectively. S2: Construct a motion-aware feature extractor to generate high-dimensional video semantic feature vectors. ; S3: Construct a supervised prototype adaptive contrastive learning module to generate a temporary fusion prototype based on the high-dimensional video semantic feature vector generated in S2. Where 'c' represents the micro-expression category, The result of the fusion; S4: Construct the prototype contrastive loss function L PCL and classification cross-entropy loss function L CE The loss function is jointly optimized to perform end-to-end training of the motion-aware feature extractor and the supervised prototype adaptive contrastive learning module. The video to be recognized is input into the trained module to obtain the micro-expression category recognition result.

[0006] Furthermore, the specific steps of S1 are as follows: S1.1: Based on the annotation information in the dataset, determine the start frame and vertex frame of the micro-expression video sequence; S1.2: The dense optical flow field between the starting frame and the vertex frame is calculated using the Total Variation-L1 algorithm to obtain the horizontal optical flow component. and vertical optical flow component ; S1.3: Based on horizontal optical flow component and vertical optical flow component Calculate optical strain diagram As shown in equation (1): (1) in: For the normal strain in the horizontal direction, The normal strain is in the vertical direction. For horizontal displacement, For vertical displacement, The rate of change in the vertical direction, The rate of change in the horizontal direction, Optical strain diagram; S1.4: Horizontal optical flow component , vertical optical flow component and optical strain diagram Normalize each part and adjust it to the preset size.

[0007] Furthermore, the specific steps of S2 are as follows: S2.1: Construct a motion-aware feature extractor, which includes three parallel VisionTransformer encoders and a mutually biased cross-attention module, with its input connected to the output of the three VisionTransformer encoders respectively. S2.2: The horizontal optical flow component , vertical optical flow component Optical strain diagram The horizontal motion features are obtained by inputting the data into three parallel Vision Transformer encoders. Vertical motion characteristics and strain characteristics ; S2.3: The cross-attention module with mutual bias includes a self-attention branch, a first cross-attention branch that introduces a first bias signal, and a second cross-attention branch that introduces a second bias signal, which will handle horizontal motion features. Vertical motion characteristics and strain characteristics The input is fed into the cross-attention module with a mutual bias, and the first fused feature is obtained in the first cross-attention branch that introduces the first bias signal. The second fusion feature is obtained in the second cross-attention branch that introduces the second bias signal. Enhanced features are obtained in the self-attention branch. ; S2.4: The first fusion feature Second fusion feature Enhanced features The data are stitched together and projected onto a multilayer perceptron to obtain a high-dimensional video semantic feature vector. .

[0008] Furthermore, the specific steps in S2.3 are as follows: S2.3.1: The first cross-attention branch with strain characteristics For query vector Characterized by horizontal movement The first key vector and the first value vector With vertical motion characteristics The signal generated by the mapping is the first bias. Calculate the weights corresponding to the first cross-attention branch. 1 As shown in equation (2): (2) in: The weights corresponding to the first attention stream are: For query vector, The first key vector The transpose of the matrix, is the scaling factor for the dimension of the key vector. The preset bias adjustment coefficient, For the first bias, for function, For transpose; The second cross-attention branch uses strain characteristics For query vector With vertical motion characteristics The second key vector Second value vector Characterized by horizontal movement The signal generated by the mapping is the second bias. Calculate the weights corresponding to the second cross-attention branch. As shown in equation (3): (3) in: The weights corresponding to the second attention stream. The second key vector The transpose of the matrix, This is the second bias; Self-attention branch is a strain characteristic The self-attention flow, where the output of the self-attention branch is the enhanced feature. As shown in equation (4): (4) in: To enhance features, The third key vector The transpose of the matrix, It is the third value vector. The dimension of the key vector in the strain feature self-attention flow; S2.3.2: Weights calculated using the first cross-attention branch For the first value vector We perform weighted aggregation to obtain the first fusion feature. As shown in equation (5), the weights are calculated using the second cross-attention branch. For the second value vector Weighted aggregation is performed to obtain the second fusion feature. As shown in equation (6): (5) in: The first fusion feature, It is the first value vector; (6) in: This is the second fusion feature. This is the second value vector.

[0009] Furthermore, the specific steps of S3 are as follows: S3.1: Initialize size to The matrix, where K is the total number of micro-expression categories and D is the dimension of the feature vector; S3.2: Define Class Adaptive Momentum As shown in equation (7): (7) Among them: CAM- For adaptive momentum, For the first Number of samples in the class The number of samples in the largest class. The upper bound of momentum for the majority class. This is the lower bound of momentum for a minority class; S3.3: In each training batch, calculate the categories belonging to micro-expressions. Local prototype and combined with global prototype and momentum coefficient Update the global prototype vector in the global repository as shown in equation (8): (8) in: Adaptive Momentum for Category momentum coefficient, For category global prototype vector , This is a local prototype of the current batch of samples. l g represents the local value, while g represents the global value. S3.4: Construct a dynamic-static prototype fusion mechanism and calculate the temporary fused prototype. .

[0010] Furthermore, the specific steps in S3.4 are as follows: S3.4.1: In each training batch, obtain the local prototype of the current batch samples. and the global prototype vector in the global repository ; S3.4.2: Based on the number of samples in category c Calculate fusion weights Utilizing fusion weights For local prototypes and global prototype vector Perform weighted fusion to obtain a temporary fusion prototype. As shown in equation (9): (9) in: For temporary fusion prototype, For weight fusion.

[0011] Furthermore, the specific steps of S4 are as follows: S4.1: Constructing the Prototype Comparative Loss Function L PCL Calculate the first Features of a sample Its fusion prototype with its category c The similarity between samples is maximized while minimizing the similarity between the sample features and all other category fusion prototypes. S4.2: Constructing the classification cross-entropy loss function L CE high-dimensional video semantic feature vectors Input the data into the multilayer perceptron and calculate the cross-entropy loss between the predicted results and the true labels. S4.3: Constructing the total loss function The motion-aware feature extractor and the supervised prototype adaptive contrastive learning module are jointly trained. S4.4: Input the video to be recognized into the trained module and output the micro-expression category prediction result.

[0012] The beneficial effects of this invention are: 1) This invention constructs a Motion-Sensing Feature Extractor (MSFE), which can explicitly capture subtle non-rigid facial movements in micro-expressions and effectively suppress the interference of rigid head movements, thereby improving the model's ability to perceive weak motion features. 2) By introducing the Supervised Prototype-Adaptive Contrastive Learning (SPACoL) module, this invention designs Class-Adaptive Momentum (CAM) and Dynamic-Static Prototype Fusion (DSPF) mechanisms, which can dynamically adjust the prototype update strategy according to the number of samples in each class, enhance the representation ability of the minority class, and significantly improve the classification performance of the model under long-tailed distribution. 3) This invention achieves excellent unweighted F1 score (UF1) and unweighted average recall (UAR) on SMIC, CASME II, SAMM and comprehensive datasets. On CASME II, the UF1 score reaches 0.9697 and the UAR reaches 0.9650, which is significantly better than the existing mainstream methods. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a supervised prototype adaptive contrastive learning video micro-expression recognition method according to the present invention; Figure 2 This is a schematic diagram of the motion sensing feature extractor (MSFE) in this invention; Figure 3 This is a schematic diagram of the supervised prototype adaptive contrastive learning module SPACoL in this invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figure 1 As shown, a specific embodiment of the present invention provides a video micro-expression recognition method based on supervised prototype adaptive contrastive learning, which specifically includes the following steps: S1: Based on the given micro-expression video sequence, the dense optical flow field is calculated using the TV-L1 algorithm based on its start frame and vertex frame to obtain the horizontal optical flow component and the vertical optical flow component. Based on the two, the optical strain map is calculated, and the above three components are normalized and size adjusted respectively. The specific steps for S1 are as follows: S1.1: Based on the annotation information provided by the dataset, determine the start frame and vertex frame of a given micro-expression video sequence; S1.2: The dense optical flow field between the starting frame and the vertex frame is calculated using the Total Variation-L1 (Total Variation - L1 Regularization) algorithm to obtain the horizontal optical flow component. and vertical optical flow component ; In a specific embodiment of the present invention, optical flow describes the apparent motion of image brightness patterns in the time domain. Its mathematical definition is based on the assumption of constant brightness, and the calculation method is as follows: set up I ( x,y,t (time) t pixels ( x,y The brightness of the pixel when it moves along the trajectory ( x ( t ), y ( t When moving, the brightness remains unchanged, as shown in equation (1): (1) in: The rate of change of brightness over time; Expanding the total derivative shown in equation (1) yields equation (2); (2) in: For the image in Spatial brightness gradient in the direction, For the image in Spatial brightness gradient in the direction, This represents the temporal brightness gradient of the image over time. The horizontal motion velocity of a pixel on the image plane, i.e., the horizontal optical flow component. , The vertical motion velocity of a pixel on the image plane, i.e., the vertical optical flow component. .

[0017] Let the optical flow vector As shown in equation (3); (3) in: For the horizontal optical flow component, v This is the vertical optical flow component; Let the spatial gradient function be as shown in equations (4) and (5): (4) in: This represents the brightness gradient of a pixel in the horizontal direction. (5) in: This represents the brightness gradient of a pixel in the vertical direction. Let the time gradient function be as shown in equation (6): (6) in: This represents the brightness gradient of a pixel over time. Combining equations (3) to (6), we can obtain the fundamental equation of optical flow, as shown in equation (7): (7) Because the displacement field is estimated directly from the image sequence This invention addresses apparent motion, which incorporates rigid head movements such as translation and rotation, as well as non-rigid facial facial expressions. To suppress interference from rigid movements and extract features more sensitive to facial expressions, this invention introduces the infinitesimal strain theory from continuum mechanics. Strain describes the relative deformation of a local region, such as tension and shear, and is invariant to the overall rigid body motion. To capture local non-rigid facial deformations, such as skin stretching, this invention calculates optical strain maps based on optical flow components. .

[0018] S1.3: Based on horizontal optical flow component and vertical optical flow component Calculate optical strain diagram As shown in equation (8): (8) in: For the normal strain in the horizontal direction, The normal strain is in the vertical direction. For horizontal displacement, For vertical displacement, The rate of change in the vertical direction, The rate of change in the horizontal direction, This is an optical strain diagram.

[0019] In a specific embodiment of this application, the normal strain in the horizontal direction Normal strain in the vertical direction The difference represents the difference in normal strain in two orthogonal directions, which can reflect a pure expansion / contraction characteristic. For example, when the corners of the mouth are stretched to the sides and down at the same time when "surprised", this difference will change.

[0020] In summary, the optical strain diagram shown in equation (8) In reality, it consists of horizontal normal strain, vertical normal strain, and engineering shear strain. The square root of the weighted sum of squares is used to construct the optical strain map. Through this combination, the present invention can uniformly integrate the subtle stretching, compression, and shear deformation of facial skin in different directions into an optical strain map. This provides the subsequent Motion-Sensing Feature Extractor (MSFE) with feature inputs that are more sensitive to changes in facial expressions and invariant to rigid motion.

[0021] In continuum mechanics, engineering shear strain As shown in equation (9): (9) in: For engineering shear strain; S1.4: Horizontal optical flow component , vertical optical flow component Optical strain diagram Normalize each part and adjust it to the preset size; In a specific embodiment of the present invention, the horizontal optical flow component is... , vertical optical flow component Optical strain diagram The three components are normalized and their dimensions are adjusted accordingly. As three independent input channels for the subsequent network, during the training phase, the horizontal optical flow component... , vertical optical flow component Optical strain diagram Data augmentation operations, such as random pruning, are performed on the three components to improve the model's generalization ability.

[0022] S2: Construct a Motion-Sensing Feature Extractor (MSFE) consisting of a Vision Transformer encoder and a cross-biased attention module to extract subtle motion features from micro-expression videos and generate high-dimensional video semantic feature vectors. ; The Motion-Aware Feature Extractor (MSFE) aims to extract raw motion features from different dimensions and accurately capture subtle local facial deformations while effectively suppressing rigid head movements through cross-dimensional feature dynamic guidance and complementary fusion.

[0023] The extractor consists of a parallel Vision Transformer encoder and a mutually biased cross-attention module, with the following structure: Figure 2As shown, motion features of different dimensions are extracted by three parallel coding branches. Then, by utilizing the complementarity between different motion components and the bias signal-guided attention mechanism, dynamic guidance and feature complementarity fusion between branches are achieved, and finally, a spatiotemporal feature vector that can comprehensively represent the subtle dynamics of micro-expressions is output.

[0024] S2.1: Construct a motion-aware feature extractor. The extractor consists of three parallel VisionTransformer encoders and a mutually biased cross-attention module. The input of the mutually biased cross-attention module is connected to the output of each of the three Vision Transformer encoders. The three parallel Vision Transformer encoders all adopt the well-known Vision Transformer architecture in the field of computer vision. The three parallel VisionTransformer encoders are referred to as the first Vision Transformer encoder. The second VisionTransformer encoder The third Vision Transformer encoder The horizontal optical flow component obtained from S1 Input to the first Vision Transformer encoder , to obtain horizontal motion characteristics The vertical optical flow component obtained from S1 Input to the second Vision Transformer encoder Vertical motion characteristics are obtained. The optical strain map obtained from S1 Input to the third Vision Transformer encoder Strain characteristics were obtained. ; S2.2: The horizontal optical flow component , vertical optical flow component Optical strain diagram The horizontal motion features are obtained by inputting the data into three parallel Vision Transformer encoders. Vertical motion characteristics and strain characteristics ; S2.3: The cross-attention module with mutual bias includes a self-attention branch, a first cross-attention branch that introduces a first bias signal, and a second cross-attention branch that introduces a second bias signal, which will handle horizontal motion features. Vertical motion characteristics and strain characteristics The input is fed into the cross-attention module with a mutual bias, and the first fused feature is obtained in the first cross-attention branch that introduces the first bias signal. The second fusion feature is obtained in the second cross-attention branch that introduces the second bias signal. Enhanced features are obtained in the self-attention branch. ; In the specific embodiments of this application, horizontal motion features are... Vertical motion characteristics and strain characteristics The input is fed into the Mutual Bias Cross-Attention Module (MBCA), which is based on horizontal motion features. Vertical motion characteristics With strain characteristics By leveraging the complementarity between the three features, a learnable bias signal is introduced to guide the computation of the cross-attention mechanism, thereby enabling the three features to mutually enhance and fuse, generating fused motion representation features. This addresses the problem that existing methods struggle to stably extract features when dealing with extremely weak and easily disturbed micro-expression muscle movements due to the lack of explicit guidance between channels. The design incorporates three parallel attention streams.

[0025] The specific steps in S2.3 are as follows: S2.3.1: The first cross-attention branch with strain characteristics For query vector Characterized by horizontal movement The first key vector and the first value vector And introduce the characteristics of vertical motion The signal generated by the mapping is used as the first bias. By explicitly adding a first bias in the weight calculation The weights corresponding to the first cross-attention branch are obtained. 1 As shown in equation (10): (10) in: The weights corresponding to the first attention stream are: For query vector, The first key vector The transpose of the matrix, is the scaling factor for the dimension of the key vector. The preset bias adjustment coefficient, For the first bias, for function; The second cross-attention branch uses strain characteristics For query vector With vertical motion characteristics The second key vector Second value vector And introduce horizontal motion features The signal generated by the mapping is used as the second bias. By explicitly adding a second bias in the weight calculation The weights corresponding to the second cross-attention branch are obtained. The function is expressed as shown in equation (11): (11) in: 2 represents the weight corresponding to the second attention flow. The second key vector The transpose of the matrix, This is the second bias; The self-attention branch is designed for strain characteristics. The self-attention flow, where the output of the self-attention branch is the enhanced feature. The function expression is shown in equation (12); (12) in: To enhance features, The third key vector The transpose of the matrix, It is the third value vector. Let be the dimension of the key vector in the strain feature self-attention stream.

[0026] S2.3.2: Weights calculated using the first cross-attention branch 1 For the first value vector Weighted aggregation is performed, that is, the vertical component is used to dynamically guide the fusion process of strain and horizontal features to obtain the first fused feature. As shown in equation (13), the weights are calculated using the second cross-attention branch. For the second value vector Weighted aggregation is performed, that is, the vertical component is used to dynamically guide the fusion process of strain and horizontal features to obtain the second fused feature. As shown in equation (14): (13) in: As the first enhancement feature, It is the first value vector; (14) in: As the second enhancement feature, It is the second value vector; S2.4: The first fusion feature Second fusion feature Enhanced features The data is concatenated along the channel dimension and projected through a multilayer perceptron to obtain a high-dimensional video semantic feature vector. .

[0027] S3: Construct a supervised prototype adaptive contrastive learning module SPACoL to dynamically adjust the update strategy of class prototypes in the feature space; like Figure 3 As shown in the specific implementation of this application, a supervised prototype-adaptive contrastive learning module SPACoL (Supervised Prototype-Adaptive Contrastive Learning) is constructed. This module consists of a global feature library, a local prototype library, a temporary fusion library, a class-adaptive momentum (CAM) and a dynamic-static prototype fusion (DSPF) unit. By adaptively weighting and fusing the static prototypes in the global feature library with the dynamic local prototypes of the current batch, a temporary fusion prototype is generated, which corrects the distortion of the representation space caused by data imbalance and significantly enhances the discrimination accuracy of minority class samples. The specific steps for S3 are as follows: S3.1: Initialize a space of size The matrix, where K is the total number of micro-expression categories and D is the vector dimension of the features, is used to store... Global prototype vectors for each microexpression category ; In a specific embodiment of the present invention, the feature vectors obtained after feature extraction from all training samples are calculated. Feature representations in the prototype space are obtained through projection layer mapping. Where p is the projection, i* is the projection of the i-th feature, and then, based on the label, the features of all samples in the same category are summed and averaged to obtain the initial global prototype vector for each category. Finally, initialize a size of A matrix used to store Global prototype vectors for each micro-expression category, where... Let g be the global prototype vector of micro-expression category c stored in the global repository, and g be the global vector.

[0028] S3.2: To balance the prototype update speed of the majority class and the minority class, define a class adaptive momentum. The function is shown in equation (15): CAM- (15) Among them: CAM- For adaptive momentum, For the first Number of samples in the class The number of samples in the largest class. The momentum upper bound for the majority class is set to 0.9 to ensure stability for the majority class. The momentum lower bound for the minority class is set to 0.1 for fast updates of the minority class.

[0029] In a specific embodiment of the present invention, adaptive momentum The momentum coefficient is dynamically calculated based on the number of samples in each category. The fewer samples a category has, the smaller its momentum coefficient, which makes its prototype update speed faster, and vice versa.

[0030] S3.3: In each training batch, calculate the categories belonging to micro-expressions. Local prototype ,in: l This represents the local area, combined with the global prototype. and momentum coefficient Update the prototype in the global repository as shown in equation (16): (16) in: Adaptive Momentum for Category momentum coefficient, For category global prototype vector , This is a local prototype of the current batch of samples. l For local purposes; S3.4: Introduce Dynamic-Static Prototype Fusion DSPF to calculate temporary fused prototypes. , f For fusion; The specific steps are as follows: S3.4.1: In each training batch, obtain the local prototype of the current batch samples. and the global prototype in the global repository ; S3.4.2: Based on the number of samples in category c Dynamic calculation of fusion weights Utilizing fusion weights For local prototypes and global prototype Perform weighted fusion to obtain a temporary fusion prototype. As shown in equation (17): (17) in: For temporary fusion prototype, For weighting; In the specific implementation of this application, the fusion weights are used in equation (17). It is not a fixed value, but depends on the number of training samples in each category. Dynamic adjustment, as shown in equation (18). (18) in: N c This represents the total number of samples in the current sample label category c.

[0031] In a specific embodiment of this invention, DSPF is essentially an adaptive fusion logic formula based on sample distribution. In terms of connectivity, the input of this unit is connected to both the global feature library (representing historical stable states) and the local prototype library of the current batch of samples (representing the current evolutionary state). It adaptively fuses the static and dynamic prototypes into a temporary fusion library by calculating fusion weights in real time. Subsequently, it uses this temporary fusion library and the original features of each sample (i.e., the features of each sample itself before constructing the local prototype library) to perform a comparative loss calculation. Compared to using only minority class features, this approach is more effective in responding to the evolutionary changes of minority class features. The unique value of this fusion mechanism lies in achieving an adaptive balance between different class prototypes: it ensures that minority class prototypes can absorb more dynamic and fresh features from the current batch to maintain the timeliness of the representation, while the majority class, with sufficient samples, relies on the global repository to ensure the smoothness and stability of the prototype. Through this "dynamic-static combination", it effectively makes up for the shortcomings of traditional methods in processing unbalanced micro-expression data due to the distortion of the representation space, thereby significantly enhancing the overall model's discrimination accuracy for minority class samples.

[0032] S4: Construct the prototype contrastive loss function L PCL and classification cross-entropy loss function L CE The network is trained end-to-end by jointly optimizing the above loss function. The video to be identified is input into the trained network to obtain the micro-expression category recognition result. S4.1: Constructing the prototype contrastive loss function L PCL Calculate the first Features of a sample Its fusion prototype with its category c The similarity between the samples is maximized while minimizing the similarity between the sample features and all other class fusion prototypes, as shown in Equation (19): (19) in: B This represents the total number of samples in the current training batch. K This represents the total number of micro-expression categories. For the first Each sample is in the prototype projection space (superscript) eigenvectors in ) For the sample The fusion prototype belonging to the real label category, For the category label of the sample, This is the result after dynamic-static adaptive fusion. For the first The fused prototype vectors of each category are used in the denominator to calculate the similarity distribution between the sample and all categories. For temperature parameters, For sample index, For category indexing.

[0033] S4.2: Constructing the classification cross-entropy loss function L CE The high-dimensional video semantic feature vector F I Inputting the data into a multilayer perceptron, the cross-entropy loss between the predicted and true labels is calculated, i.e., the classification cross-entropy loss function L. CE As shown in equation (20); (20) in: For the first The feature vector of each sample For samples in the linear classification layer Authentic Labels The corresponding weight vector, For the first linear classification layer Weight vectors for each category.

[0034] S4.3: Constructing the total loss function As shown in Equation (21), the motion-aware feature extractor and the supervised prototype adaptive contrastive learning module are jointly trained. (twenty one) S4.4: Input the video to be recognized into the trained module and output the micro-expression category prediction result.

[0035] In the data preprocessing stage, this invention uses MTCNN to crop the facial region and adjust its size. For the solver parameters, this invention uses the AdamW optimizer to minimize the objective function, where the initial learning rate is set to 5e-5, the weight decay is set to 0.01, the batch size is set to 32, and the training is conducted for a total of 500 epochs. The loss function uses balanced weights... The value was set to 1. The experiment adopted the Leave-One-Subject-Out (LOSO) protocol for cross-validation and used the unweighted F1 score (UF1) and unweighted average recall (UAR) as evaluation metrics.

[0036] Table 1 presents a quantitative comparison of the proposed method with state-of-the-art methods on a comprehensive dataset and three independent datasets. The results show that the proposed method demonstrates high competitiveness in all evaluations.

[0037] In the most challenging comprehensive dataset evaluation, the proposed method achieved the best performance on both UF1 and UAR metrics, reaching 0.8729 and 0.8760, respectively. In single dataset evaluation, MSPCoL also performed outstandingly—especially on the CASME II (Chinese Academy of Sciences Micro-expression II) dataset, where UF1 and UAR reached 0.9697 and 0.9650, respectively. In addition, on the SAMM (Spontaneous Actions and Micro-expressions Macro) dataset, the model obtained the highest UF1 score of 0.8612.

[0038] Despite variations in specific datasets and metrics, it's worth emphasizing that this method neither relies on large-scale vision-language pre-training like CLIP nor employs the unlabeled, proprietary workflow used by SODA4MER (Spontaneous Optical Database for Automatic Micro-Expression Recognition). The core advantage of this method lies in its joint design of a motion-aware front-end and an imbalance-aware back-end. This method achieves overall leadership on comprehensive datasets and CASME II, and attains the best UF1 score on SAMM. These results clearly demonstrate the effectiveness and robustness of this framework in collaboratively addressing the challenges of subtle motion perception and data imbalance in micro-expression recognition.

[0039] Table 1. Performance comparison of the proposed method with state-of-the-art methods on both integrated and independent datasets.

[0040] To verify the effectiveness of each component in this method, an ablation study was conducted. As shown in Table 2, the complete model was compared with three variants: ViT (Vision Transformer), ViT+SPACoL, and MSF. The results show that MSFE enhances the dynamic feature extraction capability, SPACoL effectively alleviates the data imbalance problem, and the combination of the two achieves the best overall performance, confirming that they have a strong complementary and synergistic effect.

[0041] Table 2 Results of the effectiveness verification experiment of MSFE and SPACoL modules

[0042] As shown in Table 3, this invention conducted an ablation study on two key components of SPACoL—Class Adaptive Momentum CAM and Dynamic-Static Prototype Fusion DSPF. The baseline model adopts a fixed momentum mechanism. Adding CAM or DSPF alone can improve performance, but the combination of the two has the best effect. This shows that CAM can balance the prototype update rate under data imbalance conditions, while DSPF coordinates stability and timeliness, and together enhances the robustness of SPACoL.

[0043] Table 3. Ablation analysis results of SPACoL internal components (CAM and DSPF)

[0044] Table 4 validates the effectiveness of SPACoL in handling imbalanced data. On the SAMM and CASME II datasets, SPACoL significantly improves the recall of the minority class. For example, in SAMM, the recall of the "surprise" class increases dramatically from 0.667 to 0.867, and in CASME II, the recall of the positive class increases from 0.844 to 0.906. On the SMIC dataset, SPACoL further improves the recall of the minority class "surprise" from 0.756 to 0.805, while significantly improving the recall of the majority class "negative" from 0.754 to 0.877. This indicates that SPACoL's adaptive mechanism improves the performance of the minority class while enhancing the overall discriminability of all classes by reshaping the decision boundary.

[0045] Table 4. Experimental results of the effectiveness of SPACoL in handling class-imbalanced data.

[0046] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0047] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A video micro-expression recognition method based on supervised prototype adaptive contrastive learning, characterized in that, The specific steps are as follows: S1: Based on the start frame and vertex frame of the given micro-expression video sequence, the dense optical flow field is calculated using the Total Variation-L1 algorithm to obtain the horizontal optical flow component and the vertical optical flow component. Based on the two, the optical strain map is calculated, and the three components are normalized and size adjusted respectively. S2: Construct a motion-aware feature extractor to generate high-dimensional video semantic feature vectors. ; S3: Construct a supervised prototype adaptive contrastive learning module to generate a temporary fusion prototype based on the high-dimensional video semantic feature vector generated in S2. Where 'c' represents the micro-expression category, The result of the fusion; S4: Construct the prototype contrastive loss function L PCL and classification cross-entropy loss function L CE The loss function is jointly optimized to perform end-to-end training of the motion-aware feature extractor and the supervised prototype adaptive contrastive learning module. The video to be recognized is input into the trained module to obtain the micro-expression category recognition result.

2. The video micro-expression recognition method based on supervised prototype adaptive contrastive learning according to claim 1, characterized in that, The specific steps for S1 are as follows: S1.1: Based on the annotation information in the dataset, determine the start frame and vertex frame of the micro-expression video sequence; S1.2: The dense optical flow field between the starting frame and the vertex frame is calculated using the Total Variation-L1 algorithm to obtain the horizontal optical flow component. and vertical optical flow component ; S1.3: Based on horizontal optical flow component and vertical optical flow component Calculate optical strain diagram As shown in equation (1): （1） in: For the normal strain in the horizontal direction, The normal strain is in the vertical direction. For horizontal displacement, For vertical displacement, The rate of change in the vertical direction, The rate of change in the horizontal direction, Optical strain diagram; S1.4: Horizontal optical flow component , vertical optical flow component and optical strain diagram Normalize each part and adjust it to the preset size.

3. The video micro-expression recognition method based on supervised prototype adaptive contrastive learning according to claim 1, characterized in that, The specific steps for S2 are as follows: S2.1: Construct a motion-aware feature extractor, which includes three parallel VisionTransformer encoders and a mutually biased cross-attention module, with its input connected to the output of the three VisionTransformer encoders respectively. S2.2: The horizontal optical flow component , vertical optical flow component Optical strain diagram The horizontal motion features are obtained by inputting the data into three parallel VisionTransformer encoders. Vertical motion characteristics and strain characteristics ; S2.3: The cross-attention module with mutual bias includes a self-attention branch, a first cross-attention branch that introduces a first bias signal, and a second cross-attention branch that introduces a second bias signal, which will handle horizontal motion features. Vertical motion characteristics and strain characteristics The input is fed into the cross-attention module with a mutual bias, and the first fused feature is obtained in the first cross-attention branch that introduces the first bias signal. The second fusion feature is obtained in the second cross-attention branch that introduces the second bias signal. Enhanced features are obtained in the self-attention branch. ; S2.4: The first fusion feature Second fusion feature Enhanced features The data are stitched together and projected onto a multilayer perceptron to obtain a high-dimensional video semantic feature vector. .

4. The video micro-expression recognition method based on supervised prototype adaptive contrastive learning according to claim 3, characterized in that, The specific steps in S2.3 are as follows: S2.3.1: The first cross-attention branch with strain characteristics For query vector Characterized by horizontal movement The first key vector and the first value vector With vertical motion characteristics The signal generated by the mapping is the first bias. Calculate the weights corresponding to the first cross-attention branch. 1 As shown in equation (2): （2） in: The weights corresponding to the first attention stream are: For query vector, The first key vector The transpose of the matrix, is the scaling factor for the dimension of the key vector. The preset bias adjustment coefficient, For the first bias, for function, For transpose; The second cross-attention branch uses strain characteristics For query vector With vertical motion characteristics The second key vector Second value vector Characterized by horizontal movement The signal generated by the mapping is the second bias. Calculate the weights corresponding to the second cross-attention branch. As shown in equation (3): （3） in: The weights corresponding to the second attention stream. The second key vector The transpose of the matrix, This is the second bias; Self-attention branch is a strain characteristic The self-attention flow, where the output of the self-attention branch is the enhanced feature. As shown in equation (4): （4） in: To enhance features, The third key vector The transpose of the matrix, It is the third value vector. The dimension of the key vector in the strain feature self-attention flow; S2.3.2: Weights calculated using the first cross-attention branch For the first value vector We perform weighted aggregation to obtain the first fusion feature. As shown in equation (5), the weights are calculated using the second cross-attention branch. For the second value vector Weighted aggregation is performed to obtain the second fusion feature. As shown in equation (6): （5） in: The first fusion feature, It is the first value vector; （6） in: This is the second fusion feature. This is the second value vector.

5. The video micro-expression recognition method based on supervised prototype adaptive contrastive learning according to claim 1, characterized in that, The specific steps for S3 are as follows: S3.1: Initialize size to The matrix, where K is the total number of micro-expression categories and D is the dimension of the feature vector; S3.2: Define Class Adaptive Momentum As shown in equation (7): （7） Among them: CAM- For adaptive momentum, For the first Number of samples in the class The number of samples in the largest class. The upper bound of momentum for the majority class. This is the lower bound of momentum for a minority class; S3.3: In each training batch, calculate the categories belonging to micro-expressions. Local prototype and combined with global prototype and momentum coefficient Update the global prototype vector in the global repository as shown in equation (8): （8） in: Adaptive Momentum for Category momentum coefficient, For category global prototype vector , This is a local prototype of the current batch of samples. l g represents the local value, while g represents the global value. S3.4: Construct a dynamic-static prototype fusion mechanism and calculate the temporary fused prototype. .

6. The video micro-expression recognition method based on supervised prototype adaptive contrastive learning according to claim 5, characterized in that, The specific steps in S3.4 are as follows: S3.4.1: In each training batch, obtain the local prototype of the current batch samples. and the global prototype vector in the global repository ; S3.4.2: Based on the number of samples in category c Calculate fusion weights Utilizing fusion weights For local prototypes and global prototype vector Perform weighted fusion to obtain a temporary fusion prototype. As shown in equation (9): （9） in: For temporary fusion prototype, For weight fusion.

7. The video micro-expression recognition method based on supervised prototype adaptive contrastive learning according to claim 1, characterized in that, The specific steps for S4 are as follows: S4.1: Constructing the Prototype Comparative Loss Function L PCL Calculate the first Features of a sample Its fusion prototype with its category c The similarity between samples is maximized while minimizing the similarity between the sample features and all other category fusion prototypes. S4.2: Constructing the classification cross-entropy loss function L CE high-dimensional video semantic feature vectors Input the data into the multilayer perceptron and calculate the cross-entropy loss between the predicted results and the true labels. S4.3: Constructing the total loss function The motion-aware feature extractor and the supervised prototype adaptive contrastive learning module are jointly trained. S4.4: Input the video to be recognized into the trained module and output the micro-expression category prediction result.

Images