Micro-expression recognition model training method and system across data set domains

Abstract

The present application relates to the technical field of image recognition, and particularly relates to a micro-expression recognition model training method and system for cross-dataset domain adaptation, which comprises the following steps: obtaining a plurality of micro-expression datasets under different acquisition conditions as source domains and target domains, and performing optical flow extraction and face alignment; training a double-branch feature extraction network with the source domain labeled data and the target domain unlabeled data, the spatial domain encoding branch extracts spatial texture features by using 2D-CNN, and the time domain encoding branch extracts time sequence motion features by using 3D-CNN; performing an antagonistic domain alignment through a domain discriminator and a gradient reversal layer, and using the target domain unlabeled data for self-training with high confidence pseudo labels, and enhancing the discriminativeness of the target domain features with a supervised contrast loss; and after training, merging the double-branch network weights into a single-path reasoning network to perform micro-expression classification. The present application significantly improves the generalization performance of cross-dataset micro-expression recognition through joint optimization of domain alignment and contrast learning.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for training a micro-expression recognition model that is adaptive across data domains. Background Technology

[0002] Microexpressions are facial muscle movements that last from 1 / 25 to 1 / 5 of a second and are not subject to conscious control. They can truthfully reflect an individual's hidden emotions. Microexpression recognition has important applications in fields such as public safety, clinical diagnosis, and psychological assessment.

[0003] However, micro-expression recognition faces two core challenges: First, micro-expressions are extremely small, manifesting only as weak muscle movements in local areas of facial images, with very subtle differences in features compared to ordinary expressions; second, existing publicly available micro-expression datasets (such as CASME II, SAMM, SMIC, etc.) have extremely limited sample sizes, with each dataset typically containing only tens to hundreds of samples. Furthermore, there are significant differences between datasets in terms of frame rate of acquisition devices, lighting conditions, and the ethnicity and cultural background of the subjects, which leads to a severe performance drop when a model trained on one dataset is directly applied to other datasets.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] This invention provides a cross-data domain adaptive micro-expression recognition model training method and system, thereby effectively solving the problems in the background technology.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a cross-data domain adaptive micro-expression recognition model training method and system, comprising the following steps:

[0007] Multiple micro-expression video datasets under different acquisition conditions were acquired as the source and target domains. Optical flow feature extraction and facial region alignment preprocessing were performed on video segments in each dataset to generate a standardized optical flow feature map sequence.

[0008] A dual-branch feature extraction network is jointly trained using labeled data from the source domain and unlabeled data from the target domain. The dual-branch feature extraction network includes a spatial coding branch and a temporal coding branch with shared weights. The spatial coding branch extracts spatial texture features using a single-frame optical flow feature map as input, and the temporal coding branch extracts temporal motion features using a sequence of optical flow feature maps as input. The outputs of the two branches are concatenated to form a fused feature vector.

[0009] Adversarial domain alignment training is performed on the fused feature vectors by a domain discriminator, making the fused feature vectors of the source domain and the target domain indistinguishable in distribution; at the same time, supervised classification training is performed on the labeled data of the source domain using cross-entropy loss, and classification constraints are performed on the unlabeled data of the target domain using a self-training method based on high-confidence pseudo-labels.

[0010] During the domain alignment training process, the contrastive learning loss is constructed using the fused feature vector output by the dual-branch feature extraction network of the target domain samples. Pseudo-label samples of the same category in the target domain are positive sample pairs, and samples of different categories are negative sample pairs, thereby narrowing the distance between similar features and widening the distance between dissimilar features.

[0011] The weights of the spatial coding branch and the temporal coding branch of the trained dual-branch feature extraction network are merged into a single-path inference network.

[0012] Furthermore, the optical flow feature extraction uses the TV-L1 optical flow algorithm to calculate the horizontal and vertical optical flow components between adjacent frames, which are then superimposed into a two-channel optical flow feature map.

[0013] The facial region alignment involves locating the eye, nose, and mouth regions using facial key point detection, and then normalizing the facial image to a uniform size and pose using affine transformation.

[0014] Furthermore, the spatial coding branch is a 2D convolutional neural network, which takes a single frame two-channel optical flow feature map as input, extracts spatial texture features through four convolutional layers, and outputs a 256-dimensional spatial feature vector.

[0015] The temporal coding branch is a 3D convolutional neural network. It takes a sequence of six consecutive frames of two-channel optical flow feature maps as input, extracts temporal motion features through three 3D convolutional layers and one temporal pooling layer, and outputs a 256-dimensional temporal feature vector. The fused feature vector is 512-dimensional.

[0016] Furthermore, the adversarial domain alignment training is achieved through a gradient inversion layer:

[0017] A gradient inversion layer is inserted at the input of the domain discriminator. The gradient inversion layer does not perform any transformation during forward propagation, but multiplies the gradient by a preset negative coefficient and propagates it back during backward propagation. This causes the dual-branch feature extraction network to update its parameters in the direction of increasing the domain discrimination loss, while the domain discriminator updates its parameters in the direction of decreasing the domain discrimination loss.

[0018] Furthermore, the self-training method based on high-confidence pseudo-labels is as follows:

[0019] The preliminary classifier trained in the source domain is used to predict unlabeled samples in the target domain, and samples with a maximum predicted probability higher than the first confidence threshold are selected as high-confidence pseudo-label samples and added to the labeling pool.

[0020] The dual-branch feature extraction network is iteratively retrained using a weighted joint loss of the source domain labeled data and the target domain high-confidence pseudo-label samples. After each training round, the pseudo-label screening criteria are gradually reduced by a preset step size of decreasing the first confidence threshold, so that more target domain samples can participate in the training.

[0021] Furthermore, the contrastive learning loss is a supervised contrastive loss, which uses sample pairs of the same pseudo-label class in the target domain as positive sample pairs and sample pairs of different pseudo-label classes as negative sample pairs. The loss function is:

[0022] ;

[0023] in and This is the fused feature vector of positive sample pairs. Let be the fused feature vector of all negative and positive samples, and sim be the cosine similarity. This refers to the temperature parameter.

[0024] Furthermore, the weights of the spatial coding branch and the temporal coding branch are merged into a single-path inference network as follows:

[0025] After training, the sum of the convolution kernel weights of the spatial coding branch and the convolution kernel weights of the temporal coding branch is used as the initial weights of the single-path inference network. The single-path inference network is then fine-tuned in a few rounds using a small batch of labeled data from the source domain, so that the merged network weights are adapted to the unified representation of feature extraction and classification.

[0026] Furthermore, the method also includes:

[0027] When there are multiple source domain datasets with different acquisition conditions, the adversarial domain alignment and contrastive learning training is performed sequentially, with each source domain as the current source domain and the data of the remaining source domains and the target domain as unlabeled data. After each source domain alignment is completed, the weights of the dual-branch feature extraction network are used as the initial weights for the next source domain alignment training.

[0028] This invention also includes a cross-data domain adaptive micro-expression recognition model training system, using the method described above, wherein the system comprises:

[0029] The acquisition and preprocessing unit is used to acquire multiple micro-expression video datasets under different acquisition conditions as the source and target domains, and to perform optical flow feature extraction and facial region alignment preprocessing on the video segments in each dataset to generate a standardized optical flow feature map sequence.

[0030] A dual-branch feature extraction unit is used to jointly train a dual-branch feature extraction network with labeled data from the source domain and unlabeled data from the target domain. The dual-branch feature extraction network includes a spatial coding branch and a temporal coding branch with shared weights. The spatial coding branch extracts spatial texture features with a single-frame optical flow feature map as input, and the temporal coding branch extracts temporal motion features with a sequence of optical flow feature maps as input. The outputs of the two branches are concatenated to form a fused feature vector.

[0031] The training unit is used to perform adversarial domain alignment training on the fused feature vector through a domain discriminator, so that the fused feature vectors of the source domain and the target domain are indistinguishable in distribution; at the same time, it performs supervised classification training on the labeled data of the source domain using cross-entropy loss, and performs classification constraints on the unlabeled data of the target domain using a self-training method based on high-confidence pseudo-labels.

[0032] During the domain alignment training process, the contrastive learning loss is constructed using the fused feature vector output by the dual-branch feature extraction network of the target domain samples. Pseudo-label samples of the same category in the target domain are positive sample pairs, and samples of different categories are negative sample pairs, thereby narrowing the distance between similar features and widening the distance between dissimilar features.

[0033] The weight merging unit is used to merge the weights of the spatial coding branch and the temporal coding branch of the trained dual-branch feature extraction network into a single-path inference network.

[0034] The present invention also includes a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described above.

[0035] The present invention also includes a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0036] The beneficial effects of this invention are as follows: adversarial domain alignment effectively eliminates collection bias across datasets; supervised contrastive learning further enhances the discriminative power of target domain features. Introducing supervised contrastive loss on top of domain alignment effectively improves the target domain test accuracy. After introducing contrastive learning, the gap between different categories of target domain features significantly increases, and inter-class overlap significantly decreases. High-confidence pseudo-label self-training effectively utilizes unlabeled data in the target domain. In the initial unlabeled state of the target domain, the proportion of samples effectively utilized in response is improved. Weight merging transforms the dual-branch network into a single-path inference network, significantly reducing the number of inference parameters and achieving a good balance between accuracy and efficiency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0039] Figure 2 This is a schematic diagram of the system structure in Embodiment 1 of the present invention;

[0040] Figure 3 This is a flowchart of the method in Embodiment 2 of the present invention;

[0041] Figure 4 This is a schematic diagram of the computer device in Embodiment 3 of the present invention. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0043] Example 1:

[0044] like Figure 1 As shown: A cross-data domain adaptive micro-expression recognition model training method and system, including the following steps:

[0045] Multiple micro-expression video datasets under different acquisition conditions were acquired as the source and target domains. Optical flow feature extraction and facial region alignment preprocessing were performed on video segments in each dataset to generate a standardized optical flow feature map sequence.

[0046] A dual-branch feature extraction network is jointly trained using labeled data from the source domain and unlabeled data from the target domain. The dual-branch feature extraction network includes a spatial coding branch and a temporal coding branch with shared weights. The spatial coding branch extracts spatial texture features using a single-frame optical flow feature map as input, while the temporal coding branch extracts temporal motion features using a sequence of optical flow feature maps as input. The outputs of the two branches are concatenated to form a fused feature vector.

[0047] Adversarial domain alignment training is performed on the fused feature vectors by a domain discriminator, making the fused feature vectors of the source and target domains indistinguishable in distribution; at the same time, supervised classification training is performed on the labeled data of the source domain using cross-entropy loss, and classification constraints are performed on the unlabeled data of the target domain using a self-training method based on high-confidence pseudo-labels.

[0048] During the domain alignment training process, the contrastive learning loss is constructed using the fused feature vector output by the dual-branch feature extraction network of the target domain samples. Pseudo-label samples of the same category in the target domain are positive sample pairs, and samples of different categories are negative sample pairs, which brings the distance between similar features closer and pushes the distance between dissimilar features further apart.

[0049] The weights of the spatial coding branch and the temporal coding branch of the trained dual-branch feature extraction network are merged into a single-path inference network.

[0050] Adversarial domain alignment effectively eliminates cross-dataset sampling bias; supervised contrastive learning further enhances the discriminative power of target domain features. Introducing supervised contrastive loss on top of domain alignment significantly improves target domain test accuracy. After introducing contrastive learning, the gap between different categories of target domain features increases significantly, while inter-class overlap decreases substantially. High-confidence pseudo-label self-training effectively utilizes unlabeled target domain data. In the initial unlabeled target domain, the proportion of samples effectively utilized increases. Weight merging transforms the dual-branch network into a single-path inference network, significantly reducing the number of inference parameters and achieving a good balance between accuracy and efficiency.

[0051] In this embodiment, the optical flow feature extraction uses the TV-L1 optical flow algorithm to calculate the horizontal and vertical optical flow components between adjacent frames, which are then superimposed to form a two-channel optical flow feature map.

[0052] Facial region alignment uses facial landmark detection to locate the eye, nose, and mouth regions, and then uses affine transformation to normalize the facial image to a uniform size and pose.

[0053] Among them, the spatial coding branch is a 2D convolutional neural network, which takes a single frame two-channel optical flow feature map as input, extracts spatial texture features through four convolutional layers and outputs a 256-dimensional spatial feature vector.

[0054] The temporal coding branch is a 3D convolutional neural network. It takes six consecutive frames of two-channel optical flow feature map sequences as input, extracts temporal motion features through three 3D convolutional layers and one temporal pooling layer, and outputs a 256-dimensional temporal feature vector; the fused feature vector is 512-dimensional.

[0055] As a preferred embodiment of the above, adversarial domain alignment training is achieved through a gradient inversion layer:

[0056] A gradient reversal layer is inserted at the input of the domain discriminator. The gradient reversal layer does not perform any transformation during forward propagation, but multiplies the gradient by a preset negative coefficient and propagates it back during backward propagation. This causes the dual-branch feature extraction network to update its parameters in the direction of increasing the domain discriminant loss, while the domain discriminator updates its parameters in the direction of decreasing the domain discriminant loss.

[0057] Among them, the self-training method based on high-confidence pseudo-labels is as follows:

[0058] The preliminary classifier trained in the source domain is used to predict unlabeled samples in the target domain. Samples with the maximum predicted probability higher than the first confidence threshold are selected as high-confidence pseudo-label samples and added to the labeling pool.

[0059] The dual-branch feature extraction network is iteratively retrained using a weighted joint loss of source domain labeled data and target domain high-confidence pseudo-label samples. After each training round, the pseudo-label selection criteria are gradually reduced by decreasing the first confidence threshold by a preset step size, so that more target domain samples can participate in the training.

[0060] In this embodiment, the contrastive learning loss is a supervised contrastive loss, which uses sample pairs of the same pseudo-label class in the target domain as positive sample pairs and sample pairs of different pseudo-label classes as negative sample pairs. The loss function is:

[0061] ;

[0062] in and This is the fused feature vector of positive sample pairs. Let be the fused feature vector of all negative and positive samples, and sim be the cosine similarity. This refers to the temperature parameter.

[0063] The method for merging the weights of the spatial coding branch and the temporal coding branch into a single-path inference network is as follows:

[0064] After training, the sum of the convolution kernel weights of the spatial coding branch and the convolution kernel weights of the temporal coding branch is used as the initial weights of the single-path inference network. The single-path inference network is fine-tuned in a few rounds using a small batch of labeled data from the source domain, so that the combined network weights can adapt to the unified representation of feature extraction and classification.

[0065] As a preferred embodiment of the above, the method further includes:

[0066] When there are multiple source domain datasets with different acquisition conditions, each source domain is used as the current source domain, and the data of the remaining source domains and the target domain are used as unlabeled data to perform adversarial domain alignment and contrastive learning training. After the domain alignment of each source domain is completed, the weights of the dual-branch feature extraction network are used as the initial weights for the next source domain alignment training.

[0067] like Figure 2 As shown, this embodiment also includes a cross-data domain adaptive micro-expression recognition model training system, using the method described above. The system includes:

[0068] The acquisition and preprocessing unit is used to acquire multiple micro-expression video datasets under different acquisition conditions as the source and target domains, and to perform optical flow feature extraction and facial region alignment preprocessing on the video segments in each dataset to generate a standardized optical flow feature map sequence.

[0069] The dual-branch feature extraction unit is used to jointly train the dual-branch feature extraction network with labeled data from the source domain and unlabeled data from the target domain. The dual-branch feature extraction network includes a spatial coding branch and a temporal coding branch with shared weights. The spatial coding branch extracts spatial texture features with a single-frame optical flow feature map as input, and the temporal coding branch extracts temporal motion features with a sequence of optical flow feature maps as input. The outputs of the two branches are concatenated to form a fused feature vector.

[0070] The training unit is used to perform adversarial domain alignment training on the fused feature vector through a domain discriminator, so that the fused feature vectors of the source domain and the target domain are indistinguishable in distribution; at the same time, it performs supervised classification training on the labeled data of the source domain using cross-entropy loss, and performs classification constraints on the unlabeled data of the target domain using a self-training method based on high-confidence pseudo-labels.

[0071] During the domain alignment training process, the contrastive learning loss is constructed using the fused feature vector output by the dual-branch feature extraction network of the target domain samples. Pseudo-label samples of the same category in the target domain are positive sample pairs, and samples of different categories are negative sample pairs, which brings the distance between similar features closer and pushes the distance between dissimilar features further apart.

[0072] The weight merging unit is used to merge the weights of the spatial coding branch and the temporal coding branch of the trained dual-branch feature extraction network into a single-path inference network.

[0073] Example 2:

[0074] like Figure 3 As shown, this embodiment includes a cross-data domain adaptive micro-expression recognition model training method, specifically including:

[0075] I. Dataset and Preprocessing

[0076] This embodiment uses the CASME II dataset (26 subjects, 256 samples, 5 types of micro-expressions) as the source domain and the SAMM dataset (32 subjects, 159 samples, 5 types of micro-expressions) as the target domain. Due to significant differences between the two datasets in frame rate (CASME II is 200fps, SAMM is 100fps), resolution, lighting conditions, and subject ethnicity, domain alignment is quite challenging.

[0077] Preprocessing steps: Extract a 9-frame image sequence from each micro-expression video segment, consisting of 4 frames before and after the peak frame; use the TV-L1 optical flow algorithm to calculate the horizontal optical flow component u and the vertical optical flow component v between adjacent frames, generating an 8-frame two-channel optical flow feature map sequence; use MediaPipe facial key point detection to locate 6 key points, including the center of the eyes, the tip of the nose, and the corners of the mouth, and use affine transformation to crop and align the facial region to 224×224 pixels.

[0078] II. Two-branch feature extraction network

[0079] The spatial coding branch takes a single-frame two-channel optical flow feature map (224×224×2) as input, passes it through 4 layers of 2D convolutions (Conv2D 32→64→128→256, kernel 3×3, stride 2, batch normalization + ReLU) and global average pooling, and outputs a 256-dimensional spatial feature vector. The temporal coding branch takes a sequence of 6 consecutive frames of two-channel optical flow feature maps (6×224×224×2) as input, passes it through 3 layers of 3D convolutions (Conv3D 32→64→128, kernel 3×3×3) and temporal global average pooling, and outputs a 256-dimensional temporal feature vector. The outputs of the two branches are concatenated to form a 512-dimensional fused feature vector, which is then connected to a fully connected classification layer (5 classes, Softmax).

[0080] III. Domain Adaptive Training

[0081] The domain discriminator is a 3-layer fully connected network (512→256→128→1), with a 512-dimensional fused feature vector as input and domain labels (0=source domain, 1=target domain) as output. A gradient reversal layer is inserted at the input of the domain discriminator, with the gradient reversal coefficient set to -0.3. Cross-entropy loss is used for source domain classification. The batch size is 32, with 16 samples each from the source and target domains. The Adam optimizer is used with an initial learning rate of 0.0001, and training is performed for 200 epochs.

[0082] IV. High-Confidence Pseudo-Label Self-Training

[0083] The initial confidence threshold was set to 0.95. After the first round of training, the current model was used to predict unlabeled samples in the target domain, and samples with a maximum predicted probability > 0.95 were selected as high-confidence pseudo-label samples. After each round of full training, the confidence threshold decreased by 0.05 until it reached 0.60. Starting from the initial threshold of 0.95, only 18 high-confidence samples (accounting for 12.3% of the target domain) were selected in the first round, and a total of 115 samples (accounting for 78.6%) were selected in the final round when the threshold was 0.60.

[0084] V. Supervision and Comparative Learning

[0085] Sample pairs with the same pseudo-label in the target domain are considered positive sample pairs, and sample pairs with different pseudo-labels are considered negative sample pairs. Supervised contrastive loss is used.

[0086] ;

[0087] Temperature parameter τ = 0.07. Contrast loss weight λ_cont = 0.1. After introducing contrast loss, t-SNE visualization shows a significant improvement in the separation between the five categories of target domain features.

[0088] VI. Experimental Results

[0089] Baseline (training only in the source domain, directly testing in the target domain) 42.3% 0.397 Performance degradation is significant across datasets +Adversarial domain alignment 56.8% 0.541 Improved by 14.5pp after eliminating acquisition bias. +Pseudo-label self-training 61.4% 0.591 Using unlabeled target domain data +Supervised comparative learning 63.5% 0.614 Enhance feature discriminativeness Complete method of the present invention 67.2% 0.648 After weighting: 67.0% / 0.645

[0090] VII. Weighting

[0091] After training, the weights of the four convolutional kernels of the 2D-CNN and the three convolutional kernels of the 3D-CNN were weighted and summed at the corresponding layers, and then merged into a single-path inference network (7 convolutional layers). After 5 rounds of fine-tuning with 32 samples from the source domain, the accuracy of the merged network was 67.0% (a decrease of 0.3 percentage points from the original 67.2%), the number of parameters decreased from 3.6M to 2.2M (a reduction of 38.9%), and the inference time decreased from 12ms to 7.5ms (an improvement of 60%).

[0092] 8. Multi-source domain cascade training

[0093] Building upon the aforementioned approach, the source domain was expanded to include two datasets (CASME II and SMIC). The SMIC dataset contains 164 samples (3 categories of micro-expressions) at a frame rate of 100fps. The cascaded training steps were as follows: The first stage used CASME II as the source domain and SMIC+SAMM as unlabeled data to train until convergence; the second stage used the network weights trained in the previous stage as initialization, and continued training with SMIC as the source domain and CASME II+SAMM as unlabeled data. The average accuracy across the three cross-dataset tests (source domain → target domain) was 64.3%, an improvement of 1.8 percentage points compared to 62.5% for single-source domain training. The accuracy improvement was greatest for SMIC → CASME II (from 57.8% to 61.2%), indicating that multi-source domain cascaded training is more effective for domain alignment between datasets with significant frame rate differences.

[0094] This embodiment has the following beneficial effects:

[0095] First, adversarial domain alignment effectively eliminated the sampling bias across datasets. In the cross-dataset test from CASME II (source domain) to SAMM (target domain), the accuracy of the baseline model without domain alignment was 42.3%, which improved to 56.8% after introducing adversarial domain alignment, an improvement of 14.5 percentage points.

[0096] Second, supervised contrastive learning further improved the discriminativeness of target domain features. After introducing supervised contrastive loss on top of domain alignment, the target domain test accuracy increased from 56.8% to 63.5%, an improvement of 6.7 percentage points. t-SNE visualization results show that after introducing contrastive learning, the spacing between different categories of target domain features significantly increased, while inter-class overlap significantly decreased.

[0097] Third, high-confidence pseudo-label self-training effectively utilized unlabeled data in the target domain. With the target domain initially unlabeled, a strategy was employed to gradually utilize target domain samples, starting with a confidence threshold of 0.95 and decreasing by 0.05 each round. This ultimately increased the proportion of effectively utilized target domain samples from the initial 12.3% to 78.6%, and further improved the accuracy from 63.5% to 67.2%.

[0098] Fourth, by merging weights, the dual-branch network is transformed into a single-path inference network, reducing the number of inference parameters by about 38%, increasing the inference speed by about 1.6 times, while the accuracy only decreases by 0.3 percentage points, achieving a good balance between accuracy and efficiency.

[0099] Fifth, the overall method achieved an average accuracy of 65.8% and an average F1-score of 0.637 in three cross-dataset tests on three public datasets (CASME II, SAMM, and SMIC), which is significantly better than existing methods (average accuracy of 52.3% and average F1-score of 0.508).

[0100] Example 3:

[0101] Please see Figure 4 The diagram shows a structural schematic of a computer device provided in an embodiment of this application. An embodiment of this application provides a computer device 400, including a processor 410 and a memory 420. The memory 420 stores a computer program executable by the processor 410. When the computer program is executed by the processor 410, it performs the method described above.

[0102] This application embodiment also provides a storage medium 430, on which a computer program is stored, and the computer program is executed by a processor 410 to perform the above method.

[0103] The storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0104] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0105] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0106] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0107] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0109] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0110] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0111] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A cross-data domain adaptive micro-expression recognition model training method, characterized in that, Includes the following steps: Multiple micro-expression video datasets under different acquisition conditions were acquired as the source and target domains. Optical flow feature extraction and facial region alignment preprocessing were performed on video segments in each dataset to generate a standardized optical flow feature map sequence. A dual-branch feature extraction network is jointly trained using labeled data from the source domain and unlabeled data from the target domain. The dual-branch feature extraction network includes a spatial coding branch and a temporal coding branch with shared weights. The spatial coding branch extracts spatial texture features using a single-frame optical flow feature map as input, and the temporal coding branch extracts temporal motion features using a sequence of optical flow feature maps as input. The outputs of the two branches are concatenated to form a fused feature vector. Adversarial domain alignment training is performed on the fused feature vectors by a domain discriminator, making the fused feature vectors of the source domain and the target domain indistinguishable in distribution; at the same time, supervised classification training is performed on the labeled data of the source domain using cross-entropy loss, and classification constraints are performed on the unlabeled data of the target domain using a self-training method based on high-confidence pseudo-labels. During the domain alignment training process, the contrastive learning loss is constructed using the fused feature vector output by the dual-branch feature extraction network of the target domain samples. Pseudo-label samples of the same category in the target domain are positive sample pairs, and samples of different categories are negative sample pairs, thereby narrowing the distance between similar features and widening the distance between dissimilar features. The weights of the spatial coding branch and the temporal coding branch of the trained dual-branch feature extraction network are merged into a single-path inference network.

2. The method according to claim 1, characterized in that, The optical flow feature extraction uses the TV-L1 optical flow algorithm to calculate the horizontal and vertical optical flow components between adjacent frames, and then superimposes them into a two-channel optical flow feature map. The facial region alignment involves locating the eye, nose, and mouth regions using facial key point detection, and then normalizing the facial image to a uniform size and pose using affine transformation.

3. The method according to claim 1, characterized in that, The spatial coding branch is a 2D convolutional neural network that takes a single-frame two-channel optical flow feature map as input, extracts spatial texture features through four convolutional layers and outputs a 256-dimensional spatial feature vector. The temporal coding branch is a 3D convolutional neural network. It takes a sequence of six consecutive frames of two-channel optical flow feature maps as input, extracts temporal motion features through three 3D convolutional layers and one temporal pooling layer, and outputs a 256-dimensional temporal feature vector. The fused feature vector is 512-dimensional.

4. The method according to claim 1, characterized in that, The adversarial domain alignment training is achieved through a gradient inversion layer: A gradient inversion layer is inserted at the input of the domain discriminator. The gradient inversion layer does not perform any transformation during forward propagation, but multiplies the gradient by a preset negative coefficient and propagates it back during backward propagation. This causes the dual-branch feature extraction network to update its parameters in the direction of increasing the domain discrimination loss, while the domain discriminator updates its parameters in the direction of decreasing the domain discrimination loss.

5. The method according to claim 1, characterized in that, The self-training method based on high-confidence pseudo-labels is as follows: The preliminary classifier trained in the source domain is used to predict unlabeled samples in the target domain, and samples with a maximum predicted probability higher than the first confidence threshold are selected as high-confidence pseudo-label samples and added to the labeling pool. The dual-branch feature extraction network is iteratively retrained using a weighted joint loss of the source domain labeled data and the target domain high-confidence pseudo-label samples. After each training round, the pseudo-label screening criteria are gradually reduced by a preset step size of decreasing the first confidence threshold, so that more target domain samples can participate in the training.

6. The method according to claim 1, characterized in that, The contrastive learning loss is a supervised contrastive loss, which uses sample pairs of the same pseudo-label class in the target domain as positive sample pairs and sample pairs of different pseudo-label classes as negative sample pairs. The loss function is: ； in and This is the fused feature vector of positive sample pairs. Let be the fused feature vector of all negative and positive samples, and sim be the cosine similarity. This refers to the temperature parameter.

7. The method according to claim 1, characterized in that, The method for merging the weights of the spatial coding branch and the temporal coding branch into a single-path inference network is as follows: After training, the sum of the convolution kernel weights of the spatial coding branch and the convolution kernel weights of the temporal coding branch is used as the initial weights of the single-path inference network. The single-path inference network is then fine-tuned in a few rounds using a small batch of labeled data from the source domain, so that the merged network weights are adapted to the unified representation of feature extraction and classification.

8. The method according to claim 1, characterized in that, The method further includes: When there are multiple source domain datasets with different acquisition conditions, the adversarial domain alignment and contrastive learning training is performed sequentially, with each source domain as the current source domain and the data of the remaining source domains and the target domain as unlabeled data. After each source domain alignment is completed, the weights of the dual-branch feature extraction network are used as the initial weights for the next source domain alignment training.

9. A cross-data domain adaptive micro-expression recognition model training system, characterized in that, Using the method of any one of claims 1 to 8, the system comprises: The acquisition and preprocessing unit is used to acquire multiple micro-expression video datasets under different acquisition conditions as the source and target domains, and to perform optical flow feature extraction and facial region alignment preprocessing on the video segments in each dataset to generate a standardized optical flow feature map sequence. A dual-branch feature extraction unit is used to jointly train a dual-branch feature extraction network with labeled data from the source domain and unlabeled data from the target domain. The dual-branch feature extraction network includes a spatial coding branch and a temporal coding branch with shared weights. The spatial coding branch extracts spatial texture features with a single-frame optical flow feature map as input, and the temporal coding branch extracts temporal motion features with a sequence of optical flow feature maps as input. The outputs of the two branches are concatenated to form a fused feature vector. The training unit is used to perform adversarial domain alignment training on the fused feature vector through a domain discriminator, so that the fused feature vectors of the source domain and the target domain are indistinguishable in distribution; at the same time, it performs supervised classification training on the labeled data of the source domain using cross-entropy loss, and performs classification constraints on the unlabeled data of the target domain using a self-training method based on high-confidence pseudo-labels. During the domain alignment training process, the contrastive learning loss is constructed using the fused feature vector output by the dual-branch feature extraction network of the target domain samples. Pseudo-label samples of the same category in the target domain are positive sample pairs, and samples of different categories are negative sample pairs, thereby narrowing the distance between similar features and widening the distance between dissimilar features. The weight merging unit is used to merge the weights of the spatial coding branch and the temporal coding branch of the trained dual-branch feature extraction network into a single-path inference network.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-8.

11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images