Sample adaptive micro-expression magnification method and apparatus

Abstract

The application discloses a micro-expression amplification method and device based on sample self-adaption, which comprises the following steps: (1) obtaining a micro-expression database; (2) converting the micro-expression video into a micro-expression frame sequence and unifying the sequence length; (3) constructing a micro-expression amplification model, wherein the micro-expression amplification model comprises an intensity extraction module, an intensity change curve construction module, an intensity correction module, a face enhancement module and a micro-expression feature integration module; (4) inputting the micro-expression frame sequence with unified length and corresponding micro-expression category labels as samples into the micro-expression amplification model for training, wherein the loss function used in the training comprises a micro-expression feature vector cross-entropy loss, an intensity extraction loss and an intensity correction loss; and (5) inputting the micro-expression video to be amplified after length unification into the trained micro-expression amplification model to obtain an amplified micro-expression feature vector. The application can adaptively amplify each micro-expression video, and the effect is better.

Description

Technical Field

[0001] This invention relates to micro-expression recognition technology, and more particularly to a sample-adaptive micro-expression amplification method and apparatus. Background Technology

[0002] Microexpressions (ME) are spontaneous, fleeting facial expressions, often caused by the conscious concealment of true emotional states. Compared to macroexpressions, microexpressions are localized, low in intensity, and short in duration (less than 0.5 seconds), making them difficult to detect and identify. These characteristics make accurately classifying microexpressions extremely challenging. However, microexpression recognition plays an indispensable role in many fields, including health diagnosis, homeland security, and criminal interrogation. These potential applications have attracted increasing numbers of researchers to this field, leading to the development of numerous methods to improve the performance of microexpression recognition.

[0003] Among these methods, convolutional neural networks (CNNs) are widely used to extract discriminative micro-expression features. However, the low intensity of micro-expressions makes the representation of motion features difficult. To address this issue, many methods employ motion amplification techniques to amplify the intensity of micro-expressions, making their motion more pronounced. Some traditional amplification techniques and deep learning filters, such as Euler Motion Amplification (EMM), Global Lagrangian Motion Amplification (GLMM), and Learning-Based Motion Amplification (LMM), have shown that increasing the intensity of micro-expressions helps improve recognition accuracy.

[0004] However, because the changes in facial muscles differ among subjects exhibiting different expressions, a uniform magnification strategy cannot be suitable for all micro-expression samples. For example, setting the same magnification factor for one sample may be insufficient to extract discriminative features, while setting an excessive magnification factor for another object may introduce noise. Therefore, finding a uniform magnification factor for all micro-expression samples in a database is difficult to achieve. Furthermore, micro-expressions are characterized by localized facial movements, so some actions may be irrelevant to identifying specific micro-expression categories, such as blinking or head shaking. These limitations necessitate the search for a magnification method suitable for each specific micro-expression to improve recognition accuracy. Summary of the Invention

[0005] Purpose of the invention: This invention addresses the problems existing in the prior art by providing a method and apparatus for micro-expression magnification that can adaptively magnify each micro-expression.

[0006] Technical solution: The sample-adaptive micro-expression amplification method of the present invention includes:

[0007] (1) Obtain a micro-expression database, which contains several micro-expression videos and corresponding micro-expression category tags;

[0008] (2) Convert the micro-expression video into a micro-expression frame sequence and unify the sequence length;

[0009] (3) Constructing a micro-expression magnification model, wherein the micro-expression magnification model includes:

[0010] The intensity extraction module is used to extract intensity values ​​by utilizing the intensity differences between micro-expression frame sequences through comparative learning.

[0011] The intensity change curve construction module is used to construct intensity change curves for micro-expression frame sequences based on Gaussian distribution;

[0012] An intensity correction module is used to correct the intensity values ​​extracted by the intensity extraction module according to the trend of the intensity change curve using a rank-sum test.

[0013] The face enhancement module is used to integrate the corrected intensity values ​​with the facial features extracted from the micro-expression frame sequence to obtain an enhanced facial feature vector;

[0014] The micro-expression feature integration module is used to integrate the enhanced facial feature vectors of all frames belonging to a micro-expression video using LSTM, as the amplified micro-expression feature vector;

[0015] (4) Use a sequence of micro-expression frames of uniform length and the corresponding micro-expression category labels as samples to train the micro-expression amplification model. The loss functions used during training include cross-entropy loss of micro-expression feature vectors, loss of intensity extraction, and loss of intensity correction.

[0016] (5) After unifying the length of the micro-expression video to be magnified, input it into the trained micro-expression magnification model to obtain the magnified micro-expression feature vector.

[0017] Furthermore, the process of unifying the sequence length in step (2) specifically includes:

[0018] N frames are extracted from each micro-expression frame sequence at preset intervals to form a micro-expression frame sequence of length N, where N is a positive integer.

[0019] Furthermore, the extraction of intensity values ​​through contrastive learning, utilizing the intensity differences between micro-expression frame sequences, specifically includes:

[0020] (3-1-1) For each micro-expression frame sequence after unification, each frame x is used as the source sample, and data augmentation is applied to x to generate two new images. and As positive samples of x, select M images from the remaining frames of the sequence excluding x, denoted as Mx. As a negative sample of x, M is a positive integer less than N, where N represents the length of the micro-expression frame sequence;

[0021] (3-1-2) Convert the positive samples of each frame and negative samples The inputs are sequentially fed into a ResNet network and a fully connected (FC) linear layer to obtain the feature vectors. and

[0022] (3-1-3) The feature vector of each frame Input a linear layer and obtain the corresponding intensity value;

[0023] (3-1-4) The loss for intensity extraction is calculated using the following formula, thereby optimizing the extraction of intensity information during training:

[0024]

[0025] In the formula, i represents the frame index, D represents the number of micro-expression videos in the micro-expression database, and k represents the index of the micro-expression video. These represent the feature vectors of the i-th frame after the k-th micro-expression video in the micro-expression database has been standardized in length.

[0026] Furthermore, the intensity change curve of the micro-expression frame sequence constructed based on the Gaussian distribution specifically includes:

[0027] (3-2-1) Set the range of intensity variation of the micro-expression frame sequence to [∈, 1], where ∈ > 0;

[0028] (3-2-2) Divide the micro-expression frame sequence into two segments with the peak frame as the boundary. The segment from the start frame to the peak frame is called the On-A segment, and the segment from the peak frame to the end frame is called the A-Off segment. The On-A segment contains N1 frames, and the A-Off segment contains N2 frames. N1+N2+1=N, where N represents the length of the micro-expression frame sequence.

[0029] (3-2-3) Set the intensity value of each frame on the On-A segment to follow the following Gaussian distribution:

[0030]

[0031] The intensity values ​​for each frame in the A-Off segment are set to follow the following Gaussian distribution:

[0032]

[0033] The peak frame is set to follow the following Gaussian distribution:

[0034]

[0035] In the formula, i represents the frame index. This represents a Gaussian distribution with mean # and variance *.

[0036] (3-2-4) Construct the intensity change curve of the micro-expression frame sequence according to the Gaussian distribution of the intensity values ​​in step (3-2-3).

[0037] Furthermore, the method of using the rank-sum test to correct the intensity values ​​extracted from the micro-expression video sequence according to the trend of the intensity change curve specifically includes:

[0038] (3-3-1) After sampling the intensity change curve according to the frame order, the intensity value sequence S1=(q1,...,q N ), and obtain the intensity value sequence S2 = (k1, ..., k) extracted by the intensity extraction module. N ), k * q * Let S1 and S2 represent the intensity values ​​of the *th frame in the micro-expression frame sequence, respectively, and N represent the length of the micro-expression frame sequence. A null hypothesis is established: the intensity value sequences S1 and S2 have the same distribution, with a significance level of α = 0.05.

[0039] (3-3-2) After mixing the two intensity value sequences S1 and S2, obtain the order of each element in the mixed sequence, i.e., the rank r. i q ,

[0040] r i q =rank of q i among (q1, ..., q) N k1, ..., k N )

[0041]

[0042] (3-3-3) Calculate the sum of the ranks of each element in the intensity value sequence S1, denoted as T1; calculate the sum of the ranks of each element in S2, denoted as T2;

[0043] (3-3-4) Calculate the test statistic Z of the intensity value sequence S2, and the probability p corresponding to this Z value in the standard normal distribution, according to the following formula:

[0044]

[0045]

[0046] In the formula,

[0047] (3-3-5) The loss for intensity correction is calculated using the following formula, thereby achieving trend correction of the intensity value sequence S2 towards the intensity change curve during training:

[0048]

[0049] In the formula, The value represents the loss due to intensity correction, D represents the number of micro-expression videos in the micro-expression database, and p k Let p represent the probability corresponding to the k-th micro-expression video, and ξ represent the hyperparameter with a value of 0.1.

[0050] Furthermore, the integration of the corrected intensity value with the facial features extracted from the micro-expression frame sequence specifically involves:

[0051] (3-4-1) Extend each corrected intensity value into an intensity vector;

[0052] (3-4-2) Input each frame of the micro-expression frame sequence into a Resnet+FC linear layer network to extract the facial feature vector of each frame;

[0053] (3-4-3) Multiply the intensity vector and the corresponding frame's facial feature vector element by element to obtain the enhanced facial feature vector after the expression intensity is enhanced.

[0054] Furthermore, the loss function used during training in step (4) is as follows:

[0055]

[0056] in

[0057]

[0058] λ1+λ2+λ3=1

[0059] The total loss is given by λ1, λ2, and λ3, which are the weights corresponding to the three loss functions, respectively. This represents the loss in intensity extraction. Indicates the loss due to intensity correction. This represents the cross-entropy loss of micro-expression feature vectors, where K is the total number of training samples, C is the total number of micro-expression categories, and y kc p is the label value corresponding to the k-th sample of category c. kc Let it be the probability that it belongs to category c.

[0060] The sample-adaptive micro-expression magnification device of the present invention includes a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, it implements the above-described method.

[0061] Beneficial Effects: Compared with existing technologies, the significant advantages of this invention are: it abandons the common practice in existing micro-expression recognition technologies of amplifying expression intensity in image space, and provides a new amplification approach. The purpose of this invention is to enable the amplified information to adaptively adjust according to different samples and dynamically change as the network is optimized. Specifically, this invention extracts intensity information through contrastive learning, and then corrects the change in intensity information through a rank-sum test, thereby achieving intensity information representation in feature space. Experimental results demonstrate that this invention can improve recognition accuracy, reduce manual intervention, and increase adjustability compared to existing amplification methods. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating an embodiment of the sample adaptive micro-expression amplification method provided by the present invention;

[0063] Figure 2 This is a framework diagram of the entire design process of this invention;

[0064] Figure 3 This is a schematic diagram of intensity information extraction;

[0065] Figure 4 This is a schematic diagram of intensity information correction using the rank-sum test. Detailed Implementation

[0066] This embodiment provides a sample-adaptive micro-expression amplification method, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0067] (1) Obtain a micro-expression database, which contains several micro-expression videos and corresponding micro-expression category tags.

[0068] (2) Convert the micro-expression video into a micro-expression frame sequence and unify the sequence length.

[0069] Since each micro-expression video sample has a different number of frames when it is acquired, in order to facilitate network recognition, the length of all videos needs to be unified. Referring to the methods of most video preprocessing, N=16 frames are uniformly selected from each micro-expression video sample. The selection method is to extract frames from the sample at certain intervals according to the length of the video. For example, if the original video has N0 frames, frames are selected at intervals of N0 / 16 to ensure that frames are selected at each stage of the video.

[0070] (3) Construct a micro-expression amplification model, which includes an intensity extraction module, an intensity change curve construction module, an intensity correction module, a face enhancement module, and a micro-expression feature integration module.

[0071] The intensity extraction module utilizes the intensity differences between micro-expression frame sequences to extract intensity values ​​through contrastive learning. For example... Figure 3 As shown, it specifically includes:

[0072] (3-1-1) For each micro-expression frame sequence after unification, each frame x is used as the source sample, and data augmentation is applied to x to generate two new images. and As positive samples of x, select M images from the remaining frames of the sequence excluding x, denoted as Mx. As a negative sample of x, M is a positive integer less than N, where N represents the length of the micro-expression frame sequence;

[0073] (3-1-2) Convert the positive samples of each frame and negative samples The inputs are sequentially fed into a ResNet network and a fully connected (FC) linear layer to obtain the feature vectors. and

[0074] (3-1-3) The feature vector of each frame Input a linear layer and obtain the corresponding intensity value;

[0075] (3-1-4) The loss for intensity extraction is calculated using the following formula, thereby optimizing the extraction of intensity information during training:

[0076]

[0077] In the formula, i represents the frame index, D represents the number of micro-expression videos in the micro-expression database, and k represents the index of the micro-expression video. These represent the feature vectors of the i-th frame after the k-th micro-expression video in the micro-expression database has been standardized in length.

[0078] The intensity variation curve construction module is used to construct intensity variation curves for micro-expression frame sequences based on a Gaussian distribution. In the original micro-expression video sequences, the intensity typically follows a trend of first increasing and then decreasing, a fact confirmed by numerous studies on micro-expressions. Based on this trend, a variation curve is constructed for each micro-expression sequence. The specific values ​​of the curves are not important; what matters is simulating the trend of first increasing and then decreasing. Therefore, the construction method specifically includes:

[0079] (3-2-1) Set the range of intensity variation of the micro-expression frame sequence to [∈, 1], where ∈ > 0;

[0080] (3-2-2) Divide the micro-expression frame sequence into two segments with the peak frame as the boundary. The segment from the start frame to the peak frame is called the On-A segment, and the segment from the peak frame to the end frame is called the A-Off segment. The On-A segment contains N1 frames, and the A-Off segment contains N2 frames. N1+N2+1=N, where N represents the length of the micro-expression frame sequence.

[0081] (3-2-3) Set the intensity value of each frame on the On-A segment to follow the following Gaussian distribution:

[0082]

[0083] The intensity values ​​for each frame in the A-Off segment are set to follow the following Gaussian distribution:

[0084]

[0085] The peak frame is set to follow the following Gaussian distribution:

[0086]

[0087] In the formula, i represents the frame index. This represents a Gaussian distribution with mean # and variance *.

[0088] (3-2-4) Construct the intensity change curve of the micro-expression frame sequence according to the Gaussian distribution of the intensity values ​​in step (3-2-3).

[0089] The intensity correction module uses a rank-sum test to correct the intensity values ​​extracted by the intensity extraction module according to the trend of the intensity change curve. For example... Figure 4 As shown, it specifically includes:

[0090] (3-3-1) After sampling the intensity change curve according to the frame order, the intensity value sequence S1=(q1,...,q N ), and obtain the intensity value sequence S2 = (k1, ..., k) extracted by the intensity extraction module. N ), k * q * Let S1 and S2 represent the intensity values ​​of the *th frame in the micro-expression frame sequence, respectively, and N represent the length of the micro-expression frame sequence. A null hypothesis is established: the intensity value sequences S1 and S2 have the same distribution, with a significance level of α = 0.05.

[0091] (3-3-2) After mixing the two intensity value sequences S1 and S2, obtain the order of each element in the mixed sequence, i.e., the rank r. i q ,

[0092] ri q =rank of q i among (q1, ..., q) N k1, ..., k N )

[0093]

[0094] (3-3-3) Calculate the sum of the ranks of each element in the intensity value sequence S1, denoted as T1; calculate the sum of the ranks of each element in S2, denoted as T2;

[0095] (3-3-4) Calculate the test statistic Z of the intensity value sequence S2, and the probability p corresponding to this Z value in the standard normal distribution, according to the following formula:

[0096]

[0097]

[0098] In the formula,

[0099] The calculation of the Z-value mainly utilizes the principle that, when the number of samples in the sequence N > 10, the total rank and T follow a Gaussian distribution as follows:

[0100]

[0101] (3-3-5) The loss for intensity correction is calculated using the following formula, thereby achieving trend correction of the intensity value sequence S2 towards the intensity change curve during training:

[0102]

[0103] In the formula, The value represents the loss due to intensity correction, D represents the number of micro-expression videos in the micro-expression database, and p k Let p represent the probability corresponding to the k-th micro-expression video, and ξ represent the hyperparameter with a value of 0.1.

[0104] The face enhancement module integrates the corrected intensity values ​​with the facial features extracted from the micro-expression frame sequence to obtain an enhanced facial feature vector.

[0105] The micro-expression feature integration module uses LSTM to integrate the enhanced facial feature vectors from all frames belonging to a micro-expression video, resulting in an amplified micro-expression feature vector. Specifically, it includes:

[0106] (3-4-1) Extend each corrected intensity value into an intensity vector;

[0107] (3-4-2) Input each frame of the micro-expression frame sequence into a Resnet+FC linear layer network to extract the facial feature vector of each frame;

[0108] (3-4-3) Multiply the intensity vector and the corresponding frame's facial feature vector element by element to obtain the enhanced facial feature vector after the expression intensity is enhanced.

[0109] (4) Use micro-expression frame sequences of uniform length and corresponding micro-expression category labels as samples to train the micro-expression amplification model. The loss functions used during training include cross-entropy loss of micro-expression feature vectors, loss of intensity extraction, and loss of intensity correction.

[0110] The loss function used is as follows:

[0111]

[0112] in

[0113]

[0114] λ1+λ2+λ3=1

[0115] The total loss is given by λ1, λ2, and λ3, which are the weights corresponding to the three loss functions, respectively. This represents the loss in intensity extraction. Indicates the loss due to intensity correction. This represents the cross-entropy loss of micro-expression feature vectors, where K is the total number of training samples, C is the total number of micro-expression categories, and y kc p is the label value corresponding to the k-th sample of category c. kc Let it be the probability that it belongs to category c.

[0116] (5) After unifying the length of the micro-expression video to be magnified, input it into the trained micro-expression magnification model to obtain the magnified micro-expression feature vector.

[0117] This embodiment also provides a sample-adaptive micro-expression recognition device, including a processor and a computer program stored in a memory and executable on the processor, wherein the processor implements the above-described method when executing the program.

[0118] To verify the effectiveness of this invention, micro-expression recognition was performed on the CAME2 micro-expression database, SAMM micro-expression database, and SMIC-HS database. The verification results and comparison with other state-of-the-art methods are shown in Table 1.

[0119] Table 1

[0120] Method Year CASME2 SAMM SMIC-HS LBP-SIP 2014 45.36 36.76 42.12 MagGA 2018 63.30 N / A N / A DSSN 2019 70.78 57.35 63.41 TSCNN-I 2020 74.05 63.53 72.74 LBPAccP u2 ]] 2021 69.03 N / A 76.59 AU-GCN 2021 74.27 74.26 N / A The method 2022 78.05 78.68 79.27

[0121] In Table 1, N / A indicates that there are no relevant records.

[0122] The facial expressions in the CASME2 database are processed as follows: only categories with more than 10 samples are retained, namely happy, regression, disgust, fear, and surprise. Similarly, for the SAMM database, only categories with more than 10 samples are retained, namely happiness, angry, disgust, fear, and surprise. The SMIC database has categories positive, negative, and surprise, so a three-class classification task is performed on this database.

[0123] Experimental results show that the micro-expression recognition method proposed in this invention has good micro-expression recognition performance. Compared with traditional micro-expression amplification methods, this invention is better able to adapt to the differences between different micro-expression samples and is also more convenient to implement.

[0124] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A sample-adaptive micro-expression amplification method, characterized in that... The method includes: (1) Obtain a micro-expression database, which contains several micro-expression videos and corresponding micro-expression category tags; (2) Convert the micro-expression video into a micro-expression frame sequence and unify the sequence length; (3) Construct a micro-expression magnification model, wherein the micro-expression magnification model includes: The intensity extraction module is used to extract intensity values ​​by utilizing the intensity differences between micro-expression frame sequences through comparative learning. The intensity change curve construction module is used to construct intensity change curves for micro-expression frame sequences based on Gaussian distribution; An intensity correction module is used to correct the intensity values ​​extracted by the intensity extraction module according to the trend of the intensity change curve using a rank-sum test. The face enhancement module is used to integrate the corrected intensity values ​​with the facial features extracted from the micro-expression frame sequence to obtain an enhanced facial feature vector; The micro-expression feature integration module is used to integrate the enhanced facial feature vectors of all frames belonging to a micro-expression video using LSTM, as the amplified micro-expression feature vector; (4) Use a sequence of micro-expression frames of uniform length and the corresponding micro-expression category labels as samples to train the micro-expression amplification model. The loss functions used during training include cross-entropy loss of micro-expression feature vectors, loss of intensity extraction, and loss of intensity correction. (5) After unifying the length of the micro-expression video to be magnified, input it into the trained micro-expression magnification model to obtain the magnified micro-expression feature vector; The method of extracting intensity values ​​by utilizing the intensity differences between micro-expression frame sequences through contrastive learning specifically includes: (3-1-1) For each unified micro-expression frame sequence, each frame As the source sample, for Two new images were generated using data augmentation. and As Positive samples, from the sequence divided Select from the remaining frames outside One picture, recorded as As The negative samples, M is a positive integer less than N, and N represents the length of the micro-expression frame sequence; (3-1-2) Take the positive samples of each frame and negative samples The inputs are sequentially fed into a ResNet network and a fully connected (FC) linear layer to obtain the feature vectors. and ; (3-1-3) The feature vector of each frame , Input a linear layer and obtain the corresponding intensity value; (3-1-4) The loss for intensity extraction is calculated using the following formula, thereby optimizing the extraction of intensity information during training: ， In the formula, Here, D represents the number of micro-expression videos in the micro-expression database, and k represents the index of the micro-expression video. These represent the feature vectors of the i-th frame after the k-th micro-expression video in the micro-expression database has been standardized in length. .

2. The sample-adaptive micro-expression amplification method according to claim 1, characterized in that: The process of unifying the sequence length in step (2) specifically includes: N frames are extracted from each micro-expression frame sequence at preset intervals to form a micro-expression frame sequence of length N, where N is a positive integer.

3. The sample-adaptive micro-expression amplification method according to claim 1, characterized in that: The intensity change curves of the micro-expression frame sequence constructed based on the Gaussian distribution specifically include: (3-2-1) Set the range of intensity variation of the micro-expression frame sequence within ,in ; (3-2-2) Divide the micro-expression frame sequence into two segments using the peak frame as the boundary. The segment from the start frame to the peak frame is denoted as On-A, and the segment from the peak frame to the end frame is denoted as A-Off. The On-A segment contains Frame, A-Off segment contains frame, , N Indicates the length of the micro-expression frame sequence; (3-2-3) Set the intensity value of each frame on the On-A segment to follow the following Gaussian distribution: ， The intensity values ​​for each frame in the A-Off segment are set to follow the following Gaussian distribution: ， The peak frame is set to follow the following Gaussian distribution: ， In the formula, i Indicates the frame index. This represents a Gaussian distribution with mean # and variance *. (3-2-4) Construct the intensity change curve of the micro-expression frame sequence according to the Gaussian distribution of the intensity value in step (3-2-3).

4. The sample-adaptive micro-expression amplification method according to claim 1, characterized in that: The method of using the rank-sum test to correct the intensity values ​​extracted from the micro-expression video sequence according to the trend of the intensity change curve specifically includes: (3-3-1) The intensity value sequence is obtained by sampling the intensity change curve according to the frame order. And obtain the intensity value sequence extracted by the intensity extraction module. , , Let N represent the intensity value of the *th frame in the micro-expression frame sequence, and let N represent the length of the micro-expression frame sequence. A null hypothesis is established: the intensity value sequence... and intensity value sequence Having the same distribution, significance level =0.05; (3-3-2) Combine two intensity value sequences After mixing, obtain the intensity value sequence. The order of each element in the mixed sequence, i.e., the rank , : ， ， (3-3-3) Calculate the intensity value sequence The sum of the rank of each element in the set is denoted as . ;calculate The sum of the rank of each element in the set is denoted as . ; (3-3-4) Calculate the intensity value sequence according to the following formula. Test statistic and the The probability corresponding to the value in the standard normal distribution : ， ， In the formula, , ; (3-3-5) The loss for intensity correction is calculated using the following formula, thereby realizing the intensity value sequence during training. Gradually correct towards the trend of the intensity change curve: ， In the formula, This represents the loss due to intensity correction, where D represents the number of micro-expression videos in the micro-expression database. This represents the probability corresponding to the k-th micro-expression video. , This represents a hyperparameter, with a value of 0.

1.

5. The sample-adaptive micro-expression amplification method according to claim 1, characterized in that: The specific steps of integrating the corrected intensity value with the facial features extracted from the micro-expression frame sequence are as follows: (3-4-1) Extend each corrected intensity value into an intensity vector; (3-4-2) Input each frame of the micro-expression frame sequence into a ResNet + FC linear layer network to extract the facial feature vector of each frame; (3-4-3) Multiply the intensity vector and the corresponding facial feature vector of the frame element by element to obtain the enhanced facial feature vector after the expression intensity is enhanced.

6. The sample-adaptive micro-expression amplification method according to claim 1, characterized in that: The loss function used during training in step (4) is as follows: ， in ， For the total loss, These are the weights corresponding to the three loss functions. This represents the loss in intensity extraction. Indicates the loss due to intensity correction. This represents the cross-entropy loss of micro-expression feature vectors. The total number of training samples. This represents the total number of micro-expression categories. For category The The label value corresponding to each sample It belongs to the category The probability of.

7. A sample-adaptive micro-expression magnification device, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, it implements the method described in any one of claims 1-6.

Images