Micro-expression analysis method and system based on adaptive pseudo mark and attention mechanism

Abstract

The invention relates to the field of image processing, provides a micro-expression analysis method and system based on an adaptive pseudo mark and an attention mechanism, and aims to solve the problem of large pseudo mark error in an existing method by designing an adaptive pseudo mark algorithm and a micro-expression analysis network based on a fusion attention mechanism. According to the self-adaptive pseudo-marking algorithm, self-adaptive marking is carried out by combining a sliding window based on the proportion of the micro-expressions in the whole video interval, so that marking errors are reduced, the micro-expression analysis accuracy is improved, and the method can be used for solving the problem that subtle changes of the micro-expressions are difficult to accurately capture in an existing method. According to the method, a micro-expression analysis network based on a fusion attention mechanism is designed to seamlessly complete micro-expression positioning and recognition tasks, the micro-expression analysis network adopts a double-layer convolutional neural network to extract features, a double attention mechanism is introduced, channel and space information is captured respectively, significant features are focused, and the positioning accuracy of the micro-expression is improved. The accuracy of micro-expression positioning and recognition is improved, and the accuracy of micro-expression analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a micro-expression analysis method and system based on adaptive pseudo-labeling and attention mechanism. Background Art

[0002] Micro-expressions refer to unconscious facial expressions that last between 1 / 25 and 1 / 3 of a second and are produced by humans in specific situations. They often contain real emotions that people try hard to conceal, are non-deceptive, and are a form of non-verbal communication. However, the accuracy rate of professionals who have been trained in micro-expression recognition in effectively identifying micro-expressions is only less than 47%. Therefore, highly accurate micro-expression analysis methods are crucial.

[0003] In the existing technology, the core of micro-expression analysis is to accurately capture and interpret the true emotions of an individual, and the accuracy and effectiveness of pseudo-labeling are crucial to this process. However, labeling micro-expressions frame by frame is not only an extremely time-consuming and labor-intensive task, but also inefficient in actual operation and difficult to meet the needs of large-scale data analysis. Since existing micro-expression analysis methods usually only focus on the recognition of a single micro-expression frame, while ignoring the distribution ratio and continuity characteristics of micro-expressions in the entire time series, the pseudo-labeling results often cannot fully and accurately reflect the true situation of micro-expressions. At the same time, because micro-expressions appear for a short time and change slightly, they are difficult to accurately locate and identify, further affecting the accuracy of the analysis.

[0004] Therefore, how to design a micro-expression analysis method to improve the accuracy of micro-expression positioning and recognition has become an urgent problem to be solved. Summary of the Invention

[0005] Based on this, the present invention proposes a micro-expression analysis method and system based on adaptive pseudo-labeling and attention mechanism. In order to more accurately and efficiently locate micro-expression intervals and identify emotional categories in videos, an adaptive pseudo-labeling algorithm and a micro-expression analysis network based on a fusion attention mechanism are designed. For the problem of large pseudo-labeling errors in existing methods, the adaptive pseudo-labeling algorithm is based on the proportion of micro-expressions in the entire video interval and is combined with a sliding window for adaptive labeling to reduce labeling errors and improve the accuracy of micro-expression analysis. For the problem that existing methods are difficult to accurately capture subtle changes in micro-expressions, the present invention designs a micro-expression analysis network based on a fusion attention mechanism to seamlessly complete micro-expression positioning and recognition tasks, including a shared subnetwork, a positioning subnetwork and an identification subnetwork. The micro-expression analysis network uses a two-layer convolutional neural network to extract features and introduces a dual attention mechanism to capture channel and spatial information respectively, focus on significant features, and improve the accuracy of micro-expression positioning and recognition. The present invention improves the accuracy of micro-expression analysis.

[0006] The present invention proposes a micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism, comprising: Obtaining and preprocessing a target face video frame image, performing optical flow feature extraction on the preprocessed target face video frame image, and generating an optical flow triplet based on the optical flow feature, wherein the optical flow feature extraction is based on the TV-L1 algorithm, and the optical flow triplet includes the optical flow feature and optical strain; Resampling the key areas in the target face video frame image and annotating the micro-expression frames according to an adaptive pseudo-labeling algorithm to obtain pseudo labels, wherein the key areas are the face areas containing micro-expression information, and the adaptive pseudo-labeling algorithm dynamically adjusts the comparison parameters according to the length of the sliding window and the true interval; Inputting the optical flow triplet and the pseudo label into a micro-expression analysis network respectively, wherein the micro-expression analysis network includes a sharing sub-network, a positioning sub-network, and a recognition sub-network. The micro-expression analysis network is based on a fusion attention mechanism and has a dual three-stream neural network structure; The shared sub-network adjusts the attention weights of the optical flow triples according to a dual attention mechanism to obtain attention-enhanced optical flow features, wherein the dual attention mechanism is based on a channel attention mechanism and a spatial attention mechanism; The localization sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, wherein the feature processing is used to reduce the feature dimension of the attention-enhanced optical flow feature and reduce the data size; The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probability.

[0007] In summary, according to the above-mentioned micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism, in order to more accurately and efficiently locate micro-expression intervals and identify emotional categories in videos, an adaptive pseudo-labeling algorithm and a micro-expression analysis network based on a fusion attention mechanism are designed. For the problem of large pseudo-labeling errors in existing methods, the adaptive pseudo-labeling algorithm is based on the proportion of micro-expressions in the entire video interval, combined with a sliding window for adaptive labeling to reduce labeling errors and improve the accuracy of micro-expression analysis. For the problem that existing methods are difficult to accurately capture subtle changes in micro-expressions, the present invention designs a micro-expression analysis network based on a fusion attention mechanism to achieve seamless completion of micro-expression positioning and recognition tasks, including a shared sub-network, a positioning sub-network and an identification sub-network. The micro-expression analysis network uses a two-layer convolutional neural network to extract features, and introduces a dual attention mechanism to capture channel and spatial information respectively, focus on significant features, and improve the accuracy of micro-expression positioning and recognition. The present invention improves the accuracy of micro-expression analysis. Specifically, the target face video frame image is obtained and preprocessed, the optical flow feature is extracted from the preprocessed target face video frame image, and an optical flow triplet is generated according to the optical flow feature, the optical flow feature extraction is based on the TV-L1 algorithm, the optical flow triplet includes optical flow feature and optical strain, the key area in the target face video frame image is resampled, and the micro-expression frame is annotated according to the adaptive pseudo-labeling algorithm to obtain a pseudo-label, the key area is the face area containing micro-expression information, the adaptive pseudo-labeling algorithm dynamically adjusts the comparison parameter according to the length of the sliding window and the true interval, for the problem of large pseudo-labeling error in the existing method, the adaptive pseudo-labeling algorithm is based on the proportion of micro-expressions in the entire video interval, combined with the sliding window for adaptive labeling to reduce the labeling error and improve the accuracy of micro-expression analysis, and the optical flow triplet and the pseudo-label are respectively input into the micro-expression analysis network The micro-expression analysis network includes a sharing sub-network, a positioning sub-network and an identification sub-network. The micro-expression analysis network is based on a fusion attention mechanism. The sharing sub-network adjusts the attention weight of the optical flow triplet according to the dual attention mechanism to obtain attention-enhanced optical flow features. The dual attention mechanism is based on the channel attention mechanism and the spatial attention mechanism. A two-layer convolutional neural network is used to extract features, and a dual attention mechanism is introduced to capture channel and spatial information respectively, focus on significant features, and improve the accuracy of micro-expression positioning and recognition. The positioning sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow features to obtain a confidence score. The feature processing is used to reduce the feature dimension of the attention-enhanced optical flow features and reduce the data scale. The identification sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probability. The present invention improves the accuracy of micro-expression analysis.

[0008] Furthermore, the steps of obtaining and preprocessing the target face video frame image, extracting optical flow features from the preprocessed target face video frame image, and generating optical flow triples based on the optical flow features specifically include: Obtain a target face video frame image and perform face cropping, wherein the face cropping uses the first frame image after face detection as the cropping standard and marks the key facial area points in the target face video frame image; Optical flow feature extraction is performed according to the V-L1 algorithm to obtain horizontal optical flow features and vertical optical flow features respectively. The specific formula for optical flow feature extraction is as follows: , , in, represents the horizontal optical flow feature, represents the vertical optical flow feature, and Represent the horizontal positive direction and the vertical positive direction respectively. Indicates the number of time frames; The optical strain and the optical strain amplitude are calculated to obtain the facial deformation intensity. The specific formulas for calculating the optical strain and the optical strain amplitude are as follows: , , in, represents the optical strain, 、 、 、 represent the optical strain in different directions, represents the optical strain amplitude value; An optical flow triplet (u, v, ϵ) is constructed according to the horizontal optical flow features, the vertical optical flow features, and the optical strain, and the optical flow triplet is input into a micro-expression analysis network.

[0009] Furthermore, the step of resampling the key areas in the target face video frame image and annotating the micro-expression frames according to the adaptive pseudo-labeling algorithm to obtain pseudo-labels specifically includes: Performing ROI selection and resampling processing on key areas in the target face video frame image, wherein the key areas include the left eye and left eyebrow area, the right eye and right eyebrow area, and the mouth area; Micro-expression frames are annotated in key areas of the target face video frame image according to the adaptive pseudo-labeling algorithm to obtain pseudo labels. The formula of the adaptive pseudo-labeling algorithm is as follows: , , , in, represents half of the average length of the micro-expression frame, 𝑁 represents the average length of the micro-expression frame, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number, and Respectively represent the micro-expression end frame and micro-expression start frame, IoU represents the intersection-over-union ratio of the sliding window and the true interval, where the sliding window starts from any frame and has a length of The real interval is the frame sequence from the micro-expression start frame to the micro-expression end frame, represents a sliding window, represents the true interval.

[0010] Furthermore, the shared subnetwork adjusts the attention weights of the optical flow triples according to the dual attention mechanism to obtain attention-enhanced optical flow features, specifically comprising: The optical flow triples are divided and input into different parallel channels of the shared sub-network respectively, where each parallel channel of the shared sub-network includes a convolutional layer, an attention enhancement layer, a maximum pooling layer, and a splicing layer; The shared sub-network is connected to the positioning sub-network and the identification sub-network respectively, and the positioning sub-network and the identification sub-network are connected through a shared layer; After convolution processing of the optical flow triplet, channel attention enhancement and spatial attention enhancement are performed respectively according to the dual attention mechanism to obtain attention-enhanced features, and the attention-enhanced features are again convolved and max-pooled to perform significant feature screening and enhancement on the attention-enhanced features; The attention-enhanced features output by all parallel channels are concatenated to obtain the attention-enhanced optical flow features.

[0011] Furthermore, the positioning sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, specifically comprising: The localization subnetwork performs convolution and maximum pooling on the attention-enhanced optical flow features based on the pseudo-labels to reduce feature dimensionality and data size, using the pseudo-labels as weak supervisory signals. During the convolution process, the attention-enhanced optical flow features are feature-focused based on the micro-expression intervals marked in the pseudo-labels. The feature focusing targets the areas of slight facial muscle deformation in the attention-enhanced optical flow features, and the dynamic trajectory of micro-expressions is enhanced based on the feature focusing. Then, the attention-enhanced optical flow features are tiled to convert the attention-enhanced optical flow features into one-dimensional data; The positioning confidence score is predicted based on the pseudo-label. The specific formula of the positioning confidence score is as follows: , in, represents the location reliability score, represents the activation function value, represents the activation function value weight, and b represents the bias; The positioning loss is then calculated based on the positioning confidence score. The specific formula for the positioning loss is as follows: , in, represents the positioning loss, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number, Indicates the positioning reliability.

[0012] Furthermore, the recognition subnetwork performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain a step of predicting emotion probability, specifically including: The recognition subnetwork freezes the shared feature extraction layer between the localization subnetwork and the recognition subnetwork based on the cross-task learning mechanism. The shared feature extraction layer is used to extract common features related to micro-expression localization in the localization subnetwork and perform convolution and maximum pooling on the attention-enhanced optical flow features to reduce feature dimensionality and data size. The attention-enhanced optical flow features are then tiled to calculate the predicted emotion probability. The specific formula for calculating the predicted emotion probability is as follows: , , Among them, p represents the predicted emotion probability, Represents the prediction score, C represents the total number of categories, c represents the category ordinal number, j Represents a collection of categories; determining an emotion category of the micro-expression based on the predicted emotion probability; The recognition loss is calculated based on the predicted emotion probability. The specific formula for calculating the recognition loss is as follows: , in, represents the recognition loss, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number.

[0013] Furthermore, the step of calculating the predicted emotion probability further includes: Post-processing is performed based on the peak discovery mechanism to identify micro-expression intervals by locating the maximum local peak and the segments around the peak; The positioning reliability score of each single video frame is smoothed, and the positioning threshold is calculated based on the average positioning reliability score and the maximum positioning reliability score of each video frame. The positioning threshold is used to divide multiple micro-expression intervals from the score sequence. The specific formula for calculating the positioning threshold is as follows: , , in, represents the smoothed location confidence score, represents half of the average length of a micro-expression frame, Represents a video frame, j Represents a collection of categories, Indicates the micro-expression frame number, 𝑇 represents the positioning threshold, represents the average location reliability score, represents the maximum location reliability score, 𝑝 Represents preset positioning parameters.

[0014] The present invention proposes a micro-expression analysis system based on adaptive pseudo-labeling and attention mechanism, comprising: A preprocessing module is used to obtain and preprocess the target face video frame image, extract optical flow features from the preprocessed target face video frame image, and generate an optical flow triplet based on the optical flow features. The optical flow feature extraction is based on the TV-L1 algorithm, and the optical flow triplet includes optical flow features and optical strain; An adaptive pseudo-labeling module is configured to resample key areas in the target face video frame image and annotate micro-expression frames according to an adaptive pseudo-labeling algorithm to obtain pseudo labels. The key areas are face regions containing micro-expression information. The adaptive pseudo-labeling algorithm dynamically adjusts comparison parameters based on the length of the sliding window and the true interval. a micro-expression analysis network module, configured to input the optical flow triples and the pseudo labels into a micro-expression analysis network, wherein the micro-expression analysis network includes a sharing sub-network, a positioning sub-network, and a recognition sub-network, and is based on a fusion attention mechanism; The shared sub-network adjusts the attention weights of the optical flow triples according to a dual attention mechanism to obtain attention-enhanced optical flow features, wherein the dual attention mechanism is based on a channel attention mechanism and a spatial attention mechanism; The localization sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, wherein the feature processing is used to reduce the feature dimension of the attention-enhanced optical flow feature and reduce the data size; The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probability.

[0015] The present invention also provides a storage medium storing one or more programs, which, when executed by a processor, implement the micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism as described above.

[0016] The present invention further provides a computer device, comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of the micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism proposed in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a micro-expression analysis system based on adaptive pseudo-labeling and attention mechanism proposed in the second embodiment of the present invention; Figure 3 Schematic diagram of the adaptive pseudo-labeling algorithm of the present invention; Figure 4 This is a schematic diagram of the micro-expression analysis network logic of the present invention; Figure 5 Schematic diagram of the network structure of the shared sub-network in the micro-expression analysis network of the present invention; Figure 6 This is a network structure relationship diagram between the positioning sub-network and the recognition sub-network in the micro-expression analysis network of the present invention; The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0018] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0019] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0021] See also Figure 1 , which is a flowchart of a micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism proposed in the first embodiment of the present invention, the micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism includes steps S01 to S06, wherein: Step S01: obtaining a target face video frame image and preprocessing it, extracting optical flow features from the preprocessed target face video frame image, and generating an optical flow triplet based on the optical flow features; It should be noted that in this embodiment, the optical flow feature extraction is based on the TV-L1 algorithm, and the optical flow triplet includes optical flow features and optical strain. The target face video frame image is obtained and the face is cropped. The face cropping uses the first frame image after face detection as the cropping standard and marks the key facial area points in the target face video frame image; Optical flow feature extraction is performed according to the V-L1 algorithm to obtain horizontal optical flow features and vertical optical flow features respectively. The specific formula for optical flow feature extraction is as follows: , , in, represents the horizontal optical flow feature, represents the vertical optical flow feature, and Represent the horizontal positive direction and the vertical positive direction respectively. Indicates the number of time frames; The optical strain and the optical strain amplitude are calculated to obtain the facial deformation intensity. The specific formulas for calculating the optical strain and the optical strain amplitude are as follows: , , in, represents the optical strain, 、 、 、 represent the optical strain in different directions, represents the optical strain amplitude value; An optical flow triplet (u, v, ϵ) is constructed according to the horizontal optical flow features, the vertical optical flow features, and the optical strain, and the optical flow triplet is input into a micro-expression analysis network.

[0022] Step S02: resampling the key areas in the target face video frame image, and annotating the micro-expression frames according to the adaptive pseudo-labeling algorithm to obtain pseudo-labels; It should be noted that in this embodiment, the key area is the facial area containing micro-expression information. The adaptive pseudo-labeling algorithm dynamically adjusts the comparison parameters according to the length of the sliding window and the true interval, and performs ROI selection and resampling processing on the key areas in the target face video frame image. The key areas include the left eye and left eyebrow area, the right eye and right eyebrow area, and the mouth area; For details on the operation of the adaptive pseudo-labeling algorithm, please refer to Figure 3 , where the four-pointed star area is the actual micro-expression frame, the five-pointed star marking result is the marking result of the adaptive pseudo-marking algorithm of the present invention, and the six-pointed star marking result is the marking result of the existing pseudo-marking algorithm. It can be clearly seen from the figure that the marking effect of the adaptive pseudo-marking method proposed by the present invention is only compared with the existing automatic pseudo-marking method. The existing automatic pseudo-marking method will exaggerate the proportion of the micro-expression interval in the entire video interval. In contrast, the new marking method proposed in this article shows a stronger coping ability when facing this situation, and can effectively reduce the micro-expression positioning error caused by marking deviation; Micro-expression frames are annotated in key areas of the target face video frame image according to the adaptive pseudo-labeling algorithm to obtain pseudo labels. The formula of the adaptive pseudo-labeling algorithm is as follows: , , , in, represents half of the average length of the micro-expression frame, 𝑁 represents the average length of the micro-expression frame, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number, and Respectively represent the micro-expression end frame and micro-expression start frame, IoU represents the intersection-over-union ratio of the sliding window and the true interval, where the sliding window starts from any frame and has a length of The real interval is the frame sequence from the micro-expression start frame to the micro-expression end frame, represents a sliding window, represents the true interval.

[0023] Step S03: inputting the optical flow triplet and pseudo label into the micro-expression analysis network respectively; It should be noted that in this embodiment, the micro-expression analysis network includes a sharing sub-network, a positioning sub-network and a recognition sub-network. The micro-expression analysis network is based on a fusion attention mechanism. For the specific logical process of the micro-expression analysis network, please refer to Figure 4 .

[0024] Step S04: the shared sub-network adjusts the attention weights of the optical flow triples according to the dual attention mechanism to obtain attention-enhanced optical flow features; It should be noted that in this embodiment, the dual attention mechanism is based on the channel attention mechanism and the spatial attention mechanism. The optical flow triples are divided and input into different parallel channels of the shared subnetwork respectively. Each parallel channel of the shared subnetwork includes a convolution layer, an attention enhancement layer, a maximum pooling layer and a splicing layer. The shared sub-network is connected to the positioning sub-network and the identification sub-network respectively, and the positioning sub-network and the identification sub-network are connected through a shared layer; After convolution processing of the optical flow triplet, channel attention enhancement and spatial attention enhancement are performed respectively according to the dual attention mechanism to obtain attention-enhanced features, and the attention-enhanced features are again convolved and max-pooled to perform significant feature screening and enhancement on the attention-enhanced features; The attention-enhanced features output by all parallel channels are concatenated to obtain the attention-enhanced optical flow features; For the specific network structure of the shared sub-network, please refer to Figure 5 ,The detailed structural parameters of the shared sub-network are shown in Table 1: Table 1 Shared sub-network structure parameters Among them, Conv represents the convolutional layer, each convolutional layer uses relu as the activation function, and the weight regularization is set to 0.001, CBAM represents the attention enhancement layer, based on the dual attention mechanism, Bn represents the batch normalization layer (BatchNormalization), each pool is a maximum pooling layer (Maxpooling), Dropout represents the dropout layer, and the proportion of randomly dropped neurons in each Dropout is set to 0.3.

[0025] Step S05: The localization sub-network performs feature processing on the pseudo-labels and attention-enhanced optical flow features to obtain a confidence score; It should be noted that in this embodiment, the feature processing is used to reduce the feature dimension of the attention-enhanced optical flow feature and reduce the data scale. The positioning subnetwork performs convolution processing and maximum pooling processing on the attention-enhanced optical flow feature according to the pseudo-label to reduce the feature dimension and data scale, and uses the pseudo-label as a weak supervision signal. In the convolution processing, the attention-enhanced optical flow feature is feature-focused according to the micro-expression interval marked in the pseudo-label. The target of the feature focusing is the area of ​​slight deformation of facial muscles in the attention-enhanced optical flow feature, and the dynamic trajectory of micro-expressions is enhanced based on the feature focusing. In this embodiment, the addition of pseudo-labels suppresses the interference of static facial background or irrelevant areas. The introduction of pseudo-labels effectively alleviates the problem of scarcity of real annotations and high annotation costs in micro-expression datasets. The weak supervision signals generated by pseudo-labels expand the diversity of training data, allowing the convolutional layer to learn more generalized spatiotemporal features during the feature extraction process. For example, pseudo-labels are used to guide the convolutional layer to learn the dynamic characteristics of muscle contraction speed and amplitude when micro-expressions occur, thereby improving the robustness of the positioning network under low signal-to-noise ratio conditions. After the feature extraction process guided by pseudo-labels, the generated feature map is significantly improved in spatiotemporal resolution and dynamic response sensitivity, providing more suitable feature input for modules such as key point trajectory prediction and motor unit activation detection in subsequent positioning networks, effectively improving the accuracy and stability of micro-expression positioning. Then, the attention-enhanced optical flow features are tiled to convert the attention-enhanced optical flow features into one-dimensional data; The positioning confidence score is predicted based on the pseudo-label. The specific formula of the positioning confidence score is as follows: , in, represents the location reliability score, represents the activation function value, represents the activation function value weight, and b represents the bias; The positioning loss is then calculated based on the positioning confidence score. The specific formula for the positioning loss is as follows: , in, represents the positioning loss, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number, Indicates the positioning reliability; The detailed structural parameters of the positioning sub-network are shown in Table 2: Table 2 Positioning sub-network structure parameters Among them, Shared represents the shared feature extraction layer, the ratio of randomly dropped neurons in each Dropout is set to 0.2, Maxpooling represents the maximum pooling layer, Flatten represents the tiling layer, and Dense represents the fully connected layer.

[0026] Step S06: The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain the predicted emotion probability; It should be noted that in this embodiment, the structural relationship between the positioning sub-network and the identification sub-network can be found in Figure 6 The recognition sub-network freezes the shared feature extraction layer between the localization sub-network and the recognition sub-network according to the cross-task learning mechanism. The shared feature extraction layer is used to extract common features related to micro-expression localization in the localization sub-network, and performs convolution and maximum pooling on the attention-enhanced optical flow features to reduce feature dimensions and data size. This embodiment adopts a cross-task learning mechanism, namely the ITL strategy. By freezing the shared feature extraction layer between the localization sub-network and the recognition sub-network, the features learned by the localization sub-network are directly transferred to the recognition sub-network. Specifically, the localization sub-network focuses on micro-expression interval prediction, and the shared feature extraction layer has extracted common features related to micro-expression localization. When training the recognition sub-network, the parameters of these shared feature extraction layers are fixed to avoid destroying the localization features due to retraining. At the same time, only the classification head unique to the recognition sub-network is updated, enabling it to efficiently learn target category information based on the shared features. This not only achieves the effective transfer of localization knowledge and recognition capabilities, but also improves training efficiency by reducing the amount of parameter updates and reduces the risk of overfitting. The attention-enhanced optical flow features are then tiled to calculate the predicted emotion probability. The specific formula for calculating the predicted emotion probability is as follows: , , Among them, p represents the predicted emotion probability, Represents the prediction score, C represents the total number of categories, c represents the category ordinal number, j Represents a collection of categories; determining an emotion category of the micro-expression based on the predicted emotion probability; The recognition loss is calculated based on the predicted emotion probability. The specific formula for calculating the recognition loss is as follows: , in, represents the recognition loss, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number; Post-processing is performed based on the peak discovery mechanism to identify micro-expression intervals by locating the maximum local peak and the segments around the peak; The positioning reliability score of each single video frame is smoothed, and the positioning threshold is calculated based on the average positioning reliability score and the maximum positioning reliability score of each video frame. The positioning threshold is used to divide multiple micro-expression intervals from the score sequence. The specific formula for calculating the positioning threshold is as follows: , , in, represents the smoothed location confidence score, represents half of the average length of a micro-expression frame, Represents a video frame, j Represents a collection of categories, Indicates the micro-expression frame number, 𝑇 represents the positioning threshold, represents the average location reliability score, represents the maximum location reliability score, 𝑝 Indicates preset positioning parameters; The detailed parameters of the identification sub-network structure are shown in Table 3: Table 3 Identification sub-network structure parameters The ratio of randomly discarded neurons in each Dropout is set to 0.2.

[0027] In summary, according to the above-mentioned micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism, in order to more accurately and efficiently locate micro-expression intervals and identify emotional categories in videos, an adaptive pseudo-labeling algorithm and a micro-expression analysis network based on a fusion attention mechanism are designed. For the problem of large pseudo-labeling errors in existing methods, the adaptive pseudo-labeling algorithm is based on the proportion of micro-expressions in the entire video interval, combined with a sliding window for adaptive labeling to reduce labeling errors and improve the accuracy of micro-expression analysis. For the problem that existing methods are difficult to accurately capture subtle changes in micro-expressions, the present invention designs a micro-expression analysis network based on a fusion attention mechanism to achieve seamless completion of micro-expression positioning and recognition tasks, including a shared sub-network, a positioning sub-network and an identification sub-network. The micro-expression analysis network uses a two-layer convolutional neural network to extract features, and introduces a dual attention mechanism to capture channel and spatial information respectively, focus on significant features, and improve the accuracy of micro-expression positioning and recognition. The present invention improves the accuracy of micro-expression analysis. Specifically, the target face video frame image is obtained and preprocessed, the optical flow feature is extracted from the preprocessed target face video frame image, and an optical flow triplet is generated according to the optical flow feature, the optical flow feature extraction is based on the TV-L1 algorithm, the optical flow triplet includes optical flow feature and optical strain, the key area in the target face video frame image is resampled, and the micro-expression frame is annotated according to the adaptive pseudo-labeling algorithm to obtain a pseudo-label, the key area is the face area containing micro-expression information, the adaptive pseudo-labeling algorithm dynamically adjusts the comparison parameter according to the length of the sliding window and the true interval, for the problem of large pseudo-labeling error in the existing method, the adaptive pseudo-labeling algorithm is based on the proportion of micro-expressions in the entire video interval, combined with the sliding window for adaptive labeling to reduce the labeling error and improve the accuracy of micro-expression analysis, and the optical flow triplet and the pseudo-label are respectively input into the micro-expression analysis network The micro-expression analysis network includes a sharing sub-network, a positioning sub-network, and a recognition sub-network. The micro-expression analysis network is based on a fusion attention mechanism. The sharing sub-network adjusts the attention weights of the optical flow triples according to a dual-attention mechanism to obtain attention-enhanced optical flow features. The dual-attention mechanism is based on a channel attention mechanism and a spatial attention mechanism. It uses a two-layer convolutional neural network to extract features and introduces a dual-attention mechanism to capture channel and spatial information, focusing on salient features, thereby improving the accuracy of micro-expression positioning and recognition. The positioning sub-network performs feature processing on the pseudo-labels and the attention-enhanced optical flow features to obtain confidence scores. This feature processing is used to reduce the feature dimension of the attention-enhanced optical flow features and reduce the data size. The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probabilities. This invention improves the accuracy of micro-expression analysis.

[0028] See also Figure 3, which is a schematic diagram of the structure of a micro-expression analysis system based on adaptive pseudo-labeling and attention mechanism proposed in the third embodiment of the present invention, comprising: A preprocessing module 10 is configured to obtain and preprocess a target face video frame image, extract optical flow features from the preprocessed target face video frame image, and generate an optical flow triplet based on the optical flow features. The optical flow feature extraction is based on the TV-L1 algorithm, and the optical flow triplet includes the optical flow features and optical strain. An adaptive pseudo-labeling module 20 is configured to resample key regions in the target face video frame image and annotate micro-expression frames according to an adaptive pseudo-labeling algorithm to obtain pseudo labels. The key regions are face regions containing micro-expression information. The adaptive pseudo-labeling algorithm dynamically adjusts comparison parameters based on the length of the sliding window and the true interval. a micro-expression analysis network module 30, configured to input the optical flow triples and the pseudo labels into a micro-expression analysis network, wherein the micro-expression analysis network includes a sharing sub-network, a positioning sub-network, and a recognition sub-network, and the micro-expression analysis network is based on a fusion attention mechanism; The shared sub-network adjusts the attention weights of the optical flow triples according to a dual attention mechanism to obtain attention-enhanced optical flow features, wherein the dual attention mechanism is based on a channel attention mechanism and a spatial attention mechanism; The localization sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, wherein the feature processing is used to reduce the feature dimension of the attention-enhanced optical flow feature and reduce the data size; The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probability.

[0029] The present invention also proposes a computer storage medium on which one or more programs are stored, which, when executed by a processor, implement the above-mentioned micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism.

[0030] The present invention also proposes a computer device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism.

[0031] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0032] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0033] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0034] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0035] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism, characterized in that: include: Obtaining and preprocessing a target face video frame image, performing optical flow feature extraction on the preprocessed target face video frame image, and generating an optical flow triplet based on the optical flow feature, wherein the optical flow feature extraction is based on the TV-L1 algorithm, and the optical flow triplet includes the optical flow feature and optical strain; Resampling the key areas in the target face video frame image and annotating the micro-expression frames according to an adaptive pseudo-labeling algorithm to obtain pseudo labels, wherein the key areas are the face areas containing micro-expression information, and the adaptive pseudo-labeling algorithm dynamically adjusts the comparison parameters according to the length of the sliding window and the true interval; Inputting the optical flow triplet and the pseudo label into a micro-expression analysis network respectively, wherein the micro-expression analysis network includes a sharing sub-network, a positioning sub-network, and a recognition sub-network. The micro-expression analysis network is based on a fusion attention mechanism and has a dual three-stream neural network structure; The shared sub-network adjusts the attention weights of the optical flow triples according to a dual attention mechanism to obtain attention-enhanced optical flow features, wherein the dual attention mechanism is based on a channel attention mechanism and a spatial attention mechanism; The localization sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, wherein the feature processing is used to reduce the feature dimension of the attention-enhanced optical flow feature and reduce the data size; The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probability.

2. The micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism according to claim 1 is characterized in that: The steps of obtaining and preprocessing the target face video frame image, extracting optical flow features from the preprocessed target face video frame image, and generating optical flow triples based on the optical flow features specifically include: Obtain a target face video frame image and perform face cropping, wherein the face cropping uses the first frame image after face detection as the cropping standard and marks the key facial area points in the target face video frame image; Optical flow feature extraction is performed according to the V-L1 algorithm to obtain horizontal optical flow features and vertical optical flow features respectively. The specific formula for optical flow feature extraction is as follows: ， ， in, represents the horizontal optical flow feature, represents the vertical optical flow feature, and Represent the horizontal positive direction and the vertical positive direction respectively. Indicates the number of time frames; The optical strain and the optical strain amplitude are calculated to obtain the facial deformation intensity. The specific formulas for calculating the optical strain and the optical strain amplitude are as follows: ， ， in, represents the optical strain, 、 、 、 represent the optical strain in different directions, represents the optical strain amplitude value; An optical flow triplet (u, v, ϵ) is constructed according to the horizontal optical flow features, the vertical optical flow features, and the optical strain, and the optical flow triplet is input into a micro-expression analysis network.

3. The micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism according to claim 1, characterized in that: The step of resampling the key areas in the target face video frame image and annotating the micro-expression frames according to the adaptive pseudo-labeling algorithm to obtain pseudo-labels specifically includes: Performing ROI selection and resampling processing on key areas in the target face video frame image, wherein the key areas include the left eye and left eyebrow area, the right eye and right eyebrow area, and the mouth area; Micro-expression frames are annotated in key areas of the target face video frame image according to the adaptive pseudo-labeling algorithm to obtain pseudo labels. The formula of the adaptive pseudo-labeling algorithm is as follows: ， ， ， in, represents half of the average length of the micro-expression frame, 𝑁 represents the average length of the micro-expression frame, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number, and Respectively represent the micro-expression end frame and micro-expression start frame, IoU represents the intersection-over-union ratio of the sliding window and the true interval, where the sliding window starts from any frame and has a length of The real interval is the frame sequence from the micro-expression start frame to the micro-expression end frame, represents a sliding window, represents the true interval.

4. The micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism according to claim 1, characterized in that: The shared sub-network adjusts the attention weights of the optical flow triples according to the dual attention mechanism to obtain attention-enhanced optical flow features, specifically comprising: The optical flow triples are divided and input into different parallel channels of the shared sub-network respectively, where each parallel channel of the shared sub-network includes a convolutional layer, an attention enhancement layer, a maximum pooling layer, and a splicing layer; The shared sub-network is connected to the positioning sub-network and the identification sub-network respectively, and the positioning sub-network and the identification sub-network are connected through a shared layer; After convolution processing of the optical flow triplet, channel attention enhancement and spatial attention enhancement are performed respectively according to the dual attention mechanism to obtain attention-enhanced features, and the attention-enhanced features are again convolved and max-pooled to perform significant feature screening and enhancement on the attention-enhanced features; The attention-enhanced features output by all parallel channels are concatenated to obtain the attention-enhanced optical flow features.

5. The micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism according to claim 1, characterized in that: The positioning sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, specifically comprising: The localization subnetwork performs convolution and maximum pooling on the attention-enhanced optical flow features based on the pseudo-labels to reduce feature dimensionality and data size, using the pseudo-labels as weak supervisory signals. During the convolution process, the attention-enhanced optical flow features are feature-focused based on the micro-expression intervals marked in the pseudo-labels. The feature focusing targets the areas of slight facial muscle deformation in the attention-enhanced optical flow features, and the dynamic trajectory of micro-expressions is enhanced based on the feature focusing. Then, the attention-enhanced optical flow features are tiled to convert the attention-enhanced optical flow features into one-dimensional data; The positioning confidence score is predicted based on the pseudo-label. The specific formula of the positioning confidence score is as follows: ， in, represents the location reliability score, represents the activation function value, represents the activation function value weight, and b represents the bias; The positioning loss is then calculated based on the positioning confidence score. The specific formula for the positioning loss is as follows: ， in, represents the positioning loss, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number, Indicates the positioning reliability.

6. The micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism according to claim 1, characterized in that: The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain the predicted emotion probability, specifically comprising: The recognition subnetwork freezes the shared feature extraction layer between the localization subnetwork and the recognition subnetwork based on the cross-task learning mechanism. The shared feature extraction layer is used to extract common features related to micro-expression localization in the localization subnetwork and perform convolution and maximum pooling on the attention-enhanced optical flow features to reduce feature dimensionality and data size. The attention-enhanced optical flow features are then tiled to calculate the predicted emotion probability. The specific formula for calculating the predicted emotion probability is as follows: ， ， Among them, p represents the predicted emotion probability, Represents the prediction score, C represents the total number of categories, c represents the category ordinal number, j Represents a collection of categories; determining an emotion category of the micro-expression based on the predicted emotion probability; The recognition loss is calculated based on the predicted emotion probability. The specific formula for calculating the recognition loss is as follows: ， in, represents the recognition loss, Indicates the total number of micro-expression frames, Indicates the micro-expression frame number.

7. The micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism according to claim 6, characterized in that: The step of calculating the predicted emotion probability further includes: Post-processing is performed based on the peak discovery mechanism to identify micro-expression intervals by locating the maximum local peak and the segments around the peak; The positioning reliability score of each single video frame is smoothed, and the positioning threshold is calculated based on the average positioning reliability score and the maximum positioning reliability score of each video frame. The positioning threshold is used to divide multiple micro-expression intervals from the score sequence. The specific formula for calculating the positioning threshold is as follows: ， ， in, represents the smoothed location confidence score, represents half of the average length of a micro-expression frame, Represents a video frame, j Represents a collection of categories, Indicates the micro-expression frame number, 𝑇 represents the positioning threshold, represents the average location reliability score, represents the maximum location reliability score, 𝑝 Represents preset positioning parameters.

8. A micro-expression analysis system based on adaptive pseudo-labeling and attention mechanism, characterized in that: include: A preprocessing module is used to obtain and preprocess the target face video frame image, extract optical flow features from the preprocessed target face video frame image, and generate an optical flow triplet based on the optical flow features. The optical flow feature extraction is based on the TV-L1 algorithm, and the optical flow triplet includes optical flow features and optical strain; An adaptive pseudo-labeling module is configured to resample key areas in the target face video frame image and annotate micro-expression frames according to an adaptive pseudo-labeling algorithm to obtain pseudo labels. The key areas are face regions containing micro-expression information. The adaptive pseudo-labeling algorithm dynamically adjusts comparison parameters based on the length of the sliding window and the true interval. A micro-expression analysis network module is used to input the optical flow triples and the pseudo labels into a micro-expression analysis network, wherein the micro-expression analysis network includes a sharing sub-network, a positioning sub-network, and a recognition sub-network. The micro-expression analysis network is based on a fusion attention mechanism and has a dual three-stream neural network structure. The shared sub-network adjusts the attention weights of the optical flow triples according to a dual attention mechanism to obtain attention-enhanced optical flow features, wherein the dual attention mechanism is based on a channel attention mechanism and a spatial attention mechanism; The localization sub-network performs feature processing on the pseudo-label and the attention-enhanced optical flow feature to obtain a confidence score, wherein the feature processing is used to reduce the feature dimension of the attention-enhanced optical flow feature and reduce the data size; The recognition sub-network performs micro-expression emotion recognition based on the attention-enhanced optical flow features to obtain predicted emotion probability.

9. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the micro-expression analysis method based on adaptive pseudo-labeling and attention mechanism described in any one of claims 1-7.

Images