Micro-expression recognition method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and image processing, can be applied to the field of intelligent medical treatment and finance, and discloses a micro-expression recognition method, device, equipment and medium, and the method comprises the steps: obtaining a to-be-recognized micro-expression video, and carrying out the normalization processing of the to-be-recognized micro-expression video, and obtaining a first video; performing feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features; carrying out attention calculation on the multi-scale features of each layer based on a gating attention mechanism to generate spatial enhancement features, carrying out time attention calculation on the spatial enhancement features of each layer along a time dimension to generate time sequence enhancement features, and carrying out multi-scale fusion on the time sequence enhancement features of each layer to obtain fusion features; and performing expression category recognition on the fusion features based on a pre-trained classification model to obtain a micro-expression recognition result. Accurate capture of micro-expression dynamic features is realized through cascade attention design, focusing on an effective area and a key frame is realized, and redundant calculation is avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and image processing technology, and in particular to a micro-expression recognition method, device, equipment and medium. Background Art

[0002] Microexpressions are subtle and fleeting expressions of human facial expression. These expressions are extremely brief (typically lasting only 40 to 200 milliseconds), yet they play a crucial role in human emotional expression and social interaction. Microexpressions often occur when individuals attempt to mask their emotional reactions during social interactions, unconsciously expressing their emotions. The difficulty of controlling and disguising microexpressions makes their emotional expressions more credible, making them applicable in fields such as clinical medicine and financial risk management.

[0003] Because micro-expressions are low-intensity, instantaneous, and sparse, traditional methods based on handcrafted features (such as local binary patterns) and machine learning (such as support vector machines) struggle to effectively capture their dynamic characteristics. In recent years, with the development of deep learning technology, deep learning has significantly improved micro-expression recognition performance through end-to-end learning. In particular, three-dimensional convolutional neural networks and long short-term memory (LSTM) networks can simultaneously model temporal and spatial features, extracting the spatiotemporal features of micro-expressions, becoming mainstream solutions. However, this approach treats all regions equally when extracting features, while micro-expressions only involve local facial regions (such as around the eyes or corners of the mouth), resulting in noise interference. Furthermore, only a few frames in a micro-expression video contain valid information, and existing temporal modeling methods (such as LSTM) do not fully suppress the interference of irrelevant frames, resulting in computational redundancy. Summary of the Invention

[0004] The present invention provides a micro-expression recognition method, device, equipment and medium to solve the technical problems of noise interference and computational redundancy in micro-expression recognition in the prior art.

[0005] In a first aspect, a micro-expression recognition method is provided, comprising:

[0006] Obtaining a micro-expression video to be recognized, and performing normalization processing on the micro-expression video to be recognized to obtain a first video;

[0007] Performing feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features;

[0008] Based on the gated attention mechanism, attention calculation is performed on the multi-scale features of each layer to generate spatial enhancement features, and temporal attention calculation is performed on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features. The temporal enhancement features of each layer are then multi-scale fused to obtain fused features.

[0009] The expression category is recognized based on the fusion features based on the pre-trained classification model to obtain the micro-expression recognition results.

[0010] In a second aspect, a micro-expression recognition device is provided, comprising:

[0011] a preprocessing module, configured to obtain a micro-expression video to be identified, and perform normalization processing on the micro-expression video to be identified to obtain a first video;

[0012] A feature hierarchical extraction module, configured to extract features from a video frame sequence of the first video to obtain multi-layer multi-scale features;

[0013] The hierarchical selective attention module is used to perform attention calculation on the multi-scale features of each layer based on the gated attention mechanism to generate spatial enhancement features, and perform temporal attention calculation on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features, and then perform multi-scale fusion on the temporal enhancement features of each layer to obtain fused features;

[0014] The classification module is used to identify expression categories based on the fusion features based on the pre-trained classification model to obtain micro-expression recognition results.

[0015] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned micro-expression recognition method when executing the computer program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned micro-expression recognition method are implemented.

[0017] In the scheme implemented by the above-mentioned micro-expression recognition method, device, equipment and medium, the micro-expression video to be recognized can be obtained through the client, and the micro-expression video to be recognized is normalized to obtain a first video; the video frame sequence of the first video is subjected to feature extraction to obtain multi-layer multi-scale features; then the multi-scale features of each layer are subjected to attention calculation based on the gated attention mechanism to generate spatial enhancement features, and the spatial enhancement features of each layer are subjected to time attention calculation along the time dimension to generate temporal enhancement features, and the temporal enhancement features of each layer are subjected to multi-scale fusion to obtain fusion features; finally, the fusion features are subjected to expression category recognition based on the pre-trained classification model to obtain micro-expression recognition results. In the present invention, medical imaging videos in the field of smart medical care or fund review videos in the field of financial technology are subjected to When recognizing users' micro-expressions, multi-level and multi-scale feature extraction can be performed on the video frame sequence to obtain multi-layer multi-scale features. Then, based on the gated attention mechanism, the regions of interest of each layer are quickly located, and content enhancement features focusing on local facial areas (such as the periphery of the eyes and the corners of the mouth) are generated. Then, based on the temporal attention mechanism, instantaneous changes are accurately captured, and timing enhancement features focusing on key timing frames are generated. The timing enhancement features of each layer are fused at multi-scale to obtain fused features, so that the micro-expression category can be identified based on the fused features, which can adapt to multi-scale micro-expression changes. The cascaded attention design can realize the accurate capture of micro-expression dynamic features, and the computing resources can be concentrated on the effective areas and key frames to avoid noise interference in other areas. At the same time, redundant calculations can be avoided, and the computing efficiency is greatly improved and the recognition accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 2. It is a schematic diagram of an application environment of a micro-expression recognition method according to an embodiment of the present invention;

[0020] Figure 2 1 is a flow chart of a micro-expression recognition method according to an embodiment of the present invention;

[0021] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0022] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0023] Figure 5 is a structural diagram of a micro-expression recognition device according to one embodiment of the present invention;

[0024] Figure 6 is a structural diagram of a computer device in one embodiment of the present invention;

[0025] Figure 7 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] The micro-expression recognition method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the client communicates with the server through a network. The server can obtain the micro-expression video to be identified through the client, normalize the micro-expression video to be identified, and obtain a first video; perform feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features; then perform attention calculation on the multi-scale features of each layer based on the gated attention mechanism to generate spatial enhancement features, and perform temporal attention calculation on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features, and perform multi-scale fusion on the temporal enhancement features of each layer to obtain fusion features; finally, perform expression category recognition on the fusion features based on the pre-trained classification model to obtain micro-expression recognition results. In the present invention, user micro-expression recognition is performed on medical imaging videos in the field of smart medical care (such as medical communication images) or fund review videos in the field of financial technology. When performing multi-level and multi-scale feature extraction on the video frame sequence, a multi-layer multi-scale feature is obtained. Then, based on the gated attention mechanism, the region of interest of the multi-scale feature of each layer is quickly located, and content enhancement features focusing on the local facial area (such as the eye area and the corner of the mouth) are generated. Then, based on the temporal attention mechanism, instantaneous changes are accurately captured, and temporal enhancement features focusing on key time frames are generated. The temporal enhancement features of each layer are multi-scale fused to obtain fused features, thereby identifying micro-expression categories based on the fused features, which can adapt to multi-scale micro-expression changes, and accurately capture the dynamic features of micro-expressions through cascaded attention design. Computing resources can be concentrated on effective areas and key frames to avoid noise interference in other areas, while avoiding redundant calculations, greatly improving computing efficiency and improving recognition accuracy. Among them, the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0028] See also Figure 2 As shown, Figure 2 A schematic flow chart of a micro-expression recognition method provided in an embodiment of the present invention includes the following steps S10-S40:

[0029] S10: Obtain a micro-expression video to be recognized, and perform normalization processing on the micro-expression video to be recognized to obtain a first video.

[0030] In the present invention, the micro-expression video to be identified can be a short video with a duration of less than 30 seconds, or a video with a duration of more than 30 seconds. For example, it can be an image video of a patient and a doctor communicating during a medical consultation, or an image video of a bank staff and a user confirming information during the financial funds review and confirmation process.

[0031] Specifically, if Figure 3As shown, step S10 includes the following steps S11-S13:

[0032] S11: Obtain a micro-expression video to be recognized, and perform facial region positioning on an image of each frame in the micro-expression video to be recognized.

[0033] In this step, obtain the micro-expression video to be recognized In this embodiment, I t ∈R H×W×3 represents the RGB image of the t-th frame, and T is the total number of frames of the micro-expression video to be recognized.

[0034] In the present invention, a face detector (such as MTCNN (MultiTask Cascaded Convolutional Networks)) can be used to locate the facial area.

[0035] S12: Aligning the positioned image of each frame to a preset size.

[0036] In this step, each frame of the image after facial region positioning is aligned to a preset size (eg, 112×112 pixels) through affine transformation.

[0037] S13: performing normalization processing on the aligned images of each frame to obtain a first video.

[0038] In this step, the aligned images are normalized, and the pixel values ​​are mapped to the interval [-1, 1]. The normalized multiple frames of images constitute the first video.

[0039] S20: Perform feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features.

[0040] In this step, a pre-trained backbone network (e.g., a ResNet-50 network with fully connected layers removed and convolutional layers retained) can be used to perform multi-level and multi-layer feature extraction on the video frame sequence. For example, in this embodiment, for the input frame I t , after passing through the above ResNet-50 network, it outputs multi-layer multi-scale features in To represent the features of the lth layer, the resolution can be halved layer by layer while the number of channels is doubled (for example, 56×56×256 when l=1).

[0041] S30: Based on the gated attention mechanism, attention calculation is performed on the multi-scale features of each layer to generate spatial enhancement features, and temporal attention calculation is performed on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features. The temporal enhancement features of each layer are then multi-scale fused to obtain fused features.

[0042] In this step, spatial / temporal attention and multi-scale feature fusion work together. First, gated spatial attention is used to quickly locate the region of interest, and then temporal attention is used to accurately capture instantaneous changes. Finally, the dual-path feature pyramid model and gating mechanism are combined to achieve adaptive fusion of multi-scale features, which greatly improves computational efficiency and significantly improves the feature discriminability in subsequent classification and recognition. It can solve the problem of feature extraction in micro-expression recognition caused by subtle and short expressions, thereby improving the accuracy and robustness of micro-expression recognition detection.

[0043] Specifically, if Figure 4 As shown, step S30 includes the following steps S31-S39:

[0044] S31: Perform temporal context encoding on the multi-scale features of each layer of each video frame in the video frame sequence to obtain a feature vector containing context information.

[0045] In this step, the multi-scale features are temporally context encoded. For example, for the l-th layer feature vector According to the formula The features of the current frame t and the previous K frames are flattened into vectors and concatenated to obtain context information, H l 、W l 、C l are the height, width, and channels of the current frame t at the current layer l, respectively; Flatten(·) means flattening the three-dimensional tensor into a one-dimensional vector, and the semicolon indicates the concatenation operation.

[0046] S32: Perform attention calculation on the feature vector of each layer based on the gated attention mechanism to obtain the spatial weight matrix.

[0047] Specifically, in this step, the gate mechanism is used to update the hidden state of the feature vector of each layer, that is, using the formula Update the hidden state, where is the hidden state dimension, the initial state is a zero vector; according to the formula Perform attention calculation to obtain the spatial weight matrix in, and is a learnable parameter, is the hidden state, and σ is the Sigmoid function.

[0048] S33: Perform spatial weighting on the multi-scale features according to the spatial weight matrix to obtain spatial enhancement features.

[0049] In this step, based on the spatial weight matrix Perform spatial weighting on multi-scale features, that is, according to the formula The spatial enhanced features are obtained, where ⊙ represents element-wise multiplication.

[0050] For steps S31-S33, the temporal context is modeled, the weights are adaptively adjusted according to the temporal context based on the gated attention mechanism, the facial region weights are dynamically generated, and each layer of features is Applying gated-guided attention weights Generate spatial enhancement features focused on micro-expression sensitive areas such as the eye area and mouth corners To concentrate computing resources in the effective area.

[0051] S34: performing feature compression on the spatial enhancement features of each layer of each video frame in the video frame sequence to obtain frame-level features.

[0052] In this step, the spatial enhancement features of each frame are averaged and pooled along the spatial dimension to obtain the frame-level features, i.e., the frame-level features

[0053] S35: Add learnable position encoding to the frame-level features to obtain frame-level features that retain temporal information.

[0054] In this step, the position encoding can be learned The final frame-level features that retain the timing information are

[0055] S36: Perform attention calculation on the frame-level features that retain the timing information through a multi-head attention mechanism to obtain an attention matrix, and calculate the average attention of the attention matrix along the time dimension to obtain a frame weight matrix.

[0056] In this step, the inter-frame correlation is calculated through the multi-head attention mechanism (the number of heads is M) to obtain the attention matrix, that is, the attention matrix Where Q is a learnable query vector, key K, and value V are frame-level features that retain temporal information. Through three weight matrices W Q , W K , W V Linear transformation yields d k =C l / M is the dimension of each head, T is the transpose operation of the K matrix; and the attention matrix is ​​averaged along the time dimension to obtain the frame importance weight, that is, the frame weight matrix

[0057] S37: Weighting the spatial enhancement features according to the frame weight matrix to obtain temporal enhancement features.

[0058] In this step, take the diagonal elements of the frame weight matrix As the weight of each frame, use the formula Perform frame-level weighting on spatial enhancement features to obtain temporal enhancement features

[0059] For S34-S37, for the l-th layer spatial enhancement feature sequence A Transformer-driven temporal attention mechanism is used to quantify the importance of each frame, suppress the interference of irrelevant frames, and highlight the key phases of micro-expressions (such as the onset, peak, and offset stages of micro-expressions), thereby achieving key frame selection and generating temporal enhancement features focused on key timing frames.

[0060] S38: A dual-path feature pyramid model is used to extract temporal enhancement features of each layer along the top-down and bottom-up paths respectively.

[0061] Specifically, in this step, according to the formula and The temporal enhancement features of each layer are extracted along the top-down and bottom-up paths, where P l Represents the features extracted from the l-th layer temporal enhancement feature along the top-down path, Initialize the features of the top layer and use bilinear interpolation for upsampling; l It represents the features extracted from the l-th layer of temporal enhancement features along the bottom-up path, N0 is the bottom-level initialization feature with zero padding, and the maximum pooling is used for downsampling.

[0062] S39: The features extracted from the two paths are fused based on the gated attention mechanism to obtain fused features.

[0063] Specifically, in this step, according to the formula The features extracted by the two paths are fused to obtain preliminary fusion features Among them, P l Represents the features extracted from the l-th layer temporal enhancement feature along the top-down path, N l Represents the features extracted from the l-th layer temporal enhancement feature along the bottom-up path, σ is the Sigmoid function, ⊙ represents element-by-element multiplication; and the initial fusion features of each layer After upsampling to a uniform size and then splicing, the fusion feature is obtained.

[0064] For steps S38-S39, in this embodiment, for each layer of timing enhancement features Feature fusion is achieved through top-down (semantic) and bottom-up (detail) dual paths, and the adaptive fusion of multi-scale features is achieved by combining the gating mechanism. That is, in the field of micro-expressions, a dynamic and balanced fusion of high-frequency muscle tremors and semantic features is achieved, taking into account both high-frequency details and semantic information.

[0065] From the above, for steps S31-S39, the region of interest is quickly located through coarse-grained gated spatial attention, and then the instantaneous changes are accurately captured through fine-grained temporal attention. The dual-path feature pyramid model and the gating mechanism are combined to achieve adaptive fusion of multi-scale features, which provides an efficient and robust feature extraction solution for micro-expression recognition and analysis, and achieves rapid and accurate capture of dynamic features of micro-expressions (such as focusing on capturing features around the patient's eyes and mouth corners during online remote consultations), suppressing interference from irrelevant areas (such as the forehead and nose of the face) and irrelevant frames (such as frames without micro-expressions). Compared with the global equal processing strategy of features, it can reduce the computational overhead by about 35%, thereby improving computational efficiency and recognition accuracy. For example, in the diagnosis of emotional disorders in smart medical care, if the patient appears in the 3S (for example, the 75th frame of the micro-expression video to be identified), A brief "nervous" micro-expression, lasting about 0.2 seconds, involves pursing the corners of the mouth and rapid blinking of the eyelids. Based on the above steps of the present invention, the weights of the corners of the mouth and the right eye area are continuously enhanced in the 75th-81st frames of the micro-expression video to be identified (corresponding to the onset-apex stage of the micro-expression), and the importance weight of the frame corresponding to the apex stage (for example, the 78th frame) is increased to be significantly higher than the background frame. In addition, the dual-path feature pyramid model can be used to retain muscle tremor details in the level-1 feature (56×56 resolution) and capture the overall expression semantics in the level-4 feature, thereby obtaining a fusion feature that focuses on the effective area and key frames and dynamically balances the fusion of high-frequency muscle tremor and semantic features, which facilitates auxiliary judgment of the severity of the patient's symptoms, thereby facilitating the subsequent rapid and accurate judgment of the severity of the patient's symptoms.

[0066] S40: Based on the pre-trained classification model, the fused features are used to identify the expression category and obtain the micro-expression recognition results.

[0067] In this step, the fusion features are used to classify expressions based on the pre-trained classification model constructed by the temporal shift module and the fully connected layer, and the temporal classification results are integrated to generate the final micro-expression category and confidence.

[0068] It can be seen that in the above scheme, when performing user micro-expression recognition on videos such as medical consultation imaging videos of smart medical care and loan information approval of financial technology, multi-level and multi-scale feature extraction can be performed on the video frame sequence to obtain multi-layer multi-scale features. Then, based on the gated attention mechanism, the regions of interest of the multi-scale features of each layer are quickly located, and content enhancement features focusing on local facial areas (such as the periphery of the eyes and the corners of the mouth) are generated. Then, based on the temporal attention mechanism, instantaneous changes are accurately captured, and temporal enhancement features focusing on key time frames are generated. The dual-path feature pyramid model is combined with the gating mechanism to enhance the temporal enhancement of each layer. Strong features are adaptively fused at multiple scales to obtain fused features, and then micro-expression categories are identified based on the fused features, which can adapt to multi-scale micro-expression changes, and accurately capture the dynamic features of micro-expressions through cascaded attention design. In the field of micro-expressions, dynamic balanced fusion of high-frequency muscle tremors and semantic features can be achieved, taking into account both high-frequency details and semantic information. It can be seen that not only can computing resources be concentrated on effective areas and key frames, while avoiding redundant calculations while avoiding noise interference in other areas, but also the feature discriminability can be significantly improved, the computing efficiency is greatly improved, and the recognition accuracy is improved, which is suitable for end-to-end training.

[0069] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0070] In one embodiment, a micro-expression recognition device is provided, which corresponds one-to-one with the micro-expression recognition method in the above embodiment. Figure 5 As shown, the micro-expression recognition device includes a pre-processing module 110, a feature layer extraction module 120, a hierarchical selection attention module 130 and a classification module 140. The functional modules are described in detail as follows:

[0071] A preprocessing module 110 is configured to obtain a micro-expression video to be recognized, and perform normalization processing on the micro-expression video to be recognized to obtain a first video;

[0072] A feature layer extraction module 120 is configured to extract features from a video frame sequence of the first video to obtain multi-layer multi-scale features;

[0073] The hierarchical selection attention module 130 includes a spatial attention unit 131, a temporal attention unit 132, and a feature fusion unit 133. The spatial attention unit 131 is used to perform attention calculation on the multi-scale features of each layer based on the gated attention mechanism to generate spatial enhancement features; the temporal attention unit 132 is used to perform temporal attention calculation on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features; the feature fusion unit 133 is used to perform multi-scale fusion on the temporal enhancement features of each layer to obtain fused features;

[0074] The classification module 140 is used to perform expression category recognition on the fusion features based on the pre-trained classification model to obtain micro-expression recognition results.

[0075] In one embodiment, the spatial attention unit 131 is specifically configured to:

[0076] Perform temporal context encoding on the multi-scale features of each layer of each video frame in the video frame sequence to obtain a feature vector containing context information;

[0077] Based on the gated attention mechanism, the feature vector of each layer is paid attention to and the spatial weight matrix is ​​obtained;

[0078] The multi-scale features are spatially weighted according to the spatial weight matrix to obtain spatial enhancement features.

[0079] In one embodiment, the spatial attention unit 131 is specifically configured to:

[0080] Use the gating mechanism to update the hidden state of the feature vector of each layer;

[0081] According to the formula Perform attention calculation to obtain the spatial weight matrix in, and is a learnable parameter, is the hidden state, and σ is the Sigmoid function.

[0082] In one embodiment, the temporal attention unit 132 is specifically configured to:

[0083] Compress the spatial enhancement features of each layer of each video frame in the video frame sequence to obtain frame-level features;

[0084] Add learnable position encoding to frame-level features to obtain frame-level features that retain temporal information;

[0085] Attention is calculated on the frame-level features retaining the time sequence information through a multi-head attention mechanism to obtain an attention matrix, and the average attention of the attention matrix along the time dimension is calculated to obtain a frame weight matrix;

[0086] The spatial enhancement features are weighted according to the frame weight matrix to obtain the temporal enhancement features.

[0087] In one embodiment, the feature fusion unit 133 is specifically configured to:

[0088] The dual-path feature pyramid model is used to extract the temporal enhancement features of each layer along the top-down and bottom-up paths respectively;

[0089] The features extracted from the two paths are fused based on the gated attention mechanism to obtain the fused features.

[0090] In one embodiment, the feature fusion unit 133 is specifically configured to:

[0091] According to the formula The features extracted by the two paths are fused to obtain preliminary fusion features Among them, P l Represents the features extracted from the l-th layer temporal enhancement feature along the top-down path, N l It represents the features extracted from the l-th layer temporal enhancement feature along the bottom-up path, σ is the Sigmoid function, and ⊙ represents element-by-element multiplication;

[0092] Preliminary fusion features of each layer After upsampling, the images are spliced ​​together to obtain fused features.

[0093] It can be understood that the hierarchical selective attention module 130 integrates a composite structure of spatial / temporal attention and multi-scale feature fusion, and realizes the hierarchical attention collaboration of GRU (Gated Recurrent Unit) and Transformer for the first time. Compared with the traditional single-path feature extraction method, the computational efficiency is improved and the accuracy is improved.

[0094] In one embodiment, the pre-processing module 110 is specifically configured to:

[0095] Acquire a micro-expression video to be recognized, and perform facial region positioning on an image of each frame in the micro-expression video to be recognized;

[0096] Align the positioned image of each frame to the preset size;

[0097] The aligned images of each frame are normalized to obtain a first video.

[0098] The present invention provides a micro-expression recognition device, which obtains a micro-expression video to be recognized, normalizes the micro-expression video to be recognized, and obtains a first video; performs feature extraction on a video frame sequence of the first video to obtain multi-layer multi-scale features; then performs attention calculation on the multi-scale features of each layer based on a gated attention mechanism to generate spatial enhancement features, and performs temporal attention calculation on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features, and performs multi-scale fusion on the temporal enhancement features of each layer to obtain fusion features; finally, performs expression category recognition on the fusion features based on a pre-trained classification model to obtain a micro-expression recognition result, that is, when working, the video frame sequence can be subjected to multi-level and multi-scale feature extraction. The multi-scale features of multiple layers are extracted, and then the regions of interest of each layer are quickly located based on the gated attention mechanism, and content enhancement features focusing on local facial areas (such as the periphery of the eyes and the corners of the mouth) are generated. Then, the instantaneous changes are accurately captured based on the temporal attention mechanism, and timing enhancement features focusing on key timing frames are generated. The timing enhancement features of each layer are fused at multiple scales to obtain fused features, so that the micro-expression categories can be identified according to the fused features, which can adapt to the changes of multi-scale micro-expressions. The cascaded attention design can realize the accurate capture of the dynamic features of micro-expressions, and the computing resources can be concentrated on the effective areas and key frames to avoid noise interference in other areas. At the same time, redundant calculations can be avoided, and the computing efficiency is greatly improved and the recognition accuracy is improved.

[0099] The specific definitions of the micro-expression recognition device can be found in the definitions of the micro-expression recognition method above and will not be repeated here. Each module in the aforementioned micro-expression recognition device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0100] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a micro-expression recognition method.

[0101] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of a micro-expression recognition method.

[0102] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0103] Obtaining a micro-expression video to be recognized, and performing normalization processing on the micro-expression video to be recognized to obtain a first video;

[0104] Performing feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features;

[0105] Based on the gated attention mechanism, attention calculation is performed on the multi-scale features of each layer to generate spatial enhancement features, and temporal attention calculation is performed on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features. The temporal enhancement features of each layer are then multi-scale fused to obtain fused features.

[0106] The expression category is recognized based on the fusion features based on the pre-trained classification model to obtain the micro-expression recognition results.

[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0108] Obtaining a micro-expression video to be recognized, and performing normalization processing on the micro-expression video to be recognized to obtain a first video;

[0109] Performing feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features;

[0110] Based on the gated attention mechanism, attention calculation is performed on the multi-scale features of each layer to generate spatial enhancement features, and temporal attention calculation is performed on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features. The temporal enhancement features of each layer are then multi-scale fused to obtain fused features.

[0111] The expression category is recognized based on the fusion features based on the pre-trained classification model to obtain the micro-expression recognition results.

[0112] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0113] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0114] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] The above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention and are therefore intended to be included within the scope of protection of the present invention. Furthermore, any software tools or components not owned by the Company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A micro-expression recognition method, characterized in that: include: Obtaining a micro-expression video to be recognized, and performing normalization processing on the micro-expression video to be recognized to obtain a first video; Performing feature extraction on the video frame sequence of the first video to obtain multi-layer multi-scale features; Based on the gated attention mechanism, attention calculation is performed on the multi-scale features of each layer to generate spatial enhancement features, and temporal attention calculation is performed on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features. The temporal enhancement features of each layer are then multi-scale fused to obtain fused features. The expression category is recognized based on the fusion features based on the pre-trained classification model to obtain the micro-expression recognition results.

2. The micro-expression recognition method according to claim 1, wherein: The gated attention mechanism is used to calculate the multi-scale features of each layer to generate spatial enhancement features, including: Perform temporal context encoding on the multi-scale features of each layer of each video frame in the video frame sequence to obtain a feature vector containing context information; Based on the gated attention mechanism, the feature vector of each layer is paid attention to and the spatial weight matrix is ​​obtained; The multi-scale features are spatially weighted according to the spatial weight matrix to obtain spatial enhancement features.

3. The micro-expression recognition method according to claim 2, wherein: The gated attention mechanism is used to calculate the attention of the feature vectors of each layer to obtain a spatial weight matrix, including: Use the gating mechanism to update the hidden state of the feature vector of each layer; According to the formula Perform attention calculation to obtain the spatial weight matrix in, and is a learnable parameter, is the hidden state, and σ is the Sigmoid function.

4. The micro-expression recognition method according to claim 1, wherein: The temporal attention calculation is performed on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features, including: Compress the spatial enhancement features of each layer of each video frame in the video frame sequence to obtain frame-level features; Add learnable position encoding to frame-level features to obtain frame-level features that retain temporal information; Attention is calculated on the frame-level features retaining the time sequence information through a multi-head attention mechanism to obtain an attention matrix, and the average attention of the attention matrix along the time dimension is calculated to obtain a frame weight matrix; The spatial enhancement features are weighted according to the frame weight matrix to obtain the temporal enhancement features.

5. The micro-expression recognition method according to claim 1, wherein: The multi-scale fusion of the temporal enhancement features of each layer is performed to obtain fused features, including: The dual-path feature pyramid model is used to extract the temporal enhancement features of each layer along the top-down and bottom-up paths respectively; The features extracted from the two paths are fused based on the gated attention mechanism to obtain the fused features.

6. The micro-expression recognition method according to claim 5, wherein: The gated attention mechanism is used to fuse the features extracted from the two paths to obtain fused features, including: According to the formula The features extracted by the two paths are fused to obtain preliminary fusion features Among them, P l Represents the features extracted from the l-th layer temporal enhancement feature along the top-down path, N l It represents the features extracted from the l-th layer temporal enhancement feature along the bottom-up path, σ is the Sigmoid function, and ⊙ represents element-by-element multiplication; Preliminary fusion features of each layer After upsampling, the images are spliced ​​together to obtain fused features.

7. The micro-expression recognition method according to claim 1, wherein: The step of obtaining a micro-expression video to be identified and performing normalization processing on the micro-expression video to be identified to obtain a first video includes: Acquire a micro-expression video to be recognized, and perform facial region positioning on an image of each frame in the micro-expression video to be recognized; Align the positioned image of each frame to the preset size; The aligned images of each frame are normalized to obtain a first video.

8. A micro-expression recognition device, characterized in that: include: a preprocessing module, configured to obtain a micro-expression video to be identified, and perform normalization processing on the micro-expression video to be identified to obtain a first video; A feature hierarchical extraction module, configured to extract features from a video frame sequence of the first video to obtain multi-layer multi-scale features; The hierarchical selective attention module is used to perform attention calculation on the multi-scale features of each layer based on the gated attention mechanism to generate spatial enhancement features, and perform temporal attention calculation on the spatial enhancement features of each layer along the time dimension to generate temporal enhancement features, and then perform multi-scale fusion on the temporal enhancement features of each layer to obtain fused features; The classification module is used to identify expression categories based on the fusion features based on the pre-trained classification model to obtain micro-expression recognition results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the micro-expression recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the micro-expression recognition method according to any one of claims 1 to 7 are implemented.

Images