A micro-expression recognition method and device based on AU guidance

Abstract

The invention discloses a micro-expression recognition method and device based on AU guidance. The micro-expression recognition method comprises: preprocessing a micro-expression sample video to obtain a micro-expression sequence; generating a facial muscle movement text description for each micro-expression sample through an AU coding label of a public data set and a facial action coding system; sending the processed micro-expression sequence to a visual encoder to obtain a coded visual motion feature; sending the motion text description to a text encoder to obtain a text motion feature; by generating AU-guided fine-grained micro-expression facial muscle movement text description, and using CLIP contrast loss to perform cross-modal semantic alignment, the model's perception and capture of weak local micro-expression movements can be improved, significant spatiotemporal micro-expression movement features can be learned, the movement features are effectively converted into emotional features, and the Focal Loss classification loss is used to solve the problem of micro-expression data class imbalance, thereby greatly improving the recognition performance of the model.

Description

Technical Field

[0001] The present application relates to the technical field of micro-expression recognition, and more specifically to a method and device for micro-expression recognition based on AU guidance. Background Art

[0002] Micro-expression (ME) is a type of facial expression on the human face, characterized by short duration and low intensity. Compared with the common facial expressions, or macro-expressions, that we are familiar with, micro-expressions cannot be forged and can reflect a person's most authentic psychological emotions. Therefore, micro-expressions are mostly used for lie detection. In the past few years, the task of micro-expression recognition (MER) has attracted much attention due to its important application value in many fields such as criminal investigation and interrogation, financial investment, psychological diagnosis, and education. Specifically, a micro-expression recognition model classifies a micro-expression image or a micro-expression video to obtain its discrete emotion label. However, since micro-expression movements are very weak and there are large intra-class differences and small inter-class differences, in the micro-expression recognition task, how to accurately capture and distinguish small and fast facial muscle movements and model their relationship with emotions is a very important and difficult challenge.

[0003] Based on the anatomical basis of facial movement, the Facial Action Coding System (FACS) was established. It can encode any facial expression based on the facial action units (AUs) activated by the observed muscles or muscle groups. In the past few decades, FACS has played an important role in various aspects of facial expression analysis. Since AUs provide a description of subtle movements in local areas of the face, they are essential for capturing subtle changes in micro-expressions. Therefore, in order to facilitate micro-expression related analysis, most micro-expression datasets also provide relatively reliable manually annotated AU labels based on FACS. So far, some MER works have attempted to introduce AUs to assist in the learning of micro-expression movement features. These methods are usually limited to customizing some facial regions of interest (ROIs) based on common AUs, and then specifically extracting traditional manual features or deep learning features from these regions for subsequent micro-expression movement representation learning. It should be noted that the performance of these methods is often greatly affected by prior information and has high computational complexity. In addition, some researchers have introduced AUs into graph structures and used graph convolutional neural networks (GCNs) for micro-expression recognition, but they are often prone to overfitting due to the complexity of graph networks and have poor generalization. Therefore, how to fully and effectively utilize the AU labels of micro-expression data to assist in the learning of fine-grained motion features remains a big challenge. Summary of the invention

[0004] In this embodiment, a micro-expression recognition method, device, electronic device and storage medium based on AU guidance are provided to solve the problem in the related art that due to the complexity of the graph network, it is easy to overfit and the generalization is poor.

[0005] In a first aspect, an embodiment of the present invention provides a micro-expression recognition method based on AU guidance, the micro-expression recognition method comprising:

[0006] Acquire a micro-expression sample video, and pre-process the micro-expression sample video to obtain a micro-expression sequence;

[0007] Generate a text description of facial muscle movements for each micro-expression sample using the AU coding labels of the public dataset and the facial action coding system;

[0008] The processed micro-expression sequence is fed into the visual encoder to obtain the encoded visual motion features; the motion text description is fed into the text encoder to obtain the text motion features;

[0009] The visual motion features and text motion features are mapped to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector. The visual embedding vector and the text embedding vector are aligned using the text encoder contrast loss function to enhance the model's ability to understand micro-expression details.

[0010] Emotion classification is performed by feeding visual motion features into the model emotion inference module, and focal loss is used as the target classification loss function;

[0011] For model training, the weights of the text encoder contrast loss function and the target classification loss function are dynamically adjusted, and the trained model is used for micro-expression recognition tasks.

[0012] In an optional embodiment, obtaining an original micro-expression video, preprocessing the original micro-expression video, and obtaining a micro-expression sequence includes:

[0013] Obtaining original micro-expression videos, uniformly downsampling each micro-expression video to form a micro-expression picture sequence, and performing face alignment and cropping on the micro-expression picture sequence to obtain a micro-expression sequence;

[0014] The micro-expression sequence is processed by basic data enhancement and local face mixing.

[0015] In an optional embodiment, the processed micro-expression sequence is fed into a visual encoder to obtain encoded visual motion features, including:

[0016] The micro-expression sequence is passed through a layer of 3D convolution of the visual encoder to perform temporal and spatial downsampling to obtain a spatiotemporal image block;

[0017] Extracting features from the spatiotemporal image blocks to obtain global features and local features;

[0018] A multi-level fusion method is used to obtain the significant visual motion features of the entire micro-expression sequence.

[0019] In an optional embodiment, the motion text description is fed into a text encoder to obtain text motion features, including:

[0020] The motion text description is sent to a pre-trained text encoder whose main architecture is Transformer to obtain the text motion features corresponding to each micro-expression sample video.

[0021] In an optional embodiment, visual motion features and text motion features are mapped to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector; and semantic alignment of the visual embedding vector and the text embedding vector is performed using a text encoder contrast loss function, including:

[0022] defining two learnable projection heads, and using the two learnable projection heads to map the visual motion features and the text motion features to a joint multimodal space to obtain a visual embedding vector and a text embedding vector, wherein the visual embedding vector and the text embedding vector are directly comparable;

[0023] The visual embedding vector and the textual embedding vector are aligned through a text encoder contrastive loss.

[0024] In an optional embodiment, emotion classification is performed by feeding visual motion features into a model emotion inference module, including:

[0025] Adding a Transformer head to the visual encoder, wherein the Transformer head includes two transformer blocks with a width of 512 and a fully connected linear classification layer;

[0026] Emotional features are learned from visual motion features through Transformer blocks and emotion classification is performed.

[0027] In an optional embodiment, the model is trained, the weights of the text encoder contrast loss function and the target classification loss function are dynamically adjusted, and the trained model is used for the micro-expression recognition task, including:

[0028] First, we focus on AU-guided cross-modal alignment to learn fine-grained motion features of micro-expressions;

[0029] Gradually increase the weight of the target classification loss function and reduce the weight of the text encoder contrast loss function.

[0030] Compared with the prior art, the beneficial effects of the micro-expression recognition method based on AU guidance of the present invention are as follows:

[0031] By generating AU-guided fine-grained text descriptions of micro-expression facial muscle movements and using CLIP contrast loss for cross-modal semantic alignment, the model can be improved to perceive and capture weak local micro-expression movements, and learn significant spatiotemporal micro-expression movement features. In addition, by introducing the emotion reasoning module, the motion features are effectively converted into emotion features, and the Focal Loss classification loss is used to solve the problem of imbalanced micro-expression data classes, which greatly improves the recognition performance of the model. At the same time, the use of the local face mixed data enhancement operation proposed in the present invention can increase the diversity of faces without destroying the subtle movement features of micro-expressions, effectively alleviating the model overfitting problem caused by the scarcity of data, and improving the generalization ability of the model.

[0032] In a second aspect, an embodiment of the present invention provides a micro-expression recognition device based on AU guidance, comprising:

[0033] A preprocessing module, used to obtain a micro-expression sample video, pre-process the micro-expression sample video, and obtain a micro-expression sequence;

[0034] The encoding module is used to generate a text description of facial muscle movements for each micro-expression sample using the AU encoding labels of the public dataset and the facial action encoding system;

[0035] The feature extraction module is used to send the processed micro-expression sequence into the visual encoder to obtain the encoded visual motion features; send the motion text description into the text encoder to obtain the text motion features;

[0036] The multimodal alignment module is used to map the visual motion features and text motion features to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector; the visual embedding vector and the text embedding vector are aligned using the text encoder contrast loss function to enhance the model's ability to understand micro-expression details; the visual motion features are fed into the model's emotion inference module for emotion classification, with focal loss as the target classification loss function;

[0037] The training recognition module is used to train the model and dynamically adjust the weights of the text encoder contrast loss function and the target classification loss function. The trained model is used for micro-expression recognition tasks.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the AU-guided micro-expression recognition method as described in the first aspect.

[0040] Compared with the prior art, the beneficial effects of the AU-guided micro-expression recognition device, electronic device and storage medium of the present invention are the same as those of the AU-guided micro-expression recognition method described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 Flow chart of the micro-expression recognition method based on AU guidance in an embodiment of the present invention;

[0043] Figure 2 : is a model diagram of a micro-expression recognition method based on AU guidance in an embodiment of the present invention;

[0044] Figure 3 This is a diagram showing the effect of a local face mixed data enhancement operation in an embodiment of the present invention;

[0045] Figure 4 : is a structural block diagram of a micro-expression recognition device based on AU guidance in an embodiment of the present invention;

[0046] Figure 5 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] In an embodiment of the present invention, a micro-expression recognition method based on AU guidance is provided. Figure 1 : is a flow chart of the micro-expression recognition method based on AU guidance of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0050] S10, obtaining a micro-expression sample video, and pre-processing the micro-expression sample video to obtain a micro-expression sequence;

[0051] Specifically, an original micro-expression video is obtained, and the original micro-expression video is pre-processed to obtain a micro-expression sequence, including:

[0052] Get the original micro-expression video, uniformly downsample each micro-expression video to form a micro-expression picture sequence, and perform face alignment and cropping on the micro-expression picture sequence to obtain a micro-expression sequence ;

[0053] The micro-expression sequence After basic data enhancement and local face mixing processing, the mixed micro-expression sequence is obtained It should be noted that the mixed micro-expression sequence The original key micro-expression motion features are retained while increasing the diversity of samples.

[0054] In this embodiment, each micro-expression video sample is first uniformly downsampled to Frames, and perform face alignment and cropping on each frame of the micro-expression image sequence , get the micro-expression sequence ,in In order to alleviate the overfitting problem caused by insufficient data, the micro-expression sequence The data has been enhanced using basic data enhancement methods such as color perturbation and horizontal flipping, as well as local face mixing.

[0055] like Figure 3 As shown, in this embodiment, the local face blending operation combines a static face image with a certain probability. Partial regions and micro-expression sequences Every frame Specifically, the static face image From another micro-expression sequence The first frame of the expression is the static frame before the expression begins. The face in the image is roughly divided into four regions: forehead (upper quarter), chin (lower quarter), left cheek (left half of the middle two quarters), and right cheek (right half of the middle two quarters). One region is randomly selected for blending for each sample to reduce excessive damage to the original motion pattern. The formalization of the operation is as follows:

[0056] ;

[0057] in is the mask matrix representing the selected blending region. is a coefficient used to control the mixing ratio of the two. , Represents element-wise multiplication.

[0058] S11, generate a text description of facial muscle movement for each micro-expression sample through the AU coding label of the public dataset and the facial action coding system;

[0059] Specifically, using the AU encoding label (such as AU1+AU2) of each micro-expression sample provided by the existing public dataset and the meaning of each AU defined in FACS (such as AU1 represents inner eyebrow raising; AU2 represents outer eyebrow raising), a fine-grained text language description of facial muscle movement is generated for each micro-expression sample. (For example, "The movement characteristics of this micro-expression mainly include the inner eyebrow raising and the outer eyebrow raising").

[0060] S12, sending the processed micro-expression sequence to the visual encoder to obtain the encoded visual motion features;

[0061] Furthermore, the processed micro-expression sequence Feed into the visual encoder , get the encoded visual motion features ,include:

[0062] The micro-expression sequence Vision encoder One layer of 3D convolution performs downsampling in time and space to obtain spatiotemporal image patches;

[0063] Extracting features from the spatiotemporal image blocks to obtain global features and local features;

[0064] A multi-level fusion method is used to obtain the significant visual motion features of the entire micro-expression sequence. .

[0065] Specifically, in this embodiment, the micro-expression sequence Send to the visual encoder with UniformerV2 as the main architecture In the process of extracting spatiotemporal motion features, the encoded visual features are finally obtained. :

[0066] ;

[0067] Specifically, micro-expression sequences First, a layer of 3D convolution is performed to downsample in time and space to obtain The dimension is These image blocks are then fed into a main architecture consisting of 12 serial local UniBlocks and 8 parallel global UniBlocks to extract local and global features. Finally, a multi-level fusion method is used to obtain the significant spatiotemporal motion features of the entire micro-expression image sequence. .

[0068] Furthermore, the motion text description is fed into the text encoder to obtain the text motion feature ;

[0069] Feed the motion text description into the text encoder , get the text motion features, including:

[0070] The motion text description is fed into a pre-trained text encoder whose main architecture is Transformer. In the example above, we get the text motion features corresponding to each micro-expression sample video. .

[0071] Combination Figure 2 As shown in FIG. 1 , in the AU-guided motion learning module of the model of the present invention, cross-modal visual text semantic alignment is used to learn the fine-grained motion features of micro-expressions. Specifically, the AU-guided micro-expression motion text description Feed the pre-trained text encoder whose main architecture is Transformer Text encoder It consists of 12 layers of transformer blocks with heads=8 and width=512, and extracts the text motion features corresponding to each micro-expression sample. It should be noted that the text encoder The parameters are frozen throughout.

[0072] S13, mapping the visual motion features and the text motion features to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector; using a text encoder contrast loss function to align the visual embedding vector and the text embedding vector to enhance the model's ability to understand micro-expression details;

[0073] Furthermore, the visual motion features and text motion features are mapped to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector; the text encoder contrast loss function is used to semantically align the visual embedding vector and the text embedding vector, including:

[0074] defining two learnable projection heads, and using the two learnable projection heads to map the visual motion features and the text motion features to a joint multimodal space to obtain a visual embedding vector and a text embedding vector, wherein the visual embedding vector and the text embedding vector are directly comparable;

[0075] The visual embedding vector and the textual embedding vector are aligned through a text encoder contrastive loss.

[0076] In this embodiment, two learnable projection heads are defined and , which are two sets of learnable parameters, respectively transforming the visual spatiotemporal motion feature vector (spatial-temporal motion feature) extracted from the video and the text motion feature vector extracted from the AU description (text motion feature) Mapping to a joint multimodal space yields new visual embedding vectors that can be directly compared and text embedding vector , where the visual embedding vector , text embedding vector .

[0077] Furthermore, we use the text encoder (CLIP) contrast loss To achieve the visual embedding vector representing the micro-expression motion features and text embedding vector The alignment between and encourages the model to associate specific visual cues with the corresponding text descriptions.

[0078] Furthermore, CLIP contrast loss The calculation method is as follows: In a training batch with N samples, define the sample Visual embedding vectors and samples of The cosine similarity between text embedding vectors is:

[0079] ,but Defined as:

[0080] ;

[0081] in, is a temperature hyperparameter.

[0082] S14, emotion classification is performed by feeding the visual motion features into the model emotion reasoning module, and focal loss is used as the target classification loss function;

[0083] Furthermore, emotion classification is performed by feeding the visual motion features into the model emotion reasoning module, including:

[0084] Adding a Transformer head to the visual encoder, wherein the Transformer head includes two transformer blocks with a width of 512 and a fully connected linear classification layer;

[0085] Emotional features are learned from visual motion features through Transformer blocks and emotion classification is performed.

[0086] Specifically, in this embodiment, another module of the present invention is an emotion reasoning module, which aims to learn the emotion characteristics of micro-expressions and perform emotion classification. Specifically, this module is in the visual encoder Later, a TransformerHead (Transformer head) was introduced , which consists of two transformer blocks with a width of 512 and a fully connected linear classification layer. Therefore, the visual micro-expression motion features obtained in Feed into Transformer head , get the predicted sentiment score , is the number of emotion categories. These scores are obtained by The operation is converted into probability to obtain the predicted probability distribution It should be noted that in order to solve the class imbalance problem, the present invention uses Focalloss as the target classification loss function .

[0087] In this embodiment, the Focal Loss classification loss function The definition is as follows:

[0088] ;

[0089] in It is a sample Belongs to the real category The predicted probability of . It is a parameter that emphasizes samples that are difficult to classify. It is set to the default value of 2 here, and N is the number of samples in the batch.

[0090] S15. Train the model, dynamically adjust the weights of the text encoder contrast loss function and the target classification loss function, and use the trained model for micro-expression recognition tasks.

[0091] Furthermore, the model is trained by dynamically adjusting the weights of the text encoder contrast loss function and the target classification loss function, and the trained model is used for the micro-expression recognition task, including:

[0092] First, we focus on AU-guided cross-modal alignment to learn fine-grained motion features of micro-expressions;

[0093] Gradually increase the weight of the target classification loss function and reduce the weight of the text encoder contrast loss function.

[0094] It should be noted that the present invention uses a progressive training strategy to dynamically balance the learning of the two modules. The model training first focuses on AU-guided cross-modal alignment to learn the fine-grained motion features of micro-expressions, and then gradually shifts to the learning of emotional features to improve the performance of emotion classification. This strategy is achieved by assigning an optimization loss of 1 to 2. and Dynamic weights are used to achieve this, and the formal expression is as follows:

[0095] ;

[0096] In this embodiment, the total weight of the two losses is Fixed to 2.0, Refers to the first Round time The weight size. It decreases linearly during the training process, formalized as:

[0097] ;

[0098] in refer to During training, we set the minimum weight to 1.0. Refers to the total number of training rounds.

[0099] The model is trained using the above method to significantly improve the recognition performance of the model.

[0100] Next, we use the public dataset CAS(ME) 3 , CASME II and SMIC-HS datasets as an example, the recognition results are shown in Table 1:

[0101] Table 1

[0102]

[0103] It can be seen from the technical solution provided by the present invention that by generating AU-guided fine-grained text descriptions of micro-expression facial muscle movements and using CLIP contrast loss for cross-modal semantic alignment, the model's perception and capture of weak local micro-expression movements can be improved, and significant spatiotemporal micro-expression movement features can be learned. In addition, by introducing an emotional reasoning module, motion features are effectively converted into emotional features, and Focal Loss classification loss is used to solve the problem of imbalanced micro-expression data classes, which greatly improves the recognition performance of the model. At the same time, the use of the local face mixed data enhancement operation proposed by the present invention can increase the diversity of faces without destroying the subtle movement features of micro-expressions, effectively alleviating the problem of model overfitting caused by scarce data and improving the generalization ability of the model.

[0104] The embodiment of the present invention also provides a micro-expression recognition device based on AU guidance, which is used to implement the above method embodiment, and the description has been made no further. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0105] like Figure 4 As shown, Figure 4 : is a structural block diagram of a micro-expression recognition device based on AU guidance in the present invention, the device comprises:

[0106] A preprocessing module 101 is used to obtain a micro-expression sample video, pre-process the micro-expression sample video, and obtain a micro-expression sequence;

[0107] An encoding module 102, for generating a facial muscle movement text description for each micro-expression sample by using the AU encoding labels of the public dataset and the facial action encoding system;

[0108] The feature extraction module 103 is used to send the processed micro-expression sequence to the visual encoder to obtain the encoded visual motion features; send the motion text description to the text encoder to obtain the text motion features;

[0109] The multimodal alignment module 104 is used to map the visual motion features and the text motion features to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector; align the visual embedding vector and the text embedding vector using a text encoder contrast loss function to enhance the model's ability to understand micro-expression details; and classify emotions by sending the visual motion features to the model emotion reasoning module, using focal loss as the target classification loss function;

[0110] The training recognition module 105 is used to train the model and dynamically adjust the weights of the text encoder contrast loss function and the target classification loss function. The trained model is used for the micro-expression recognition task.

[0111] Figure 5 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0112] Acquire a micro-expression sample video, and pre-process the micro-expression sample video to obtain a micro-expression sequence;

[0113] Generate a text description of facial muscle movements for each micro-expression sample using the AU coding labels of the public dataset and the facial action coding system;

[0114] The processed micro-expression sequence is fed into the visual encoder to obtain the encoded visual motion features; the motion text description is fed into the text encoder to obtain the text motion features;

[0115] The visual motion features and text motion features are mapped to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector. The visual embedding vector and the text embedding vector are aligned using the text encoder contrast loss function to enhance the model's ability to understand micro-expression details.

[0116] Emotion classification is performed by feeding visual motion features into the model emotion inference module, and focal loss is used as the target classification loss function;

[0117] For model training, the weights of the text encoder contrast loss function and the target classification loss function are dynamically adjusted, and the trained model is used for micro-expression recognition tasks.

[0118] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0119] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0120] Acquire a micro-expression sample video, and pre-process the micro-expression sample video to obtain a micro-expression sequence;

[0121] Generate a text description of facial muscle movements for each micro-expression sample using the AU coding labels of the public dataset and the facial action coding system;

[0122] The processed micro-expression sequence is fed into the visual encoder to obtain the encoded visual motion features; the motion text description is fed into the text encoder to obtain the text motion features;

[0123] The visual motion features and text motion features are mapped to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector. The visual embedding vector and the text embedding vector are aligned using the text encoder contrast loss function to enhance the model's ability to understand micro-expression details.

[0124] Emotion classification is performed by feeding visual motion features into the model emotion inference module, and focal loss is used as the target classification loss function;

[0125] For model training, the weights of the text encoder contrast loss function and the target classification loss function are dynamically adjusted, and the trained model is used for micro-expression recognition tasks.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A micro-expression recognition method based on AU guidance, characterized in that: The micro-expression recognition method comprises: Acquire a micro-expression sample video, and pre-process the micro-expression sample video to obtain a micro-expression sequence; Generate a text description of facial muscle movements for each micro-expression sample using the AU coding labels of the public dataset and the facial action coding system; The processed micro-expression sequence is fed into the visual encoder to obtain the encoded visual motion features; the motion text description is fed into the text encoder to obtain the text motion features; The visual motion features and text motion features are mapped to a joint multimodal space through two learnable projection heads to obtain a visual motion vector and a text embedding vector. The visual embedding vector and the text embedding vector are aligned using the text encoder contrast loss function to enhance the model's ability to understand micro-expression details. Specifically, the following are included: defining two learnable projection heads, and using the two learnable projection heads to map the visual motion features and the text motion features to a joint multimodal space to obtain a visual embedding vector and a text embedding vector, wherein the visual embedding vector and the text embedding vector are directly comparable; aligning the visual embedding vector and the text embedding vector via a text encoder contrastive loss; Emotion classification is performed by feeding visual motion features into the model emotion inference module, and focal loss is used as the target classification loss function; For model training, the weights of the text encoder contrast loss function and the target classification loss function are dynamically adjusted, and the trained model is used for the micro-expression recognition task.

2. The micro-expression recognition method based on AU guidance according to claim 1, characterized in that: Obtaining an original micro-expression video, and preprocessing the original micro-expression video to obtain a micro-expression sequence, including: Obtaining original micro-expression videos, uniformly downsampling each micro-expression video to form a micro-expression picture sequence, and performing face alignment and cropping on the micro-expression picture sequence to obtain a micro-expression sequence; The micro-expression sequence is processed by basic data enhancement and local face mixing.

3. The micro-expression recognition method based on AU guidance according to claim 1, characterized in that: The processed micro-expression sequence is fed into the visual encoder to obtain the encoded visual motion features, including: The micro-expression sequence is passed through a layer of 3D convolution of a visual encoder to perform temporal and spatial downsampling to obtain a spatiotemporal image block; Extracting features from the spatiotemporal image blocks to obtain global features and local features; A multi-level fusion method is used to obtain the significant visual motion features of the entire micro-expression sequence.

4. The micro-expression recognition method based on AU guidance according to claim 1, characterized in that: The motion text description is fed into the text encoder to obtain text motion features, including: The motion text description is sent to a pre-trained text encoder whose main architecture is Transformer to obtain the text motion features corresponding to each micro-expression sample video.

5. The micro-expression recognition method based on AU guidance according to claim 1, characterized in that: Emotion classification is performed by feeding visual motion features into the model emotion inference module, including: Adding a Transformer head to the visual encoder, wherein the Transformer head includes two transformer blocks with a width of 512 and a fully connected linear classification layer; Emotional features are learned from visual motion features through Transformer blocks and emotion classification is performed.

6. The micro-expression recognition method based on AU guidance according to claim 1, characterized in that: The model training is performed by dynamically adjusting the weights of the text encoder contrast loss function and the target classification loss function. The trained model is used for the micro-expression recognition task, including: First, we focus on AU-guided cross-modal alignment to learn fine-grained motion features of micro-expressions; Gradually increase the weight of the target classification loss function and reduce the weight of the text encoder contrast loss function.

7. A micro-expression recognition device based on AU guidance, characterized in that: include: A preprocessing module, used to obtain a micro-expression sample video, pre-process the micro-expression sample video, and obtain a micro-expression sequence; The encoding module is used to generate a text description of facial muscle movements for each micro-expression sample using the AU encoding labels of the public dataset and the facial action encoding system; The feature extraction module is used to send the processed micro-expression sequence into the visual encoder to obtain the encoded visual motion features; send the motion text description into the text encoder to obtain the text motion features; The multimodal alignment module is used to map the visual motion features and text motion features into a joint multimodal space through two learnable projection heads to obtain the visual motion vector and text embedding vector; The contrast loss function of the text encoder is used to align the visual embedding vector and the text embedding vector to enhance the model's ability to understand micro-expression details; specifically: defining two learnable projection heads, and using the two learnable projection heads to map the visual motion features and the text motion features to a joint multimodal space to obtain a visual embedding vector and a text embedding vector, wherein the visual embedding vector and the text embedding vector are directly comparable; aligning the visual embedding vector and the text embedding vector via a text encoder contrastive loss; Emotion classification is performed by feeding visual motion features into the model emotion inference module, and focal loss is used as the target classification loss function; The training recognition module is used to train the model and dynamically adjust the weights of the text encoder contrast loss function and the target classification loss function. The trained model is used for micro-expression recognition tasks.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the micro-expression recognition method based on AU guidance as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the AU-guided micro-expression recognition method as described in any one of claims 1 to 6 are implemented.

Images