Micro-expression recognition method and device, and training method of micro-expression recognition model

Abstract

The application provides a micro-expression recognition method and device, a micro-expression recognition model training method, an electronic device and a storage medium. The micro-expression recognition method comprises: acquiring a plurality of image blocks of a to-be-recognized image; based on a multi-head self-attention mechanism, self-attention features of the plurality of image blocks are extracted respectively to obtain feature maps of the plurality of image blocks; based on a double channel, attention weights of the feature maps are learned respectively to adjust the feature maps to obtain target feature maps; and micro-expression recognition of the to-be-recognized image is performed based on the target feature maps. The technical scheme of the embodiment of the application can realize accurate micro-expression recognition.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to a micro-expression recognition method, device, training method of micro-expression recognition model, electronic device and storage medium. Background Technology

[0002] Micro-expressions, due to their short duration and low muscle fluctuations, pose a significant challenge to automatic recognition technology. Traditional micro-expression recognition methods are generally based on handcrafted features, such as local binary patterns, optical flow histograms, and gradient histograms, to achieve micro-expression analysis. However, these methods rely excessively on prior knowledge, and the extracted information is mostly superficial, lacking abstract features that characterize micro-expressions.

[0003] In recent years, methods based on convolutional neural networks have become popular and have been applied to automatically recognize facial micro-expressions. However, these methods require massive amounts of data to train the model, while micro-expression data is often relatively scarce, making it impossible to accurately recognize micro-expressions. In addition, convolutional neural networks have weak global modeling capabilities and cannot perceive changes in micro-expressions based on global facial muscle movements. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a micro-expression recognition method, apparatus, training method for a micro-expression recognition model, electronic device, and computer-readable storage medium.

[0005] According to one aspect of the embodiments of this application, a micro-expression recognition method is provided, comprising: acquiring multiple image blocks of an image to be recognized; extracting self-attention features of the multiple image blocks respectively based on a multi-head self-attention mechanism to obtain feature maps of the multiple image blocks; learning attention weights of the feature maps respectively based on dual channels to adjust the feature maps to obtain target feature maps; and performing micro-expression recognition of the image to be recognized based on the target feature maps.

[0006] In one embodiment, the step of extracting self-attention features from the multiple image patches based on a multi-head self-attention mechanism to obtain feature maps of the multiple image patches includes:

[0007] The plurality of image blocks are divided into different sets of image blocks;

[0008] Extract set self-attention features for each set of image patches separately;

[0009] The feature maps of the multiple image patches are obtained by concatenating the set self-attention features of each set of image patches.

[0010] In one embodiment, the plurality of image patches includes a first set containing facial key points and a second set not containing facial key points; the step of learning attention weights for the feature maps based on dual channels to adjust the feature maps and obtain a target feature map includes:

[0011] Based on dual-channel learning of the feature maps of the first set and the second set respectively, the first attention weight and the second attention weight are obtained accordingly.

[0012] The feature map of the first set is adjusted based on the first attention weight, and the feature map of the second set is adjusted based on the second attention weight;

[0013] The target feature map is obtained by concatenating the feature maps of the first set and the second set after adjustment.

[0014] In one embodiment, before learning the feature maps of the first set and the feature maps of the second set based on dual channels respectively, and obtaining the first attention weight and the second attention weight accordingly, the method further includes:

[0015] Locate the key facial features in the image to be identified;

[0016] The image blocks containing the facial key points are designated as the first set, and the image blocks that do not contain the facial key points are designated as the second set.

[0017] In one embodiment, locating facial key points in the image to be identified includes:

[0018] Obtain the initial facial key points in the image to be identified;

[0019] Remove facial contour-related key points from the initial facial key points to obtain the first key points;

[0020] Based on the first key point, the location of the cheek in the image to be identified is located to obtain the second key point;

[0021] The first key point and the second key point are used as the facial key points.

[0022] In one embodiment, the step of locating the cheek in the image to be identified based on the first key point to obtain the second key point includes:

[0023] Select a set of target key points from the first key point;

[0024] Calculate the center point between the key points in the target key point set;

[0025] The center point is fixedly offset, and the center point after the fixed offset and the center point are used as the second key point.

[0026] According to one aspect of the embodiments of this application, a training method for a micro-expression recognition model is provided, comprising: inputting a training image into an initial micro-expression recognition model, performing random masking deactivation processing on multiple training image blocks of the training image in the initial micro-expression recognition model, obtaining training feature maps of the multiple training image blocks with random masking deactivation processing based on a multi-head self-attention mechanism, adjusting the training feature maps based on dual channels to obtain a target training feature map, and obtaining a training prediction result based on the target training feature map; and training the initial micro-expression recognition model according to the prediction result output by a pre-trained teacher model for the training image and the training prediction result.

[0027] According to one aspect of the embodiments of this application, a micro-expression recognition device is provided, comprising: an image patch acquisition module configured to acquire multiple image patches of an image to be recognized; a feature map acquisition module configured to extract self-attention features of the multiple image patches respectively based on a multi-head self-attention mechanism to obtain feature maps of the multiple image patches; a target feature map module configured to learn attention weights of the feature maps respectively based on dual channels to adjust the feature maps to obtain a target feature map; and a micro-expression recognition module configured to perform micro-expression recognition of the image to be recognized based on the target feature map.

[0028] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the micro-expression recognition method or the micro-expression recognition model training method as described above.

[0029] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the micro-expression recognition method or the micro-expression recognition model training method as described above.

[0030] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the micro-expression recognition method or micro-expression recognition model training method provided in the various optional embodiments described above.

[0031] In the technical solution provided in the embodiments of this application, the multi-head self-attention mechanism deeply mines the features of the image to be recognized, and adaptively learns the importance of different channels of the feature vector through the dual-channel perception unit, thereby accurately realizing micro-expression recognition.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0034] Figure 1 This is a schematic diagram of one implementation environment involved in this application;

[0035] Figure 2 This is a flowchart illustrating a micro-expression recognition method in an exemplary embodiment of this application;

[0036] Figure 3 This is a structural diagram illustrating a micro-expression recognition model as shown in an exemplary embodiment of this application;

[0037] Figure 4 yes Figure 2 A flowchart of step S230 in an exemplary embodiment shown in the illustration;

[0038] Figure 5 yes Figure 2 A flowchart of step S250 in an exemplary embodiment shown in the illustration;

[0039] Figure 6 This is a structural diagram of a dual-channel sensing unit shown in an exemplary embodiment of this application;

[0040] Figure 7 This is a flowchart illustrating a micro-expression recognition method, as shown in another exemplary embodiment of this application;

[0041] Figure 8 This is a flowchart illustrating a training method for a micro-expression recognition model, as shown in an exemplary embodiment of this application.

[0042] Figure 9 This is a flowchart illustrating a training method for a micro-expression recognition model, as shown in another exemplary embodiment of this application.

[0043] Figure 10 This is a diagram illustrating a hard distillation process, as shown in an exemplary embodiment of this application.

[0044] Figure 11 This is a schematic diagram illustrating the structure of a micro-expression recognition device according to an exemplary embodiment of this application;

[0045] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0048] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0049] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0050] Advantages and disadvantages of existing micro-expression recognition methods:

[0051] Advantages of using traditional methods to identify micro-expressions:

[0052] The hand-designed feature extractor extracts features such as image texture and edges, which is easy to implement and the model is highly interpretable.

[0053] Shortcomings of traditional methods for identifying micro-expressions:

[0054] It relies too heavily on prior knowledge and the extracted information is mostly superficial, lacking abstract features that characterize micro-expressions; it requires complex experimental design and tedious parameter adjustments to obtain the ideal result model.

[0055] Advantages of using neural networks to recognize micro-expressions:

[0056] Neural networks have powerful feature extraction capabilities that can effectively extract facial features and recognize micro-expressions in a fully intelligent way.

[0057] The shortcomings of using neural networks to identify micro-expressions:

[0058] The model relies on massive amounts of micro-expression training data. Convolutional neural networks have weak global relationship modeling capabilities and cannot perceive micro-expression changes based on global facial muscle movements.

[0059] The following will provide a detailed description of the micro-expression recognition method, apparatus, training method for micro-expression recognition model, electronic device, and storage medium proposed in the embodiments of this application.

[0060] Please refer to the following first. Figure 1 , Figure 1 This is a schematic diagram of an implementation environment related to this application. The implementation environment includes a terminal 100 and a server 200, which communicate with each other via a wired or wireless network.

[0061] Terminal 100 is used to receive an image to be identified, which should contain the facial image of the task, so as to perform micro-expression recognition based on the facial image. The image to be identified can be an image frame in a video. When it is necessary to perform micro-expression recognition on members in a video, micro-expression recognition can be performed on the image frames of the video.

[0062] Terminal 100 also sends the image to be recognized to server 200. Server 200 is equipped with a pre-trained micro-expression recognition model so that the micro-expression recognition model in server 200 can recognize the micro-expressions of the members in the image to be recognized and obtain the recognition results. Finally, the recognition results can be visualized and displayed through the display module built into terminal 100.

[0063] For example, after receiving the image to be recognized, the terminal 100 sends the image to the server 200; after receiving the image to be recognized, the server 200 obtains multiple image patches of the image to be recognized; based on a multi-head self-attention mechanism, it extracts the self-attention features of the multiple image patches respectively to obtain feature maps of the multiple image patches; it learns the attention weights of the feature maps based on dual channels respectively to adjust the feature maps to obtain target feature maps; and it performs micro-expression recognition of the image to be recognized based on the target feature maps.

[0064] The terminal 100 can be any electronic device capable of data visualization, such as a smartphone, tablet, laptop, or computer; no restrictions are imposed here. The server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. Multiple servers can form a blockchain, with the server being a node on the blockchain. The server 200 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms; no restrictions are imposed here either.

[0065] Of course, the micro-expression recognition method proposed in this embodiment can also be implemented independently in the terminal 100.

[0066] Figure 2 This is a flowchart illustrating a micro-expression recognition method according to an exemplary embodiment. This micro-expression recognition method can be applied to… Figure 1 The implementation environment shown is specifically executed by the server 200 in that implementation environment. It should be understood that the method can also be used in other exemplary implementation environments and specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0067] like Figure 2 As shown, in an exemplary embodiment, the method may include steps S210 to S270, which are described in detail below:

[0068] Step S210: Obtain multiple image blocks of the image to be identified.

[0069] In this embodiment, micro-expression recognition is performed using a pre-trained micro-expression recognition model. The structure of this micro-expression recognition model can be found in [reference]. Figure 3 It includes a preprocessing module, a Transformer module (Transformer: a machine learning model) and a prediction module, wherein the Transformer module also includes a multi-head self-attention unit and a dual-channel perception unit.

[0070] In one specific embodiment, the image to be identified first enters the preprocessing module, which segments the image to be identified to obtain multiple image blocks.

[0071] Step S230: Based on the multi-head self-attention mechanism, extract the self-attention features of multiple image patches respectively to obtain feature maps of multiple image patches.

[0072] When multiple image patches arrive at the Transformer module, its multi-head self-attention unit first encodes these patches to obtain a two-dimensional image patch sequence. Then, two learnable vectors x are introduced. class and x distill The two vectors are concatenated with a sequence of two-dimensional image patches to obtain a concatenated two-dimensional image patch sequence. The subsequent micro-expression recognition model then uses x... class and x distill Perform micro-expression recognition.

[0073] The multi-head self-attention unit includes multiple heads. The image blocks in the stitched two-dimensional image block sequence are sequentially assigned to heads with different values, and self-attention feature extraction is performed on each head. That is, each head can obtain a corresponding feature sequence. Then, the feature sequences calculated by each head are stitched together and linearly mapped to obtain a feature map of the same size as the input (i.e., the stitched two-dimensional image block sequence).

[0074] In this embodiment, the "multi-head" in the multi-head self-attention unit can be regarded as multiple channels. That is, by segmenting and stitching the two-dimensional image block sequence, the multiple image blocks obtained after segmentation are sent to different channels for self-attention feature acquisition. Finally, the feature sequences of each channel are stitched together to obtain the feature map of the stitched two-dimensional image block sequence.

[0075] Step S250: Learn the attention weights of the feature maps based on the dual channels respectively, so as to adjust the feature maps and obtain the target feature map.

[0076] Of course, there are also some unit structures in the Transformer module. Figure 3 The LN layer (used for normalization operations) and residual connection unit are not shown in the figure. After the multi-head self-attention unit outputs the feature map, the LN layer will perform a normalization operation on the output feature map. The residual connection unit will concatenate the output feature map with the data input to the multi-head self-attention unit and send the concatenated feature map to the dual-channel perception unit.

[0077] In this embodiment, the multiple image blocks of the image to be identified include a first set consisting of image blocks containing facial key points and a second set consisting of image blocks not containing facial key points. The dual-channel perception unit adaptively learns the channel attention weights of the input feature sequence from the first set and the second set, respectively.

[0078] This is equivalent to a dual-channel perception unit containing two channels. One channel learns the feature map corresponding to the second set, and the other set learns the features corresponding to the first set. The feature map corresponding to the first set is the feature map obtained by the image block containing facial key points through the steps shown in step S230.

[0079] In this way, the two channels will output the first attention weight and the second attention weight respectively. Then, the feature map of the first set is adjusted by the first attention weight, and the feature map of the second set is adjusted based on the second attention weight. The adjusted feature maps of the first set and the adjusted feature maps of the second set are restored according to their positions when they were input into the dual-channel perception unit, so as to obtain the target feature map with the same size as the input dual-channel perception unit.

[0080] Step S270: Perform micro-expression recognition on the image to be recognized based on the target feature map.

[0081] Similar to the processing of the feature map output by the multi-head self-attention unit, after the target feature map of the same size as the dual-channel perception unit, the LN layer is used to normalize the output target feature map. The residual connection unit then concatenates the output target feature map with the data input to the dual-channel perception unit and sends the concatenated target feature map to the prediction unit for micro-expression recognition. The vector x in this target feature map... class and x distill The numerical values ​​have been learned, and can be obtained through the vector x in the target feature map. class and x distill To perform micro-expression recognition.

[0082] In this embodiment, the prediction unit can be an MLP Head classifier to recognize micro-expressions using the MLP Head.

[0083] In this embodiment, a multi-head self-attention mechanism is set up to deeply mine the features of the image to be identified, providing richer feature data for subsequent micro-expression recognition. On the other hand, the image blocks are divided into different sets based on facial key points, and the dual-channel perception unit adaptively learns the importance of different channels of the feature vector based on different sets. Then, the feature vector is re-evaluated according to the importance value so that the model can adapt to various micro-expression recognition scenarios and achieve the effect of accurately recognizing facial micro-expression changes.

[0084] The micro-expression recognition method in this embodiment can be applied to scenarios such as interview communication, online education, and fatigued driving. It can capture the unconscious "micro-expressions" displayed by the person being identified in real time, perceive their true feelings and emotional conflicts, and thus help the observer or system to take effective intervention measures or improve their own communication skills. For example, in the scenario of fatigued driving, by detecting the driver's facial micro-expressions in real time, it is possible to accurately determine whether there is fatigued driving, inattention or other abnormal driving states, reduce the occurrence of traffic accidents, and accelerate the pace of building smart and safe cities.

[0085] Figure 4 It is aimed at Figure 2The flowchart of step S230 in the illustrated embodiment is shown in an exemplary embodiment. Figure 4 As shown, in an exemplary embodiment, step S230, based on a multi-head self-attention mechanism, extracts self-attention features from multiple image patches to obtain feature maps of multiple image patches. This process may include steps S410 to S450, which are detailed below:

[0086] Step S410: Divide multiple image blocks into different image block sets.

[0087] In this embodiment, for an input size x∈R 224×224×3 The image to be recognized can be divided into 196 image blocks. The image to be recognized is then encoded into x. p ∈R 196×768 The sequence of two-dimensional image patches, where 196 is the number of image patches and 768 is the length of the one-dimensional image patch vector, further introduces x. class ∈R 768 and x distill ∈R 768 Two learnable vectors are concatenated with a sequence of image patches to obtain x. p ∈R 198×768 As input to the multi-head self-attention unit.

[0088] For x p ∈R 198×768 Then, it can be divided into different image patch sets. The number of image patch sets is the same as the number of heads in the multi-head self-attention unit. For example, if there are 8 heads, then there are 8 image patch sets, and they are allocated according to the order of the heads. For example, the image patch in the image patch set corresponding to the first head is x1∈R. 198×96 The image patch in the set of image patches corresponding to the second head is x2∈R 198×96 The 96 corresponding to the second head is the 96 units following the 96 corresponding to the first head. In this way, the set of image blocks corresponding to each head can be allocated.

[0089] Step S430: Extract the set self-attention features of each set of image patches for different sets of image patches.

[0090] Each head processes the corresponding set of image patches to obtain the corresponding set of self-attention features.

[0091] In one specific embodiment, each head contains three fully connected layers. The three fully connected layers calculate q, k, and v respectively on the input image patch set. For example, for the first head, the calculation process is [q, k, v] = x1(W q W k W v ), W q Wk W v The weights of the three fully connected layers are used as inputs, and then the set of self-attention features for the image patch set are calculated using the q, k, and v values ​​computed through the three fully connected layers.

[0092]

[0093] Where d k Let k be the dimension of the sequence, and let O be a constant. h It is a set of self-attention features for a certain set of image patches.

[0094] Step S450: Concatenate the set self-attention features of each image patch set to obtain feature maps of multiple image patches.

[0095] In this embodiment, the feature sequences (collection of self-attention features) calculated by each head are concatenated and then linearly mapped to obtain a feature map O of the same size as the input. final :

[0096] O final =concat(O h1 O h2 , ..., O h8 W H

[0097] Among them, W H These are the weights of the fully connected layer, O h1 It is the set of self-attention features output by the first head.

[0098] Of course, the above is an example of setting up 8 heads. In other embodiments, other values ​​of heads can also be set to achieve self-attention features of multiple image blocks. No specific limitations are made here.

[0099] This embodiment proposes a method for self-attention feature extraction based on a multi-head self-attention mechanism. It directly models the global relationship of all image blocks, prompting the model to extract micro-expression features from the difference information subspace of different image blocks. By setting multiple heads, the features of image blocks are extracted separately to obtain richer and more accurate features, providing reference data for subsequent accurate micro-expression recognition.

[0100] Figure 5 It is aimed at Figure 2 The flowchart of step S250 in the illustrated embodiment is shown in an exemplary embodiment. Figure 5As shown, in an exemplary embodiment, the plurality of image blocks include a first set containing facial key points and a second set not containing facial key points; step S250, which learns attention weights of the feature maps based on dual channels to adjust the feature maps and obtain the target feature map, may include steps S510 to S550, which are described in detail below:

[0101] Step S510: Based on dual channels, learn the feature maps of the first set and the feature maps of the second set respectively, and obtain the first attention weight and the second attention weight accordingly.

[0102] For multiple image patches, facial landmarks can be used to divide them into a first set and a second set. The dual-channel perception unit adaptively learns the channel attention weights of the input feature map from both the first and second sets, and then dynamically adjusts the values ​​in the feature map. This process can be referenced... Figure 6 Regarding x p ∈R 198×768 Given the input, the first set can be x1∈R N1N1×768 The second set can be x2∈R N2×768 .

[0103] The dropout probabilities (dropout is the process of deactivating certain neurons with a certain probability) differ between the first and second sets. The dropout probability for the first set is 0.2, and the dropout probability for the second set is 0.1. The formula for calculating the attention weights is as follows:

[0104] z h =sigmoid(conv2(dropout(Gelu(conv1(z in )))))

[0105] Among them, z h These are attention weights, conv is convolution, Gelu is the activation function, and z... in It is the feature map of the first set or the feature map of the second set, and sigmoid is a function.

[0106] Step S530: Adjust the feature map of the first set based on the first attention weight, and adjust the feature map of the second set based on the second attention weight.

[0107] After obtaining the attention weights, the values ​​at corresponding positions in the feature map can be adjusted based on these attention weights:

[0108] z out =z h ×z in +z in

[0109] Among them, z outIt is either the feature map of the first set after adjustment or the feature map of the second set after adjustment.

[0110] Step S550: Concatenate the feature maps of the adjusted first set and the adjusted second set to obtain the target feature map.

[0111] In this embodiment, the adjusted feature maps of the first set and the adjusted feature maps of the second set are concatenated and fused (MERQE) to restore a target feature map of the same size as the data input to the dual-channel sensing unit.

[0112]

[0113] Among them, z OUT For the target feature map, These are the feature maps of the first set after adjustment and the feature maps of the second set after adjustment, respectively.

[0114] In this embodiment, the importance of different channels of the feature map is adaptively learned by the dual-channel perception unit, and then the feature map is re-evaluated according to the importance value in order to obtain accurate prediction results in the future.

[0115] Figure 7 This is a flowchart illustrating a micro-expression recognition method according to another exemplary embodiment. The method can be run on... Figure 5 Prior to step S510, specifically, the method may be... Figure 2 The process is completed in step S210, that is, in the preprocessing module. This process may include steps S710 to S730, which are described in detail below:

[0116] Step S710: Locate the facial key points in the image to be recognized.

[0117] In this embodiment, the first set and the second set are determined in the preprocessing module.

[0118] Specifically, 68 initial facial landmarks are first located on the image to be identified using 2D-FAN (based on human pose estimation architecture), and 18 landmarks of the facial contour are discarded, leaving the remaining 50 initial facial landmarks, namely the 50 first landmarks.

[0119] To effectively characterize muscle changes in the cheek area, four key points need to be added to mark the location of the cheek. These key points can be obtained manually or calculated from the first key point. For example, a target key point set can be selected from the first key point set; the center point between the key points in the target key point set can be calculated, thus obtaining one key point for each of the left and right cheeks; and the center points of the left and right cheeks can be fixedly offset to obtain two more key points with fixed offsets.

[0120] In one specific embodiment, a key point is selected on the brow bone and the lips on the left and right sides of the face respectively to obtain a target key point set. The target key point set includes 4 key points, 2 of which are on the left side of the image to be recognized, and the other two are on the right side of the image to be recognized.

[0121] For the two key points on the left side of the face, one key point is at the brow bone on the left side of the face, and the other key point is at the lip on the left side of the face. For example, the key point at the brow bone is the second key point at the brow bone of the image to be identified, and the key point at the lip on the left side of the face is the first key point of the lower left lip. Then, the center point is calculated based on these two key points on the left side of the face, which gives us a key point for marking the location of the cheek on the left side of the face.

[0122] For the two key points on the right side of the face, the same process of obtaining the key points for marking the location of the cheek on the left side of the face can be used as a reference. In this way, a key point for marking the location of the cheek on the right side of the face can also be obtained.

[0123] Then, based on the fixed offset (x, y) of the left and right corner points of the mouth... left =(x left -16, y left -16, x, yright = xright + 16, yright + 16, calculate the other two points, thus obtaining 4 key points, where (x, y) left The coordinates of the keypoint obtained by fixing the offset of a keypoint on the left cheek, x left and y left Let (x, y) be the coordinates of a key point on the left cheek that marks its location. right The coordinates of the keypoint obtained by fixing the offset of a keypoint on the right cheek, x right and yright The coordinates of a key point on the right cheek that marks its location.

[0124] The four key points and 50 first key points obtained are collectively referred to as facial key points in the image to be identified.

[0125] Step S730: Select the image blocks containing facial key points from the multiple image blocks as the first set, and select the image blocks that do not contain facial key points from the multiple image blocks as the second set.

[0126] For input size x ∈R H×W×C The image to be identified, where H, W, and C represent the height, width, and number of channels (typically 3 channels) of the image, is divided into 16×16 image blocks, resulting in 196 image blocks. Image blocks containing facial key points are assigned to the first set, and other image blocks are assigned to the second set.

[0127] gather.

[0128] This embodiment proposes a method of dividing image blocks into a first set and a second set. By dividing image blocks with and without facial key points, it is easier to process the image blocks through multiple channels and improve the accuracy of subsequent micro-expression recognition.

[0129] based on Figures 2 to 7 Micro-expression recognition methods in China Figure 8 This is a flowchart illustrating a training method for a micro-expression recognition model according to an exemplary embodiment. Figure 8 As shown, in an exemplary embodiment, the method may include steps S810 to S830, which are described in detail below:

[0130] Step S810: Input the image to be trained into the initial micro-expression recognition model, and perform random masking deactivation processing on multiple training image blocks of the image to be trained in the initial micro-expression recognition model. Based on the multi-head self-attention mechanism, obtain the training feature map of multiple training image blocks that have undergone random masking deactivation processing, and adjust the training feature map based on dual channels to obtain the target training feature map, and obtain the training prediction result based on the target training feature map.

[0131] The training method for the micro-expression recognition model in this embodiment can be referred to Figure 9 First, the image to be trained is input into the initial micro-expression recognition model. The initial micro-expression recognition model preprocesses the input data, such as data augmentation: the center of the input image is cropped to R. 224×224×3 Horizontal flipping, colorjitter (adjusting brightness, contrast, saturation, and hue); blending enhancement: mixup (an algorithm used in computer vision to enhance images by mixing classes), cutmix (overlay), with the probability of mixup set to 0.8 and the probability of cutmix set to 1.0; data normalization: calculating the mean μ and standard deviation σ of the dataset of images to be trained, and standardizing the data using Z-Score, etc.

[0132] Similarly, in the preprocessing module of the initial micro-expression recognition model, multiple training image blocks of the images to be trained are also divided into a first training set containing facial key points and a second training set not containing facial key points. The difference is that, during the training phase, the preprocessing module also performs random masking deactivation processing on the first training set and the second training set respectively.

[0133] During the training phase, a mask deactivation mechanism is set up to address the overfitting problem of neural networks in micro-expression tasks. This problem can be divided into two categories: first, neural networks tend to classify micro-expressions through a small number of salient regions; second, there are interdependent and interactive relationships between the feature extractors of the neural network, resulting in poor generalization ability.

[0134] Specifically, each time, image blocks are randomly deactivated in the first training set and the second training set at a ratio of 1 / 8 (deactivation means setting the pixels of the image block to 0). The first training set deactivates image blocks with a high deactivation rate of 0.5, while the second training set deactivates image blocks with a low deactivation rate of 0.3. Deactivating different image blocks in different images can force the model to learn the overall facial features more comprehensively, rather than relying only on local regions to output the discrimination result.

[0135] Subsequently, multiple training image patches that have undergone random masking deactivation are fed into the Transformer module of the initial micro-expression recognition model. The processing within the Transformer module is the same as that in practical applications of the micro-expression recognition model; for details, please refer to [link / reference]. Figures 2 to 7 .

[0136] At this point, the Transformer module outputs the target training feature map, which contains the learned vector x. class and x distill Unlike micro-expression recognition models used in practical applications, this embodiment adds a knowledge distillation module to the prediction module to ensure the initial training level of the micro-expression recognition model.

[0137] Step S830: Train the initial micro-expression recognition model based on the prediction results output by the pre-trained teacher model for the image to be trained, and the training prediction results.

[0138] The self-attention mechanism in self-attention networks lacks inductive bias, while convolutional neural networks (CNNs) inherently possess strong inductive bias: local similarity and translation equivariance. This allows CNNs to achieve better results with less data. This embodiment introduces a knowledge distillation module, enabling the initial micro-expression recognition model to learn the inductive bias of a pre-trained teacher model (RegNetY 16GF, a machine learning model), reducing dependence on data volume. Furthermore, it is demonstrated that hard distillation achieves better results in this task.

[0139] Distillation mechanisms include soft distillation and hard distillation. Soft distillation is achieved by minimizing the KL divergence of the softmax results of the teacher and student models. Its calculation formula is as follows:

[0140]

[0141] Among them, Z s and Z t These refer to the predicted outputs of the student network and the teacher network, respectively; Ψ refers to the softmax operation; τ refers to the distillation temperature; λ refers to the balance coefficient; KL refers to the Kullback-Leibler divergence loss; and L... CE y refers to the cross-entropy loss, and y refers to the true label.

[0142] In this embodiment, hard distillation is used during the training process. The hard distillation process can be found in [reference needed]. Figure 10 It directly uses the discrimination result of the teacher model as another true label, and its formula is as follows:

[0143]

[0144] Among them, y t =argmax(Z t (c) refers to the discrimination result output by the teacher model, y t It has the same function as y.

[0145] In this embodiment, the student network is the initial micro-expression recognition model. However, unlike the micro-expression recognition model, during distillation, the prediction result output by the student network is not based on the vector x in the target training feature map. class and x distill What is obtained is not just for x. distill The result obtained is the prediction output by the teacher model, which is the prediction output for the image to be trained.

[0146] Figure 10 In this context, `patch tokens` are 768-dimensional features obtained by linearly encoding the image after it has been concatenated and cropped in a 16x16 format. `classtoken` and `distillationtoken` are learnable embedding vectors of the same dimension as `patch tokens`. `classtoken` is used to generate the discriminant layer that calculates the loss function between the final image and the ground truth label, while `distillationtoken` is used to generate the discriminant layer that calculates the loss function between the final image and the teacher network output. teacher This is the loss function for the teacher model.

[0147] In this embodiment, the initial micro-expression recognition model is pre-trained on ImageNet1K using the AdamW optimization algorithm. The minimum batch size is 64, the iteration period is 200 rounds, the initial learning rate is 0.0005, and the learning rate decays using a cosine annealing strategy, decreasing every 30 rounds at a decay rate of 0.05. The loss function is... Calculate the hard distillation loss for the initial micro-expression recognition model. Backpropagation recursively reduces this loss, transferring the inductive bias capability of the teacher network to the student network, i.e., the initial micro-expression recognition model. When the loss function is minimized, the prediction result is output; otherwise, the weight matrix is ​​iteratively updated to obtain the trained micro-expression recognition model.

[0148] Regarding the training method for the initial micro-expression recognition model proposed in this embodiment, this embodiment also conducts a comparative analysis of related training methods. First, the first type of method is... Figure 8 The training methods for the initial micro-expression recognition model are as follows: the first method uses random masking deactivation and hard distillation; the second method uses only hard distillation; the third method uses both random masking deactivation and hard distillation, but replaces the dual-channel perceptual unit of the initial micro-expression recognition model with an MLP layer; the fourth method uses only random masking deactivation; and the fifth method uses both random masking deactivation and soft distillation. The accuracy metrics (ACC) are shown in Table 1.

[0149] Training methods ACC (%) First type of method 90.3 Second type of method 87.6 Third type of method 89.4 Fourth type of method 88.2 Fifth type of method 88.2

[0150] Table 1

[0151] As shown in Table 1, random mask deactivation, hard distillation mechanism, and addition of dual-channel sensing units can significantly improve the accuracy of micro-expression recognition model. Hard distillation mechanism also proves that micro-expression recognition model has learned the inductive bias ability of teacher model.

[0152] On the other hand, to verify the model's performance, the trained micro-expression recognition model was sampled from data of over 2000 members on the visual network platform, capturing a total of 20,000 facial images. Micro-expressions were categorized into seven types: happy, angry, neutral, surprised, disgusted, afraid, and sad. Each of the 20,000 facial images was labeled according to these seven expression categories. 16,000 images were selected as the training set, and 4,000 as the test set. The test data was input into the model for verification. The verification showed that the micro-expression recognition model significantly improved in both accuracy and recall. The performance of each algorithm model on the test set is shown in Table 2 below.

[0153] Model Accuracy (%) Recall rate (%) F-Measure (F-score, %) Resnet 50 93.43 82.76 83.1 RegNetY 16GF 86.6 85.45 86.02 ViT-B / 16 79.2 76.3 77.7 DeiT-B 87.6 85.42 86.5 This model 90.3 88.7 89.5

[0154] Table 2

[0155] Among them, ResNet 50, RegNetY 16GF, ViT-B / 16, and DeiT-B are all machine learning models.

[0156] Figure 11This is a schematic diagram illustrating the structure of a micro-expression recognition device according to an exemplary embodiment. Figure 11 As shown, in one exemplary embodiment, the device includes:

[0157] Image block acquisition module 1110 is configured to acquire multiple image blocks of the image to be recognized;

[0158] The feature map acquisition module 1130 is configured to extract the self-attention features of multiple image patches based on a multi-head self-attention mechanism, thereby obtaining feature maps of multiple image patches.

[0159] The target feature map module 1150 is configured to learn attention weights for the feature maps based on dual channels to adjust the feature maps and obtain the target feature map.

[0160] The micro-expression recognition module 1170 is configured to perform micro-expression recognition on the image to be recognized based on the target feature map.

[0161] The micro-expression recognition device proposed in this embodiment can be used for accurate micro-expression recognition.

[0162] In one embodiment, the feature map acquisition module includes:

[0163] The set partitioning unit is configured to divide multiple image blocks into different image block sets;

[0164] The self-attention feature acquisition unit is configured to extract the set self-attention features of each image patch set for different image patch sets;

[0165] The feature map acquisition unit is configured to concatenate the set of self-attention features of each set of image patches to obtain feature maps of multiple image patches.

[0166] In one embodiment, the plurality of image blocks include a first set containing facial key points and a second set not containing facial key points; the target feature map module includes:

[0167] The attention weight acquisition unit is configured to learn the feature maps of the first set and the feature maps of the second set based on dual channels, and obtain the first attention weight and the second attention weight accordingly.

[0168] The feature adjustment unit is configured to adjust the feature maps of the first set based on the first attention weight, and adjust the feature maps of the second set based on the second attention weight;

[0169] The target feature map unit is configured to concatenate the feature maps of the first set and the feature maps of the second set that have been adjusted to obtain the target feature map.

[0170] In one embodiment, the micro-expression recognition device further includes:

[0171] The key point localization module is configured to locate facial key points in the image to be recognized.

[0172] The set partitioning module is configured to take image blocks containing facial key points as the first set and image blocks not containing facial key points as the second set.

[0173] In one embodiment, the key point localization module includes:

[0174] The initial key point localization unit is configured to acquire the initial facial key points in the image to be recognized.

[0175] The contour removal unit is configured to remove facial contour-related key points from the initial facial key points to obtain the first key points;

[0176] The cheek localization unit is configured to locate the cheek in the image to be recognized based on the first key point, and obtain the second key point;

[0177] The facial landmark acquisition unit is configured to use the first landmark and the second landmark as facial landmarks.

[0178] In one embodiment, the cheek positioning unit includes:

[0179] The set is determined by the segment, and configured to select the target key point set from the first key point;

[0180] The center point acquisition module is configured to calculate the center point between key points in the target key point set;

[0181] The cheek positioning module is configured to fix and offset the center point, and use the fixed offset center point and the center point as the second key point.

[0182] It should be noted that the micro-expression recognition device provided in the above embodiments and the micro-expression recognition method provided in the above embodiments belong to the same concept. The specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0183] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the micro-expression recognition method provided in the above embodiments.

[0184] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0185] It should be noted that, Figure 12 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0186] like Figure 12 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0187] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0188] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0189] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0190] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0191] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0192] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the micro-expression recognition method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0193] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the micro-expression recognition method provided in the various embodiments described above.

[0194] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A micro-expression recognition method, characterized in that, include: Obtain multiple image patches from the image to be identified; Based on the multi-head self-attention mechanism, the self-attention features of the multiple image patches are extracted respectively to obtain the feature maps of the multiple image patches; Attention weights for the feature maps are learned separately using two channels to adjust the feature maps and obtain the target feature map. Micro-expression recognition of the image to be identified is performed based on the target feature map; The plurality of image blocks include a first set containing facial key points and a second set not containing facial key points. The step of learning attention weights for the feature maps based on dual channels to adjust the feature maps and obtain a target feature map includes: learning feature maps of the first set and the second set based on dual channels to obtain a first attention weight and a second attention weight; adjusting the feature map of the first set based on the first attention weight and adjusting the feature map of the second set based on the second attention weight; and concatenating the adjusted feature map of the first set and the adjusted feature map of the second set to obtain the target feature map.

2. The method of claim 1, wherein, The multi-head self-attention mechanism extracts self-attention features from the multiple image patches to obtain feature maps of the multiple image patches, including: The plurality of image blocks are divided into different sets of image blocks; Extract set self-attention features for each set of image patches separately; The feature maps of the multiple image patches are obtained by concatenating the set self-attention features of each set of image patches.

3. The method of claim 1, wherein, Before learning the feature maps of the first set and the feature maps of the second set based on dual channels respectively, and obtaining the first attention weight and the second attention weight accordingly, the method further includes: Locate the key facial features in the image to be identified; The image blocks containing the facial key points are designated as the first set, and the image blocks that do not contain the facial key points are designated as the second set.

4. The method of claim 3, wherein, The process of locating facial key points in the image to be identified includes: Obtain the initial facial key points in the image to be identified; Remove facial contour-related key points from the initial facial key points to obtain the first key points; Based on the first key point, the location of the cheek in the image to be identified is located to obtain the second key point; The first key point and the second key point are used as the facial key points.

5. The method of claim 4, wherein, The step of locating the cheek in the image to be identified based on the first key point to obtain the second key point includes: Select a set of target key points from the first key point; Calculate the center point between the key points in the target key point set; The center point is fixedly offset, and the center point after the fixed offset and the center point are used as the second key point.

6. A method for training a micro-expression recognition model, comprising: include: The image to be trained is input into an initial micro-expression recognition model. In the initial micro-expression recognition model, multiple training image blocks of the image to be trained are randomly masked and deactivated. Based on a multi-head self-attention mechanism, training feature maps of multiple training image blocks that have undergone random masking and deactivation are obtained. The training feature maps are then adjusted based on dual channels to obtain a target training feature map. Finally, a training prediction result is obtained based on the target training feature map. The initial micro-expression recognition model is trained based on the prediction results output by the pre-trained teacher model for the image to be trained, and the training prediction results. The plurality of training image blocks include a first training set containing facial key points and a second training set not containing facial key points. The step of adjusting the training feature maps based on dual-channel learning to obtain a target training feature map includes: learning the training feature maps of the first training set and the second training set respectively based on dual-channel learning, and correspondingly obtaining a first attention weight and a second attention weight; adjusting the training feature map of the first training set based on the first attention weight, and adjusting the training feature map of the second training set based on the second attention weight; and concatenating the adjusted training feature map of the first training set and the adjusted training feature map of the second training set to obtain the target training feature map.

7. A micro-expression recognition apparatus, characterized by, include: The image patch acquisition module is configured to acquire multiple image patches of the image to be recognized; The feature map acquisition module is configured to extract the self-attention features of the multiple image blocks based on a multi-head self-attention mechanism, thereby obtaining the feature maps of the multiple image blocks. The target feature map module is configured to learn the attention weights of the feature map based on dual channels respectively, so as to adjust the feature map and obtain the target feature map; The micro-expression recognition module is configured to perform micro-expression recognition on the image to be recognized based on the target feature map; The plurality of image blocks include a first set containing facial key points and a second set not containing facial key points; the target feature map module is further configured to: learn the feature maps of the first set and the second set respectively based on dual channels, and obtain a first attention weight and a second attention weight accordingly; adjust the feature map of the first set based on the first attention weight, and adjust the feature map of the second set based on the second attention weight; and concatenate the adjusted feature map of the first set and the adjusted feature map of the second set to obtain the target feature map.

8. An electronic device, comprising: include: One or more processors; A storage device for storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a computer's processor, cause the computer to perform the method described in any one of claims 1-6.

Images