A micro-expression recognition method based on feature fusion and structure reparameterization

Abstract

The application provides a micro-expression recognition method based on feature fusion and structure reparameterization, comprising the following steps: S1, acquiring at least one micro-expression video to be detected; S2, inputting into a structure reparameterization model to obtain a micro-expression recognition result; the training process of the structure reparameterization model comprises the following steps: A1, acquiring at least one historical micro-expression video and obtaining a preprocessed image; A2, extracting a face image of a starting frame and a peak value frame and obtaining an optical flow image and a texture image, and then splicing to obtain a spliced image; A3, inputting the spliced image into a feature fusion module to obtain a fused feature map; A4, inputting the fused feature map into a multi-branch network module to obtain a historical recognition result; and A5, training to obtain the structure reparameterization model. The beneficial effect is that the application can fuse multiple features of micro-expression, and accurately analyze and obtain a recognition result through the structure reparameterization model.

Description

Technical Field

[0001] This invention relates to the technical field of micro-expression recognition, and more specifically, to a micro-expression recognition method based on feature fusion and structural reparameterization. Background Technology

[0002] Facial expressions are a way to display an individual's emotional state without verbal communication. However, a person's facial expressions do not always accurately reflect their emotions, as they may intentionally suppress facial expressions they deem inappropriate. Microexpressions, which occur when a person tries to hide their true feelings, are very brief in duration. Because microexpressions can reveal a person's true emotions, they can be applied in many fields, such as depression treatment, lie detection, and criminal interrogation. Compared to normal facial expressions, microexpressions typically last very short, ranging from 1 / 25 to 1 / 5 of a second. In addition to their short duration, microexpressions also have another significant characteristic: low intensity and small range. Due to these three characteristics, microexpressions are difficult to identify with the naked eye.

[0003] Traditional micro-expression recognition methods use operators such as LBP, LBP-TOP, and LBP-SIP to extract texture feature information, or obtain feature information based on geometric transformations through optical flow. Such manually calculated feature information mainly relies on the designer's knowledge and experience, lacks sufficient reliability, and is therefore limited in terms of accuracy.

[0004] In recent years, with the rapid development of deep learning, convolutional neural networks have been able to automatically extract features for classification end-to-end. Micro-expression recognition methods based on deep learning have also received more and more attention and achieved increasingly better performance. However, the micro-expression recognition methods currently used generally only select a single manually extracted feature information as input, often focusing only on one aspect of the micro-expression features and making it difficult to estimate other aspects, resulting in inaccurate recognition results. Summary of the Invention

[0005] The problem this invention aims to solve is to provide a micro-expression recognition method based on feature fusion and structural reparameterization, which can effectively improve the accuracy of recognition results.

[0006] To address the above problems, this invention provides a micro-expression recognition method based on feature fusion and structural reparameterization, comprising:

[0007] Step S1: Obtain at least one micro-expression video to be detected;

[0008] Step S2: Input the micro-expression video to be detected into the pre-trained structural reparameterization model to obtain the corresponding micro-expression recognition result;

[0009] The training process of the structure reparameterization model includes:

[0010] Step A1: Obtain at least one historical micro-expression video, perform face recognition on each video frame in the historical micro-expression video to obtain the corresponding frame image containing the face region, and perform face alignment and cropping preprocessing operations on each frame image to obtain the corresponding preprocessed image.

[0011] Step A2: For each preprocessed image, extract the face images corresponding to the start frame and the peak frame in the preprocessed image, process the two face images to obtain the corresponding optical flow map, and process the face image corresponding to the peak frame to obtain the corresponding texture image. Then, stitch the optical flow map and the texture image together to obtain the stitched image.

[0012] Step A3: Input the stitched image into a pre-constructed feature fusion module to extract time dimension feature maps and spatial dimension feature maps, and perform feature fusion on the time dimension feature maps and spatial dimension feature maps to obtain a fused feature map;

[0013] Step A4: Input the fused feature map into a multi-branch network module of multiple stages to obtain the corresponding local spatial information and long-distance spatial information, and integrate the local spatial information and the long-distance spatial information to obtain the corresponding historical recognition result;

[0014] Step A5: Using the historical micro-expression video as input and the historical recognition result as output, a multi-branch network model is trained, and the multi-branch network model is structurally reparameterized to obtain the structurally reparameterized model.

[0015] Preferably, in step A1, a mixed dataset containing multiple micro-expression sequences is selected as the historical micro-expression video, and each micro-expression sequence is sequentially subjected to face recognition, face alignment, and cropping to obtain the corresponding preprocessed image.

[0016] Preferably, the number of each micro-expression sequence in the mixed dataset is 442, and each micro-expression sequence includes 109 positive expressions, 83 surprised expressions and 250 negative expressions. In step A1, each positive expression, each surprised expression and each negative expression is sequentially subjected to face recognition, face alignment and cropping to obtain the corresponding preprocessed image.

[0017] Preferably, in step A2, the two face images are processed by the TV-L1 optical flow method to obtain the corresponding optical flow maps, and the face image corresponding to the peak frame is processed by the local binary mode texture operator of equivalent mode to obtain the texture image.

[0018] Preferably, the image scale of each optical flow map and the texture image is 256x256x3. Then, in step A2, each optical flow map and the texture image are stitched together to obtain the stitched image with an image scale of 256x256x6.

[0019] Preferably, the feature fusion module includes a spatial attention submodule and a channel attention submodule. In step A3, the stitched images are concatenated and processed serially through the spatial attention submodule and the channel attention submodule to obtain the temporal dimension feature map and the spatial dimension feature map.

[0020] Preferably, each of the multi-branch network modules includes a spatial awareness submodule and a channel awareness submodule. In step A4, the spatial awareness submodule obtains the local spatial information at different scales in the fused feature map, and the channel awareness submodule obtains the long-distance spatial information of each channel in the fused feature map.

[0021] Preferably, the multi-branch network model includes convolutional layers and fully connected layers. In step A5, the parameters of the convolutional layers are injected into the parameters of the fully connected layers using a structural reparameterization method to form the structural reparameterization model.

[0022] The present invention has the following beneficial effects: The present invention introduces a feature fusion module, which performs early fusion of the temporal and spatial features of micro-expressions through an attention mechanism, so that the fused feature map input into the multi-branch network module contains the geometric transformation features and texture features of micro-expressions, and has sufficient spatiotemporal features. The accuracy of the recognition result is improved by the fusion of multiple features. Furthermore, based on the structural reparameterization model, features at various scales of the fused feature map are obtained through multiple branches during training, and the multi-branch model is compressed into a single branch during inference, thereby improving the ability to discriminate micro-expression sequences and thus improving the accuracy of the recognition result. Attached Figure Description

[0023] Figure 1 This is a flowchart of the steps of the present invention;

[0024] Figure 2 This is a flowchart illustrating the training process of the structural reparameterization model of the present invention.

[0025] Figure 3This is a schematic diagram of the feature fusion module of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of the multi-branch network module of the present invention. Detailed Implementation

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0028] In a preferred embodiment of the present invention, based on the above-mentioned problems existing in the prior art, a micro-expression recognition method based on feature fusion and structural reparameterization is provided, such as... Figure 1 As shown, it includes:

[0029] Step S1: Obtain at least one micro-expression video to be detected;

[0030] Step S2: Input the micro-expression video to be detected into the pre-trained structural reparameterization model to obtain the corresponding micro-expression recognition result;

[0031] The training process of the structural reparameterization model is as follows: Figure 2 As shown, it includes:

[0032] Step A1: Obtain at least one historical micro-expression video, perform face recognition on each video frame in the historical micro-expression video to obtain the corresponding frame image containing the face region, and perform face alignment and cropping preprocessing operations on each frame image to obtain the corresponding preprocessed image.

[0033] Step A2: For each preprocessed image, extract the face images corresponding to the start frame and peak frame in the preprocessed image, process the two face images to obtain the corresponding optical flow map, and process the face image corresponding to the peak frame to obtain the corresponding texture image. Then, stitch the optical flow map and texture image together to obtain the stitched image.

[0034] Step A3: Input the stitched image into a pre-built feature fusion module to extract the temporal and spatial feature maps, and perform feature fusion on the temporal and spatial feature maps to obtain the fused feature map;

[0035] Step A4: Input the fused feature map into the multi-branch network module of multiple stages to obtain the corresponding local spatial information and long-range spatial information, and integrate the local spatial information and long-range spatial information to obtain the corresponding historical recognition result;

[0036] Step A5: Using historical micro-expression videos as input and historical recognition results as output, a multi-branch network model is trained, and the multi-branch network model is structurally reparameterized to obtain a structurally reparameterized model.

[0037] In a preferred embodiment of the present invention, in step A1, a mixed dataset containing multiple micro-expression sequences is selected as a historical micro-expression video, and each micro-expression sequence is sequentially subjected to face recognition, face alignment, and cropping to obtain the corresponding preprocessed image.

[0038] In a preferred embodiment of the present invention, the number of micro-expression sequences in the mixed dataset is 442, and each micro-expression sequence includes 109 positive expressions, 83 surprised expressions and 250 negative expressions. In step A1, each positive expression, each surprised expression and each negative expression is sequentially subjected to face recognition, face alignment and cropping to obtain the corresponding preprocessed image.

[0039] In a preferred embodiment of the present invention, in step A2, the two face images are processed by the TV-L1 optical flow method to obtain the corresponding optical flow maps, and the face image corresponding to the peak frame is processed by the local binary mode texture operator of the equivalent mode to obtain the texture image.

[0040] Specifically, in this embodiment, optical flow refers to the instantaneous velocity of pixel motion of a moving object in space on the imaging plane. It is a method that uses the changes in pixels in the image sequence and the correlation between adjacent frames to find the correspondence between the pixels of the previous frame and the current frame, thereby calculating the motion information of pixels between adjacent frames. Micro-expressions themselves include some subtle facial movements. The TV-L1 optical flow method helps to represent the local image motion of micro-expressions. The TV-L1 optical flow method improves the original optical flow method and can improve the robustness to noise.

[0041] Specifically, in this embodiment, Local Binary Pattern (LCB) is a simple and efficient texture operator. It labels the pixels of an image by thresholding the neighborhood of each pixel and treats the result as a binary number. Due to its discriminative power and computational simplicity, LCB texture operator has become a popular method in various applications. However, with the increase of sampling points in the neighborhood, the number of binary patterns in traditional LCB increases exponentially, leading to an increase in data volume. To solve the problem of too many binary patterns in traditional LCB, this invention adopts an equivalent pattern LCB. This is mainly achieved by re-encoding the LCB to reduce dimensionality and enhance its discriminativeness. By using the equivalent pattern LCB, the dimensionality can be reduced, making the feature vector more compact, thereby reducing computational and storage costs. Furthermore, it can enhance discriminativeness, making it easier to distinguish different texture patterns in classification and recognition tasks. It can also improve noise resistance, becoming more robust to noise and subtle changes in the image, thus improving the algorithm's noise resistance.

[0042] In a preferred embodiment of the present invention, the image scale of each optical flow map and texture image is 256x256x3. In step A2, the optical flow map and texture image are stitched together to obtain a stitched image with an image scale of 256x256x6.

[0043] In a preferred embodiment of the present invention, such as Figure 3 As shown, the feature fusion module includes a spatial attention submodule and a channel attention submodule. In step A3, the stitched images are concatenated and processed serially through the spatial attention submodule and the channel attention submodule to obtain the temporal dimension feature map and the spatial dimension feature map.

[0044] Specifically, in this embodiment, in step A3, an attention mechanism is used to fuse the feature maps representing the time dimension and the spatial dimension respectively. The feature fusion module consists of two attention mechanism modules, namely the spatial attention submodule and the channel attention submodule. The stitched image is sequentially input into these two modules to obtain the fused feature map.

[0045] Preferably, the spatial attention submodule contains two 13×13 convolutional layers, with GELU as the activation function. The channel attention submodule is modified from the SE module, with the activation function also changed to GELU.

[0046] Preferably, the feature fusion module is an improvement on the traditional SE module. The SE module is an attention mechanism used to enhance the performance of deep neural networks. It aims to adaptively learn the weights of input features to better capture useful feature information, thereby improving the network's expressiveness and generalization ability. The SE module is designed for the intermediate feature maps of the model and uses global features to effectively promote redistribution. However, it is worth noting that the global feature operator of the SE module is computed using GAP, which aggressively compresses the H×W matrix to 1×1. For key features such as micro-expression movement changes, which usually exist at a small scale, using the SE module alone for multi-feature fusion will lead to the loss of significant local change information, which is unacceptable for micro-expression recognition. Therefore, in this invention, the SE module performs attention operations on global features while introducing local information to achieve accurate channel weight redistribution.

[0047] Preferably, the spatial attention submodule is used first. Early fusion methods used raw, unprocessed images as input feature maps. To achieve better spatial feature extraction, convolutional neural networks (CNNs) with relatively large kernels were employed to increase the effective receptive field and obtain more information. To ensure the module's lightweight design, a bottleneck structure was adopted. The following formula can be used to calculate X∈R C×H×W Spatial attention information:

[0048] S(X) = Conv2(GELU(Conv1(X)))

[0049] The kernel sizes of Conv1 and Conv2 are respectively and To maintain consistency in feature map size between input and output, both convolutions have a stride of 6 and padding of 0.

[0050] Preferably, the next preferred approach is the channel attention submodule. Given an intermediate feature map with C channels and a feature map H×W, the first step is to extract global information for a specific channel through a squeezing operation. This squeezing operation is implemented using global average pooling, which can be represented by averaging the H×W matrix of the z-th channel.

[0051]

[0052] Furthermore, to capture the dependencies between different channels, activation operations are performed through a bottleneck containing two fully connected layers:

[0053] G(X)=σ(W2δ(W1(g)))

[0054] Finally, the channel attention submodule calculates the weight value of each channel broadcast subsequently, and then multiplies it by the original feature map. Essentially, the channel attention submodule performs attention operations on the channel dimension, enabling the model to pay more attention to the channels with the most informative features, while suppressing less important features.

[0055] Preferably, in order to obtain spatial features and channel features from the feature map, they are concatenated through the spatial attention submodule and the channel attention submodule in this invention, thereby allowing these features to be computed independently. The advantage of using serial processing instead of parallel processing is that it decouples spatial features and channel features and avoids interference between their computation. The results show that serial processing is more effective than parallel processing.

[0056] In a preferred embodiment of the present invention, such as Figure 4 As shown, each multi-branch network module includes a spatial awareness submodule and a channel awareness submodule. In step A4, the spatial awareness submodule obtains local spatial information at different scales in the fused feature map, and the channel awareness submodule obtains long-distance spatial information of each channel in the fused feature map.

[0057] Specifically, in this embodiment, the spatial perception submodule uses the original input feature map, i.e. the fused feature map. In order to expand its effective receptive field, two convolutional layers with kernel sizes of 25×25 and 13×13 are used, and depthwise convolution is used to obtain consistent input and output sizes, and padding is set to half of the kernel size (rounded down).

[0058] Preferably, the channel-aware submodule is an indispensable component for processing the input of the original feature map, i.e., the fused feature map. It contains a fully connected layer. To reduce the problem of excessive parameters in the fully connected layer, the concept of a shared partition set is introduced, whereby the fused feature map is divided into sets of equal dimensions (s shared sets). Each shared set is input to the fully connected layer using the same weights, thereby reducing the parameters of the fully connected layer to s×C×H×W. In addition, for ease of operation, the shared sets are flattened and convolved with 1×1 convolutions with the number of groups s. The output is reshaped back to the original feature map input shape (n, c, h, w).

[0059] In a preferred embodiment of the present invention, the multi-branch network model includes convolutional layers and fully connected layers. In step A5, the parameters of the convolutional layers are injected into the parameters of the fully connected layers through a structural reparameterization method to form a structural reparameterization model.

[0060] Specifically, in this embodiment, structural reparameterization of the multi-branch network model can reduce the increase in inference time and memory caused by multiple branches, and enable the fully connected layer to contain the local inductive bias of the convolutional layer.

[0061] Preferably, for spatial injection, the convolutional layer parameters and weight matrix W of kernel size K(n, c, k, k) are... FC The input parameter ehw and the output parameter ohw are fused together when F in When the input is used, the final weight matrix W of the fully connected layer is obtained. out The equation is as follows:

[0062] MATMUL(F in W out ) = MATMUL(F in W FC )+CONV(F in K)

[0063] Based on the distributional property of matrix multiplication relative to addition, for a given input matrix M, there exist two matrices W of the same size. 1 and W 2 , so that:

[0064] MATMUL(M, W) 1 )+MATMUL(M,W 2 =MATMUL(M, W) 1 +W 1 )

[0065] In order to fuse the parameters of the convolutional layer into the fully connected layer, there needs to exist a parameter that is related to W. FC The parameter matrix W of the same shape CONV It meets the following conditions:

[0066] MATMUL(F in W CONV ) = CONV(F in K)

[0067] Convolution is a type of sparse matrix multiplication, which can be represented by a sparse and weight-shared Toplitz matrix. Therefore, in spatially aware channel sensing, the parameters of the convolutional layer can be seamlessly integrated into the parameter matrix of the fully connected layer.

[0068] Preferably, the structural reparameterization model uses SGD with momentum, where the momentum is set to 0.9 and the weight decay is set to 10. -4 The learning rate was set to 0.1, and for LOSO evaluation, each fold was trained for 30 epochs.

[0069] While the disclosure is as stated above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the protection scope of this invention.

Claims

1. A micro-expression recognition method based on feature fusion and structural reparameterization, characterized in that, Includes the following steps: Step S1: Obtain at least one micro-expression video to be detected; Step S2: Input the micro-expression video to be detected into the pre-trained structural reparameterization model to obtain the corresponding micro-expression recognition result; The training process of the structure reparameterization model includes: Step A1: Obtain at least one historical micro-expression video, perform face recognition on each video frame in the historical micro-expression video to obtain the corresponding frame image containing the face region, and perform face alignment and cropping preprocessing operations on each frame image to obtain the corresponding preprocessed image. Step A2: For each preprocessed image, extract the face images corresponding to the start frame and the peak frame in the preprocessed image, process the two face images to obtain the corresponding optical flow map, and process the face image corresponding to the peak frame to obtain the corresponding texture image. Then, stitch the optical flow map and the texture image together to obtain the stitched image. Step A3: Input the stitched image into a pre-constructed feature fusion module to extract time dimension feature maps and spatial dimension feature maps, and perform feature fusion on the time dimension feature maps and spatial dimension feature maps to obtain a fused feature map; Step A4: Input the fused feature map into a multi-branch network module of multiple stages to obtain the corresponding local spatial information and long-distance spatial information, and integrate the local spatial information and the long-distance spatial information to obtain the corresponding historical recognition result; Step A5: Using the historical micro-expression video as input and the historical recognition result as output, a multi-branch network model is trained, and the multi-branch network model is structurally reparameterized to obtain the structurally reparameterized model.

2. The micro-expression recognition method according to claim 1, characterized in that, In step A1, a mixed dataset containing multiple micro-expression sequences is selected as the historical micro-expression video, and each micro-expression sequence is sequentially subjected to face recognition, face alignment, and cropping to obtain the corresponding preprocessed image.

3. The micro-expression recognition method according to claim 2, characterized in that, The number of micro-expression sequences in the mixed dataset is 442, and each micro-expression sequence includes 109 positive expressions, 83 surprised expressions and 250 negative expressions. In step A1, each positive expression, each surprised expression and each negative expression is sequentially subjected to face recognition, face alignment and cropping to obtain the corresponding preprocessed image.

4. The micro-expression recognition method according to claim 1, characterized in that, In step A2, the two face images are processed by the TV-L1 optical flow method to obtain the corresponding optical flow maps, and the face image corresponding to the peak frame is processed by the local binary mode texture operator of the equivalent mode to obtain the texture image.

5. The micro-expression recognition method according to claim 1, characterized in that, If the image scale of each optical flow map and the texture image is 256x256x3, then in step A2, each optical flow map and the texture image are stitched together to obtain the stitched image with an image scale of 256x256x6.

6. The micro-expression recognition method according to claim 1, characterized in that, The feature fusion module includes a spatial attention submodule and a channel attention submodule. In step A3, the stitched images are concatenated and processed serially through the spatial attention submodule and the channel attention submodule to obtain the temporal dimension feature map and the spatial dimension feature map.

7. The micro-expression recognition method according to claim 1, characterized in that, Each of the multi-branch network modules includes a spatial awareness submodule and a channel awareness submodule. In step A4, the spatial awareness submodule obtains the local spatial information at different scales in the fused feature map, and the channel awareness submodule obtains the long-distance spatial information of each channel in the fused feature map.

8. The micro-expression recognition method according to claim 1, characterized in that, The multi-branch network model contains convolutional layers and fully connected layers. In step A5, the parameters of the convolutional layers are injected into the parameters of the fully connected layers using a structural reparameterization method to form the structural reparameterization model.

Images