Micro-expression recognition method and system based on state space model and double-flow fusion

Abstract

The invention discloses a micro-expression recognition method and system based on a state space model and double-flow fusion, and relates to the technical field of expression recognition, and the method comprises the steps: extracting a starting frame image and a top frame image of a micro-expression continuous video frame sequence; inputting the initial frame image and the top frame image into a micro-expression recognition model based on a double-flow fusion network, extracting an optical flow image, and performing face key point detection and clipping operation on the top frame image and the optical flow image respectively; carrying out image block embedding operation to obtain texture features of the top frame image after cutting and motion features of the optical flow image; the two features learn the global dependency relationship and the local correlation between the image blocks from the positive direction and the negative direction through a parallel multi-layer bidirectional Mama fusion module, and deep image features and motion features are extracted after feature fusion; and the deep features of the two modes are interacted through a double-flow cross attention classification module, comprehensive features are learned, and a plurality of classifiers are combined to identify and output the micro-expression type more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of expression recognition, and in particular to a micro-expression recognition method and system based on a state space model and dual-stream fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Facial expressions of human face are divided into macro-expressions and micro-expressions. Macro-expressions are the most common expressions in daily life and are also easy to disguise. On the contrary, micro-expressions are not easy to disguise and can reflect the most real human emotions. Micro-expressions are characterized by low muscle movement intensity and short duration, usually only lasting 1 / 25 to 1 / 2 second, and only local facial muscle movement. In recent years, the important practical value of micro-expression analysis technology has received widespread attention. This technology is often used in disease diagnosis, lie detection and business negotiations.

[0004] The task of micro-expression recognition is to automatically detect and classify these usually imperceptible emotional expressions by analyzing the brief and subtle changes in human facial expressions, so as to understand the underlying true emotional state. In the early stage of micro-expression task research, it was proposed to design a micro-expression recognition framework based on manual methods. A commonly used method is the local binary pattern method, which can well extract texture features in images. After combining the texture features with time information, the model can simultaneously extract texture information and spatial information; in addition to the local binary pattern method, the gradient histogram is also widely used to describe the texture, structure and edge distribution of micro-expression images; in addition, the optical flow method can learn the micro-expression muscle movement representation from the information between video frames for micro-expression classification. However, the above-mentioned manually designed methods need to be designed and adjusted by experts based on prior knowledge, which limits the accuracy of the model when processing complex expressions.

[0005] With the development of deep learning, convolutional neural networks have been widely used in the field of micro-expression recognition. First, a micro-expression recognition method based on convolutional neural networks was proposed. This method uses operations such as convolution and pooling to extract discriminative features from the input image for classification; then, the convolutional neural network is combined with the recurrent neural network to extract micro-expression features from the two dimensions of space and time for more accurate classification; the Transformer model initially achieved good performance in natural language processing tasks. It only uses the self-attention mechanism to capture the relationship between tokens in the sequence. The Transformer model was introduced into computer vision tasks due to its high performance. The Transformer model has made great progress in micro-expression recognition tasks. In addition, the current selective state space model and hardware perception algorithm are integrated into an end-to-end neural network structure, called the Mamba model, which has fast reasoning capabilities and linear complexity. On this basis, a variety of visual backbone networks based on the Mamba model are proposed, such as Vim, VMamba and other networks, which promote the application of the Mamba model in the field of computer vision.

[0006] However, for micro-expression recognition tasks, existing deep learning models have not yet achieved good recognition accuracy. The main problems are as follows:

[0007] (1) Although the recognition method based on the Transformer model is superior to most methods based on convolutional neural networks, matrix multiplication is required to calculate the attention matrix in the self-attention mechanism, which leads to quadratic complexity and high computational overhead;

[0008] (2) Existing micro-expression recognition frameworks fail to take advantage of the two-stream architecture and often use only a single image as input, which limits the accuracy of micro-expression recognition methods;

[0009] (3) The existing two-stream deep learning framework simply concatenates or superimposes the features of different modalities, and the existing fusion method is not fully suitable for micro-expression recognition tasks. Summary of the invention

[0010] To address the deficiencies of the above-mentioned prior art, the present invention provides a method and system for micro-expression recognition based on a state-space model and dual-stream fusion. Taking into account the advantages of the Mamba model, the Mamba model is applied to the dual-stream architecture, and a dual-stream fusion network based on a bidirectional Mamba fusion module is proposed. The motion features and facial detail features of the subject's micro-expressions are learned from the optical flow image and the top frame image, respectively, and the two types of features are fused using a specific fusion method. The fused features are classified into micro-expressions through a dual-stream cross-attention classification module, so as to achieve more accurate micro-expression recognition, and the entire recognition model has a faster reasoning speed and less computational complexity.

[0011] In a first aspect, the present invention provides a micro-expression recognition method based on a state-space model and dual-stream fusion.

[0012] A micro-expression recognition method based on state space model and dual-stream fusion, comprising:

[0013] Obtain the micro-expression video to be recognized, and extract the starting frame and top frame image of the continuous video frame sequence;

[0014] The starting frame and the top frame images are input into the micro-expression recognition model based on the dual-stream fusion network, and the recognized micro-expression type is output; wherein, the optical flow image is extracted from the starting frame and the top frame images, and the facial key point detection and cropping operations are performed on the top frame image and the optical flow image respectively; and then the texture features of the cropped top frame image and the motion features of the optical flow image are obtained through the image block embedding operation;

[0015] The two features are respectively fused through a parallel multi-layer bidirectional Mamba fusion module to learn the global dependency and local correlation between each image block from the positive and negative directions. After feature fusion, deep image features and motion features are extracted.

[0016] The deep features of the two modalities are interacted through the dual-stream cross-attention classification module to learn comprehensive features, and multiple classifiers are jointly used to identify micro-expression types.

[0017] In a second aspect, the present invention provides a micro-expression recognition system based on a state-space model and dual-stream fusion.

[0018] A micro-expression recognition system based on state space model and dual-stream fusion, comprising:

[0019] An image extraction module is used to obtain the micro-expression video to be identified and extract the starting frame and top frame images of the continuous video frame sequence;

[0020] A micro-expression recognition module is used to input the start frame and the top frame images into a micro-expression recognition model based on a dual-stream fusion network, and output the recognized micro-expression type; wherein, an optical flow image is extracted from the start frame and the top frame images, and facial key point detection and cropping operations are performed on the top frame image and the optical flow image respectively; and then, through an image block embedding operation, the texture features of the cropped top frame image and the motion features of the optical flow image are obtained;

[0021] The two features are respectively fused through a parallel multi-layer bidirectional Mamba fusion module to learn the global dependency and local correlation between each image block from the positive and negative directions. After feature fusion, deep image features and motion features are extracted.

[0022] The deep features of the two modalities are interacted through the dual-stream cross-attention classification module to learn comprehensive features, and multiple classifiers are jointly used to identify micro-expression types.

[0023] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0024] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0025] One or more of the above technical solutions have the following beneficial effects:

[0026] 1. The present invention provides a method and system for micro-expression recognition based on a state-space model and dual-stream fusion. Taking into account the advantages of the Mamba model, the Mamba model is applied to the dual-stream architecture, and a dual-stream fusion network based on a bidirectional Mamba fusion module is proposed. The dual-stream fusion network is used as a micro-expression recognition model to perform more accurate micro-expression recognition. The bidirectional Mamba fusion module is used to extract features from optical flow images and top frame images respectively, learn the dependency between global patches (image blocks) and the correlation between local patches in the image, extract micro-expression motion features and subject facial texture features, and then fuse these two features to obtain more comprehensive and discriminative micro-expression features, thereby improving the accuracy of final recognition. Moreover, compared with the traditional Transformer model, the bidirectional Mamba model has faster reasoning speed and less computational complexity, and can use fewer parameters and faster speed to train a better model, thereby reducing computational complexity and computational overhead.

[0027] 2. The present invention proposes a dual-stream cross-attention classification network, which uses the network to make deep motion features and facial texture features interact and act on each other, and fuse to obtain a more comprehensive and richer feature representation; at the same time, multiple classifiers are used in combination to integrate the classification results, making full use of the features extracted by the model to further improve the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 It is an overall block diagram of the micro-expression recognition method based on the state space model and dual-stream fusion according to an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a bidirectional Mamba module in a bidirectional Mamba fusion module in an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of a fusion module in a bidirectional Mamba fusion module in an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of a dual-stream cross-attention classification module in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] Embodiment 1

[0035] This embodiment provides a micro-expression recognition method based on a state space model and dual-stream fusion. The Mamba model and fusion method are used to learn micro-expression discriminative features from optical flow images and top frame images to achieve micro-expression classification. The overall framework of the method is as follows: Figure 1 As shown, Figure 1The facial images used are images from existing public facial image collections and do not involve privacy issues. In this embodiment, the dual-stream fusion network includes a preprocessing module, a parallel multi-layer bidirectional Mamba fusion module, and a dual-stream cross-attention classification module, wherein the preprocessing module is used to extract an optical flow image representing the micro-expression motion characteristics from the start frame and top frame image of the subject's micro-expression video, and then crop the top frame image and the optical flow image by the facial key point detection method to remove redundant information and retain the subject's facial information; the bidirectional Mamba fusion module is used to extract the subject's facial muscle movement characteristics and facial texture characteristics from the optical flow image and the top frame image, respectively. The module includes a bidirectional Mamba module and a fusion module. The bidirectional Mamba module learns from both forward and reverse directions. The global dependency between patches, and the fusion module learns the correlation between local patches and fuses the features by using convolution and pooling operations to remove unimportant features and retain important features; the two-stream cross-attention classification network includes two parallel cross-attention modules and a multi-classifier. The two parallel cross-attention modules are used to fuse the two extracted features. Through the interaction between motion features and image features (i.e., facial texture features), the motion features contain image features, and the image features contain motion features, so as to achieve a better fusion effect. Multiple classifiers are jointly classified to make full use of the fused features and further improve the accuracy of the model.

[0036] The micro-expression recognition method based on the state space model and dual-stream fusion proposed in this embodiment specifically includes the following steps:

[0037] Step S1: Obtain a micro-expression video to be identified, and extract the starting frame and top frame image of the continuous video frame sequence. That is, generate a micro-expression dataset based on the micro-expression video, and extract the starting frame / starting frame image x of the subject from the micro-expression dataset. onset and the top frame image x apex .

[0038] In this embodiment, four datasets, MEGC2019, SMIC, CASME II and SAMM, are used to extract the start frame images x of 68 subjects from the datasets. onset and the top frame image x apex .

[0039] Step S2: input the start frame and the top frame image into the micro-expression recognition model based on the dual-stream fusion network, and output the recognized micro-expression type, including:

[0040] Step S2.1, preprocessing stage, includes: extracting optical flow images from the starting frame and the top frame images, and performing facial key point detection and cropping operations on the top frame image and the optical flow image respectively; and then obtaining the texture features of the cropped top frame image and the motion features of the optical flow image through image block embedding operations.

[0041] Step S2.1.1, extracting an optical flow image from the starting frame and the top frame image, comprising: extracting an optical flow feature from the difference between the starting frame and the top frame image using an optical flow method, the optical flow feature comprising a horizontal optical flow x representing a horizontal motion feature u , vertical optical flow x representing vertical motion features v and the optical strain x representing the image deformation characteristics o ; Then extract the optical flow features (including horizontal optical flow x u , vertical optical flow x v and light strain x o ) After stacking in the spatial dimension, the optical flow image x is obtained optical =Concat(x u ,x v ,x o ).

[0042] Step S2.1.2: top frame image x apex and the optical flow image x optical The facial key point detection and cropping operations are performed respectively, and redundant operations are removed to retain the facial area of ​​the subject, including: using a face detection model based on a multi-task cascaded convolutional neural network to extract the coordinate points of the facial features in the top frame image. In this embodiment, the multi-task cascaded convolutional neural network (MTCNN) in the existing facenet_pytorch library is used for face detection, and the top frame image x is obtained. apex The positions of facial key points are extracted from the image, including the left eye, right eye, left lip corner and right lip corner. Based on the coordinate points of the facial features, the top frame image and the optical flow image are cropped to obtain the facial features area image. That is, for the optical flow image x optical and the top frame image x apex , taking each coordinate point extracted in the previous step as the center, cropping out the corresponding square area image; finally, splicing the facial features area images according to the positions of the facial features to obtain the updated top frame image x r ∈R C×H×W and the optical flow image x o ∈R C×H×W , where H, W, and C represent the length, width, and channel dimensions of the top frame image and the optical flow image, respectively.

[0043] Step S2.1.3, perform image block embedding (i.e., Patch embedding) operation on the cropped top frame image and optical flow image, convert the image from pixel space to feature space, and convert the three-dimensional features of the image into sequence features to adapt to the input of the subsequent bidirectional Mamba fusion module. Through this operation, the texture features of the cropped top frame image and the motion features of the optical flow image are obtained.

[0044] Step S2.1.3.1: Through the convolution operation, the input top frame image and the optical flow image are divided into multiple non-overlapping image blocks of the same size, and the image is mapped to a higher dimension in the spatial dimension. In this embodiment, a convolution operation Conv with D convolution kernel sizes and stride lengths of P is used. p (·) Divide the two input images into L non-overlapping image blocks of size P×P, where At the same time, the spatial dimension of the input image is mapped from the C-dimensional feature space to the D-dimensional feature space.

[0045] Step S2.1.3.2, by flattening and transposing operations, the mapping features are converted into sequence features to obtain texture features and motion features of the input image. In this embodiment, the features are transformed into sequence features suitable for processing by the Mamba model by using Flatten(·) and Transpose(·) operations to obtain motion features after the Patch embedding operation. and facial texture features

[0046] Furthermore, the Patch embedding operation process can be expressed as:

[0047]

[0048] Among them, Conv p (·) refers to the convolution operation with kernel size and stride of P, Flatten(·) means flattening the last two dimensions of the feature, and Transpose(·) means matrix transposition operation. and They are respectively the motion features and facial texture features (also called image features) after Patch embedding.

[0049] Step S2.2: The two features are respectively fused through a parallel multi-layer bidirectional Mamba fusion module to learn the global dependency and local correlation between each image block from both the positive and negative directions, and the deep image features and motion features are extracted after feature fusion.

[0050] Specifically, the motion characteristics and image features As input, they are input into N parallel bidirectional Mamba fusion modules BMFB i(·), i = 1, 2, ..., N, using N bidirectional Mamba fusion modules to perform the motion features after the patch embedding operation. and texture features The dependencies between global patches (i.e., image blocks) and the correlations between local patches are learned and feature fusion is performed.

[0051] like Figure 1 As shown in Figure 1, the entire micro-expression recognition framework contains N parallel bidirectional Mamba fusion modules. Each bidirectional Mamba fusion module includes a bidirectional Mamba module BMB(·) and a fusion module Fusion(·), which can effectively capture the feature interaction between different regions of the image. Among them, the i-th bidirectional Mamba fusion module BMFB i The input features of (·) are or , the input of the i-th bidirectional Mamba fusion module is expressed as Then the i-th bidirectional Mamba fusion module BMFB i Bidirectional Mamba module BMB in (·) i (·) The steps of extracting features include:

[0052] like Figure 2 As shown, step S2.2.1, the feature Input into the bidirectional Mamba module, first pass through the feature normalization layer and the linear layer, and map the features to the high-dimensional space D i ′Get the high-dimensional features, and then divide the high-dimensional features into two tensors z with the same dimension 1 ,z 2 , the formula is as follows:

[0053]

[0054] Among them, Norm(·) refers to the normalization operation, Linear(·) refers to the linear layer, and Chunk(·) refers to the feature segmentation operation. Represents two tensors with the same dimension after the split.

[0055] Step S2.2.2, the first tensor is convolved through two convolutional layers and then input into the forward and backward state space models to learn the dependencies between global image blocks from both the forward and reverse directions, capture the forward and reverse spatial context information, and output the two initial global temporal features.

[0056] Specifically, the first tensor z 1 Used to perform bidirectional Mamba operations to efficiently learn the relationship between input features and capture fine-grained spatial context information, including: The first tensor z 1After two one-dimensional convolutional layers, the input features of the forward state space model and the reverse state space model are obtained, and they are input into the forward state space model SSM respectively. forward (·) and the inverse state space model SSM backward In (·), the dependency between global image patches is learned from both positive and negative directions to capture the spatial context information in both positive and negative directions. The formula is as follows:

[0057]

[0058] Among them, Conv forward (·) and Conv backward (·) denotes the convolution operations corresponding to the forward state space model and the reverse state space model, respectively. forward (·) and SSM backward (·) refer to the forward state space model and the reverse state space model, respectively. They represent the features after the forward state space model and the backward state space model respectively.

[0059] In particular, the structured state-space model is a deep learning sequence model designed from a linear time-invariant system that is constructed by N Mapping a one-dimensional function or sequence x(t)∈R to another one-dimensional function or multidimensional sequence y(t)∈R can be expressed as: The system can be described using the state space model:

[0060]

[0061] Among them, the hidden state h(t) is used to capture the cumulative impact of inputs from the past to the present on the system; x(t) represents the input of the system; A, B and C are system matrices that can be obtained through model training; y(t) is the output of the system.

[0062] Furthermore, in this embodiment, the forward state space model and the reverse state space model used are discretized structured state space sequence models, which are discretized as follows:

[0063] Considering that the zero-order holding (ZOH) is a commonly used discretization method for converting a continuous-time system into a discrete-time system, the sampling period Δ is applied in the zero-order holding to discretize the system matrices A and B. The zero-order holding can be expressed as:

[0064]

[0065] in, and Refers to the system matrix after discretization.

[0066] Based on the above content, a discretization method based on a zero-order holder is used to discretize the state space model, which is: the first tensor after convolution is input into the state space sequence model, and the sampling period Δ and state space matrices B and C are extracted respectively through two linear layers; the sampling period Δ is applied to the zero-order holder, and the state space matrices A and B are discretized using the zero-order holder; based on the discretized state space matrix, the spatial context information is captured, and the initial global timing features are extracted and output.

[0067] Among them, the discretized state space model can be expressed as:

[0068]

[0069] Step S2.2.3: After the second tensor is activated by the activation function, it is multiplied with the two initial global time series features respectively. The two features obtained are then added element by element, and the final global time series feature is output through the linear layer.

[0070] Specifically, the second tensor z 2 Input to the gating branch, which is used to dynamically adjust the output of the forward state space model and the reverse state space model. 2 After the activation function and the above two features Multiply to enhance or suppress the feature. The two multiplied features are added element by element and then input into a linear layer to restore the feature to the same value as the input feature. The same size can be expressed as:

[0071]

[0072] Among them, σ(·) refers to the activation function, z 2 ′ refers to z 2 The features after activation function are refers to element-by-element multiplication, ⊕ refers to addition, Refers to input features Features after passing the bidirectional Mamba module.

[0073] Furthermore, the bidirectional Mamba module BMB i The operation in (·) can be expressed as:

[0074] Step S2.2.4: Input the global temporal features into the fusion module, convert them into three-dimensional features after rearrangement operation, and perform convolution, layer normalization, convolution, layer normalization, maximum pooling, and rearrangement operations on the three-dimensional features in sequence. Through the convolution operation, learn and integrate the correlation between local image blocks, and output the final fusion features.

[0075] like Figure 3 As shown, considering After passing through the bidirectional Mamba module It is still a sequence feature. To facilitate the convolution operation, first Rearrange to three-dimensional data through rearrange operation (·) Use a convolution operation with a kernel size of 3 To learn the correlation between local patches, and then use a convolution operation with a convolution kernel size of 1 To further integrate features, a layer normalization operation is performed after each convolution operation to normalize the output of the convolution layer to improve the training stability of the model, accelerate the convergence speed, and enhance the generalization ability; then, the maximum pooling Maxpool(·) operation is performed on the features to retain the most significant features in the local area and remove redundant features; finally, the fused features are rearranged to restore to sequence features to meet the input of the bidirectional Mamba fusion module at the next level.

[0076] After the above operations, the correlation between local patches can be established and integrated. The formula is as follows:

[0077]

[0078] Among them, rearrange(·) refers to the rearrangement operation, which can realize the conversion between sequence features and three-dimensional features. L i =S i ×S i refer to After rearrangement, the features Fusion module i The convolution operations in (·) have kernel sizes of 3 and 1 respectively. LN(·) refers to layer normalization operation, and Maxpool(·) refers to maximum pooling operation. j∈{o,r} represents the i-th bidirectional Mamba fusion module BMFB i The output feature of (·) is also the i+1th bidirectional Mamba fusion module BMFB i+1 (·)’s input features.

[0079] That is, the i-th bidirectional Mamba fusion module BMFB i (·) Fusion module i The operation of (·) can be expressed as:

[0080] In summary, After the i-th bidirectional Mamba fusion module BMFBi (·)get The overall process can be expressed as:

[0081] Step S2.3: The deep features of the two modalities are interacted through the dual-stream cross-attention classification module to learn comprehensive features, and multiple classifiers are jointly used to identify micro-expression types.

[0082] After the above steps, the motion features and image features After N bidirectional Mamba fusion modules, the corresponding features are The motion feature focuses on the facial muscle motion features of micro-expressions, while the image feature focuses more on the facial attributes and texture features of the subject. The fusion of the two features can effectively improve the classification performance of the model by extracting the fusion features including the motion features and the facial features of the subject. To this end, this embodiment proposes a dual-stream cross-attention classification module DCCB(·) for fusing the two modal features and realizing micro-expression classification, such as Figure 4 As shown in Figure 2, the dual-stream cross-attention classification module DCCB(·) includes two parallel cross-attention modules CAM(·) and a multi-classifier MC(·) to capture the correlation between the two modal features and utilize the complementary information of the two feature streams for classification.

[0083] Step S2.3.1, input the deep image features and deep motion features into the parallel cross-attention modules respectively, calculate the query matrix and key matrix of the two features through convolution operation, and then calculate the attention weight matrix corresponding to the two features.

[0084] Specifically, and Input two parallel cross attention modules, and calculate the attention weight to obtain the correlation between the two features for feature fusion. First, the convolution operation is performed to calculate and The corresponding query,key matrix is ​​expressed as:

[0085]

[0086] in, and Respectively represent the calculation motion features Query Q o , key K o The corresponding convolution operation is and Respectively represent the calculated image features Query Q r , key K r The corresponding convolution operation.

[0087] Furthermore, the attention weight matrix corresponding to the motion features and image features is calculated through the above query and key, where the query Q of the motion feature is o and the key K of the image features r Calculate the attention weight matrix A of the motion feature o , similarly, based on the query Q of the image features r and the key K of the motion feature o Calculate the attention weight matrix of image features, which can be expressed as:

[0088]

[0089] Among them, Attention(·) refers to the process of calculating the attention weight matrix, Softmax(·) refers to the normalization operation, is a scaling factor used to prevent gradient explosion, A o and A r Represent the attention weight matrices of motion features and image features respectively.

[0090] Step S2.3.2: Multiply and fuse the deep image features and deep motion features with the corresponding attention weight matrices to obtain updated image features and motion features.

[0091] Specifically, and The fused features are obtained by multiplying the corresponding attention weight matrix. The calculation process is as follows:

[0092]

[0093] in, and They respectively refer to the motion features and image features after fusion by the cross-attention module.

[0094] Then, the operation in the criss-cross attention module CAM(·) can be expressed as:

[0095]

[0096] Step S2.3.3: Use two classifiers to classify micro-expressions on the updated image features and motion features respectively, and obtain two initial classification results.

[0097] Specifically, in order to better utilize the fused features and improve the classification accuracy of the model, and Input to the multi-classifier MC(·) to realize the classification of the model. First, the two classifiers are respectively and After micro-expression classification, two preliminary classification results are obtained, which can be expressed as follows:

[0098]

[0099] Among them, Linear(·) refers to the linear layer used for classifier, class o ,class r Represents motion characteristics and image features The classification result after the classifier.

[0100] Step S2.3.4: Based on the two initial classification results, they are stacked in the channel dimension and input into the final classifier to obtain the final classification result.

[0101] Specifically, the two classification results are stacked in the channel dimension and input into the final classifier for classification to obtain the final classification result, which can effectively improve the discrimination ability of the model and enhance the model performance, which can be expressed as:

[0102] y=Linear(Concat(class o ,class r ));

[0103] Among them, Concat(·) refers to the operation of stacking two features in the channel dimension, and y refers to the final classification result of the model.

[0104] Then, the operation in the multi-classifier MC(·) can be expressed as:

[0105]

[0106] Further, and The overall process of feature fusion and classification can be expressed as follows:

[0107]

[0108] Embodiment 2

[0109] This embodiment provides a micro-expression recognition system based on a state space model and dual-stream fusion, including:

[0110] An image extraction module is used to obtain the micro-expression video to be identified and extract the starting frame and top frame images of the continuous video frame sequence;

[0111] A micro-expression recognition module is used to input the start frame and the top frame images into a micro-expression recognition model based on a dual-stream fusion network and output the recognized micro-expression type;

[0112] The optical flow image is extracted from the starting frame and the top frame image, and the facial key point detection and cropping operations are performed on the top frame image and the optical flow image respectively; then, the texture features of the cropped top frame image and the motion features of the optical flow image are obtained through the image block embedding operation;

[0113] The two features are respectively fused through a parallel multi-layer bidirectional Mamba fusion module to learn the global dependency and local correlation between each image block from the positive and negative directions. After feature fusion, deep image features and motion features are extracted.

[0114] The deep features of the two modalities are interacted through the dual-stream cross-attention classification module to learn comprehensive features, and multiple classifiers are jointly used to identify micro-expression types.

[0115] Embodiment 3

[0116] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the micro-expression recognition method based on the state space model and dual-stream fusion as described above are completed.

[0117] Embodiment 4

[0118] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the micro-expression recognition method based on the state-space model and dual-stream fusion as described above are completed.

[0119] The steps involved in the above embodiments 2 to 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0120] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0121] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A micro-expression recognition method based on state space model and dual-stream fusion, characterized in that: include: Obtain the micro-expression video to be recognized, and extract the starting frame and top frame image of the continuous video frame sequence; The starting frame and the top frame images are input into the micro-expression recognition model based on the dual-stream fusion network, and the recognized micro-expression type is output; wherein, the optical flow image is extracted from the starting frame and the top frame images, and the facial key point detection and cropping operations are performed on the top frame image and the optical flow image respectively; and then the texture features of the cropped top frame image and the motion features of the optical flow image are obtained through the image block embedding operation; The two features are respectively fused through a parallel multi-layer bidirectional Mamba fusion module to learn the global dependency and local correlation between each image block from the positive and negative directions. After feature fusion, deep image features and motion features are extracted. The deep features of the two modalities are interacted through the dual-stream cross-attention classification module to learn comprehensive features, and multiple classifiers are jointly used to identify micro-expression types.

2. A micro-expression recognition method based on state space model and dual stream fusion as claimed in claim 1, characterized in that: Extract optical flow images from the starting frame and the top frame image, including: The optical flow method is used to extract optical flow features from the starting frame and the top frame images, including horizontal optical flow representing horizontal motion features, vertical optical flow representing vertical motion features, and optical strain representing image deformation features; After stacking the extracted optical flow features in the spatial dimension, an optical flow image is obtained.

3. The micro-expression recognition method based on state space model and dual stream fusion as claimed in claim 1, characterized in that: Perform facial key point detection and cropping operations on the top frame image and the optical flow image, respectively, and retain the facial area image, including: A face detection model based on a multi-task cascade convolutional neural network is used to extract the coordinate points of the facial features in the top frame image; Based on the coordinate points of the facial features, the top frame image and the optical flow image are cropped to obtain the facial features area images; The facial feature area images are spliced ​​by position to obtain the updated top frame image and optical flow image.

4. The micro-expression recognition method based on state space model and dual stream fusion as claimed in claim 1, characterized in that: Perform image block embedding on the cropped top frame image and optical flow image, convert the image from pixel space to feature space, and convert the three-dimensional features of the image into sequence features, including: Through the convolution operation, the input top frame image and optical flow image are divided into multiple non-overlapping image blocks of the same size, and the image is mapped to a higher dimension in spatial dimension at the same time; Through flattening and transposition operations, the mapping features are converted into sequence features to obtain the texture features and motion features of the input image.

5. The micro-expression recognition method based on state space model and dual stream fusion as claimed in claim 1, characterized in that: The bidirectional Mamba fusion module includes a bidirectional Mamba module and a fusion module; The features are input into the bidirectional Mamba module, and the features are first mapped to a high-dimensional space through a normalization layer and a linear layer to obtain high-dimensional features, and then the high-dimensional features are divided into two tensors of the same dimension. The first tensor is convolved through two convolutional layers and then input into the forward and backward state space modules to learn the dependencies between global image blocks from both the forward and reverse directions, capture the forward and reverse spatial context information, and output the two initial global temporal features. After the second tensor is activated by the activation function, it is multiplied with the two initial global time series features respectively. The two features are then added element by element, and the final global time series features are output through the linear layer. The global temporal features are input into the fusion module and converted into three-dimensional features after rearrangement. The three-dimensional features are sequentially subjected to convolution, layer normalization, convolution, layer normalization, maximum pooling, and rearrangement operations. The correlation between local image blocks is learned and integrated through the convolution operation to output the final fusion features.

6. A micro-expression recognition method based on state space model and dual stream fusion as claimed in claim 5, characterized in that: The state space module is a discretized state space sequence model, and the discretization is: The first tensor after convolution is input into the state-space sequence model, and the sampling period and state-space matrix are extracted respectively through two linear layers; Applying the sampling period to a zero-order holder, and using the zero-order holder to discretize the state space matrix; Based on the discretized state space matrix, the spatial context information is captured, and the initial global time series features are extracted and output.

7. A micro-expression recognition method based on state space model and dual stream fusion as claimed in claim 1, characterized in that: The dual-stream cross-attention classification module includes two parallel cross-attention modules and multiple classifiers; The deep image features and the deep motion features are respectively input into the parallel cross attention module, and the query matrix and the key matrix of the two features are calculated by convolution operation, and then the attention weight matrix corresponding to the two features is calculated; wherein, the attention weight matrix of the deep motion features is calculated based on the query matrix of the deep motion features and the key matrix of the deep image features; the attention weight matrix of the deep image features is calculated based on the query matrix of the deep image features and the key matrix of the deep motion features; The deep image features and deep motion features are respectively multiplied and fused with the corresponding attention weight matrices to obtain updated image features and motion features; Two classifiers are used to classify micro-expressions based on the updated image features and motion features, respectively, and two initial classification results are obtained; Based on the two initial classification results, they are stacked in the channel dimension and input into the final classifier to obtain the final classification result.

8. A micro-expression recognition system based on state space model and dual stream fusion, characterized in that: include: An image extraction module is used to obtain the micro-expression video to be identified and extract the starting frame and top frame images of the continuous video frame sequence; A micro-expression recognition module is used to input the start frame and the top frame images into a micro-expression recognition model based on a dual-stream fusion network, and output the recognized micro-expression type; wherein, an optical flow image is extracted from the start frame and the top frame images, and facial key point detection and cropping operations are performed on the top frame image and the optical flow image respectively; and then, through an image block embedding operation, the texture features of the cropped top frame image and the motion features of the optical flow image are obtained; The two features are respectively fused through a parallel multi-layer bidirectional Mamba fusion module to learn the global dependency and local correlation between each image block from the positive and negative directions. After feature fusion, deep image features and motion features are extracted. The deep features of the two modalities are interacted through the dual-stream cross-attention classification module to learn comprehensive features, and multiple classifiers are jointly used to identify micro-expression types.

9. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a micro-expression recognition method based on a state space model and dual-stream fusion as described in any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a micro-expression recognition method based on a state-space model and dual-stream fusion as described in any one of claims 1-7.

Images