A dynamic micro-expression recognition method based on frame weight and related device

Abstract

The application discloses a dynamic micro-expression recognition method based on frame weight and a related device, and relates to the technical field of image processing. The method comprises the following steps: acquiring a micro-expression image sequence to be recognized; inputting the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result; wherein the dynamic micro-expression recognition model comprises an initial convolution module, a frame attention module, a dynamic and static combination module and a time sequence processing module. By adopting the frame attention module in the dynamic micro-expression recognition model, the calculation of channel attention weight and the calculation of frame attention weight can be simultaneously performed, the recognition of key frames in the dynamic micro-expression image sequence is realized, and by adopting the dynamic and static combination module, dynamic information between feature maps is added on the basis of the processing of the static feature map, so that the model is more sensitive to the change details of the dynamic micro-expression, and the accuracy of micro-expression recognition is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a dynamic micro-expression recognition method based on frame weights and related devices. Background Art

[0002] Research data shows that facial expressions, as one of the most direct forms of human expression, convey 55% of all information. However, macro-facial expressions can be artificially disguised and manipulated, making them incapable of truly reflecting a person's inner emotions and mental activities. Therefore, researchers have turned their attention to studying micro-expressions, the subconscious expressions displayed by the human face. Micro-expressions, as the name suggests, are subtle, short-lived facial changes, lasting only tens to hundreds of milliseconds, yet they contain a wealth of emotional and intentional information. Compared to regular macro-expressions, micro-expressions are often unconscious and spontaneous, making the emotional and intentional information they convey more authentic and reliable. Therefore, micro-expressions hold great research value in emotion recognition.

[0003] Because microexpressions in static images resemble neutral expressions, researchers often study entire frame sequences containing microexpression variations to more clearly demonstrate and identify these variations, effectively recognizing dynamic microexpressions. Current research methods for microexpression recognition can be categorized as either handcrafted features or deep learning-based approaches. However, handcrafted feature-based approaches only extract surface information from images and fail to capture subtle variations in facial expressions. With the rapid development of deep learning technology in recent years, related techniques and methods have also been applied to microexpression recognition. Compared with traditional handcrafted feature extraction methods, deep learning-based approaches are more effective in extracting and classifying microexpression video features. Deep learning models can also further improve microexpression recognition performance through various attention mechanisms. However, due to the large number of frames in dynamic microexpression image sequences, redundant information can easily overwhelm valuable information. This makes it difficult for existing methods to extract valuable feature information from the large number of features, making microexpression recognition challenging. Summary of the Invention

[0004] The purpose of this application is to provide a dynamic micro-expression recognition method and related devices based on frame weights, which can effectively extract valuable feature information from dynamic micro-expression image sequences and improve the accuracy of micro-expression recognition by combining static features and dynamic features.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for dynamic micro-expression recognition based on frame weights, comprising:

[0007] obtaining a micro-expression image sequence to be recognized;

[0008] inputting the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result, wherein the dynamic micro-expression recognition model comprises an initial convolution module, a frame attention module, a dynamic-static combination module, and a time sequence processing module;

[0009] The frame attention module is configured to calculate a feature map with channel attention weights and frame attention weights according to a feature map output by the initial convolution module.

[0010] The dynamic-static combination module is configured to:

[0011] process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain a static feature.

[0012] calculate a difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and a feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain a dynamic feature.

[0013] process the static feature and the dynamic feature to obtain a feature map containing the static feature and the dynamic feature.

[0014] The time sequence processing module is configured to process the feature map containing the static feature and the dynamic feature output by the dynamic-static combination module to obtain the micro-expression recognition result.

[0015] In a second aspect, a computer device is provided, which comprises a memory, a processor, a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the dynamic micro-expression recognition method based on frame weights in the above.

[0016] In a third aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the dynamic micro-expression recognition method based on frame weights in the above.

[0017] In a fourth aspect, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the dynamic micro-expression recognition method based on frame weights in the above.

[0018] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0019] The present application provides a dynamic micro-expression recognition method and related devices based on frame weights, the method comprising: obtaining a micro-expression image sequence to be recognized, inputting the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model, and obtaining a micro-expression recognition result; wherein, the present application adopts a frame attention module in the dynamic micro-expression recognition model, thereby calculating the frame attention weight while calculating the channel attention weight based on the idea of ​​the channel attention mechanism, thereby realizing the recognition of key frames (frames that have a greater impact on the accuracy of the recognition results) in the dynamic micro-expression image sequence, and can more effectively extract valuable feature information, and also by adopting a dynamic and static combination module, on the basis of processing static feature maps, adds dynamic information between feature maps, making the model more sensitive to the changing details of dynamic micro-expressions, greatly improving the accuracy of micro-expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a diagram of an application environment of a dynamic micro-expression recognition method based on frame weights in one embodiment of the present application;

[0022] Figure 2 A flowchart of a method for dynamic micro-expression recognition based on frame weights provided in one embodiment of the present application;

[0023] Figure 3 A schematic diagram of a dynamic micro-expression recognition model based on frame weights provided in one embodiment of the present application;

[0024] Figure 4 A schematic diagram of an initial convolution unit provided in one embodiment of the present application;

[0025] Figure 5 A schematic diagram of a residual convolution unit provided in one embodiment of the present application;

[0026] Figure 6 A schematic diagram of a frame attention unit provided in one embodiment of the present application;

[0027] Figure 7 A schematic diagram of a dynamic-static combination module provided in one embodiment of the present application;

[0028] Figure 8 A schematic diagram of a timing processing module provided in one embodiment of the present application;

[0029] Figure 9 A schematic flow chart of another method for dynamic micro-expression recognition based on frame weights provided in one embodiment of the present application;

[0030] Figure 10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] Currently, micro-expression recognition methods based on handcrafted features rely on the manual design of image features. The designed feature representation directly affects the accuracy of micro-expression recognition. In terms of handcrafted feature extraction, a variety of effective micro-expression recognition methods have been developed. For example, the Local Binary Pattern of Three Orthogonal Planes (LBP-TOP) feature, based on the traditional LBP (Local Binary Pattern) feature, incorporates temporal information, expanding the feature dimension from two to three dimensions, thereby enabling feature extraction from video image sequences. Furthermore, due to the feature's ease of computation and adaptability, researchers have designed variants such as ELBPTOP and DCP-TOP, further enhancing the feature's expressive power by incorporating additional information or improving computational methods. Furthermore, optical flow-based features are also a representative micro-expression recognition method based on handcrafted features. The basic optical flow algorithm calculates pixel displacements within each image in a video sequence to capture motion information between images, making it naturally adaptable to micro-expression variations. Therefore, researchers proposed a micro-expression recognition method based on directional optical flow histograms, which can simultaneously represent motion information in the temporal dimension and the size and direction of the motion. They also introduced the Facial Action Coding System (FACS) to divide the facial area into 36 regions of interest (ROIs). Finally, the optical flow features of each region were input into a support vector machine for micro-expression recognition. Due to the effectiveness of optical flow methods in micro-expression recognition, researchers subsequently proposed sparse optical flow features (Spare MDMO) that focus on the underlying structure of the feature space, dynamic facial features (FDM) that extracts the optical flow features of video sequences after segmentation, and a dual-weighted feature descriptor (Bi-WOOF) that uses the starting frame and peak frame to obtain optical flow features.

[0033] Methods based on handcrafted features can only extract surface information from images and fail to capture subtle changes in facial expressions. With the rapid development of deep learning technology in recent years, related techniques and methods have also been applied to micro-expression recognition. Convolutional neural networks (CNNs) are widely considered to be effective image feature extraction methods. Considering the importance of dynamic information between micro-expression sequences for micro-expression recognition, a recognition method based on small-scale spatiotemporal features has emerged. This method first extracts spatial features of micro-expressions using a convolutional neural network, then extracts temporal features using a long short-term memory network (LSTM), thereby enabling micro-expression type recognition. Other researchers have used graph structures to represent facial features, using graph nodes to represent key facial nodes and edge features to represent the motion of facial muscle groups. Finally, a graph convolutional neural network (GCNN) is used to extract the graph structure, thereby enabling micro-expression recognition. Furthermore, with the development of related technologies, deep recurrent convolutional networks (STRCNs) based on multiple recurrent convolutional layers have emerged. Micro-expression recognition methods that extract and recognize global and local dual-channel features from amplified peak frames have also emerged. Three-dimensional convolutional neural networks (3D convolutional neural networks) simultaneously convolute both spatial and temporal image information. In addition to various applications of convolutional neural networks, researchers have also noted the significant effect of attention mechanisms on improving the accuracy of micro-expression recognition. For example, the micro-attention residual network extracts features from the peak frames of a micro-expression sequence and then extracts multi-scale features by combining the attention mechanism with a residual network; the micro-expression attention unit combines the spatial attention mechanism with optical flow features to assign weights to areas where micro-expressions may occur; and the spatiotemporal attention module combines the spatial attention module with the temporal attention module to focus on the location and timing of large muscle movements.

[0034] In summary, compared with traditional manual feature extraction methods, methods based on deep learning can more effectively extract micro-expression video features and classify them. At the same time, deep learning models can further improve micro-expression recognition performance through various attention mechanisms. However, the number of frames in dynamic micro-expression image sequences is relatively large, which can easily affect the training speed of deep learning models. At the same time, too many frames can easily cause redundant information to overwhelm valuable classification information, bringing difficulties to micro-expression recognition. When faced with the above problems, existing methods find it difficult to extract valuable features from a large number of features, and lack the use of dynamic information between frames in dynamic micro-expression sequences. Therefore, the present invention designs a dynamic micro-expression recognition method based on frame weights, that is, a dynamic micro-expression recognition method based on a frame attention module and a dynamic and static combination module to solve the above problems.

[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] The dynamic micro-expression recognition method based on frame weights provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send a micro-expression image sequence to be recognized to the server 104. After the server 104 receives the micro-expression image sequence to be recognized, the server 104 inputs the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result. The server 104 can feed back the obtained micro-expression recognition result to the terminal 102. In addition, in some embodiments, the dynamic micro-expression recognition method can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform recognition on the micro-expression image sequence to be recognized, or the server 104 can obtain the micro-expression image sequence to be recognized from the data storage system and perform recognition on the micro-expression image sequence to be recognized.

[0037] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0038] In an exemplary embodiment, Figure 2 As shown, a dynamic micro-expression recognition method based on frame weight is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps 201 to 202.

[0039] Step 201: Acquire a sequence of micro-expression images to be recognized.

[0040] Step 202: Input the micro-expression image sequence to be recognized into the trained dynamic micro-expression recognition model to obtain micro-expression recognition results; wherein, the dynamic micro-expression recognition model includes an initial convolution module, a frame attention module, a dynamic and static combination module, and a time sequence processing module, see Figure 3 .

[0041] The frame attention module is used to calculate a feature map with channel attention weights and frame attention weights based on the feature map output by the initial convolution module.

[0042] The dynamic and static combination module is used for:

[0043] Process the feature map with channel attention weight and frame attention weight corresponding to the current frame to obtain static features;

[0044] Calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features;

[0045] The static features and the dynamic features are processed to obtain a feature map including the static features and the dynamic features.

[0046] The time sequence processing module is used to process the feature graph containing static features and dynamic features output by the dynamic and static combination module to obtain the micro-expression recognition result.

[0047] In an exemplary embodiment, step 202 specifically includes:

[0048] Step 202.1: pre-process the micro-expression image sequence to be recognized to obtain a pre-processed micro-expression image sequence.

[0049] Step 202.2: Input the pre-processed micro-expression image sequence into the trained dynamic micro-expression recognition model to obtain micro-expression recognition results.

[0050] Furthermore, step 202.1 specifically includes:

[0051] Step 202.1.1: Convert each frame of the micro-expression image in the micro-expression image sequence to be identified into a three-channel RGB image with a size of 112×112 pixels according to a linear interpolation method to obtain an interpolated micro-expression image sequence.

[0052] Step 202.1.2, performing a frame extraction operation on the interpolated micro-expression image sequence to obtain 16 extracted frames of micro-expression images; the 16 extracted frames of micro-expression images are images uniformly distributed in the interpolated micro-expression image sequence.

[0053] Step 202.1.3: Randomly crop the edges of the extracted 16-frame micro-expression image sequence and perform left-right mirror flipping to obtain the pre-processed micro-expression image sequence. Images in the same sequence need to undergo the same operation.

[0054] In an exemplary embodiment, the initial convolution module includes an initial convolution unit and a residual convolution unit; the input of the initial convolution unit is the dynamic micro-expression image, the output of the initial convolution unit is the input of the residual convolution unit, and the output of the residual convolution unit is the input of the frame attention module.

[0055] Referring to Figure 4 In the initial convolution unit, a convolution layer (1), a batch normalization layer (1), an activation layer (1), a convolution layer (2), a batch normalization layer (2), and an activation layer (2) are included. Among them, the convolution layer (1) has 64 convolution kernels, the convolution size is 3x3, the size of a single convolution kernel is 64x3x3, the convolution kernel moving step is 2, the image edge padding size is 1, and no hole convolution is set. The batch normalization layer (1), the activation layer (1) are sequentially arranged after the convolution layer (1), and the activation layer (1) uses the ReLU function as the activation function. The convolution layer (2) is arranged after the activation layer (1), the convolution layer (2) has 64 convolution kernels, the convolution size is 3x3, the size of a single convolution kernel is 64x3x3, the convolution kernel moving step is 1, the image edge padding size is 1, and no hole convolution is set. The batch normalization layer (2) and the activation layer (2) are sequentially arranged after the convolution layer (2), and the second activation layer uses the ReLU function as the activation function.

[0056] Through the initial convolution unit, 64-channel image features can be extracted from the original three-channel image data, and more channel numbers are helpful to better describe the image features.

[0057] Referring to Figure 5 In the residual convolution unit, a convolution layer (3), a batch normalization layer (3), an activation layer (3), a convolution layer (4), a batch normalization layer (4), an activation layer (4), and a convolution layer (5) are included. Among them, the convolution layer (3) has 64 convolution kernels, the convolution size is 3x3, the size of a single convolution kernel is 64x3x3, the convolution kernel moving step is 1, the image edge padding size is 1, and no hole convolution is set. The batch normalization layer (3) and the activation layer (3) are sequentially arranged after the convolution layer (3), and the activation layer (3) uses the ReLU activation function for activation. The convolution layer (4) is arranged after the activation layer (3), the convolution layer (4) has 64 convolution kernels, the convolution size is 3x3, the size of a single convolution kernel is 64x3x3, the convolution kernel moving step is 1, the image edge padding size is 1, and no hole convolution is set. The batch normalization layer (4) and the activation layer (4) are sequentially arranged after the convolution layer (4), and the activation layer (4) uses the ReLU function for activation. The convolution layer (5) is arranged in the residual connection channel, the convolution layer (5) has 64 convolution kernels, the convolution size is 1x1, the size of a single convolution kernel is 64x1x1, the starting point of the residual connection is located before the convolution layer (3), and the ending point is located after the batch normalization layer (4).

[0058] The residual convolution unit can further extract hidden information in the feature map, and the residual connection structure can enhance the flow and utilization efficiency of the feature information and inhibit gradient disappearance during neural network training.

[0059] In an exemplary embodiment, the frame attention module includes one frame attention unit or two frame attention units; when the frame attention module includes two frame attention units, the input of one frame attention unit is the output of the initial convolution module, the output of one frame attention unit is the input of the other frame attention unit, and the output of the other frame attention unit is the input of the dynamic-static combination module. When the frame attention module includes one frame attention unit, the input of the frame attention unit is the output of the initial convolution module, and the output of the frame attention unit is the input of the dynamic-static combination module.

[0060] Stacking similar structures is a common method for constructing neural network models, which can help the model extract deeper features and improve model performance. Stacking another frame attention unit after one frame attention unit helps the model better allocate attention weights to key frames.

[0061] Referring to Figure 6 , the frame attention unit includes a first convolution subunit, a channel attention weight calculation subunit, a frame attention weight calculation subunit, a comprehensive calculation subunit, and an activation output subunit.

[0062] The first convolution subunit is configured to process the input feature map to obtain a first feature map.

[0063] The channel attention weight calculation subunit is configured to process the first feature map to obtain a channel attention weight.

[0064] The frame attention weight calculation subunit is configured to process the first feature map to obtain a frame attention weight. The frame attention module helps the model allocate higher attention weights to valuable frames (key frames), so that the model better recognizes dynamic micro-expressions.

[0065] The comprehensive calculation subunit is configured to perform a product operation on the channel attention weight and the frame attention weight to obtain a first product, and perform a product operation on the first product and the first feature map to obtain a second feature map.

[0066] The activation output subunit is configured to perform activation processing on the second feature map to obtain a feature map with channel attention weights and frame attention weights, and output the feature map with channel attention weights and frame attention weights to the dynamic-static combination module.

[0067] In an exemplary embodiment, the first convolution subunit comprises: a first convolution layer, a first batch normalization layer, a first activation layer, a second convolution layer, and a second batch normalization layer; wherein the first convolution layer has 128 convolution kernels, a convolution size of 3x3, a single convolution kernel size of 64x3x3, a convolution kernel moving step of 2, an edge padding size of 1, and no hollow convolution is set; the first batch normalization layer, the first activation layer are sequentially arranged after the first convolution layer, and the first activation layer uses a ReLU function for activation; the second convolution layer is arranged after the first activation layer, the second convolution layer has 128 convolution kernels, a convolution size of 3x3, a single convolution kernel size of 128x3x3, a convolution kernel moving step of 1, an edge padding size of 1, and no hollow convolution is set; the second batch normalization layer is arranged after the second convolution layer; refer to Figure 6 .

[0068] Through the first convolution subunit, a feature map with a relatively smaller size can be obtained. Then, based on the size of the feature map, the channel attention weight calculation subunit convolves each channel of the feature map into a single numerical value, so that the single numerical value corresponds to a single channel, thereby forming a channel attention weight through an activation function; the frame attention weight calculation subunit convolves the feature map of each frame into a single numerical value in a similar manner, so that the single numerical value corresponds to a single frame, thereby forming a channel attention weight through an activation function; then, the integrated calculation subunit multiplies the product of the channel attention weight and the frame attention weight by the feature map with a relatively smaller size, to obtain a feature map based on the channel attention weight and the frame attention weight.

[0069] Wherein, the inputs of the channel attention weight calculation subunit and the frame attention weight calculation subunit are all the feature maps of the frames output by the first convolution subunit.

[0070] In an exemplary embodiment, the channel attention weight calculation subunit comprises: a third convolution layer, a fourth convolution layer, a second activation layer, a fifth convolution layer, and a third activation layer; wherein, the input of the third convolution layer is the output of the first convolution subunit; the output of the third convolution layer is the input of the fourth convolution layer; the output of the fourth convolution layer is the input of the second activation layer; the output of the second activation layer is the input of the fifth convolution layer; the output of the fifth convolution layer is the input of the third activation layer; refer to Figure 6 .

[0071] The frame attention weight calculation subunit includes: a sixth convolutional layer, a seventh convolutional layer, a fourth activation layer, an eighth convolutional layer and a fifth activation layer; wherein the input of the sixth convolutional layer is the output of the first convolutional subunit; the output of the sixth convolutional layer is the input of the seventh convolutional layer; the output of the seventh convolutional layer is the input of the fourth activation layer; the output of the fourth activation layer is the input of the eighth convolutional layer; the output of the eighth convolutional layer is the input of the fifth activation layer; refer to Figure 6 .

[0072] In an exemplary embodiment, when the frame attention module includes two frame attention units, in one frame attention unit:

[0073] The third convolutional layer has 128 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 128×14×14, the convolution kernel movement step is 1, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 128 groups;

[0074] The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no void convolution is set;

[0075] The second activation layer adopts the ReLU function;

[0076] The fifth convolutional layer has 128 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 8×1×1, the edge filling size is 0, and no void convolution is set;

[0077] The third activation layer adopts Sigmoid function;

[0078] The sixth convolutional layer has 2048 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 2048×14×14, the edge filling size is 0, no void convolution is set, and the convolution grouping is 16 groups;

[0079] The seventh convolutional layer has 128 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 2048×1×1, the edge filling size is 0, and no void convolution is set;

[0080] The fourth activation layer adopts the ReLU function;

[0081] The eighth convolutional layer has 2048 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no void convolution is set;

[0082] The fifth activation layer adopts the Sigmoid function.

[0083] Furthermore, when the frame attention module includes two frame attention units, in the other frame attention unit:

[0084] The first convolutional layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 128×3×3, the convolution kernel movement step is 2, the edge filling size is 1, and no void convolution is set;

[0085] The first activation layer adopts the ReLU function;

[0086] The second convolutional layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 256×3×3, the convolution kernel movement step is 1, the edge filling size is 1, and no void convolution is set;

[0087] The third convolutional layer has 256 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 256×14×14, the convolution kernel movement step is 1, the edge filling size is 0, no void convolution is set, and the convolution grouping is 256 groups;

[0088] The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no void convolution is set;

[0089] The second activation layer adopts the ReLU function;

[0090] The fifth convolutional layer has 256 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 16×1×1, the edge filling size is 0, and no void convolution is set;

[0091] The third activation layer adopts Sigmoid function;

[0092] The sixth convolutional layer has 4096 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 4096×14×14, the edge filling size is 0, no void convolution is set, and the convolution grouping is 16 groups;

[0093] The seventh convolutional layer has 256 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 4096×1×1, the edge filling size is 0, and no void convolution is set;

[0094] The fourth activation layer adopts the ReLU function;

[0095] The eighth convolutional layer has 4096 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no void convolution is set;

[0096] The fifth activation layer adopts the Sigmoid function.

[0097] In this embodiment, the channel attention weight calculation subunit first extracts the features of a single channel through grouped convolution, where the number of groups is equal to the number of channels of the feature map, and the size of the convolution kernel is equal to the size of the feature map. This allows a single channel of the feature map to be convolved into a numerical value rather than a two-dimensional tensor. At the same time, grouped convolution also ensures that the feature map of a single channel will not be affected by other channels when it is convolved into a numerical value. The subsequent two convolutional layers are used to perform deep feature learning on the numerical values ​​convolved from a single channel, and ReLU activation functions and Sigmoid activation functions are inserted after the two convolutional layers respectively. The ReLU activation function is used to accelerate training and avoid gradient disappearance, and the Sigmoid activation function is used to compress the weight value to between 0 and 1. Similarly, the frame attention weight calculation subunit first extracts the features of a single frame through grouped convolution, where the number of groups is equal to the number of extracted frames, and the size of the convolution kernel is equal to the size of the feature map. This allows the feature map corresponding to a single frame to be convolved into a single value rather than a high-dimensional tensor. At the same time, grouped convolution also ensures that the feature map of a single frame will not be affected by the features of other frames when it is convolved into a value. The subsequent two convolutional layers are used to perform deep feature learning on the values ​​convolved from a single frame, and the ReLU activation function and Sigmoid activation function are inserted after the two convolutional layers respectively. The ReLU activation function is used to accelerate training and avoid gradient disappearance, and the Sigmoid activation function is used to compress the weight value to between 0 and 1.

[0098] In an exemplary embodiment, the activation output subunit includes a fifth activation layer, which is activated using a Sigmoid function; see Figure 6 .

[0099] See Figure 7 In an exemplary embodiment, the dynamic-static combination module includes a static feature extraction unit, a dynamic feature extraction unit, and a combination unit.

[0100] The static feature extraction unit is used to process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain static features.

[0101] The dynamic feature extraction unit is used to calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features.

[0102] The combining unit is used to process the static features and the dynamic features to obtain a feature map containing the static features and the dynamic features.

[0103] In an exemplary embodiment, the combining unit combines the static feature and the dynamic feature, thereby achieving a combination of static and dynamic features.

[0104] In an exemplary embodiment, the static feature extraction unit includes one Transformer layer or two Transformer layers with the same structure; when the static feature extraction unit includes two Transformer layers with the same structure, the input of one Transformer layer is the output of the frame attention module, the output of one Transformer layer is the input of another Transformer layer, and the output of the other Transformer layer is the input of the combination unit; see Figure 7 When the static feature extraction unit includes a Transformer layer, the input of the Transformer layer is the output of the frame attention module, and the output of the Transformer layer is the input of the combination unit.

[0105] In one exemplary embodiment, the Transformer layer has four heads for the multi-head self-attention mechanism, a single-head has 32 dimensions, and the query, key, and value vectors are all 128-dimensional. When two Transformer layers are included, the two layers are identical. The positional encoding tensor is 256×7×7 in size. The positional encoding tensor is initialized randomly and directly added to the input feature map tensor.

[0106] In an exemplary embodiment, the dynamic feature extraction unit includes a difference calculation layer, a convolution layer and an activation layer; wherein the input of the difference calculation layer is the output of the frame attention module; the output of the difference calculation layer is the input of the convolution layer; the output of the convolution layer is the input of the activation layer; the output of the activation layer is the input of the combination unit; refer to Figure 7 .

[0107] The difference calculation layer is used to calculate the difference between the feature map with channel attention weight and frame attention weight corresponding to the current frame and the feature map with channel attention weight and frame attention weight corresponding to the previous frame;

[0108] The convolution layer is used to perform a convolution operation on the difference to obtain features after the convolution operation;

[0109] The activation layer is used to perform an activation operation on the features after the convolution operation to obtain the dynamic features.

[0110] In an exemplary embodiment, in the dynamic feature extraction unit, the convolution layer has 256 convolution kernels, and the convolution kernel size is 256×7×7; the activation layer adopts the ReLU activation function.

[0111] In other words, the dynamic-static combination module can form a dual-channel structure consisting of two Transformer layers, a convolutional layer and an activation layer. The static features of the input feature map are further extracted through the two Transformer layers, and the dynamic features are obtained by the difference between the feature maps corresponding to two adjacent frames. The dynamic features are further extracted through the convolutional layer and the activation layer; the static features are added to the dynamic features to achieve dynamic-static combination.

[0112] See Figure 8 The temporal processing module includes a temporal processing unit and a fully connected layer. The input of the temporal processing unit is the output of the dynamic-static combination module, and the output of the temporal processing unit is the input of the fully connected layer. The temporal processing unit includes two Transformer layers and an average calculation layer. The position encoding tensor corresponding to the Transformer layer has a size of 15×4096 and is directly added to the input feature map tensor. The multi-head self-attention mechanism in the first Transformer layer has four heads, each head has a dimension of 32 dimensions, and the query vector, key vector, and value vector dimensions are all 128. The multi-head self-attention mechanism in the second Transformer layer after the first Transformer layer has four heads, each head has a dimension of 32 dimensions, and the query vector, key vector, and value vector dimensions are all 128. Further feature extraction is performed on the feature map containing static and dynamic features through two consecutive Transformer layers. The average calculation layer is used to average the feature maps of all frames output by the second Transformer layer to obtain the averaged feature map, achieving averaging in the temporal dimension. The fully connected layer in the time series processing module is used to classify the feature map after the averaging operation to obtain a classification result.

[0113] In some embodiments, the fully connected layer has an input dimension of 4096 and an output dimension of 7. The fully connected layer ultimately outputs a seven-dimensional tensor, corresponding to seven emotion categories, where the emotion category corresponding to the dimension with the largest value is the recognition result of the model.

[0114] In an exemplary embodiment, the process of determining the trained dynamic micro-expression recognition model is as follows:

[0115] Step 301 , obtaining a dynamic micro-expression dataset; each sample data in the dynamic micro-expression dataset includes a dynamic micro-expression image sequence consisting of continuous frames of facial micro-expression images and a corresponding micro-expression label.

[0116] In some embodiments, the dynamic micro-expression dataset used is the SAMM dynamic micro-expression dataset.

[0117] In step 302, the dynamic micro-expression dataset is divided into a training set and a test set. Preprocessing is performed on the training set and the test set, respectively, to obtain preprocessed training and test sets. The preprocessing performed on each sample in this step is the same as in the previous embodiment and will not be further described.

[0118] Step 303 : performing several rounds of deep learning training on the dynamic micro-expression recognition model based on the pre-processed training set to obtain several trained dynamic micro-expression recognition models.

[0119] Step 304 : Testing the trained dynamic micro-expression recognition model based on the pre-processed test set to obtain a test result of the trained dynamic micro-expression recognition model.

[0120] Step 305 : Determine the trained dynamic micro-expression recognition model corresponding to the optimal test result as the trained dynamic micro-expression recognition model.

[0121] In some embodiments, the number of training rounds is set to 1000 rounds, the learning rate is 0.0001, and each round of training is performed using the training set. After each round of training is completed, the dynamic micro-expression recognition model is tested using the test set without gradient calculation, and the loss curve and accuracy are output.

[0122] During the training process, the dynamic micro-expression recognition model with the best rounds is recorded, and the T-SNE image and confusion matrix image are output to facilitate the observation of the recognition effect of the dynamic micro-expression recognition model.

[0123] In another exemplary embodiment, a method for dynamic micro-expression recognition based on a frame attention and dynamic-static combination module includes a step of constructing a neural network model based on the frame attention and dynamic-static combination module, and a step of dynamic micro-expression recognition; wherein the step of constructing a neural network model based on the frame attention and dynamic-static combination module includes:

[0124] Step 401 : extracting micro-expression samples from a dynamic micro-expression dataset. The sample format is a dynamic micro-expression image sequence consisting of continuous frames of facial micro-expression images, wherein the facial micro-expression images are three-channel RGB images.

[0125] In step 402, the dynamic micro-expression sequence data is divided into a training set and a test set, and preprocessing operations are performed on each set. The preprocessing operations specifically include: converting each frame of each sample in the dataset into a three-channel RGB image of 112×112 pixels using linear interpolation; extracting 16 frames of micro-expression images from each sample in the dataset, with the 16 frames of images being uniformly spaced and spaced in the original position of the sample sequence; and finally, performing random edge cropping and left-right mirror flipping on each sample of the extracted image sequence. Images in the same sequence must undergo the same operations.

[0126] Step 403: construct a dynamic micro-expression recognition model based on frame attention and dynamic and static combined modules.

[0127] Step 404 , deep learning is performed to obtain a dynamic micro-expression recognition model that can assign more weights to key frames and integrate dynamic and static micro-expression information.

[0128] The steps of dynamic micro-expression recognition include:

[0129] Step 405: Acquire a sequence of micro-expression images to be recognized and perform pre-processing.

[0130] Step 406 , load the trained dynamic micro-expression recognition model, input the pre-processed micro-expression image sequence, and obtain dynamic micro-expression detection results.

[0131] like Figure 9 As shown, in another exemplary embodiment, a dynamic micro-expression recognition method based on frame attention and dynamic and static combined modules includes the following steps:

[0132] Step 501: Record micro-expression changes through a camera, and extract the micro-expression video frame by frame to form an image sequence.

[0133] Step 502: pre-process the acquired image sequence.

[0134] In some embodiments, step 502 specifically includes: first, for each frame in the image sequence obtained in step 501, converting it into a three-channel RGB image with a size of 112×112 pixels by linear interpolation; and extracting 16 frames of micro-expression images from the converted image sequence, where the positions of the 16 frames of micro-expression images in the original sequence are the same and evenly distributed; finally, performing random edge cropping and left-right mirror flipping on the extracted 16-frame micro-expression image sequence, and each image in the sequence needs to perform the same operation.

[0135] In some embodiments, the converted image sequence is subjected to frame extraction to extract 16 frames of micro-expression images, specifically comprising: evenly dividing the converted image sequence into 16 parts according to the time dimension, extracting the first frame in each part, and forming an extracted image sequence.

[0136] Step 503: Select the dynamic micro-expression dataset SAMM, train the neural network model, and obtain a dynamic micro-expression recognition model.

[0137] In some embodiments, seven micro-expression samples from the SAMM dynamic micro-expression dataset are selected: anger, surprise, sadness, fear, happiness, disgust, and contempt. The image sequence of each selected sample is subjected to the same preprocessing operations as in the previous embodiment.

[0138] Step 504 : Recognize the acquired dynamic micro-expression sequence using a dynamic micro-expression recognition model.

[0139] Step 505: output the recognition result of the dynamic micro-expression recognition model.

[0140] Steps 504 and 505 are to convert the dynamic micro-expression image sequence processed in step 502 into RGB three-channel image data, and input them into the dynamic micro-expression recognition model through data loading operation, and finally obtain the recognition result output by the model.

[0141] The present application proposes a dynamic micro-expression recognition method based on frame attention and dynamic-static combination modules. A neural network model is constructed by combining the frame attention module and the dynamic-static combination module, and the recognition of dynamic micro-expressions is realized through the model. The frame attention module is based on the idea of ​​the channel attention mechanism. It calculates the frame attention weight while calculating the channel attention weight, and multiplies the two to form a frame attention module, which improves the model's ability to recognize dynamic micro-expression key frames. The dynamic-static combination module adds dynamic information between feature maps based on the processing of static feature maps, making the model more sensitive to the changing details of dynamic micro-expressions.

[0142] This application realizes the extraction of dynamic micro-expression features based on the designed frame attention module, and realizes the combination of dynamic micro-expressions in static images and dynamic change information based on the dynamic-static combination module, and combines the two for time series processing, and finally realizes the recognition of dynamic micro-expressions.

[0143] To further demonstrate the effectiveness and practical application prospects of this application, the dynamic micro-expression recognition accuracy of the ResNet model was compared in the ablation experiment, as shown in Table 1.

[0144] Table 1. Dynamic micro-expression recognition accuracy

[0145] Model structure Accuracy (%) ResNet+Time Series Processing 60.5 Frame attention module + temporal processing 62.4 Dynamic and static combined module + timing processing 65.5 Frame attention module + dynamic and static combination module + timing processing 66.0

[0146] As shown in Table 1, both the frame attention module and the dynamic-static combination module can improve the model's recognition accuracy for dynamic micro-expressions. Using only the frame attention module and temporal processing improves the accuracy by 1.9% compared to ResNet with temporal processing; using only the dynamic-static combination module and temporal processing improves the accuracy by 5% compared to ResNet with temporal processing; and using both the frame attention module and the dynamic-static combination module with temporal processing improves the accuracy by 5.5% compared to ResNet with temporal processing.

[0147] In addition, to further demonstrate the accuracy of the method of this application for various types of dynamic micro-expressions, the UF1 scores of dynamic micro-expression recognition in the ablation experiment of the ResNet model were compared, as shown in Table 2.

[0148] Table 2. UF1 scores for dynamic micro-expression recognition

[0149] Model structure UF1 score ResNet+Time Series Processing 0.258 Frame attention module + temporal processing 0.225 Dynamic and static combined module + timing processing 0.285 Frame attention module + dynamic and static combination module + timing processing 0.286

[0150] It can be seen that the UF1 score of using only the frame attention module and time series processing is actually lower than that of ResNet and time series processing. Combined with the results of the previous increase in the accuracy of the frame attention module, it can be found that the frame attention module has an improvement effect on the types of micro-expressions with large expression changes in key frames, but the recognition performance improvement effect on the types of expressions with no obvious changes in key frames is not obvious, thus reducing the average recognition performance. The dynamic and static combined module has a better effect on the recognition accuracy of each dynamic micro-expression, and the UF1 score is improved by 0.025 compared to ResNet. Finally, the model structure based on the frame attention module and the dynamic and static combined module of this application has relatively better performance than a single frame attention module or a dynamic and static combined module, and the UF1 score reaches 0.286.

[0151] The present application also provides an application scenario, which applies the above-mentioned dynamic micro-expression recognition method based on frame weights. Specifically: the dynamic micro-expression recognition method based on frame weights provided in this embodiment can be applied in dynamic micro-expression recognition scenarios. The dynamic micro-expression recognition scenario includes a dynamic micro-expression image shooting link, a dynamic micro-expression image transmission link, and a dynamic micro-expression recognition link; the dynamic micro-expression image shooting link shoots a dynamic micro-expression image, and the shot dynamic micro-expression image enters the dynamic micro-expression recognition link through the dynamic micro-expression image transmission link. The dynamic micro-expression recognition method based on frame weights provided in this embodiment belongs to the dynamic micro-expression recognition link. Specifically in the dynamic micro-expression recognition link, the obtained micro-expression image sequence to be recognized can be input into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result.

[0152] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store dynamic micro-expression image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a dynamic micro-expression recognition method based on frame weights is implemented.

[0153] Those skilled in the art will understand that Figure 10 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

[0154] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0155] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0157] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0158] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0159] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A dynamic micro-expression recognition method based on frame weights, characterized in that: The dynamic micro-expression recognition method comprises: Obtain a sequence of micro-expression images to be recognized; Inputting the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain micro-expression recognition results; wherein the dynamic micro-expression recognition model includes an initial convolution module, a frame attention module, a dynamic and static combination module, and a time sequence processing module; The frame attention module is used to calculate a feature map with channel attention weights and frame attention weights based on the feature map output by the initial convolution module; The dynamic and static combination module is used for: Process the feature map with channel attention weight and frame attention weight corresponding to the current frame to obtain static features; Calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features; Processing the static features and the dynamic features to obtain a feature map including the static features and the dynamic features; The time sequence processing module is used to process the feature graph containing static features and dynamic features output by the dynamic and static combination module to obtain the micro-expression recognition result; The frame attention module includes one frame attention unit or two frame attention units; when the frame attention module includes two frame attention units, the input of one frame attention unit is the output of the initial convolution module, the output of one frame attention unit is the input of the other frame attention unit, and the output of the other frame attention unit is the input of the dynamic-static combination module; The frame attention unit includes a first convolution subunit, a channel attention weight calculation subunit, a frame attention weight calculation subunit, a comprehensive calculation subunit and an activation output subunit; The first convolution subunit is used to process the input feature map to obtain a first feature map; The channel attention weight calculation subunit is used to process the first feature map to obtain a channel attention weight; The frame attention weight calculation subunit is used to process the first feature map to obtain a frame attention weight; The comprehensive calculation subunit is configured to perform a product operation on the channel attention weight and the frame attention weight to obtain a first product, and perform a product operation on the first product and the first feature map to obtain a second feature map; The activation output subunit is used to perform activation processing on the second feature map to obtain the feature map with channel attention weights and frame attention weights, and output the feature map with channel attention weights and frame attention weights to the dynamic and static combination module; The first convolution subunit includes: a first convolution layer, a first batch of normalization layers, a first activation layer, a second convolution layer, and a second batch of normalization layers; wherein the first convolution layer has 128 convolution kernels, a convolution size of 3×3, a single convolution kernel size of 64×3×3, a convolution kernel movement step of 2, an edge filling size of 1, and no hole convolution is set; the first convolution layer is followed by the first batch of normalization layers and the first activation layer, and the first activation layer is activated using the ReLU function; the first activation layer is followed by the second convolution layer, and the second convolution layer has 128 convolution kernels, a convolution size of 3×3, a single convolution kernel size of 128×3×3, a convolution kernel movement step of 1, an edge filling size of 1, and no hole convolution is set; the second convolution layer is followed by the second batch of normalization layers; The channel attention weight calculation subunit includes: a third convolutional layer, a fourth convolutional layer, a second activation layer, a fifth convolutional layer and a third activation layer; wherein the input of the third convolutional layer is the output of the first convolutional subunit; the input of the fourth convolutional layer is the output of the third convolutional layer; the input of the second activation layer is the output of the fourth convolutional layer; the input of the fifth convolutional layer is the output of the second activation layer; and the input of the third activation layer is the output of the fifth convolutional layer; The frame attention weight calculation subunit includes: a sixth convolutional layer, a seventh convolutional layer, a fourth activation layer, an eighth convolutional layer and a fifth activation layer; wherein the input of the sixth convolutional layer is the output of the first convolutional subunit; the input of the seventh convolutional layer is the output of the sixth convolutional layer; the input of the fourth activation layer is the output of the seventh convolutional layer; the input of the eighth convolutional layer is the output of the fourth activation layer; and the input of the fifth activation layer is the output of the eighth convolutional layer. The dynamic-static combination module includes a static feature extraction unit, a dynamic feature extraction unit and a combination unit; The static feature extraction unit is used to process the feature map with channel attention weight and frame attention weight corresponding to the current frame to obtain static features; The dynamic feature extraction unit is used to calculate the difference between the feature map with channel attention weight and frame attention weight corresponding to the current frame and the feature map with channel attention weight and frame attention weight corresponding to the previous frame, and process the difference to obtain dynamic features; The combining unit is used to process the static features and the dynamic features to obtain a feature map containing the static features and the dynamic features; The static feature extraction unit includes one Transformer layer or two Transformer layers with the same structure; when the static feature extraction unit includes two Transformer layers with the same structure, the input of one Transformer layer is the output of the frame attention module, the output of one Transformer layer is the input of the other Transformer layer, and the output of the other Transformer layer is the input of the combination unit; The dynamic feature extraction unit includes a difference calculation layer, a convolution layer and an activation layer; wherein the input of the difference calculation layer is the output of the frame attention module; the output of the difference calculation layer is the input of the convolution layer; the output of the convolution layer is the input of the activation layer; and the output of the activation layer is the input of the combination unit; The difference calculation layer is used to calculate the difference between the feature map with channel attention weight and frame attention weight corresponding to the current frame and the feature map with channel attention weight and frame attention weight corresponding to the previous frame; The convolution layer is used to perform a convolution operation on the difference to obtain features after the convolution operation; The activation layer is used to perform an activation operation on the features after the convolution operation to obtain the dynamic features.

2. The dynamic micro-expression recognition method based on frame weights according to claim 1, wherein When the frame attention module includes two frame attention units, in one frame attention unit: The third convolutional layer has 128 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 128×14×14, the convolution kernel movement step is 1, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 128 groups; The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no void convolution is set; The second activation layer adopts the ReLU function; The fifth convolutional layer has 128 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 8×1×1, the edge filling size is 0, and no void convolution is set; The third activation layer adopts Sigmoid function; The sixth convolutional layer has 2048 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 2048×14×14, the edge filling size is 0, no void convolution is set, and the convolution grouping is 16 groups; The seventh convolutional layer has 128 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 2048×1×1, the edge filling size is 0, and no void convolution is set; The fourth activation layer adopts the ReLU function; The eighth convolutional layer has 2048 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no void convolution is set; The fifth activation layer adopts the Sigmoid function.

3. The dynamic micro-expression recognition method based on frame weights according to claim 2, wherein When the frame attention module includes two frame attention units, in the other frame attention unit: The first convolutional layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 128×3×3, the convolution kernel movement step is 2, the edge filling size is 1, and no void convolution is set; The first activation layer adopts the ReLU function; The second convolutional layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 256×3×3, the convolution kernel movement step is 1, the edge filling size is 1, and no void convolution is set; The third convolutional layer has 256 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 256×14×14, the convolution kernel movement step is 1, the edge filling size is 0, no void convolution is set, and the convolution grouping is 256 groups; The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no void convolution is set; The second activation layer adopts the ReLU function; The fifth convolutional layer has 256 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 16×1×1, the edge filling size is 0, and no void convolution is set; The third activation layer adopts Sigmoid function; The sixth convolutional layer has 4096 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 4096×14×14, the edge filling size is 0, no void convolution is set, and the convolution grouping is 16 groups; The seventh convolutional layer has 256 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 4096×1×1, the edge filling size is 0, and no void convolution is set; The fourth activation layer adopts the ReLU function; The eighth convolutional layer has 4096 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no void convolution is set; The fifth activation layer adopts the Sigmoid function.

4. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the frame weight-based dynamic micro-expression recognition method according to any one of claims 1 to 3.

5. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the frame weight-based dynamic micro-expression recognition method according to any one of claims 1 to 3 is implemented.

Images