Dynamic micro-expression recognition method based on frame weight and related device

Abstract

The invention discloses a frame weight-based dynamic micro-expression recognition method and a related device, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-recognized micro-expression image sequence, inputting the to-be-recognized micro-expression image sequence into a trained dynamic micro-expression recognition model, and obtaining a micro-expression recognition result; the dynamic micro-expression recognition model comprises an initial convolution module, a frame attention module, a dynamic and static combination module and a time sequence processing module. According to the method, the frame attention module is adopted in the dynamic micro-expression recognition model, the frame attention weight can be calculated while the channel attention weight is calculated, recognition of the key frame in the dynamic micro-expression image sequence is achieved, and by adopting the dynamic and static combination module, the recognition accuracy of the key frame in the dynamic micro-expression image sequence is improved. Dynamic information between the feature maps is added on the basis of static feature map processing, so that the model is more sensitive to change details of dynamic micro-expressions, and the micro-expression recognition accuracy is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a dynamic micro-expression recognition method based on frame weights and related devices. Background Art

[0002] Research data shows that facial expressions, as one of the most direct ways of expression for humans, convey 55% of the total amount of information people convey. However, macro facial expressions may be artificially disguised and controlled, resulting in them not being able to truly reflect a person's inner emotions and psychological activities. Therefore, researchers have turned their attention to the study of micro expressions that are subconsciously displayed on the face. Micro expressions, as the name suggests, refer to facial expressions with subtle facial changes and short durations, which may last only tens to hundreds of milliseconds, but contain a lot of emotional and intentional information. Compared with conventional macro expressions, micro expressions are often unconscious and spontaneous, which also makes the emotional and intentional information they express more authentic and reliable. Therefore, micro expressions also have extremely high research value in emotion recognition.

[0003] Since micro-expressions in static images are similar to neutral expressions, researchers often take a whole frame sequence containing micro-expression changes as the research object, so as to more significantly show the changes and recognition of micro-expressions, that is, to achieve the recognition of dynamic micro-expressions. At present, the relevant research methods of micro-expression recognition can be mainly divided into two methods: those based on manual features and those based on deep learning. However, the method based on manual features can only extract the surface information of the image and cannot further capture the subtle deformation of the expression. With the rapid development of deep learning technology in recent years, related technologies and methods have also been applied to micro-expression recognition. Compared with traditional manual feature extraction methods, the method based on deep learning can more effectively extract micro-expression video features and classify them. At the same time, the deep learning model can further improve the micro-expression recognition performance through various attention mechanisms. However, due to the large number of frames in the dynamic micro-expression image sequence, it is easy for redundant information to overwhelm valuable information, resulting in the existing methods being difficult to extract valuable feature information from a large number of features, which brings difficulties to micro-expression recognition. Summary of the invention

[0004] The purpose of this application is to provide a dynamic micro-expression recognition method and related devices based on frame weights, which can effectively extract valuable feature information from dynamic micro-expression image sequences and improve the accuracy of micro-expression recognition by combining static features and dynamic features.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for dynamic micro-expression recognition based on frame weights, comprising:

[0007] Obtain a sequence of micro-expression images to be recognized;

[0008] Input the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result; wherein the dynamic micro-expression recognition model includes an initial convolution module, a frame attention module, a dynamic and static combination module, and a timing processing module;

[0009] The frame attention module is used to calculate a feature map with channel attention weights and frame attention weights according to the feature map output by the initial convolution module;

[0010] The dynamic and static combination module is used for:

[0011] Process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain static features;

[0012] Calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features;

[0013] Processing the static features and the dynamic features to obtain a feature map including the static features and the dynamic features;

[0014] The timing processing module is used to process the feature graph containing static features and dynamic features output by the dynamic-static combination module to obtain the micro-expression recognition result.

[0015] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned frame weight-based dynamic micro-expression recognition method.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned frame weight-based dynamic micro-expression recognition method.

[0017] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned frame weight-based dynamic micro-expression recognition method.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present application provides a method and related devices for dynamic micro-expression recognition based on frame weights, the method comprising: obtaining a sequence of micro-expression images to be recognized, inputting the sequence of micro-expression images to be recognized into a trained dynamic micro-expression recognition model, and obtaining a micro-expression recognition result; wherein, the present application adopts a frame attention module in the dynamic micro-expression recognition model, thereby calculating the frame attention weight while calculating the channel attention weight based on the idea of ​​the channel attention mechanism, thereby realizing the recognition of key frames (frames that have a greater impact on the accuracy of the recognition results) in the dynamic micro-expression image sequence, and can more effectively extract valuable feature information, and also adopts a dynamic and static combination module, on the basis of the processing of static feature maps, adds dynamic information between feature maps, makes the model more sensitive to the changing details of dynamic micro-expressions, and greatly improves the accuracy of micro-expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is an application environment diagram of a dynamic micro-expression recognition method based on frame weights in one embodiment of the present application;

[0022] Figure 2 A schematic diagram of a flow chart of a dynamic micro-expression recognition method based on frame weights provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a dynamic micro-expression recognition model based on frame weights provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of an initial convolution unit provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a residual convolution unit provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a frame attention unit provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a dynamic-static combination module provided in one embodiment of the present application;

[0028] Figure 8 A schematic diagram of a timing processing module provided in an embodiment of the present application;

[0029] Fig. 9 A schematic flow chart of another method for dynamic micro-expression recognition based on frame weights provided in one embodiment of the present application;

[0030] Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] At present, the micro-expression recognition method based on manual features relies on the manual design of image features, and the designed feature representation directly affects the accuracy of micro-expression recognition. In terms of manual feature extraction, a variety of effective micro-expression recognition methods have been developed. For example, the local binary pattern (LBP-TOP) feature of three orthogonal planes introduces time domain information on the basis of the traditional LBP (local binary pattern) feature, and expands the feature dimension from two dimensions to three dimensions, thereby realizing feature extraction for video image sequences. At the same time, because of the characteristics of simple calculation and strong adaptability of this feature, researchers have also designed variants such as ELBPTOP and DCP-TOP on its basis, and further improved the expressive ability of the feature by introducing additional information or improving the calculation method. In addition, the feature based on optical flow is also a representative micro-expression recognition method based on manual features. The basic algorithm of optical flow is to obtain the motion information between images by calculating the pixel displacement on each image in the video sequence, which has a natural adaptability to the changes in micro-expressions. Therefore, researchers proposed a micro-expression recognition method based on directional optical flow histograms, which can simultaneously represent the motion information in the time dimension and the size and direction of the motion. At the same time, the facial action coding system (FACS) was introduced to divide the facial area into 36 regions of interest (ROIs). Finally, the optical flow features of each region were input into the support vector machine for micro-expression recognition. Due to the effectiveness of the optical flow method in micro-expression recognition, researchers subsequently proposed sparse optical flow features (Spare MDMO) that focus on the underlying structure of the feature space, dynamic face features (FDM) that extracts the optical flow features of video sequences after segmentation, and dual weighted feature descriptors (Bi-WOOF) that use the start frame and peak frame to obtain optical flow features.

[0033] Methods based on manual features can only extract surface information of images and cannot further capture subtle changes in facial expressions. With the rapid development of deep learning technology in recent years, related technologies and methods have also been applied to micro-expression recognition, among which convolutional neural network (CNN) is widely considered to be an effective image feature extraction method. Considering that the dynamic information between micro-expression sequences plays an important role in identifying micro-expressions, a recognition method based on small-scale spatiotemporal features has emerged. First, the spatial features of micro-expressions are extracted through convolutional neural networks, and then the temporal features of micro-expressions are extracted through long short-term memory networks (LSTM), thereby realizing the recognition of micro-expression types. Some researchers also use graph structures to represent facial features, use graph nodes to represent key facial nodes, use edge features to represent the movement state of facial muscle groups, and finally extract graph structures through graph convolutional neural networks to achieve micro-expression recognition. In addition, with the development of related technologies, deep recursive convolutional networks (STRCNs) based on multiple recurrent convolutional layers have also emerged, micro-expression recognition methods that extract and recognize global and local dual-channel features of the amplified peak frame, and three-dimensional convolutional neural networks that convolve image spatial information and temporal information at the same time. In addition to various applications of convolutional neural networks, researchers have also noticed the significant effect of attention mechanisms on improving the accuracy of micro-expression recognition. For example, the peak frame of the micro-expression sequence is feature extracted, and then the micro-attention mechanism (Micro-Attention) residual network is combined with the attention mechanism and the residual network to extract multi-scale features; the micro-expression attention unit combines the spatial attention mechanism with the optical flow features and assigns weights to the areas where micro-expressions may occur; the spatiotemporal attention module combines the spatial attention module with the temporal attention module to achieve attention to the position and time of large muscle movements, etc.

[0034] In summary, compared with the traditional manual feature extraction method, the method based on deep learning can more effectively extract micro-expression video features and classify them. At the same time, the deep learning model can further improve the micro-expression recognition performance through various attention mechanisms. However, the number of frames in the dynamic micro-expression image sequence is large, which is easy to affect the training speed of the deep learning model. At the same time, too many frames can easily make redundant information submerge valuable classification information, bringing difficulties to micro-expression recognition. When facing the above problems, the existing methods are difficult to extract valuable features from a large number of features, and lack the use of dynamic information between frames of the dynamic micro-expression sequence. Therefore, the present invention designs a dynamic micro-expression recognition method based on frame weights, that is, a dynamic micro-expression recognition method based on a frame attention module and a dynamic and static combination module to solve the above problems.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] The dynamic micro-expression recognition method based on frame weights provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the micro-expression image sequence to be identified to the server 104. After the server 104 receives the micro-expression image sequence to be identified, for the micro-expression image sequence to be identified, the server 104 inputs the micro-expression image sequence to be identified into the trained dynamic micro-expression recognition model to obtain the micro-expression recognition result. The server 104 can feed back the obtained micro-expression recognition result to the terminal 102. In addition, in some embodiments, the dynamic micro-expression recognition method can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly identify the micro-expression image sequence to be identified, or the server 104 can obtain the micro-expression image sequence to be identified from the data storage system and identify the micro-expression image sequence to be identified.

[0037] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0038] In an exemplary embodiment, Figure 2 As shown, a dynamic micro-expression recognition method based on frame weight is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 202. Among them:

[0039] Step 201, obtaining a sequence of micro-expression images to be recognized.

[0040] Step 202, input the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain micro-expression recognition results; wherein the dynamic micro-expression recognition model includes an initial convolution module, a frame attention module, a dynamic and static combination module, and a timing processing module, see Figure 3 .

[0041] The frame attention module is used to calculate a feature map with channel attention weights and frame attention weights based on the feature map output by the initial convolution module.

[0042] The dynamic and static combination module is used for:

[0043] Process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain static features;

[0044] Calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features;

[0045] The static features and the dynamic features are processed to obtain a feature map including the static features and the dynamic features.

[0046] The timing processing module is used to process the feature graph containing static features and dynamic features output by the dynamic-static combination module to obtain the micro-expression recognition result.

[0047] In an exemplary embodiment, the step 202 specifically includes:

[0048] Step 202.1, preprocessing the micro-expression image sequence to be recognized to obtain a preprocessed micro-expression image sequence.

[0049] Step 202.2, input the pre-processed micro-expression image sequence into the trained dynamic micro-expression recognition model to obtain micro-expression recognition results.

[0050] Furthermore, step 202.1 specifically includes:

[0051] Step 202.1.1, according to the linear interpolation method, convert each frame of the micro-expression image in the micro-expression image sequence to be identified into a three-channel RGB image with a size of 112×112 pixels, to obtain an interpolated micro-expression image sequence.

[0052] Step 202.1.2, performing a frame extraction operation on the interpolated micro-expression image sequence to obtain 16 extracted frames of micro-expression images; the extracted 16 frames of micro-expression images are images uniformly distributed in the interpolated micro-expression image sequence.

[0053] Step 202.1.3, randomly cropping the edges of the extracted 16-frame micro-expression image sequence and flipping the left and right images to obtain the pre-processed micro-expression image sequence. Images in the same sequence need to undergo the same operation.

[0054] In an exemplary embodiment, the initial convolution module includes an initial convolution unit and a residual convolution unit; the input of the initial convolution unit is a dynamic micro-expression image, the output of the initial convolution unit is the input of the residual convolution unit, and the output of the residual convolution unit is the input of the frame attention module.

[0055] See also Figure 4 In the initial convolution unit, there are convolution layer (1), batch normalization layer (1), activation layer (1), convolution layer (2), batch normalization layer (2) and activation layer (2). Among them, convolution layer (1) has 64 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 64×3×3, the convolution kernel moving step is 2, the image edge filling size is 1, and no hole convolution is set. After convolution layer (1), batch normalization layer (1) and activation layer (1) are set in sequence. The activation layer (1) uses the ReLU function as the activation function. After activation layer (1), convolution layer (2) is set. Convolution layer (2) has 64 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 64×3×3, the convolution kernel moving step is 1, the image edge filling size is 1, and no hole convolution is set. After the convolution layer (2), a batch normalization layer (2) and an activation layer (2) are sequentially set. The second activation layer uses the ReLU function as the activation function.

[0056] Through the initial convolution unit, 64-channel image features can be extracted from the original three-channel image data. More channels help to better describe the image features.

[0057] See also Figure 5 In the residual convolution unit, there are convolution layer (3), batch normalization layer (3), activation layer (3), convolution layer (4), batch normalization layer (4), activation layer (4), and convolution layer (5). Among them, convolution layer (3) has 64 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 64×3×3, the convolution kernel moving step is 1, the image edge filling size is 1, and no hole convolution setting is performed. After convolution layer (3), batch normalization layer (3) and activation layer (3) are sequentially set, and activation layer (3) is activated using ReLU activation function. After activation layer (3), convolution layer (4) is set, and convolution layer (4) has 64 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 64×3×3, the convolution kernel moving step is 1, the image edge filling size is 1, and no hole convolution setting is performed. After the convolution layer (4), a batch normalization layer (4) and an activation layer (4) are sequentially set. The activation layer (4) uses the ReLU function for activation. A convolution layer (5) is set in the residual connection channel. The convolution layer (5) has 64 convolution kernels, the convolution size is 1×1, and the size of a single convolution kernel is 64×1×1. The starting point of the residual connection is before the convolution layer (3) and the end point is after the batch normalization layer (4).

[0058] The residual convolution unit can further extract the hidden information in the feature map. At the same time, the residual connection structure can enhance the fluidity and utilization efficiency of feature information and suppress the gradient disappearance during neural network training.

[0059] In an exemplary embodiment, the frame attention module includes one frame attention unit or two frame attention units; when the frame attention module includes two frame attention units, the input of one frame attention unit is the output of the initial convolution module, the output of one frame attention unit is the input of another frame attention unit, and the output of the other frame attention unit is the input of the dynamic-static combination module. When the frame attention module includes one frame attention unit, the input of the frame attention unit is the output of the initial convolution module, and the output of the frame attention unit is the input of the dynamic-static combination module.

[0060] Stacking similar structures is a common method for building neural network models, which can help the model extract deeper features and thus improve model performance. Stacking another frame attention unit after a frame attention unit helps the model better distribute attention weights to key frames.

[0061] See also Figure 6 The frame attention unit includes a first convolution subunit, a channel attention weight calculation subunit, a frame attention weight calculation subunit, a comprehensive calculation subunit and an activation output subunit.

[0062] The first convolution subunit is used to process the input feature map to obtain a first feature map.

[0063] The channel attention weight calculation subunit is used to process the first feature map to obtain the channel attention weight.

[0064] The frame attention weight calculation subunit is used to process the first feature map to obtain the frame attention weight. The frame attention module helps the model to assign higher attention weights to the features of valuable frames (key frames), so that the model can better recognize dynamic micro-expressions.

[0065] The comprehensive calculation subunit is used to perform a product operation on the channel attention weight and the frame attention weight to obtain a first product, and to perform a product operation on the first product and the first feature map to obtain a second feature map.

[0066] The activation output subunit is used to perform activation processing on the second feature map to obtain the feature map with channel attention weights and frame attention weights, and output the feature map with channel attention weights and frame attention weights to the dynamic and static combination module.

[0067] In an exemplary embodiment, the first convolution subunit includes: a first convolution layer, a first batch of normalized layers, a first activation layer, a second convolution layer, and a second batch of normalized layers; wherein the first convolution layer has 128 convolution kernels, a convolution size of 3×3, a single convolution kernel size of 64×3×3, a convolution kernel moving step of 2, an edge filling size of 1, and no hole convolution is set; the first convolution layer is followed by the first batch of normalized layers and the first activation layer, and the first activation layer is activated using the ReLU function; the first activation layer is followed by the second convolution layer, and the second convolution layer has 128 convolution kernels, a convolution size of 3×3, a single convolution kernel size of 128×3×3, a convolution kernel moving step of 1, an edge filling size of 1, and no hole convolution is set; the second batch of normalized layers is followed by the second convolution layer; see Figure 6 .

[0068] A relatively smaller feature map can be obtained through the first convolution subunit. Based on the size of the feature map, the channel attention weight calculation subunit convolves each channel of the feature map into a single value, so that the single value corresponds to the single channel, thereby forming a channel attention weight through the activation function; the frame attention weight calculation subunit convolves the feature map of each frame into a single value in a similar way, so that the single value corresponds to the single frame, thereby forming a channel attention weight through the activation function; then, the comprehensive calculation subunit multiplies the product of the channel attention weight and the frame attention weight by the relatively smaller feature map to obtain a feature map based on the channel attention weight and the frame attention weight.

[0069] Among them, the inputs of the channel attention weight calculation subunit and the frame attention weight calculation subunit are both feature maps of all frames output by the first convolution subunit.

[0070] In an exemplary embodiment, the channel attention weight calculation subunit includes: a third convolutional layer, a fourth convolutional layer, a second activation layer, a fifth convolutional layer and a third activation layer; wherein the input of the third convolutional layer is the output of the first convolutional subunit; the output of the third convolutional layer is the input of the fourth convolutional layer; the output of the fourth convolutional layer is the input of the second activation layer; the output of the second activation layer is the input of the fifth convolutional layer; the output of the fifth convolutional layer is the input of the third activation layer; refer to Figure 6 .

[0071] The frame attention weight calculation subunit includes: a sixth convolutional layer, a seventh convolutional layer, a fourth activation layer, an eighth convolutional layer and a fifth activation layer; wherein the input of the sixth convolutional layer is the output of the first convolutional subunit; the output of the sixth convolutional layer is the input of the seventh convolutional layer; the output of the seventh convolutional layer is the input of the fourth activation layer; the output of the fourth activation layer is the input of the eighth convolutional layer; the output of the eighth convolutional layer is the input of the fifth activation layer; refer to Figure 6 .

[0072] In an exemplary embodiment, when the frame attention module includes two frame attention units, in one frame attention unit:

[0073] The third convolution layer has 128 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 128×14×14, the convolution kernel moving step is 1, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 128 groups;

[0074] The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no hole convolution is set;

[0075] The second activation layer adopts the ReLU function;

[0076] The fifth convolutional layer has 128 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 8×1×1, the edge filling size is 0, and no hole convolution is set;

[0077] The third activation layer adopts Sigmoid function;

[0078] The sixth convolutional layer has 2048 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 2048×14×14, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 16 groups;

[0079] The seventh convolutional layer has 128 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 2048×1×1, the edge filling size is 0, and no hole convolution is set;

[0080] The fourth activation layer adopts the ReLU function;

[0081] The eighth convolutional layer has 2048 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no hole convolution is set;

[0082] The fifth activation layer adopts the Sigmoid function.

[0083] Further, when the frame attention module includes two frame attention units, in the other frame attention unit:

[0084] The first convolution layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 128×3×3, the convolution kernel movement step is 2, the edge filling size is 1, and no hole convolution is set;

[0085] The first activation layer adopts the ReLU function;

[0086] The second convolution layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 256×3×3, the convolution kernel movement step is 1, the edge filling size is 1, and no hole convolution is set;

[0087] The third convolution layer has 256 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 256×14×14, the convolution kernel moving step is 1, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 256 groups;

[0088] The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no hole convolution is set;

[0089] The second activation layer adopts the ReLU function;

[0090] The fifth convolutional layer has 256 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 16×1×1, the edge filling size is 0, and no hole convolution is set;

[0091] The third activation layer adopts Sigmoid function;

[0092] The sixth convolution layer has 4096 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 4096×14×14, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 16 groups;

[0093] The seventh convolutional layer has 256 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 4096×1×1, the edge filling size is 0, and no hole convolution is set;

[0094] The fourth activation layer adopts the ReLU function;

[0095] The eighth convolutional layer has 4096 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no hole convolution is set;

[0096] The fifth activation layer adopts the Sigmoid function.

[0097] In this embodiment, the channel attention weight calculation subunit first extracts the features of a single channel through grouped convolution, where the number of groups is equal to the number of channels of the feature map, and the size of the convolution kernel is equal to the size of the feature map, so that a single channel of the feature map can be convolved into a numerical value instead of a two-dimensional tensor. At the same time, grouped convolution also makes it possible to convolve the feature map of a single channel into a numerical value without being affected by other channels. The subsequent two convolutional layers are used for deep feature learning of the numerical values ​​convolved from a single channel, and ReLU activation functions and Sigmoid activation functions are inserted after the two convolutional layers, respectively. The ReLU activation function is used to accelerate training and avoid gradient disappearance, and the Sigmoid activation function is used to compress the weight value to between 0 and 1. Similarly, the frame attention weight calculation subunit first extracts the features of a single frame through grouped convolution, where the number of groups is equal to the number of extracted frames, and the size of the convolution kernel is equal to the size of the feature map, so that the feature map corresponding to a single frame can be convolved into a numerical value instead of a high-dimensional tensor. At the same time, grouped convolution also makes it possible to convolve the feature map of a single frame into a numerical value without being affected by the features of other frames. The subsequent two convolutional layers are used for deep feature learning of the numerical values ​​convolved from a single frame, and the ReLU activation function and the Sigmoid activation function are inserted after the two convolutional layers. The ReLU activation function is used to accelerate training and avoid gradient disappearance, and the Sigmoid activation function is used to compress the weight value to between 0 and 1.

[0098] In an exemplary embodiment, the activation output subunit includes a fifth activation layer, which is activated using a Sigmoid function; see Figure 6 .

[0099] See also Figure 7 In an exemplary embodiment, the dynamic-static combination module includes a static feature extraction unit, a dynamic feature extraction unit and a combination unit.

[0100] The static feature extraction unit is used to process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain static features.

[0101] The dynamic feature extraction unit is used to calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features.

[0102] The combining unit is used to process the static features and the dynamic features to obtain a feature map including the static features and the dynamic features.

[0103] In an exemplary embodiment, the combining unit combines the static feature and the dynamic feature to achieve a combination of static and dynamic features.

[0104] In an exemplary embodiment, the static feature extraction unit includes one Transformer layer or two Transformer layers with the same structure; when the static feature extraction unit includes two Transformer layers with the same structure, the input of one Transformer layer is the output of the frame attention module, the output of one Transformer layer is the input of another Transformer layer, and the output of the other Transformer layer is the input of the combination unit; see Figure 7 When the static feature extraction unit includes a Transformer layer, the input of the Transformer layer is the output of the frame attention module, and the output of the Transformer layer is the input of the combination unit.

[0105] In an exemplary embodiment, in the Transformer layer, the number of heads of the multi-head self-attention mechanism is 4, the dimension of the single head is 32, the dimensions of the query vector, key vector and value vector are all 128, and when two Transformer layers are included, the two Transformer layers are the same; the position encoding tensor size is 256×7×7. The initialization method of the position encoding tensor is random initialization and directly added to the input feature map tensor.

[0106] In an exemplary embodiment, the dynamic feature extraction unit includes a difference calculation layer, a convolution layer and an activation layer; wherein the input of the difference calculation layer is the output of the frame attention module; the output of the difference calculation layer is the input of the convolution layer; the output of the convolution layer is the input of the activation layer; the output of the activation layer is the input of the combination unit; see Figure 7 .

[0107] The difference calculation layer is used to calculate the difference between the feature map with channel attention weight and frame attention weight corresponding to the current frame and the feature map with channel attention weight and frame attention weight corresponding to the previous frame;

[0108] The convolution layer is used to perform a convolution operation on the difference to obtain features after the convolution operation;

[0109] The activation layer is used to perform an activation operation on the features after the convolution operation to obtain the dynamic features.

[0110] In an exemplary embodiment, in the dynamic feature extraction unit, the convolution layer has 256 convolution kernels, and the convolution kernel size is 256×7×7; the activation layer adopts the ReLU activation function.

[0111] In other words, the dynamic-static combination module can form a dual-channel structure consisting of two Transformer layers, a convolutional layer and an activation layer. The static features of the input feature map are further extracted through the two Transformer layers, and the dynamic features are obtained by the difference between the feature maps corresponding to two adjacent frames. The dynamic features are further extracted through the convolutional layer and the activation layer; the static features are added to the dynamic features to achieve dynamic-static combination.

[0112] See also Figure 8 , the time series processing module includes a time series processing unit and a fully connected layer; the input of the time series processing unit is the output of the dynamic and static combination module, and the output of the time series processing unit is the input of the fully connected layer. Among them, the time series processing unit includes two Transformer layers and an average calculation layer. The size of the position encoding tensor corresponding to the Transformer layer is 15×4096, and the position encoding tensor is directly added to the input feature map tensor. The number of heads of the multi-head self-attention mechanism in the first Transformer layer is 4, the dimension of a single head is 32 dimensions, and the dimensions of the query vector, key vector and value vector are all 128 dimensions. The number of heads of the multi-head self-attention mechanism in the second Transformer layer after the first Transformer layer is 4, the dimension of a single head is 32 dimensions, and the dimensions of the query vector, key vector and value vector are all 128 dimensions. The feature map containing static features and dynamic features is further extracted through two consecutive Transformer layers. The average calculation layer is used to average the feature maps of all frames output by the second Transformer layer to obtain the feature map after the average operation, so as to achieve averaging in the time dimension. The fully connected layer in the time series processing module is used to classify the feature map after the averaging operation to obtain a classification result.

[0113] In some embodiments, the fully connected layer has an input dimension of 4096 and an output dimension of 7. The fully connected layer finally outputs a seven-dimensional tensor corresponding to seven emotion categories, wherein the emotion category corresponding to the dimension with the largest value is the recognition result of the model.

[0114] In an exemplary embodiment, the determination process of the trained dynamic micro-expression recognition model is as follows:

[0115] Step 301, obtaining a dynamic micro-expression data set; each sample data in the dynamic micro-expression data set includes a dynamic micro-expression image sequence composed of continuous frames of facial micro-expression images and a corresponding micro-expression label.

[0116] In some embodiments, the dynamic micro-expression dataset used is the SAMM dynamic micro-expression dataset.

[0117] Step 302, the dynamic micro-expression data set is divided into a training set and a test set, and preprocessing operations are performed on the training set and the test set to obtain a preprocessed training set and a test set. In this step, the preprocessing operation for each sample in the data set is the same as the preprocessing operation in the previous embodiment, and will not be repeated.

[0118] Step 303: Perform several rounds of deep learning training on the dynamic micro-expression recognition model according to the preprocessed training set to obtain several trained dynamic micro-expression recognition models.

[0119] Step 304 , testing the trained dynamic micro-expression recognition model according to the preprocessed test set to obtain a test result of the trained dynamic micro-expression recognition model.

[0120] Step 305 , determining the trained dynamic micro-expression recognition model corresponding to the optimal test result as the trained dynamic micro-expression recognition model.

[0121] In some embodiments, the number of training rounds is set to 1000 rounds, the learning rate is 0.0001, and each round of training is trained using the training set. After each round of training is completed, the dynamic micro-expression recognition model is tested using the test set without gradient calculation, and the loss curve and accuracy are output.

[0122] During the training process, the dynamic micro-expression recognition model with the best rounds is recorded, and the T-SNE image and confusion matrix image are output to facilitate the observation of the recognition effect of the dynamic micro-expression recognition model.

[0123] In another exemplary embodiment, a dynamic micro-expression recognition method based on a frame attention and a dynamic and static combination module includes a neural network model construction step based on the frame attention and the dynamic and static combination module, and a dynamic micro-expression recognition step; wherein the neural network model construction step based on the frame attention and the dynamic and static combination module includes:

[0124] Step 401, extracting micro-expression samples from a dynamic micro-expression dataset, wherein the sample form is a dynamic micro-expression image sequence composed of continuous frames of facial micro-expression images, wherein the facial micro-expression images are three-channel RGB images.

[0125] Step 402, the dynamic micro-expression sequence data is divided into a training set and a test set, and the training set and the test set are preprocessed. The preprocessing operation specifically includes: converting each frame of each sample in the data set into a three-channel RGB image with a size of 112×112 pixels by linear interpolation; extracting frames for each sample in the data set, and extracting 16 frames of micro-expression images for each sample, and the 16 frames of images are evenly distributed at the same original position interval in the sample sequence; finally, performing random edge cropping and left-right mirror flipping on each sample of the extracted image sequence, and the images in the same sequence need to perform the same operation.

[0126] Step 403, construct a dynamic micro-expression recognition model based on frame attention and dynamic and static combination modules.

[0127] Step 404 , deep learning is performed to obtain a dynamic micro-expression recognition model that can assign more weights to key frames and integrate dynamic and static information of micro-expressions.

[0128] The steps of dynamic micro-expression recognition include:

[0129] Step 405: Acquire a sequence of micro-expression images to be recognized and perform pre-processing.

[0130] Step 406 , load the trained dynamic micro-expression recognition model, input the pre-processed micro-expression image sequence, and obtain the dynamic micro-expression detection result.

[0131] like Fig. 9 As shown, in another exemplary embodiment, a dynamic micro-expression recognition method based on frame attention and dynamic and static combination modules includes the following steps:

[0132] Step 501, recording micro-expression changes through a camera, and extracting the micro-expression video frame by frame to form an image sequence.

[0133] Step 502: pre-process the acquired image sequence.

[0134] In some embodiments, step 502 specifically includes: first, for each frame in the image sequence obtained in step 501, converting it into a three-channel RGB image with a size of 112×112 pixels by linear interpolation; and extracting 16 frames of micro-expression images from the converted image sequence, and the positions of the 16 frames of micro-expression images in the original sequence are the same and evenly distributed; finally, performing random edge cropping and left-right mirror flipping on the extracted 16-frame micro-expression image sequence, and each image in the sequence needs to perform the same operation.

[0135] In some embodiments, the converted image sequence is subjected to frame extraction to extract 16 frames of micro-expression images, specifically comprising: evenly dividing the converted image sequence into 16 parts according to the time dimension, extracting the first frame in each part, and forming an extracted image sequence.

[0136] Step 503, select the dynamic micro-expression data set SAMM, train the neural network model, and obtain a dynamic micro-expression recognition model.

[0137] In some embodiments, seven types of micro-expression samples are selected from the SAMM dynamic micro-expression data set, namely, anger, surprise, sadness, fear, happiness, disgust, and contempt. The same preprocessing operation as in the previous embodiment is performed on the image sequence of each selected sample.

[0138] Step 504: Recognize the acquired dynamic micro-expression sequence using a dynamic micro-expression recognition model.

[0139] Step 505: output the recognition result of the dynamic micro-expression recognition model.

[0140] Step 504 and step 505 are to convert the dynamic micro-expression image sequence processed in step 502 into RGB three-channel image data, and input it into the dynamic micro-expression recognition model through data loading operation, and finally obtain the recognition result output by the model.

[0141] The present application proposes a dynamic micro-expression recognition method based on frame attention and dynamic-static combination module, which combines the frame attention module and the dynamic-static combination module to build a neural network model, and realizes the recognition of dynamic micro-expressions through the model. The frame attention module is based on the idea of ​​the channel attention mechanism. It calculates the frame attention weight while calculating the channel attention weight, and multiplies the two to form a frame attention module, which improves the model's recognition ability for dynamic micro-expression key frames. The dynamic-static combination module adds dynamic information between feature maps based on the processing of static feature maps, making the model more sensitive to the changing details of dynamic micro-expressions.

[0142] This application realizes the extraction of dynamic micro-expression features based on the designed frame attention module, and realizes the combination of dynamic micro-expressions in static images and dynamic change information based on the dynamic-static combination module, and combines the two for time series processing, and finally realizes the recognition of dynamic micro-expressions.

[0143] To further demonstrate the effectiveness and practical application prospects of this application, the dynamic micro-expression recognition accuracy of the ResNet model in the ablation experiment is compared, as shown in Table 1.

[0144] Table 1. Dynamic micro-expression recognition accuracy

[0145] Model structure Correct rate (%) ResNet+ timing processing 60.5 Frame attention module + timing processing 62.4 Dynamic and static combination module + timing processing 65.5 Frame attention module + dynamic and static combination module + timing processing 66.0

[0146] It can be seen from Table 1 that both the frame attention module and the dynamic and static combination module can improve the recognition accuracy of the model for dynamic micro-expressions. Among them, only using the frame attention module and timing processing improves the accuracy by 1.9% compared with ResNet and timing processing; only using the dynamic and static combination module and timing processing improves the accuracy by 5% compared with ResNet and timing processing; using the frame attention module and the dynamic and static combination module at the same time and through timing processing improves the accuracy by 5.5% compared with ResNet and timing processing.

[0147] In addition, to further demonstrate the accuracy of the method of the present application for various types of dynamic micro-expressions, the UF1 scores of dynamic micro-expression recognition in the ablation experiment of the ResNet model were compared, as shown in Table 2.

[0148] Table 2. UF1 scores for dynamic micro-expression recognition

[0149] Model structure UF1 score ResNet+ timing processing 0.258 Frame attention module + timing processing 0.225 Dynamic and static combination module + timing processing 0.285 Frame attention module + dynamic and static combination module + timing processing 0.286

[0150] It can be seen that the UF1 score of using only the frame attention module and timing processing is lower than that of ResNet and timing processing. Combined with the results of the previous increase in the accuracy of the frame attention module, it can be found that the frame attention module has an improvement effect on micro-expressions with large expression changes in key frames, but the recognition performance improvement effect on expression types with unclear key frame changes is not obvious, thus reducing the average recognition performance. The dynamic and static combined module has a good effect on the recognition accuracy of each dynamic micro-expression, and the UF1 score is improved by 0.025 compared to ResNet. Finally, the model structure based on the frame attention module and the dynamic and static combined module of the present application has relatively better performance than a single frame attention module or a dynamic and static combined module, and the UF1 score reaches 0.286.

[0151] The present application also provides an application scenario, which applies the above-mentioned dynamic micro-expression recognition method based on frame weights. Specifically: the dynamic micro-expression recognition method based on frame weights provided in this embodiment can be applied in dynamic micro-expression recognition scenarios. The dynamic micro-expression recognition scenario includes a dynamic micro-expression image shooting link, a dynamic micro-expression image transmission link, and a dynamic micro-expression recognition link; the dynamic micro-expression image shooting link captures a dynamic micro-expression image, and the captured dynamic micro-expression image enters the dynamic micro-expression recognition link through the dynamic micro-expression image transmission link. The dynamic micro-expression recognition method based on frame weights provided in this embodiment belongs to the dynamic micro-expression recognition link. Specifically in the dynamic micro-expression recognition link, the acquired micro-expression image sequence to be recognized can be input into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result.

[0152] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store dynamic micro-expression image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a dynamic micro-expression recognition method based on frame weights is implemented.

[0153] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0154] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0155] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0157] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0158] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A dynamic micro-expression recognition method based on frame weights, characterized in that: The dynamic micro-expression recognition method comprises: Obtain a sequence of micro-expression images to be recognized; Input the micro-expression image sequence to be recognized into a trained dynamic micro-expression recognition model to obtain a micro-expression recognition result; wherein the dynamic micro-expression recognition model includes an initial convolution module, a frame attention module, a dynamic and static combination module, and a timing processing module; The frame attention module is used to calculate a feature map with channel attention weights and frame attention weights according to the feature map output by the initial convolution module; The dynamic and static combination module is used for: Process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain static features; Calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features; Processing the static features and the dynamic features to obtain a feature map including the static features and the dynamic features; The timing processing module is used to process the feature graph containing static features and dynamic features output by the dynamic-static combination module to obtain the micro-expression recognition result.

2. The frame weight-based dynamic micro-expression recognition method according to claim 1, characterized in that: The frame attention module includes one frame attention unit or two frame attention units; when the frame attention module includes two frame attention units, the input of one frame attention unit is the output of the initial convolution module, the output of one frame attention unit is the input of another frame attention unit, and the output of the other frame attention unit is the input of the dynamic-static combination module; The frame attention unit includes a first convolution subunit, a channel attention weight calculation subunit, a frame attention weight calculation subunit, a comprehensive calculation subunit and an activation output subunit; The first convolution subunit is used to process the input feature map to obtain a first feature map; The channel attention weight calculation subunit is used to process the first feature map to obtain a channel attention weight; The frame attention weight calculation subunit is used to process the first feature map to obtain a frame attention weight; The comprehensive calculation subunit is used to perform a product operation on the channel attention weight and the frame attention weight to obtain a first product, and to perform a product operation on the first product and the first feature map to obtain a second feature map; The activation output subunit is used to perform activation processing on the second feature map to obtain the feature map with channel attention weights and frame attention weights, and output the feature map with channel attention weights and frame attention weights to the dynamic and static combination module.

3. The frame weight-based dynamic micro-expression recognition method according to claim 2, characterized in that: The channel attention weight calculation subunit includes: a third convolutional layer, a fourth convolutional layer, a second activation layer, a fifth convolutional layer and a third activation layer; wherein the input of the third convolutional layer is the output of the first convolutional subunit; the input of the fourth convolutional layer is the output of the third convolutional layer; the input of the second activation layer is the output of the fourth convolutional layer; the input of the fifth convolutional layer is the output of the second activation layer; the input of the third activation layer is the output of the fifth convolutional layer; The frame attention weight calculation subunit includes: a sixth convolutional layer, a seventh convolutional layer, a fourth activation layer, an eighth convolutional layer and a fifth activation layer; wherein the input of the sixth convolutional layer is the output of the first convolutional subunit; the input of the seventh convolutional layer is the output of the sixth convolutional layer; the input of the fourth activation layer is the output of the seventh convolutional layer; the input of the eighth convolutional layer is the output of the fourth activation layer; and the input of the fifth activation layer is the output of the eighth convolutional layer.

4. The frame weight-based dynamic micro-expression recognition method according to claim 3, characterized in that: When the frame attention module includes two frame attention units, in one frame attention unit: The third convolution layer has 128 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 128×14×14, the convolution kernel moving step is 1, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 128 groups; The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no hole convolution is set; The second activation layer adopts the ReLU function; The fifth convolutional layer has 128 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 8×1×1, the edge filling size is 0, and no hole convolution is set; The third activation layer adopts Sigmoid function; The sixth convolutional layer has 2048 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 2048×14×14, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 16 groups; The seventh convolutional layer has 128 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 2048×1×1, the edge filling size is 0, and no hole convolution is set; The fourth activation layer adopts the ReLU function; The eighth convolutional layer has 2048 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 128×1×1, the edge filling size is 0, and no hole convolution is set; The fifth activation layer adopts the Sigmoid function.

5. The frame weight-based dynamic micro-expression recognition method according to claim 4, characterized in that: When the frame attention module includes two frame attention units, in the other frame attention unit: The first convolution layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 128×3×3, the convolution kernel movement step is 2, the edge filling size is 1, and no hole convolution is set; The first activation layer adopts the ReLU function; The second convolution layer has 256 convolution kernels, the convolution size is 3×3, the size of a single convolution kernel is 256×3×3, the convolution kernel movement step is 1, the edge filling size is 1, and no hole convolution is set; The third convolution layer has 256 convolution kernels, the convolution size is 14×14, the size of a single convolution kernel is 256×14×14, the convolution kernel moving step is 1, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 256 groups; The fourth convolutional layer has 8 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no hole convolution is set; The second activation layer adopts the ReLU function; The fifth convolutional layer has 256 convolution kernels, the convolution kernel size is 1×1, the size of a single convolution kernel is 16×1×1, the edge filling size is 0, and no hole convolution is set; The third activation layer adopts Sigmoid function; The sixth convolution layer has 4096 convolution kernels, the convolution kernel size is 14×14, the size of a single convolution kernel is 4096×14×14, the edge filling size is 0, no hole convolution is set, and the convolution grouping is 16 groups; The seventh convolutional layer has 256 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 4096×1×1, the edge filling size is 0, and no hole convolution is set; The fourth activation layer adopts the ReLU function; The eighth convolutional layer has 4096 convolution kernels, the convolution size is 1×1, the size of a single convolution kernel is 256×1×1, the edge filling size is 0, and no hole convolution is set; The fifth activation layer adopts the Sigmoid function.

6. The frame weight-based dynamic micro-expression recognition method according to claim 1, characterized in that: The dynamic-static combination module includes a static feature extraction unit, a dynamic feature extraction unit and a combination unit; The static feature extraction unit is used to process the feature map with channel attention weights and frame attention weights corresponding to the current frame to obtain static features; The dynamic feature extraction unit is used to calculate the difference between the feature map with channel attention weights and frame attention weights corresponding to the current frame and the feature map with channel attention weights and frame attention weights corresponding to the previous frame, and process the difference to obtain dynamic features; The combining unit is used to process the static features and the dynamic features to obtain a feature map including the static features and the dynamic features.

7. The frame weight-based dynamic micro-expression recognition method according to claim 6, characterized in that: The static feature extraction unit includes one Transformer layer or two Transformer layers with the same structure; when the static feature extraction unit includes two Transformer layers with the same structure, the input of one Transformer layer is the output of the frame attention module, the output of one Transformer layer is the input of another Transformer layer, and the output of the other Transformer layer is the input of the combination unit.

8. The frame weight-based dynamic micro-expression recognition method according to claim 6, characterized in that: The dynamic feature extraction unit includes a difference calculation layer, a convolution layer and an activation layer; wherein the input of the difference calculation layer is the output of the frame attention module; the output of the difference calculation layer is the input of the convolution layer; the output of the convolution layer is the input of the activation layer; the output of the activation layer is the input of the combination unit; The difference calculation layer is used to calculate the difference between the feature map with channel attention weight and frame attention weight corresponding to the current frame and the feature map with channel attention weight and frame attention weight corresponding to the previous frame; The convolution layer is used to perform a convolution operation on the difference to obtain features after the convolution operation; The activation layer is used to perform an activation operation on the features after the convolution operation to obtain the dynamic features.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the frame weight-based dynamic micro-expression recognition method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the frame weight-based dynamic micro-expression recognition method described in any one of claims 1 to 8 is implemented.

Images