Micro-expression recognition method and device, computer equipment and storage medium

Abstract

The invention discloses a micro-expression recognition method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method is applied to micro-expression recognition of a financial system or an old-age health medical system. According to the method, firstly, through face alignment and standardization processing, the face posture difference between different individuals is effectively eliminated; secondly, an attention mechanism based on a heat map is introduced in the spatial feature extraction process, key modeling is conducted on micro-expression sensitive areas such as eyebrows and eyes and mouth corners, and the self-adaptive focusing capacity for local muscle movement is improved; in the aspect of time modeling, a time sequence encoder is combined with a self-adaptive gating mechanism, micro-expression key frames are dynamically screened, redundant frame interference is avoided, and the instantaneous capturing capacity of micro-expression weak dynamic changes is improved; and finally, through fusing information of two dimensions of space and time, forming a spatio-temporal joint feature with high discrimination, and combining with a pre-training classification model to realize high-precision identification of micro-expressions.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a micro-expression recognition method and device, computer equipment and a storage medium. BACKGROUND

[0002] Micro-expression is a short (usually lasting 1 / 25 to 1 / 5 seconds) and weak facial muscle movement of a human being when trying to hide true emotions, which has important application value in the fields of psychological assessment, security screening and clinical diagnosis, such as micro-expression recognition in the financial field or the field of health care for the elderly. Traditional micro-expression detection methods mainly rely on manually designed features (such as LBP, optical flow) and rule-based classifiers, but it is difficult to capture the spatiotemporal dynamic characteristics of micro-expression.

[0003] In recent years, the performance of micro-expression detection methods based on deep learning has been significantly improved, but still faces many challenges, including the spatial feature limitation that the convolutional neural network equally processes facial regions and cannot adaptively focus on the spatial features of micro-expression related local muscle movements, the time modeling deficiency that the recurrent neural network and 3D convolution cannot accurately capture the instantaneity of micro-expression, and the low feature discriminability problem that the existing methods are easily disturbed by individual differences and illumination changes due to the low intensity of micro-expression.

[0004] Therefore, how to overcome the limitations of existing deep learning methods and improve the adaptive focusing ability and instantaneity capturing ability of micro-expression detection on local muscle movements has become a technical problem to be solved. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a micro-expression recognition method and device, computer equipment and a storage medium, and to solve the technical problem of how to improve the adaptive focusing ability and instantaneity capturing ability of micro-expression detection on local muscle movements.

[0006] To solve the above technical problem, the embodiments of the present application provide a micro-expression recognition method, which adopts the following technical solution:

[0007] A micro-expression recognition method, comprising:

[0008] obtaining an input video sequence, performing face alignment processing on the video sequence to generate standardized face image data;

[0009] performing spatial feature extraction on the standardized face image data to obtain initial spatial features;

[0010] For the initial spatial features, a preset attention mechanism enhancement mechanism is used to perform feature enhancement processing on the local spatial features related to micro-expression to obtain enhanced spatial features;

[0011] input the enhanced spatial feature into a preset temporal coding network to obtain a temporal key frame of the video sequence;

[0012] extract a temporal feature from the temporal key frame, and perform feature fusion on the enhanced spatial feature and the temporal feature to generate a micro-expression spatiotemporal feature;

[0013] input the micro-expression spatiotemporal feature into a pre-trained micro-expression classification model to obtain a micro-expression classification result.

[0014] To solve the above technical problems, the embodiment of the application further provides a micro-expression recognition device, which adopts the technical scheme as follows:

[0015] A micro-expression recognition device comprises:

[0016] a face alignment processing module configured to obtain an input video sequence, perform face alignment processing on the video sequence, and generate standardized face image data;

[0017] a spatial feature extraction module configured to perform spatial feature extraction on the standardized face image data to obtain initial spatial features;

[0018] a spatial feature enhancement module configured to perform feature enhancement processing on local spatial features related to micro-expression by using a preset attention mechanism enhancement mechanism for the initial spatial features to obtain enhanced spatial features;

[0019] a key frame identification module configured to input the enhanced spatial features into a preset temporal coding network to obtain a temporal key frame of the video sequence;

[0020] a spatiotemporal feature fusion module configured to extract a temporal feature from the temporal key frame, and perform feature fusion on the enhanced spatial feature and the temporal feature to generate a micro-expression spatiotemporal feature;

[0021] a micro-expression classification module configured to input the micro-expression spatiotemporal feature into a pre-trained micro-expression classification model to obtain a micro-expression classification result.

[0022] To solve the above technical problems, the embodiment of the application further provides a computer device, which adopts the technical scheme as follows:

[0023] A computer device comprises a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the micro-expression recognition method according to any one of the above.

[0024] To solve the above technical problems, the embodiment of the application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0025] A computer readable storage medium, the computer readable storage medium has stored thereon computer readable instructions, the computer readable instructions are executed by a processor to implement the steps of the micro-expression recognition method in any one of the above.

[0026] Compared with the prior art, the embodiments of the application have the following beneficial effects:

[0027] The application discloses a micro-expression recognition method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The micro-expression recognition method and device are applied to a financial system or an old-age health and medical system. The application first effectively eliminates facial posture differences between different individuals through face alignment and standardization processing; subsequently, an attention mechanism based on a heat map is introduced in a spatial feature extraction process, and a micro-expression sensitive area such as an eyebrow, an eye or a corner of a mouth is modeled in a key manner, so that the adaptive focusing ability for local muscle movement is improved; in terms of time modeling, a time sequence encoder is combined with an adaptive gating mechanism to dynamically filter micro-expression key frames, so that the interference of redundant frames is avoided, and the instantaneous capture ability for weak dynamic changes of micro-expression is improved; finally, information in two dimensions of space and time is fused to form a highly discriminative space-time joint feature, and a pre-training classification model is combined to realize high-precision micro-expression recognition. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the scheme in the application, the drawings needed in the description of the embodiments of the application will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0029] Figure 1 An exemplary system architecture diagram to which the application can be applied is shown;

[0030] Figure 2 A flowchart of one embodiment of the micro-expression recognition method according to the application is shown;

[0031] Figure 3 A structural schematic diagram of one embodiment of the micro-expression recognition device according to the application is shown;

[0032] Figure 4 A structural schematic diagram of one embodiment of the computer equipment according to the application is shown. DETAILED DESCRIPTION

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terms used in the specification are intended to describe the particular embodiments and are not intended to limit the application; the terms "include" and "have" and their any variations used in the specification and the claims and the above description of drawings are intended to cover the non-exclusive inclusion; the terms "first", "second" and the like used in the specification and the claims and the above description of drawings are intended to distinguish different objects, not to describe a particular order.

[0034] Reference herein to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive or alternative embodiments. It is expressly understood that the embodiments described herein are combinable with each other.

[0035] In order to make the person skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings below.

[0036] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102 and a server 103, and the terminal device 101 can be a notebook computer 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0037] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0038] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing, in addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer and a desktop computer, etc.

[0039] The server 103 can be a server providing various services, for example, a background server providing support for the page displayed on the terminal device 101.

[0040] It should be noted that the micro-expression recognition method provided by the embodiments of the present application is generally executed by a server / terminal device, and correspondingly, the micro-expression recognition apparatus is generally arranged in a server / terminal device.

[0041] It should be understood that, Figure 1 The number of terminal devices, networks and servers in the system is only illustrative, and the system can have any number of terminal devices, networks and servers according to the needs of implementation.

[0042] With reference to Figure 2 , a flow chart of one embodiment of the micro-expression recognition method according to the present application is shown. The micro-expression recognition method includes the following steps:

[0043] S201, obtaining an input video sequence, performing face alignment processing on the video sequence to generate standardized face image data;

[0044] Specifically, first, image frame extraction is performed from the input video sequence at a fixed frame rate (for example, 15 frames per second or 30 frames per second) to ensure coverage of the key process of expression change. Then, a high-precision face detection algorithm such as RetinaFace is used to detect the face region in the image frame by frame. In order to achieve consistency between the video sequences, further standardized processing is performed through a face alignment algorithm, usually based on a five-point or sixty-eight-point landmark algorithm for affine transformation, so that the face image of each frame is consistent in pose, scale and direction. Finally, the standardized face image data with a uniform size of 112x112 pixels is output.

[0045] RetinaFace is a single-stage face detection algorithm based on deep learning, which combines the advantages of SSD (Single Shot Multibox Detector) and DenseBox, and realizes high-precision and real-time face detection through multi-task learning and feature pyramid network (FPN), especially good at processing small faces and face detection under occlusion.

[0046] Landmark algorithm (represented by ERT level regression tree) is an efficient face key point detection technology, which constructs a cascade model by integrating multiple regression trees, and gradually approaches the real key point position from the initial average shape by using local features such as pixel difference to regress the residual of face shape. This algorithm combines the high-precision feature selection ability of gradient boosting tree and the fast convergence characteristics of level regression, maintains real-time while having strong robustness to complex scenes such as occlusion and pose changes, and is widely used in face alignment, expression recognition, liveness detection and 3D pose estimation fields. The pre-trained model provided by the open source library Dlib further reduces the application threshold.

[0047] This processing step greatly reduces the interference caused by the difference in head posture between individuals. Standardized face data not only improves the generalization ability of the model, but also enables the model to focus more on the subtle muscle changes in the micro-expression region.

[0048] S202, spatial feature extraction is performed on the standardized face image data to obtain initial spatial features;

[0049] Specifically, the standardized face image is input into a convolutional neural network (CNN) with ResNet-50 as the backbone to perform multi-level spatial feature extraction. The network extracts local texture, edge and structure information of the face image layer by layer, and finally outputs a high-dimensional feature map (such as 7x7x2048 dimensions), which contains texture change information in different regions of the face, especially in the eyes, eyebrows, corners of the mouth and other micro-expression related regions. The initial spatial features retain the static representation of facial muscle movement. During the extraction process, batch normalization and ReLU activation operations are used to improve the stability of the feature distribution and retain rich semantic information in the channel dimension of the feature map, forming a multi-scale representation of spatial attention mechanism.

[0050] In addition, maintaining high-resolution local region features helps to enhance the model's ability to perceive subtle muscle movement changes.

[0051] S203, for the initial spatial features, a preset attention mechanism enhancement mechanism is used to enhance the local spatial features related to micro-expression, and enhanced spatial features are obtained;

[0052] Specifically, the system introduces an attention mechanism enhancement mechanism, that is, an improved SE (Squeeze-and-Excitation) module combining regional gating and channel attention, to enhance the initial spatial features. First, the heat map generated by the landmark is used to highlight the key areas of the face, such as eyebrows, eye corners, nose wings, and mouth corners, which are active areas of micro-expression. These heat maps are embedded into the attention module to guide the attention to focus on sensitive areas, thereby suppressing the information interference of background and irrelevant areas. In the channel dimension, the Squeeze operation is used to aggregate global spatial information, and then the Excitation stage is used to adaptively adjust the response strength of each channel to improve the feature expression ability related to AU (Action Units). This biologically inspired spatial attention mechanism simulates the human visual focusing mechanism and realizes the enhanced capture of local details, which is particularly important for distinguishing low-intensity and short-time micro-expressions. The final output of the enhanced spatial features not only contains rich local semantics, but also takes into account the difference in the representation of the overall face region.

[0053] By combining the improved SE module of regional gating and channel attention, the initial spatial features are dynamically partitioned and filtered by introducing a regional gating mechanism, allowing only key region features to enter the subsequent processing flow. At the same time, the channel attention submodule is used to adaptively assign weights to each channel of the filtered feature map. The global average pooling is used to capture the inter-channel dependency, and the fully connected layer is used to generate the channel importance coefficients. Finally, the region selection and channel weighting results are multiplied to realize spatial-channel joint enhancement, effectively improving the focusing ability of the model on important feature regions and the efficiency of inter-channel information interaction.

[0054] S204, input the enhanced spatial features into a preset temporal encoding network to obtain the temporal key frames of the video sequence;

[0055] Specifically, the enhanced spatial feature sequence is input into a 12-layer stacked Transformer encoder to model the temporal dependency between frames. Transformer captures the global context information between each frame and other frames in the sequence through the self-attention mechanism (Self-Attention), thereby determining the importance of each frame in the development process of micro-expression. An adaptive temporal gating mechanism is also introduced in the system to replace the traditional fixed window or average pooling method. This mechanism can dynamically adjust the importance score of the frame according to the learned attention weight, thereby filtering out the key frames, that is, the frames containing the main changes or most representative micro-expressions.

[0056] These key frames can reflect the key turning points of emotions from neutral to changes, help to reduce the noise caused by redundant frames, and improve the discriminability of the model in time series modeling. The time series key frames extracted in this way are representative and retain the complete context logic.

[0057] S205, extracting time series features from the time series key frames, and performing feature fusion on the enhanced spatial features and the time series features to generate micro-expression spatio-temporal features;

[0058] Specifically, the system first uses a TSM (Temporal Shift Module) or a similar lightweight time series modeling network to further extract features from the key frame sequence, generating frame-level time series feature representations. This module achieves low-computational-cost modeling of temporal dynamics by transferring partial inter-frame information in the channel dimension. These time series features reflect the subtle movement trajectories of facial muscles over time, such as eye corner lifting, mouth corner depression, and other detailed dynamics. Subsequently, the enhanced spatial features and the extracted time series features are integrated through a feature fusion strategy, typically using concatenation, weighted addition, or attention-weighted fusion. The final spatio-temporal joint features not only retain the static representation of key regions, but also embed the dynamic information of micro-expression evolution over time. This fusion strategy can improve the separability of micro-expressions in multi-dimensional space, effectively addressing the shortcomings of single-dimensional features and improving the accuracy of micro-expression classification.

[0059] S206, inputting the micro-expression spatio-temporal features into a pre-trained micro-expression classification model to obtain a micro-expression classification result.

[0060] Specifically, the fused spatio-temporal features are input into a classification model pre-trained on large-scale micro-expression data. This model can be a deep neural network composed of multiple fully connected layers, or can further integrate classification modules such as Transformer-CLS head or SVM. The model outputs a class probability distribution, representing the micro-expression type (such as suppressed anger, brief surprise, etc.) corresponding to the input video. During training, a joint loss function is used for optimization, including cross-entropy loss for multi-class classification tasks and contrastive loss for enhancing the distinguishability between low-intensity micro-expressions. This training mechanism enables the model to learn to distinguish subtle differences between similar expressions while enhancing its robustness to abnormal samples. The output result not only includes the most likely expression category, but also provides a category confidence, which can be used for various decision support, such as AI face review emotion scoring, fraud warning, human-computer emotional feedback, and other practical scenarios.

[0061] For example, in the micro-expression recognition scenario of the financial system, when a customer conducts online insurance or claim video verification, the system first uses the RetinaFace algorithm to quickly locate the face area in each frame of image, and unifies and normalizes it to 112x112 pixel standard size through affine transformation. Subsequently, based on the 68-point key point detection model of Dlib, the micro-expression sensitive areas such as the eye corner and the mouth corner are accurately extracted, and a two-dimensional heat map is generated. The heat map serves as significant guiding information, which can dynamically enhance the feature weight of the nasolabial sulcus and the eyelid area, suppress irrelevant background response, and make the model focus more on the micro-expression clues such as the mouth corner and the blinking frequency when the customer lies. The finally output standardized face image not only maintains the spatiotemporal consistency of the biological features, but also enhances the discriminative features of the micro-expression through regional saliency marking, thereby providing high-credibility AI-assisted decision basis for the anti-fraud of the insurance industry, the authenticity verification of health information, and other businesses.

[0062] Further, the step of performing face alignment processing on the video sequence to generate standardized face image data specifically includes:

[0063] For each frame of image of the video sequence, a face detection technology is used to locate the face area;

[0064] The face area is subjected to size normalization processing to generate a standard size face image;

[0065] A heat map data corresponding to the standard size face image is generated through key point detection, wherein the heat map data is used to mark the micro-expression sensitive area;

[0066] The standard size face image is subjected to regional calibration according to the heat map data, and standardized face image data is output.

[0067] In this embodiment, for each frame of image of the video sequence, a face detection algorithm with high precision and robustness such as RetinaFace is first used to quickly and accurately locate the position of the face in the image. Then, the detected face area is subjected to size normalization processing, and an affine transformation is usually used to align the face image to a standard size of 112x112 pixels, so as to reduce the interference caused by different postures, scales or angles. Subsequently, a face key point detection method (such as the 68-point Dlib or deep learning model) is used to extract face key positions such as the eye corner, the eyebrow and the mouth corner, and a corresponding two-dimensional heat map is generated to highlight the micro-expression sensitive area. The heat map data serves as significant guiding information, which can be further used for regional calibration, that is, by enhancing or suppressing the response intensity of different regions in the standard face image, the model pays more attention to the local area containing rich micro-expression information. The finally output standardized face image data fuses the biological heuristic regional saliency marking on the basis of maintaining the uniform scale and alignment state.

[0068] Through the above steps, the positioning accuracy of the model on the key area of micro-expression and the feature expression ability can be improved, and a stable, standardized and biologically prior enhanced face image input for micro-expression recognition is provided.

[0069] Further, the step of generating heat map data corresponding to the standard size face image through key point detection specifically includes:

[0070] inputting the standard size face image into a preset key point detection model, determining face key points on the standard size face image through face key point detection;

[0071] determining a micro-expression sensitive area according to the distribution of the face key points, and marking the micro-expression sensitive area;

[0072] recognizing the position coordinates of the micro-expression sensitive area to obtain sensitive area coordinates;

[0073] initializing a preset heat map matrix, and assigning values to elements corresponding to positions of the heat map matrix according to the sensitive area coordinates and a preset distribution function to obtain heat map data.

[0074] In the embodiment, first, the standardized face image is input into a preset face key point detection model. The model can be a deep convolutional network based on regression, which has high precision face landmark positioning capability. The model output contains the coordinate information of 68 key points, covering the face contour, eyebrows, eyes, nose and mouth, etc. According to the geometric distribution of these key points, combined with the AU definition in FACS (Facial Action Coding System), the typical micro-expression sensitive areas are determined, such as the glabella, the corner of the eye, the sides of the nose and the corners of the mouth, etc. These areas usually accompany subtle muscle contractions when micro-expression occurs, and are the key to recognizing micro-expression. Then, according to the center coordinates of the sensitive area, the corresponding image position is extracted, and a two-dimensional heat map matrix with the same size as the face image is initialized. In the matrix, the values are assigned by a Gaussian distribution function or other kernel function with the sensitive area coordinates as the center, so that the values of the corresponding areas in the heat map show a pattern of local high response and gradual attenuation of other areas, and finally a region saliency map that can be perceived by the model is generated.

[0075] Through the above steps, explicit modeling of the salient region of micro-expression can be realized, the attention and discrimination ability of the model on the key region are improved, and the feature extraction effect is enhanced.

[0076] Further, for the initial spatial features, a preset attention mechanism enhancement mechanism is used to perform feature enhancement processing on the local spatial features related to micro-expression to obtain enhanced spatial features.

[0077] For the initial spatial features, the local area related to micro-expression is marked according to the heat map data, and the micro-expression sensitive area is obtained;

[0078] For the spatial features in the micro-expression sensitive area, the feature enhancement processing is performed by using the attention mechanism enhancement mechanism, and the initial enhanced features are obtained;

[0079] The region gating processing is performed on the heat map data, and the feature weight distribution in the micro-expression sensitive area is obtained;

[0080] The feature weighting is performed on the initial enhanced features using the feature weight distribution, and the enhanced spatial features are obtained.

[0081] In this embodiment, first, for the initial spatial feature map extracted by the backbone network (such as ResNet-50), the positions corresponding to the micro-expression hot area (such as the eye corner, the eyebrow, and the mouth corner) in the feature map are marked in combination with the generated heat map data, and a micro-expression sensitive area mask is formed. Next, for the feature information in these areas, a feature enhancement strategy based on the attention mechanism is used for processing, which specifically includes introducing an improved squeeze-excitation (SE) module or its variant combined with channel attention and spatial attention, dynamically adjusting the response strength of different features in the channel dimension, thereby improving the feature dimension highly related to micro-expression changes. Subsequently, according to the intensity distribution in the heat map, a region gating operation is performed: a weight coefficient is assigned to each spatial position, and these coefficients reflect the importance of the position to micro-expression expression. Finally, the initial enhanced feature map is weighted and fused position by position using the weight distribution, to suppress the features of invalid areas and amplify the features of sensitive areas, thereby obtaining an enhanced spatial feature map with stronger discrimination ability. This process realizes the selective amplification of the significant area, which helps to improve the feature discriminability and robustness in the downstream recognition task.

[0082] For example, the initial spatial features of 2048 video frame sequences F t are extracted by the backbone network ResNet-50, and the specific formula is as follows:

[0083]

[0084] In the formula, i and j take values of 1, 2, …, 2048, H is the feature weight of the i-th video frame, W is the feature weight of the j-th video frame, z t ∈R 2048 represents the channel feature vector after spatial compression.

[0085] Then, the channel attention weight is calculated by the bottleneck structure:

[0086] a t =σ(W2δ(W1z t ))

[0087] wherein W1∈R 256×2048 is the first dimension reduction matrix (compression ratio r=8), is the second dimension reduction matrix, and W1 and W2 are learnable weights; δ is a ReLU activation function, and σ is a Sigmoid function for normalizing the weights to [0, 1].

[0088] The region gating mechanism converts the landmark heat map M t into channel weights m t , which is implemented as follows:

[0089] m t = Conv 3×3 (AvgPool(M t ))∈R 2048

[0090] wherein Conv 3×3 represents a 3x3 convolution kernel, and AvgPool(M t ) represents an average pooling operation on the heat map M t .

[0091] The final spatial attention weights are as follows:

[0092]

[0093] wherein ⊙ represents a dot product operation. This design automatically increases the channel weights of the eye region by 30%-50% when detecting frowning, weights the feature tensor, and obtains the weighted result as follows:

[0094]

[0095] Through the above steps, the network is effectively guided to focus on the active area of micro-expression, and the semantic accuracy and spatial sensitivity of feature expression are improved.

[0096] Further, the step of performing region gating processing on the heat map data to obtain the feature weight distribution in the micro-expression sensitive region includes:

[0097] determining a gating function, wherein the gating function is a Sigmoid function and a Softmax function;

[0098] inputting the heat map data into the selected gating function;

[0099] obtaining a preset weight threshold, and comparing the weight values of each element in the heat map data with the weight threshold;

[0100] If the element weight value is lower than the weight threshold value, the element weight value is updated to 0 based on a gating function;

[0101] After the weight values of all elements are updated, a feature weight distribution map is obtained, and a feature weight distribution in the micro-expression sensitive region is obtained.

[0102] In the embodiment, first, a gating function suitable for feature weight normalization and noise response suppression is selected, such as a Sigmoid function for smooth activation or a Softmax function for normalization weighting, to enhance the importance difference of different regions. The heat map data generated in the previous step is input into the selected gating function, so that the response values of the micro-expression sensitive region form a continuous probability distribution between 0 and 1. Subsequently, a pre-defined weight threshold value (for example, 0.5 according to the average value of the heat map intensity or experience) is set, and the weights of each pixel or region in the heat map are compared one by one. When the weight of a certain region is lower than the threshold value, it means that its significance in micro-expression recognition is low, and at this time the corresponding value is updated to 0 to achieve the effect of region suppression; while the region whose weight value is higher than the threshold value retains the original value or enhances the value after the region gating processing, so as to highlight its contribution to the micro-expression expression. This region gating processing can effectively suppress the interference of non-target regions and retain the high-response micro-expression key region. After the update is completed, a new feature weight distribution map is output, which provides prior guidance for the spatial attention mechanism, so that the model can focus more on the region features with high discriminability.

[0103] Through the above steps, dynamic screening and response enhancement of the facial region can be realized, and the attention degree of the model to the micro-expression key region and the feature utilization efficiency are improved.

[0104] Further, the enhanced spatial features are input into a preset temporal coding network to obtain the temporal key frames of the video sequence, specifically including:

[0105] The enhanced spatial features are arranged in time sequence to form a spatial feature sequence;

[0106] The spatial feature sequence is input into a preset temporal coding network to model the inter-frame dependency of the video sequence;

[0107] According to the inter-frame dependency, a preset adaptive temporal gating mechanism is used to perform time weight distribution on the video sequence;

[0108] The video sequence after completing the time weight distribution is subjected to key frame screening to obtain time key frames related to micro-expression changes, and the temporal key frames are obtained.

[0109] In this embodiment, the spatial attention enhanced feature map is first arranged in time sequence of video frames to form a complete spatial feature sequence, fully retaining the spatial information of each frame and its time sequence. Then, the sequence is input into a preset timing encoding network, usually adopting a multi-layer Transformer encoder structure, to model the complex temporal dependence between frames by using its powerful self-attention mechanism, effectively capturing the short-term dynamic changes in the micro-expression occurrence process. The timing encoding network outputs information containing inter-frame dependence weights, which reflects the importance of features at different time points to the overall emotion recognition. Then, combined with the adaptive time gating mechanism, dynamic weight values are given to each frame according to the time weights learned by the model, and low-weight frames are removed by threshold screening, retaining key frames that significantly contribute to micro-expression changes.

[0110] For example, the spatial feature sequence is processed by a Transformer encoder (T is the number of enhanced spatial features in the spatial feature sequence), a feature projection needs to be performed as follows:

[0111]

[0112] In the formula, Z t is the feature projection result, W p ∈R 512×100352 , Flatten represents the flattening operation, represents flattening the spatial feature sequence into a 512-dimensional vector, p t is a learnable position encoding.

[0113] Then, multi-head attention calculation is performed (taking the hth head as an example), and the specific calculation is as follows:

[0114]

[0115] In the formula, head h is the attention weight of the hth head, d h = 64, which represents the dimension of the hth head, are the query transformation matrix, key transformation matrix, value transformation matrix, and weight of the Z transformation matrix, respectively.

[0116] Finally, the key frames are selected based on the learnable vector v∈R 512 through the time gating mechanism:

[0117] g t = σ(v T Z t )

[0118] In the formula, g t is the timing key frame, and v Tis a learnable vector corresponding to a certain enhanced spatial feature, and σ is an activation function of the time gating. Only g t Frames with g > 0.5 are involved in the subsequent calculation, and this method can reduce 60% of redundant frame processing.

[0119] Through the above steps, fine modeling of the time dependence of video frames and dynamic screening of key frames are achieved, and the time sensitivity and accuracy of micro-expression recognition are enhanced.

[0120] Further, the step of inputting the micro-expression spatiotemporal feature into the pre-trained micro-expression classification model to obtain a micro-expression classification result specifically includes:

[0121] The micro-expression spatiotemporal feature is input into the pre-trained micro-expression classification model, wherein the micro-expression classification model includes a time sequence offset layer, a classification layer, and an optimization layer;

[0122] In the time sequence offset layer, the micro-expression spatiotemporal feature is subjected to time sequence offset processing to obtain a spatiotemporal offset feature;

[0123] In the classification layer, the spatiotemporal offset feature is classified to output an initial predicted category of the micro-expression;

[0124] In the optimization layer, the initial predicted category is jointly optimized by using a preset contrastive loss function and a cross-entropy loss to output a micro-expression classification result.

[0125] In this embodiment, first, the micro-expression spatiotemporal feature fused with spatial and temporal information is input into the pre-trained micro-expression classification model, and the model structure includes a time sequence offset layer, a classification layer, and an optimization layer. The time sequence offset layer uses a time sequence offset module (TSM) to dynamically adjust the input feature in the time dimension, moves the feature information between adjacent time frames, enhances the model's ability to capture subtle time sequence changes of micro-expressions, and improves the continuity and expressiveness of the time sequence feature. Subsequently, the classification layer uses a fully connected layer or a lightweight neural network to classify the spatiotemporal offset feature into multiple categories of micro-expressions and outputs the predicted probability distribution of each category. The optimization layer combines the contrastive loss function and the cross-entropy loss to jointly optimize the discriminability of the classifier and the compactness of the samples in the feature space. The contrastive loss promotes the aggregation of features of the same category and the separation of features of different categories, effectively improving the discrimination of low-intensity micro-expressions; the cross-entropy loss ensures the accuracy and stability of the classification results. Through end-to-end training, the model can adaptively adjust the parameters to efficiently decode and accurately classify complex micro-expression spatiotemporal features.

[0126] The standard cross-entropy loss F ce The expression is as follows:

[0127]

[0128] In the formula, C is the number of micro-expression categories (such as happy, angry, etc.), y c represents the initial prediction result of a certain type of micro-expression, p c represents the micro-expression prediction probability of the corresponding category.

[0129] Contrast loss F c The expression is as follows:

[0130]

[0131] In the formula, f i f j are features of the same category sample, f i f k are features of different category samples, and m is a boundary parameter for controlling the intra-class or inter-class distance, m = 0.2.

[0132] Through the above steps, the accuracy and robustness of micro-expression classification are significantly improved, especially in processing low-intensity and short-duration micro-expressions.

[0133] In the above embodiment, the present application discloses a micro-expression recognition method, which belongs to the field of artificial intelligence technology. The micro-expression recognition is applied to the financial system or the old-age health and medical system. The present application first effectively eliminates the facial posture differences between different individuals through face alignment and standardization processing; then in the spatial feature extraction process, the attention mechanism based on heat map is introduced, and the sensitive areas of micro-expression such as eyebrows, eyes and corners of the mouth are modeled, the adaptive focusing ability to local muscle movement is improved; in the time modeling aspect, the time sequence encoder combined with the adaptive gating mechanism is adopted, the key frames of micro-expression are dynamically selected, the interference of redundant frames is avoided, and the instantaneous capture ability to weak dynamic changes of micro-expression is improved; finally, through the fusion of information in space and time dimensions, the spatio-temporal joint features with high discriminability are formed, and the high-precision recognition of micro-expression is realized combined with the pre-training classification model.

[0134] In the present embodiment, the electronic device (for example Figure 1 The server shown in the figure) on which the micro-expression recognition method runs can receive instructions or obtain data through wired connection or wireless connection. It should be noted that the above-mentioned wireless connection mode can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.

[0135] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned face image information, the above-mentioned face image information can also be stored in a node of a block chain.

[0136] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The blockchain is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information of a batch of network transactions, used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0137] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0138] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0139] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. The program can include the processes of the above-mentioned embodiments when executed, wherein the storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0140] It should be understood that although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0141] Further reference is made to Figure 3As an implementation of the method shown in the above Figure 2 The present application provides an embodiment of a micro-expression recognition device, which corresponds to the method embodiment shown in the above Figure 2 The device can be applied to various electronic devices.

[0142] As shown in the above Figure 3 The micro-expression recognition device 300 comprises:

[0143] A face alignment processing module 301 is configured to obtain an input video sequence, perform face alignment processing on the video sequence, and generate standardized face image data.

[0144] A spatial feature extraction module 302 is configured to perform spatial feature extraction on the standardized face image data to obtain initial spatial features.

[0145] A spatial feature enhancement module 303 is configured to perform feature enhancement processing on local spatial features related to micro-expression by using a preset attention mechanism enhancement mechanism for the initial spatial features to obtain enhanced spatial features.

[0146] A key frame identification module 304 is configured to input the enhanced spatial features into a preset temporal coding network to obtain temporal key frames of the video sequence.

[0147] A space-time feature fusion module 305 is configured to extract temporal features from the temporal key frames and perform feature fusion on the enhanced spatial features and the temporal features to generate micro-expression space-time features.

[0148] A micro-expression classification module 306 is configured to input the micro-expression space-time features into a pre-trained micro-expression classification model to obtain a micro-expression classification result.

[0149] Further, the face alignment processing module 301 specifically comprises:

[0150] A face detection unit is configured to locate a face region by using a face detection technology for each frame of image of the video sequence.

[0151] A size normalization unit is configured to perform size normalization processing on the face region to generate a standard size face image.

[0152] A key point detection unit is configured to generate heat map data corresponding to the standard size face image by key point detection, wherein the heat map data is used to mark a micro-expression sensitive region.

[0153] A region calibration unit is configured to perform region calibration on the standard size face image according to the heat map data to output standardized face image data.

[0154] Further, the key point detection unit specifically comprises:

[0155] A key point detection subunit is configured to input a standard size face image into a preset key point detection model, and determine face key points on the standard size face image through face key point detection;

[0156] A key point distribution subunit is configured to determine a micro-expression sensitive area according to the distribution of the face key points, and mark the micro-expression sensitive area;

[0157] A sensitive area coordinate subunit is configured to identify the position coordinates of the micro-expression sensitive area, and obtain sensitive area coordinates;

[0158] A heat map matrix processing subunit is configured to initialize a preset heat map matrix, assign values to elements at corresponding positions of the heat map matrix according to the sensitive area coordinates and a preset distribution function, and obtain heat map data.

[0159] Further, the spatial feature enhancement module 303 specifically includes:

[0160] A sensitive area marking unit is configured to mark a local area related to micro-expression according to the heat map data for the initial spatial feature, and obtain a micro-expression sensitive area;

[0161] A feature enhancement processing unit is configured to perform feature enhancement processing on the spatial feature in the micro-expression sensitive area by using an attention mechanism enhancement mechanism, and obtain initial enhanced features;

[0162] A region gate processing unit is configured to perform region gate processing on the heat map data, and obtain a feature weight distribution in the micro-expression sensitive area;

[0163] A feature weighting unit is configured to perform feature weighting on the initial enhanced features by using the feature weight distribution, and obtain enhanced spatial features.

[0164] Further, the region gate processing unit specifically includes:

[0165] A gate function determination subunit is configured to determine a gate function, wherein the gate function is a Sigmoid function and a Softmax function;

[0166] A heat map data input subunit is configured to input the heat map data as an input to the selected gate function;

[0167] A weight comparison subunit is configured to obtain a preset weight threshold, and compare the weight values of the elements in the heat map data with the weight threshold;

[0168] A weight value updating subunit is configured to, if the element weight value is lower than the weight threshold, update the element weight value to 0 based on the gate function;

[0169] The weight distribution map subunit is configured to obtain a feature weight distribution map in the micro-expression sensitive area after the weight values of all elements are updated.

[0170] Further, the key frame identification module 304 specifically comprises:

[0171] The time sequence arrangement unit is configured to arrange the enhanced spatial features in time sequence to form a spatial feature sequence.

[0172] The time sequence coding unit is configured to input the spatial feature sequence into a preset time sequence coding network to model an inter-frame dependency of the video sequence.

[0173] The time weight distribution unit is configured to use a preset adaptive time gating mechanism to perform time weight distribution on the video sequence according to the inter-frame dependency.

[0174] The key frame screening unit is configured to perform key frame screening on the video sequence after the time weight distribution to obtain a time key frame related to the micro-expression change, thereby obtaining the time key frame.

[0175] Further, the micro-expression classification module specifically comprises:

[0176] The space-time feature input unit is configured to input the micro-expression space-time feature into a pre-trained micro-expression classification model, wherein the micro-expression classification model comprises a time sequence offset layer, a classification layer and an optimization layer.

[0177] The time sequence offset unit is configured to perform time sequence offset processing on the micro-expression space-time feature in the time sequence offset layer to obtain a space-time offset feature.

[0178] The micro-expression classification unit is configured to perform classification on the space-time offset feature in the classification layer to output an initial prediction category of the micro-expression.

[0179] The joint optimization unit is configured to perform joint optimization on the initial prediction category in the optimization layer by using a preset contrast loss function and a cross-entropy loss of the model, thereby outputting a micro-expression classification result.

[0180] In the above embodiment, the application discloses a micro-expression recognition device, which belongs to the technical field of artificial intelligence. The micro-expression recognition device is applied to a financial system or an old-age health and medical system. The application first eliminates the facial posture differences between different individuals through face alignment and standardization processing; then in the spatial feature extraction process, an attention mechanism based on a heat map is introduced to model the sensitive areas of micro-expression such as eyebrows, eyes and corners of the mouth, thereby improving the adaptive focusing ability on local muscle movement; in the time modeling aspect, a time sequence encoder combined with an adaptive gating mechanism is adopted to dynamically select micro-expression key frames, avoid redundant frame interference, and improve the instantaneous capture ability of weak dynamic changes of micro-expression; finally, by fusing the information in the spatial and time dimensions, a highly discriminative spatio-temporal joint feature is formed, and a pre-trained classification model is combined to realize high-precision micro-expression recognition.

[0181] To solve the above technical problems, the application embodiment further provides a computer device. For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the embodiment is shown in the figure.

[0182] The computer device 4 includes a memory 41, a processor 42 and a network interface 43 which are connected to each other through a system bus. It should be pointed out that only the computer device 4 with the memory 41, the processor 42 and the network interface 43 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, which hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0183] The computer device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad or a voice control device, etc.

[0184] The memory 41 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed on the computer device 4, such as computer readable instructions of the micro-expression recognition method, etc. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.

[0185] The processor 42 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer readable instructions or process data stored in the memory 41, such as computer readable instructions of the micro-expression recognition method.

[0186] The network interface 43 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0187] The present application also provides an embodiment, i.e., to provide a computer device, which includes a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to implement the steps of the micro-expression recognition method as described above, i.e., to implement:

[0188] A micro-expression recognition method includes:

[0189] Obtaining an input video sequence, performing face alignment processing on the video sequence to generate standardized face image data;

[0190] extracting spatial features from the standardized face image data to obtain initial spatial features;

[0191] performing feature enhancement processing on local spatial features related to micro-expression by using a preset attention mechanism enhancement mechanism for the initial spatial features to obtain enhanced spatial features;

[0192] inputting the enhanced spatial features into a preset temporal coding network to obtain temporal key frames of the video sequence;

[0193] extracting temporal features from the temporal key frames, and performing feature fusion on the enhanced spatial features and the temporal features to generate micro-expression spatio-temporal features;

[0194] inputting the micro-expression spatio-temporal features into a pre-trained micro-expression classification model to obtain a micro-expression classification result.

[0195] The application also provides another embodiment, namely providing a computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions can be executed by at least one processor, so that the at least one processor executes the steps of the micro-expression recognition method as described above, namely realizes:

[0196] A micro-expression recognition method comprises:

[0197] obtaining an input video sequence, performing face alignment processing on the video sequence to generate standardized face image data;

[0198] extracting spatial features from the standardized face image data to obtain initial spatial features;

[0199] performing feature enhancement processing on local spatial features related to micro-expression by using a preset attention mechanism enhancement mechanism for the initial spatial features to obtain enhanced spatial features;

[0200] inputting the enhanced spatial features into a preset temporal coding network to obtain temporal key frames of the video sequence;

[0201] extracting temporal features from the temporal key frames, and performing feature fusion on the enhanced spatial features and the temporal features to generate micro-expression spatio-temporal features;

[0202] inputting the micro-expression spatio-temporal features into a pre-trained micro-expression classification model to obtain a micro-expression classification result.

[0203] Those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device) execute the methods described in various embodiments of the present application.

[0204] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0205] It should be noted that the non-company software tools or components appearing in various embodiments of the present application are only illustrative and do not represent actual use.

[0206] Obviously, the above-described embodiments are only a part of the embodiments of the present application, and are not all the embodiments. The preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be realized in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art

[0207] can still modify the technical solutions described in the foregoing specific embodiments, or equivalently replace some of the technical features. Any equivalent structure made by using the contents of the present application specification and drawings, directly or indirectly, is within the scope of the present application.

[0208]

[0209]

[0210] ​​The related technical fields thereof are all within the patent protection scope of the present application.

Claims

1. A micro-expression recognition method, characterized in that, include: The input video sequence is acquired, and face alignment processing is performed on the video sequence to generate standardized face image data; Spatial features are extracted from the standardized face image data to obtain initial spatial features; Based on the initial spatial features, a preset attention mechanism is used to enhance the local spatial features related to micro-expressions, resulting in enhanced spatial features. The enhanced spatial features are input into a preset temporal coding network to obtain the temporal keyframes of the video sequence; Temporal features are extracted from the temporal keyframes, and the enhanced spatial features and temporal features are fused to generate micro-expression spatiotemporal features; The spatiotemporal features of the micro-expressions are input into a pre-trained micro-expression classification model to obtain the micro-expression classification results.

2. The micro-expression recognition method as described in claim 1, characterized in that, The step of performing face alignment processing on the video sequence to generate standardized face image data specifically includes: For each frame of the video sequence, face detection technology is used to locate the face region; The face region is normalized to generate a standard-sized face image; Heatmap data corresponding to the standard-sized face image is generated by key point detection, wherein the heatmap data is used to mark the micro-expression sensitive areas; Based on the heatmap data, the standard-sized face image is calibrated regionally, and the standardized face image data is output.

3. The micro-expression recognition method as described in claim 2, characterized in that, The step of generating heatmap data corresponding to the standard-sized face image through key point detection specifically includes: The standard-sized face image is input into a preset key point detection model, and the face key points are determined on the standard-sized face image through face key point detection. Based on the distribution of the facial key points, the micro-expression sensitive area is determined and marked; Identify the location coordinates of the micro-expression sensitive area to obtain the coordinates of the sensitive area; Initialize a preset heatmap matrix, and assign values ​​to the elements at corresponding positions of the heatmap matrix according to the coordinates of the sensitive region and a preset distribution function to obtain the heatmap data.

4. The micro-expression recognition method as described in claim 3, characterized in that, The step of using a preset attention mechanism to enhance local spatial features related to micro-expressions, based on the initial spatial features, to obtain enhanced spatial features specifically includes: Based on the initial spatial features, the local regions related to micro-expressions are marked according to the heatmap data to obtain the micro-expression sensitive regions; For the spatial features within the micro-expression sensitive area, the attention mechanism enhancement mechanism is used to perform feature enhancement processing to obtain initial enhanced features; The heatmap data is subjected to region gating processing to obtain the feature weight distribution within the micro-expression sensitive area; The initial enhanced features are weighted using the feature weight distribution to obtain the enhanced spatial features.

5. The micro-expression recognition method as described in claim 4, characterized in that, The step of performing region gating processing on the heatmap data to obtain the feature weight distribution within the micro-expression sensitive region specifically includes: Determine the gating functions, wherein the gating functions are the Sigmoid function and the Softmax function; The heatmap data is used as input and passed to the selected gating function; Obtain a preset weight threshold, and compare the weight value of each element in the heatmap data with the weight threshold; If the element weight value is lower than the weight threshold, the element weight value is updated to 0 based on the gating function; Once the weight values ​​of all elements have been updated, a feature weight distribution map is generated to obtain the feature weight distribution within the micro-expression sensitive area.

6. The micro-expression recognition method as described in claim 1, characterized in that, The step of inputting the enhanced spatial features into a preset temporal coding network to obtain the temporal keyframes of the video sequence specifically includes: The enhanced spatial features are arranged in chronological order to form a spatial feature sequence; The spatial feature sequence is input into a preset temporal coding network to model the inter-frame dependencies of the video sequence; Based on the inter-frame dependencies, a preset adaptive time gating mechanism is used to allocate time weights to the video sequence; The video sequence with completed time weighting is subjected to key frame filtering to obtain time key frames related to micro-expression changes, thus obtaining the temporal key frames.

7. The micro-expression recognition method as described in claim 1, characterized in that, The step of inputting the spatiotemporal features of micro-expressions into a pre-trained micro-expression classification model to obtain micro-expression classification results specifically includes: The micro-expression spatiotemporal features are input into a pre-trained micro-expression classification model, wherein the micro-expression classification model includes a temporal offset layer, a classification layer, and an optimization layer; In the temporal offset layer, the micro-expression spatiotemporal features are subjected to temporal offset processing to obtain spatiotemporal offset features; In the classification layer, the spatiotemporal offset features are classified, and the initial predicted category of micro-expressions is output. In the optimization layer, the initial predicted category is jointly optimized using the model's preset contrast loss function and cross-entropy loss, and the micro-expression classification result is output.

8. A micro-expression recognition device, characterized in that, include: The face alignment processing module is used to acquire the input video sequence, perform face alignment processing on the video sequence, and generate standardized face image data; The spatial feature extraction module is used to extract spatial features from the standardized face image data to obtain initial spatial features. The spatial feature enhancement module is used to perform feature enhancement processing on the local spatial features related to micro-expressions using a preset attention mechanism enhancement mechanism to obtain enhanced spatial features; A keyframe recognition module is used to input the enhanced spatial features into a preset temporal coding network to obtain temporal keyframes of the video sequence; The spatiotemporal feature fusion module is used to extract temporal features from the temporal keyframes and fuse the enhanced spatial features and the temporal features to generate micro-expression spatiotemporal features; The micro-expression classification module is used to input the spatiotemporal features of the micro-expressions into a pre-trained micro-expression classification model to obtain the micro-expression classification results.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the micro-expression recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the micro-expression recognition method as described in any one of claims 1 to 7.

Images