A micro-expression recognition method based on motion learning and image feature representation fusion
By deeply fusing motion and image features through a heterogeneous information fusion mechanism, the problem of low accuracy in micro-expression recognition in existing technologies has been solved, achieving a more efficient micro-expression recognition effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES)
- Filing Date
- 2024-06-11
- Publication Date
- 2026-05-22
AI Technical Summary
Existing deep learning models suffer from noise interference, lighting variations, and resolution limitations in micro-expression recognition, resulting in low recognition accuracy. Furthermore, existing methods have failed to effectively integrate motion information and RGB image features.
A heterogeneous information fusion mechanism is adopted to deeply fuse the features of the motion learning stream and the image feature representation stream. Multi-scale motion and fine-grained image features are extracted through the Transformer model and the multilayer perceptron module, and feature interaction and classification are performed using a multi-head cross attention layer.
It improves the accuracy and generalization ability of micro-expression recognition, and enhances the model's ability to describe facial features and expression states by deeply fusing motion and image features.
Smart Images
Figure CN118609186B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of facial expression recognition, and in particular relates to a micro-expression recognition method based on motion learning and image feature representation fusion. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Microexpressions are produced by subtle movements of facial muscles. These movements are extremely weak and last only 0.25 to 0.5 seconds. Microexpression recognition aims to analyze the true emotions expressed by people by recognizing microexpressions. It has wide applications in areas such as depression detection, lie detection, and human-computer interaction.
[0004] In recent years, the significant practical value of micro-expression analysis has garnered widespread attention. In the early stages of micro-expression recognition research, many scholars extracted micro-expression features based on traditional learning methods and performed micro-expression recognition, thus promoting the development of micro-expression analysis. These methods extracted micro-expression features based on local binary patterns, gradient histograms, and optical flow, and then input these features into a classifier for classification. However, traditional methods require manual feature design or selection of appropriate methods based on expert experience, which poses challenges to feature extraction and limits the generalization ability of the model.
[0005] With the development of deep learning, micro-expression recognition methods based on deep learning have attracted increasing attention from scholars. Compared with traditional learning methods, the performance of micro-expression recognition methods based on deep learning frameworks has been further improved. Deep learning-based micro-expression recognition methods automatically learn high-level feature representations through deep neural network models and classify images containing micro-expressions, improving model performance and generalization ability. These deep learning-based micro-expression recognition methods can be divided into three categories based on network input: methods utilizing motion information, methods utilizing RGB image information, and methods combining both. Methods utilizing motion information capture subtle muscle movements in micro-expression images by analyzing pixel changes between the start and top frames; the deep learning model learns muscle movement features to classify micro-expressions. RGB images contain facial texture information, facial muscle movement information, and personal identification information, providing a large number of learnable features for deep learning models. Existing methods combining both motion and RGB image information obtain a more comprehensive and richer feature representation, improving the model's learning and generalization ability.
[0006] However, existing deep learning models have not achieved good recognition accuracy, and the main problems are as follows:
[0007] (1) Since the collected micro-expression dataset contains background information, the motion information extracted by optical flow may contain some background information. Therefore, using only motion information for micro-expression recognition may introduce noise, and the lack of facial texture information of the subjects in the network leads to poor model performance;
[0008] (2) In the field of micro-expression recognition, accurately extracting the motion features of facial muscles is crucial. However, RGB images are easily affected by changes in lighting and resolution limitations, making it impossible for the model to accurately capture facial muscle movements and affecting model performance;
[0009] (3) Existing methods that combine motion information and RGB image information simply stack optical flow images and RGB images and use deep learning models to extract features for classification, or stack the optical flow features extracted by deep learning models and RGB image features and directly classify them. They do not deeply fuse the features of the two types of images, which limits the performance of the model. Summary of the Invention
[0010] To address at least one of the technical problems in the aforementioned background art, this invention provides a micro-expression recognition method based on motion learning and image feature representation fusion. This method utilizes a heterogeneous information fusion mechanism to achieve information interaction and classification information fusion between extracted facial muscle motion features and image features, which can effectively improve the accuracy of micro-expression recognition.
[0011] To achieve the above objectives, the present invention adopts the following technical solution:
[0012] A first aspect of the present invention provides a micro-expression recognition method based on motion learning and image feature representation fusion, comprising the following steps:
[0013] Acquire RGB images containing micro-expressions and extract facial muscle movement information from the RGB images;
[0014] Facial muscle movement information is combined with a motion learning stream to extract facial muscle movement features;
[0015] By combining RGB images with an image feature representation stream, image features are extracted.
[0016] Micro-expression recognition results are obtained based on facial muscle movement features, image features, and a constructed heterogeneous information fusion mechanism; wherein, the construction process of the heterogeneous information fusion mechanism includes:
[0017] The motion learning stream and the image feature representation stream have a one-to-one hierarchical correspondence. The facial muscle motion features and image features extracted from the previous level are converted into corresponding queries, keys, and values. After passing through a multi-head cross-attention layer, the query of the image feature representation is multiplied with the key of the motion feature to determine the attention weight. The interaction between the attention weight and the value of the motion feature is used to obtain the output of the attention head. The outputs of all attention heads are concatenated to obtain the cross-fused facial muscle motion features and image features. This is added to the facial muscle motion features and image features extracted from the previous level and used as the input for the next level. The facial muscle motion features and image features output from the last level of the motion learning stream and the image feature representation stream are combined for classification to obtain the micro-expression classification result.
[0018] Furthermore, the step of combining facial muscle movement information with a motion learning flow to extract facial muscle movement features includes:
[0019] The motion learning flow contains a multi-level structure. In each level, facial muscle motion information is used as input to a Transformer model. The Transformer model includes a multi-head self-attention module and a multi-layer perceptron module. The facial muscle motion information is processed by the multi-head self-attention module to extract local muscle motion information to obtain the first facial muscle motion feature. The first facial muscle motion feature and the facial muscle motion information are fused and used as input to the multi-layer perceptron module. The multi-layer perceptron module extracts the second facial muscle motion feature. The second facial muscle motion feature and the input of the multi-layer perceptron module are combined and aggregated by an aggregation layer to obtain the local and global facial semantic information of that level.
[0020] Furthermore, the step of combining the RGB image with an image feature representation stream to extract image features includes:
[0021] The image feature representation stream contains a multi-level structure. In each level, a convolution operation is performed on the RGB image to extract fine-grained features in the local region of the face. The fine-grained features are then combined with pooling layers to aggregate the fine-grained features learned from the convolution operation, thus obtaining the image features of that level.
[0022] Furthermore, the start frame and top frame images are obtained from the RGB image, and the horizontal optical flow, vertical optical flow and optical strain are extracted using the optical flow method. The horizontal optical flow, vertical optical flow and optical strain are superimposed in the spatial dimension as facial muscle motion information.
[0023] Furthermore, the method also includes preprocessing the RGB image and facial muscle motion information, including:
[0024] Four main coordinate points of the face were detected in the top frame image: the left eye, the right eye, the left corner of the mouth, and the right corner of the mouth. Based on these four coordinate points, the RGB image and facial muscle motion information were cropped to obtain the cropped RGB image and facial muscle motion information.
[0025] Furthermore, after obtaining the facial detail features, position embedding is added to the facial detail features to obtain facial detail features with spatial position information.
[0026] Furthermore, when the facial muscle motion features and image features from the output of the last layer of the motion learning stream and image feature representation stream are combined, the normalized scores of the facial muscle motion features and image features are added proportionally.
[0027] A second aspect of the present invention provides a micro-expression recognition system based on motion learning and image feature representation fusion, comprising:
[0028] The data acquisition module is used to acquire RGB images containing micro-expressions and extract facial muscle movement information from the RGB images;
[0029] The feature extraction module is used to combine facial muscle motion information with a motion learning stream to extract facial muscle motion features; and to combine RGB images with an image feature representation stream to extract image features.
[0030] A heterogeneous information fusion mechanism is used to obtain micro-expression recognition results based on facial muscle movement features, image features, and a constructed heterogeneous information fusion mechanism; wherein, the construction process of the heterogeneous information fusion includes:
[0031] The motion learning stream and the image feature representation stream have a one-to-one hierarchical correspondence. The facial muscle motion features and image features extracted from the previous level are converted into corresponding queries, keys, and values. After passing through a multi-head cross-attention layer, the query of the image feature representation is multiplied with the key of the motion feature to determine the attention weight. The interaction between the attention weight and the value of the motion feature is used to obtain the output of the attention head. The outputs of all attention heads are concatenated to obtain the cross-fused facial muscle motion features and image features. This is added to the facial muscle motion features and image features extracted from the previous level and used as the input for the next level. The facial muscle motion features and image features output from the last level of the motion learning stream and the image feature representation stream are combined for classification to obtain the micro-expression classification result.
[0032] A third aspect of the present invention provides a computer-readable storage medium.
[0033] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a micro-expression recognition method based on motion learning and image feature representation fusion as described above.
[0034] A fourth aspect of the present invention provides a computer device.
[0035] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a micro-expression recognition method based on motion learning and image feature representation fusion as described above.
[0036] Compared with the prior art, the beneficial effects of the present invention are:
[0037] 1. This invention deeply integrates motion information and RGB image information. The motion learning stream extracts multi-scale motion features from micro-expression motion information based on the Transformer layer and the aggregation layer. The top frame image learns multi-scale detail features of the subject's face through the image feature representation stream. By combining the motion learning stream and the image feature representation stream, the advantages of motion features and facial detail features are applied to the micro-expression recognition task. The two complement each other, enabling the algorithm to extract richer features and more comprehensively and in detail describe the subject's facial features and expression state.
[0038] 2. This invention fuses motion features and facial detail features at multiple scales through a heterogeneous information fusion mechanism for micro-expression classification. The cross-fusion module aims to integrate features extracted from two streams to generate a more comprehensive and richer representation, where image features and motion features influence and reinforce each other. The result integration strategy integrates the results from the two streams as the model's classification result, further integrating different feature information.
[0039] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0041] Figure 1 This is an overall framework diagram of the micro-expression recognition method based on motion learning and image feature representation fusion provided in the embodiments of the present invention;
[0042] Figure 2 This is the hierarchical structure of the motion learning flow provided in the embodiments of the present invention;
[0043] Figure 3 This is a schematic diagram of the cross-fusion module provided in an embodiment of the present invention. Detailed Implementation
[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0045] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0047] This invention proposes a micro-expression recognition fusion method based on motion learning and image feature representation, which is used to fuse and classify micro-expression motion information and RGB image information. The overall framework of the fusion method is as follows: Figure 1 As shown, the fusion network consists of a preprocessing module, a feature extraction module, and a heterogeneous information fusion mechanism.
[0048] like Figure 1 As shown in (a), in the preprocessing stage, the original image is cropped and motion information is extracted.
[0049] like Figure 1 As shown in (b), the feature extraction module consists of a motion learning stream and an image feature representation stream, which respectively learn facial muscle motion features and facial detail features.
[0050] Figure 1 The motion learning stream in (b) uses motion information between the start frame and the vertex frame as input. Figure 1 In the image feature representation stream of (b), one of the starting frame and the vertex frame is used as input. In this invention, the vertex frame is used as the input of the model in the image feature representation stream. The heterogeneous information fusion mechanism combines multi-scale motion features and multi-scale facial detail features through the cross-fusion module, and then the two classification results are fused by the result integration strategy.
[0051] Example 1
[0052] This embodiment provides a micro-expression recognition fusion method based on motion learning and image feature representation, including the following steps:
[0053] Step 1: Acquire RGB images containing micro-expressions and extract facial muscle movement information of the subject from the images to obtain horizontal optical flow, vertical optical flow and optical strain.
[0054] In this embodiment, the RGB images containing micro-expressions are obtained from a composite dataset (MEGC2019) consisting of three datasets: SAMM, CASME II, and SMIC. The start frame and top frame images of each subject are obtained from MEGC2019.
[0055] MEGC2019 included 68 participants, with 28 from SAMM, 24 from CASME II, and 16 from SMIC. MEGC2019 comprised a total of 442 samples from all three datasets, and all samples were categorized into three classes: positive, negative, and surprised.
[0056] Using the start frame image x for each subject in the MEGC2019 dataset onset and top frame image x apex Horizontal optical flow x is extracted using the optical flow method. u Vertical optical flow x v and optical strain x o These are stacked in spatial dimension and used as the subject's facial muscle movement information x m =Concat(x) u ,x v ,x o ).
[0057] Step 2: Preprocess the RGB image and facial muscle motion information;
[0058] The preprocessing of RGB images and facial muscle motion information includes face detection and face cropping.
[0059] In this embodiment, Multi-task Cascaded Convolutional Networks (MTCNN) are used to extract data from the top frame image x of each subject. apex Four main coordinate points on the face were detected: the left eye, the right eye, the left corner of the mouth, and the right corner of the mouth. Based on these four coordinate points, the motion information x... m and top frame image x apex Perform facial cropping to obtain the cropped motion information and RGB images
[0060] Step 3: Extract the cropped motion information The input motion learning stream is used to extract multi-scale facial muscle motion features using a Transformer model and aggregation layers, thus integrating motion information. The input motion learning flow employs a three-level Transformer model. i (·) and Aggregation layeri (·) Capture multi-scale muscle movement features
[0061] The specific calculation method is as follows:
[0062] The motion information input to the motion learning stream is represented as Where H and W are the length and width of the cropped image, respectively, and C is the number of channels in the image;
[0063] like Figure 2 As shown, in each hierarchical structure of the motion learning flow, a Transformer model with a local self-attention mechanism is used to represent local muscle motion information; an aggregation layer is used to extract local and global facial semantic information, while learning the interaction of features in the eye and lip regions; the input of the Transformer model in the i-th hierarchical structure is... When i = 1, 2, 3, the process of extracting features can be represented as:
[0064]
[0065] Among them, MHSA i (·), MLP i (·) represent the multi-head self-attention module and the multilayer perceptron module of the Transformer model in the i-th hierarchical structure, respectively. i (·) represents the Transformer model in the i-th level structure. This represents the output of the Transformer model in the i-th hierarchical structure. The formula does not include the layer normalization part.
[0066] The input to the aggregation layer in the i-th hierarchical structure is Features are aggregated using convolution and pooling operations, with the convolution kernel being... The pooling operation uses max pooling with a size of 3, a step of 1, and padding of 1. With a kernel size of 3, a step size of 2, and padding of 1, the aggregate layer can be represented as:
[0067]
[0068] in This is the output of the i-th level structure.
[0069] Step 4: Input the cropped RGB image into the image feature representation stream, and use a convolutional neural network to extract local and global facial details and semantic features from the RGB image.
[0070] The specific process is as follows: RGB image The facial muscle movement intensity is divided into four parts and fed into an image feature representation stream, which consists of three stacked convolutional and aggregation layers to extract facial texture and detail features from the subject. Specifically, the convolutional operations... Using only convolutional kernels of size 1 to extract features helps to extract fine-grained features from local regions of the face within a small area; pooling layers are employed. The features learned from the convolution operation are aggregated to facilitate the model's learning of coarse-grained facial features; the model is defined as:
[0071]
[0072] in, Let represent the convolution and pooling operations in the i-th hierarchical structure, respectively. This is the output of the i-th level structure.
[0073] Step 5: The features extracted from the motion learning stream and the image feature representation stream are combined using a heterogeneous information fusion mechanism to achieve information interaction and classification information fusion between motion features and image features, and micro-expressions are divided into three categories: "positive", "negative" and "surprised".
[0074] The specific method is as follows: The heterogeneous information fusion mechanism includes two key parts: the cross-fusion module aims to combine muscle motion features with image feature representations, and the result integration strategy aims to coordinate the classification results of the two streams.
[0075] like Figure 3 As shown, the cross-fusion module is added between each level of the motion learning stream and the image feature representation stream to integrate the features of the two streams and enhance the model's understanding of micro-expressions.
[0076] The cross-fusion module includes layer normalization, linear layers, multi-head cross-attention layers, and a multilayer perceptron. The input to the cross-fusion module consists of two parts: one part comes from the motion learning stream. The other part is the facial detail features of the image feature representation stream. Position embedding Added to facial details In order to help the model understand the spatial location information of each subject's face, the facial detail features after location embedding are as follows:
[0077]
[0078] The cross-fusion module first uses layer normalization and linear layers to... and The input features are transformed into corresponding queries, keys, and values. This transformation maps the input features to a higher-dimensional space, thus better capturing the relationships between different elements. Then, the multi-head cross-attention layer utilizes the cross-attention mechanism of z attention heads to focus on information from different subspaces, capturing richer feature information. In the j-th attention head of the multi-head cross-attention layer, the query represented by the image features is... Key to motion characteristics Multiply them to determine attention weights, representing the correlation between them. Then, use the attention weights and motion features... The values interact, and the output of the j-th attention head is calculated. The output of the multi-head cross-attention layer is obtained by concatenating the outputs of the z attention heads, i.e.:
[0079]
[0080] Where j = 1, 2, ..., z are the indices of the attention heads, and n = v and n = m refer to the operations of the image feature representation stream and the motion learning stream, respectively. It is a scaling value to prevent gradient explosion.
[0081] The outputs of the two multi-head cross-attention layers are fed into a multilayer perceptron to obtain the two outputs of the cross-fusion module. The model can be expressed mathematically as follows:
[0082]
[0083] Among them, MLP m (·), MLP v (·) denotes the multilayer perceptron for motion learning flow and image feature representation flow, respectively, and LN(·) denotes layer normalization; and This represents the output of the i-th cross-fusion module.
[0084]
[0085] in and They are used as inputs to the next level of the motion learning stream and the image feature representation stream, respectively.
[0086] The result integration strategy uses two classifiers to classify the features of the two streams respectively. Then, the normalized scores of the two streams are added proportionally to obtain the model's classification result. If one stream performs poorly in certain situations, the other stream can compensate for its shortcomings, thereby improving the overall classification performance. The model is expressed by the following mathematical formula:
[0087]
[0088] in, The final output of the classification model is represented by λ1 and λ2, which are hyperparameters. and The inputs are the motion learning stream and the image feature representation stream, respectively, for the multilayer perceptron.
[0089] Example 2
[0090] This embodiment provides a micro-expression recognition system based on the fusion of motion learning and image feature representation, including:
[0091] The data acquisition module is used to acquire RGB images containing micro-expressions and extract facial muscle movement information from the RGB images;
[0092] The feature extraction module is used to combine facial muscle motion information with a motion learning stream to extract facial muscle motion features; and to combine RGB images with an image feature representation stream to extract image features.
[0093] A heterogeneous information fusion mechanism is used to obtain micro-expression recognition results based on facial muscle movement features, image features, and a constructed heterogeneous information fusion model; wherein, the construction process of the heterogeneous information fusion includes:
[0094] The motion learning stream and the image feature representation stream have a one-to-one hierarchical correspondence. The facial muscle motion features and image features extracted from the previous level are converted into corresponding queries, keys, and values. After passing through a multi-head cross-attention layer, the query of the image feature representation is multiplied with the key of the motion feature to determine the attention weight. The interaction between the attention weight and the value of the motion feature is used to obtain the output of the attention head. The outputs of all attention heads are concatenated to obtain the cross-fused facial muscle motion features and image features. This is added to the facial muscle motion features and image features extracted from the previous level and used as the input for the next level. The facial muscle motion features and image features output from the last level of the motion learning stream and the image feature representation stream are combined for classification to obtain the micro-expression classification result.
[0095] Example 3
[0096] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the micro-expression recognition method based on motion learning and image feature representation fusion as described above.
[0097] Example 4
[0098] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the micro-expression recognition method based on motion learning and image feature representation fusion as described above.
[0099] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0100] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A micro-expression recognition method based on motion learning and image feature representation fusion, characterized in that, Includes the following steps: Acquire RGB images containing micro-expressions and extract facial muscle movement information from the RGB images; Facial muscle movement information is combined with a motion learning stream to extract facial muscle movement features; By combining RGB images with an image feature representation stream, image features are extracted. Micro-expression recognition results are obtained based on facial muscle movement features, image features, and a constructed heterogeneous information fusion mechanism; wherein, the construction process of the heterogeneous information fusion includes: The motion learning stream and the image feature representation stream have a one-to-one hierarchical correspondence. The facial muscle motion features and image features extracted from the previous level are converted into corresponding queries, keys, and values. After passing through a multi-head cross-attention layer, the query of the image feature representation is multiplied with the key of the motion feature to determine the attention weight. The interaction between the attention weight and the value of the motion feature is used to obtain the output of the attention head. The outputs of all attention heads are concatenated to obtain the cross-fused facial muscle motion features and image features. This is added to the facial muscle motion features and image features extracted from the previous level and used as the input for the next level. The facial muscle motion features and image features output from the last level of the motion learning stream and the image feature representation stream are combined for classification to obtain the micro-expression classification result.
2. The micro-expression recognition method based on motion learning and image feature representation fusion as described in claim 1, characterized in that, The step of combining facial muscle movement information with a motion learning stream to extract facial muscle movement features includes: The motion learning flow contains a multi-level structure. In each level, facial muscle motion information is used as input to a Transformer model. The Transformer model includes a multi-head self-attention module and a multi-layer perceptron module. The facial muscle motion information is processed by the multi-head self-attention module to extract local muscle motion information to obtain the first facial muscle motion feature. The first facial muscle motion feature and the facial muscle motion information are fused and used as input to the multi-layer perceptron module. The multi-layer perceptron module extracts the second facial muscle motion feature. The second facial muscle motion feature and the input of the multi-layer perceptron module are combined and aggregated by an aggregation layer to obtain the local and global facial semantic information of that level.
3. The micro-expression recognition method based on motion learning and image feature representation fusion as described in claim 1, characterized in that, The step of combining RGB images with an image feature representation stream to extract image features includes: The image feature representation stream contains a multi-level structure. In each level, a convolution operation is performed on the RGB image to extract fine-grained features in the local region of the face. The fine-grained features are then combined with pooling layers to aggregate the fine-grained features learned from the convolution operation, thus obtaining the image features of that level.
4. The micro-expression recognition method based on motion learning and image feature representation fusion as described in claim 1, characterized in that, The extraction of facial muscle movement information from RGB images includes: The start frame and top frame images are obtained from the RGB image. The horizontal optical flow, vertical optical flow and optical strain are extracted using the optical flow method. The horizontal optical flow, vertical optical flow and optical strain are superimposed in the spatial dimension as facial muscle motion information.
5. The micro-expression recognition method based on motion learning and image feature representation fusion as described in claim 1, characterized in that, The method further includes preprocessing the RGB image and facial muscle motion information, including: Four main coordinate points of the face were detected in the top frame image: the left eye, the right eye, the left corner of the mouth, and the right corner of the mouth. Based on these four coordinate points, the RGB image and facial muscle motion information were cropped to obtain the cropped RGB image and facial muscle motion information.
6. The micro-expression recognition method based on motion learning and image feature representation fusion as described in claim 1, characterized in that, After obtaining the facial detail features, position embedding is added to the facial detail features to obtain facial detail features with spatial position information.
7. The micro-expression recognition method based on motion learning and image feature representation fusion as described in claim 1, characterized in that, When the facial muscle motion features and image features from the last layer of the motion learning flow and image feature representation flow are combined, the normalized scores of the facial muscle motion features and image features are added proportionally.
8. A micro-expression recognition system based on motion learning and image feature representation fusion, characterized in that, include: The data acquisition module is used to acquire RGB images containing micro-expressions and extract facial muscle movement information from the RGB images; The feature extraction module is used to combine facial muscle motion information with a motion learning stream to extract facial muscle motion features; and to combine RGB images with an image feature representation stream to extract image features. A heterogeneous information fusion mechanism is used to obtain micro-expression recognition results based on facial muscle movement features, image features, and a constructed heterogeneous information fusion model; wherein, the construction process of the heterogeneous information fusion includes: The motion learning stream and the image feature representation stream have a one-to-one hierarchical correspondence. The facial muscle motion features and image features extracted from the previous level are converted into corresponding queries, keys, and values. After passing through a multi-head cross-attention layer, the query of the image feature representation is multiplied with the key of the motion feature to determine the attention weight. The interaction between the attention weight and the value of the motion feature is used to obtain the output of the attention head. The outputs of all attention heads are concatenated to obtain the cross-fused facial muscle motion features and image features. This is added to the facial muscle motion features and image features extracted from the previous level and used as the input for the next level. The facial muscle motion features and image features output from the last level of the motion learning stream and the image feature representation stream are combined for classification to obtain the micro-expression classification result.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the micro-expression recognition method based on motion learning and image feature representation fusion as described in any one of claims 1-7.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the micro-expression recognition method based on motion learning and image feature representation fusion as described in any one of claims 1-7.