A micro-expression recognition method and system with multi-granularity cognition enhancement of facial features

Abstract

The present application relates to a kind of facial feature multi-granularity cognitive enhancement micro-expression recognition method and system, belong to emotional computing technical field.The method includes data acquisition, multidimensional preprocessing, multi-granularity global feature extraction, multi-granularity local feature extraction, multi-granularity cognitive enhancement and feature classification.Data preprocessing is carried out using multidimensional preprocessing method.Coarse / fine-grained feature vectors are obtained using multi-granularity global / local feature extraction method, and transfer learning is introduced when extracting multiple local features.In the multi-granularity cognitive enhancement process, the global feature vector of the face is extracted through the channel attention module, and then it is weighted to the local feature vector, aiming to use the coarse-grained global feature of the face to guide the fine-grained feature of the local feature.The final feature vector obtained is used as the input component of the classification model for model training to obtain the classification model.The system is realized based on the above method, which can monitor and feedback the emotional changes of students in class in real time.

Description

Technical Field

[0001] This invention belongs to the field of affective computing technology and relates to a micro-expression recognition method and system with multi-granularity cognitive enhancement of facial features. Background Technology

[0002] In today's society, emotions are a crucial component of communication and understanding between people, and one of the most widely applied biological characteristics. Microexpressions, as the outward manifestation of emotions, are classified to elucidate the true expression of human emotions. They have wide applications in education, medical diagnosis, and crime prevention, thus consistently attracting research interest.

[0003] In the field of education, students are core participants and beneficiaries, playing a crucial role in the classroom. Their emotional changes during class can reflect their level of knowledge absorption, an indicator that has attracted widespread attention in academia. However, current classroom teaching suffers from typical problems such as a lack of communication and interaction between teachers and students, leading to decreased student concentration and teachers' difficulty in understanding students' needs and difficulties. Therefore, utilizing micro-expressions to reflect emotional characteristics, it is imperative to construct a real-time, accurate, stable, and reliable emotion classification system based on micro-expressions in the classroom. This would allow teachers to perceive students' emotional changes, provide appropriate guidance, establish positive communication channels, and enable students to actively participate in class, genuinely experience the appeal of the course, and resonate with their teachers. Based on the above background, this invention proposes a micro-expression recognition method enhanced by multi-granularity facial feature cognition and constructs an efficient, precise, and robust real-time micro-expression detection system based on this method.

[0004] Micro-expression recognition technology faces two main challenges: the abundance of redundant information in the entire image that is difficult to filter out, and extremely subtle facial muscle movements. To address this redundancy, mainstream face alignment techniques using Active Shape Models (ASM) and Active Appearance Models (AAM) can align facial images to a standardized position and angle, eliminating redundant information caused by changes in head posture. In addition, extracting regions of interest (ROIs) containing only essential information from a facial image is another common approach. Common ROIs include the eyes, eyebrows, and mouth. Some researchers have proposed masking the eyes and cheeks in each image to eliminate noise caused by blinking and static areas; however, this method also filters out useful motion information around the eyes.

[0005] The most commonly used technique for capturing subtle facial muscle movements is Eulerian Video Magnification (EVM), which amplifies the dynamic content of two consecutive frames in a video, making it an important tool for enhancing the intensity of micro-expressions. However, while amplifying subtle muscle movements, EVM also introduces noise and artifacts, interfering with the extraction of the true signal. Secondly, capturing subtle facial muscle movements through local feature analysis in the spatial domain, such as local texture, shape, and gradients, is also widely used. However, these methods focus too much on fine-grained features, neglecting coarse-grained global features, resulting in incomplete feature extraction. Furthermore, analyzing facial images in the frequency domain, such as applying Discrete Wavelet Transform (DWT), can also analyze and extract subtle changes. Among the methods for obtaining the direction of muscle movement, optical flow can capture minute facial muscle movements by calculating the motion vectors between pixels in adjacent images. These methods only address one pain point in micro-expression detection. However, there is little mention of how to combine coarse and fine facial features for feature extraction, amplifying minute facial muscle movements while reducing redundant information in the image. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a micro-expression recognition method and system with multi-granularity cognitive enhancement of facial features. Besides using histogram equalization to focus on subtle facial muscle movements in classroom teaching, the most important aspect of this invention is the introduction of a multi-stream recognition network based on multi-granularity cognitive computation and incorporating an attention mechanism in the multi-granularity feature extraction stage. This allows for the extraction of facial features from global to local levels. Furthermore, to improve the generalization performance of the method, transfer learning is incorporated into the multi-granularity feature extraction process. Finally, this invention proposes a micro-expression recognition system with multi-granularity cognitive enhancement of facial features. This system uses an RGB camera to capture and analyze student faces in online classrooms, achieving enhancement of facial features from coarse to fine granular, thereby improving the accuracy and robustness of student emotion detection in the classroom.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A micro-expression recognition method with multi-granularity facial feature cognitive enhancement is disclosed. The method includes a data acquisition module, a multi-dimensional preprocessing EAOFHE (Euler Action Amplification-Optical Flow-Histogram Equalization) module, a multi-granularity global feature extraction module, a multi-granularity local feature extraction module, a multi-granularity cognitive enhancement module, and a feature classification module. Specifically, the multi-dimensional EAOFHE preprocessing method extracts the start frame and vertex frame from a video frame, uses Euler video to enlarge the two frames, then uses histogram equalization to highlight texture features, and finally performs TV-L1 optical flow processing to obtain the preprocessed image. Multi-granularity global feature extraction and multi-granularity local feature extraction methods are used to obtain feature vectors, and transfer learning is introduced when extracting multi-granularity local features. In the multi-granularity cognitive enhancement process, a channel attention module extracts the global facial feature vector, which is then weighted and added to the local feature vector, aiming to use the coarse-grained global facial features to guide the fine-grained local features. The final feature vector obtained from the aforementioned process is used as the input component of a classification model for model training to obtain the classification model.

[0008] Furthermore, during the preprocessing stage, when extracting the start frame and vertex frame from the video frames, face alignment is performed as follows:

[0009] in These represent the scaling factor, rotation matrix, and translation vector, respectively. , These refer to two different face image matrices, where each row of the matrix represents a feature point. Coordinates. Specifically, represent The first of the matrix OK, represent The first of the matrix OK.

[0010] Furthermore, when performing image magnification during preprocessing, the following formula is used:

[0011] The input video frame sequence is: ,in Represents spatial coordinates, Indicates time, It is a frequency domain representation. and Is it frequency? and Components in direction, Indicates in Take the direction The value, Indicates in Take the direction The value of .

[0012] Furthermore, by representing the phase and amplitude information in the image in the frequency domain, changes in phase information amplify the subtle movements of micro-expressions. The phase change formula for each frequency component is calculated as follows:

[0013] in It is the phase of the frequency component.

[0014] Furthermore, in the preprocessing stage, the formula for highlighting texture features in face images through histogram equalization is as follows:

[0015]

[0016]

[0017] in This indicates that the grayscale value of a pixel in the image is... The probability density function, It is the cumulative distribution function. It is the first after histogram equalization A cumulative probability density function, Indicates the total number of pixels. It refers to grayscale levels; To balance pixel values, It is a grayscale level. Indicates an integer rounded to the nearest whole number.

[0018] Furthermore, in the coarse-grained feature extraction process, the preprocessed optical flow map is processed using an attention mechanism to obtain the global feature matrix. , The average value of each channel is obtained based on the global feature matrix. :

[0019] in It is a length of The vector, This refers to the number of channels. and These refer to the height and width of the feature map, respectively.

[0020] Then As input, obtain the two channel weights. , :

[0021] in, , , , These are the weights and biases of the two fully connected layers, respectively. This represents the activation function. It is a scaling factor used to reduce computational complexity.

[0022] Finally, the obtained channel weights are multiplied by the input feature map to obtain a feature map recalibrated by the spatial attention mechanism. This recalibrated feature map is then fed into the Facial Block for feature extraction and mapped to a feature vector through a fully connected layer. .

[0023] Furthermore, during the fine-grained feature extraction process, when extracting eye features, the optical flow map around the eye is extracted from the preprocessed optical flow map and used as input to the Base Layer for preliminary feature extraction. Finally, it is fed into the Eyes Block for eye feature extraction, and after passing through a fully connected layer, it is mapped to an eye feature vector. .

[0024] Furthermore, during the fine-grained feature extraction process, when extracting mouth features, the captured optical flow map is fed into a Base Layer learned from the eye's basic extraction network for basic extraction. MouthBlock is then used for further fine-grained feature extraction, and after passing through a fully connected layer, it is mapped to a mouth feature vector. .

[0025] Furthermore, in the process of multi-granularity feature enhancement, As a guiding vector, it is weighted onto two linear layers, namely and And concatenate them together to form a feature vector , As input components for the final classification, where The calculation method is as follows:

[0026] in Indicates weighted average. and Represents parameters in the network. and This indicates the bias amount.

[0027] Furthermore, in the feature classification process, The micro-expression categories are classified using the input of the final fully connected layer. The constructed model is then trained, the parameters in the model are updated, and finally, a classification model is obtained.

[0028] This invention also proposes a micro-expression recognition system with multi-granularity facial feature recognition enhancement, the system including a teacher terminal, a student terminal, a model call, and a database.

[0029] Furthermore, the student client includes a client login module, an RGB image acquisition module, a micro-expression recognition module, and a data storage module; the client login module is used for student login; the RGB image acquisition module acquires student facial images through an RGB camera; the micro-expression recognition module internally carries a classification model trained based on a micro-expression recognition method with multi-granularity cognitive enhancement of facial features; the data storage module stores the detected results in a database; Furthermore, the teacher client includes a client login module, a lecture module, and a data viewing module; teachers log in through the client login module; teachers conduct classroom teaching through the lecture module and can view students' emotions in real time through the lecture module; the data viewing module allows teachers to read the continuous changes in students' classroom emotions.

[0030] The beneficial effects of this invention are as follows: (1) This invention proposes a micro-expression recognition method with multi-granularity cognitive enhancement of facial features by combining attention mechanism. This method draws inspiration from the process of human people distinguishing micro-expressions, adopts the "global priority" strategy, and constructs multi-granular representation of facial micro-expression features from coarse-grained to fine-grained strategy. It uses coarse-grained information of the face to guide fine-grained information around the eyes and mouth, thereby increasing the accuracy of micro-expression recognition.

[0031] (2) Compared with traditional micro-expression detection methods, the method of the present invention uses histogram equalization on the face in the preprocessing stage, which makes the brightness differences in the image more obvious, enhances the texture features of facial muscles, highlights useful information in the image, and plays an auxiliary role in the subsequent feature extraction network. This method improves the robustness of the system while increasing accuracy.

[0032] (3) This invention utilizes the similarity between the mouth and eye features of micro-expressions, using them as the source and target tasks respectively, and introduces transfer learning to improve the versatility of this method.

[0033] (4) The system of the present invention can monitor and provide feedback on students’ classroom mood changes in real time, and can store the classroom status in the database for viewing after class.

[0034] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is an overall structural diagram of a micro-expression recognition method with multi-granularity cognitive enhancement of facial features according to the present invention; Figure 2 This is a schematic diagram illustrating how the method utilizes multidimensional preprocessing EAOFHE to enhance facial details.

[0036] Figure 3 This is a framework diagram for the system application. Detailed Implementation

[0037] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0038] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0039] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0040] Please see Figures 1-3 This is a micro-expression recognition method and system that enhances facial feature recognition with multi-granularity cognition.

[0041] Example 1 The micro-expression recognition method based on multi-granularity cognitive enhancement of facial features proposed in this embodiment is mainly implemented through processes such as data acquisition, multi-dimensional preprocessing EAOFHE, multi-granularity global feature extraction, multi-granularity local feature extraction, multi-granularity cognitive enhancement, and feature classification. It is implemented based on the data acquisition module, the multi-dimensional preprocessing module EAOFHE, the multi-granularity global feature extraction module, the multi-granularity local feature extraction module, the multi-granularity cognitive enhancement module, and the feature classification module, respectively.

[0042] Based on the human cognitive principle of "global priority", this invention uses spatial attention mechanism to extract coarse-grained global information about the face. The extracted coarse-grained information is then used to effectively enhance fine-grained features in local areas such as the eyes and mouth, thereby achieving multi-granular cognitive enhancement of subtle facial micro-expression features and improving the ability to judge expression features.

[0043] Furthermore, in the data acquisition process, the present invention acquires a video dataset of people containing micro-expression classification through a data acquisition module, segments the video dataset, extracts each frame in the video, divides it into appropriate training datasets, validation datasets and test datasets, uses a face detection toolkit to locate and crop the face position, retains only the frames containing the face, and finally performs face alignment operation.

[0044] Specifically, the data acquisition module uses images from the video frame containing micro-expressions to the final frame as the dataset. Preferably, these are videos or images recorded under natural or indoor conditions using professional equipment, exhibiting clear facial emotional features and accurate classification. In this embodiment, 70% of the dataset is used as training data, 15% as validation data, and 15% as test data. Most video frames in the dataset contain background information and features below the neck. Since micro-expressions primarily appear on the face, this information is noise and needs to be masked.

[0045] Furthermore, in the data preprocessing process, this invention combines the advantages of Euler motion magnification and optical flow, and integrates histogram equalization to invent a new multidimensional image preprocessing method—the EAOFHE preprocessing method. This invention first segments each video and extracts each frame. Then, it uses the face detection toolkit (Dlib) to locate and crop the face position, retaining only the face image frames. Next, it extracts the start frame and vertex frame from the video frames, using Euler motion magnification to highlight the subtle movements of facial muscles in two consecutive frames. Then, it applies histogram equalization to the obtained images to highlight the texture features in the face images. Finally, it processes the equalized images using TV-L1 optical flow, incorporating total variation regularization to reduce the impact of image brightness variations and noise. Simultaneously, because TV-L1 optical flow has a relatively small penalty for gradient changes, it can effectively reduce errors at the boundaries of moving objects.

[0046] Specifically, histogram equalization is used to highlight the texture features in the face image, employing the following formula:

[0047]

[0048]

[0049] in This indicates that the grayscale value of a pixel in the image is... The probability density function, It is the cumulative distribution function. It is the first after histogram equalization A cumulative probability density function, Indicates the total number of pixels. It refers to grayscale levels; To balance pixel values, It is a gray level. This represents an integer rounded to the nearest whole number.

[0050] Furthermore, in the process of global facial feature extraction, the global facial feature extraction module crops the histogram-equalized facial image into... The image size is determined by a base layer that performs a preliminary extraction of the coarse-grained facial image, resulting in a tensor of depth 64 and size 28×28. The global feature matrix obtained after processing using an attention mechanism is... .

[0051]

[0052] in It is a length of The vector represents the average value of each channel. This refers to the number of channels. and These refer to the height and width of the feature map, respectively. Then, it is necessary to... As input, channel weights are learned through two fully connected layers.

[0053]

[0054] in , , , These are the weights and biases of the two fully connected layers, respectively. This represents the activation function. This is a scaling factor used to reduce computational complexity. Finally, the obtained channel weights are multiplied by the input feature map to obtain a feature map recalibrated by the spatial attention mechanism. This is then fed into the Facial Block for feature extraction and mapped to a fully connected layer. eigenvectors .

[0055] Furthermore, in the multi-granularity local feature extraction process, the multi-granularity local feature extraction module crops the generated optical flow map into... The size is [value], and the light flow map around the eye is extracted, with a size of [value]. This is used as input to the Base Layer for initial feature extraction, and finally fed into the Eye Block for eye feature extraction. After passing through a fully connected layer, it is mapped to a... Eye feature vector .

[0056] Furthermore, similar to eye feature extraction, the multi-granularity local feature extraction module crops the generated optical flow map to a specific size, and extracts the optical flow map around the mouth, cropping its size accordingly. Because there is some overlap between the features of the eyes and the mouth, this invention introduces feature transfer learning. The captured optical flow map is fed into a Base Layer learned from a basic extraction network for the eyes for basic extraction. Further detailed feature extraction is performed using a Mouth Block, and after passing through a fully connected layer, it is mapped to a... Mouth feature feature vector .

[0057] Furthermore, in the multi-granularity cognitive enhancement process, to simulate the cognitive patterns and characteristics of multi-granularity cognition, the multi-granularity cognitive enhancement module uses a multi-granularity global feature extraction module to extract coarse-grained global feature vectors of the face using a channel attention mechanism. A weighted algorithm is then designed to guide the extraction of fine-grained local features such as the eyes and mouth. The aim is to use the coarse-grained global facial features to provide different guidance for the fine-grained features around the mouth and eyes before final classification. During this process, to prevent overflow issues, the multi-granularity cognitive enhancement module needs to ensure that the dimensionality of both the global and local feature vectors remains constant. Specifically, the coarse-grained feature vector extracted from the entire face is defined as... The fine-grained feature vectors extracted from the eyes and mouth are defined as follows: and .Will As a guiding vector, it is weighted onto two linear layers, namely and And concatenate them together to form a feature vector. , As input components for the final classification.

[0058]

[0059] in Indicates weighted average. and Represents parameters in the network. and This represents the bias amount. The input to the fully connected layer is used to classify micro-expression categories, and the output is a feature vector of length 5.

[0060] Furthermore, during the feature classification process, the feature classification module will... The input from the final fully connected layer is used to classify micro-expression categories, outputting a feature vector of length 5. The training set from S2 is then used as input to S3 to train the constructed model, updating its parameters, ultimately yielding the best-performing classification model.

[0061] Example 2 This embodiment further details the implementation steps of the micro-expression recognition method based on multi-granularity cognitive enhancement of facial features according to the present invention.

[0062] according to Figure 1 The diagram shown illustrates the overall structure of the micro-expression recognition method based on multi-granularity cognitive enhancement of facial features according to the present invention. This method includes the following steps: Step 1: Obtain the dataset, which specifically includes images of all frames from the video frame showing micro-expressions to the ending frame. Ideally, the dataset should be recorded using professional equipment under natural or indoor conditions, and the facial emotional features should be obvious and the classification accurate.

[0063] This invention uses 70% of the dataset as training data, 15% as validation data, and 15% as test data. Most video frames in the dataset contain background information and features below the neck. Since micro-expressions mainly appear on the face, this information is noise and needs to be masked out. In the preprocessing stage, each video segment is segmented and each frame is extracted. Then, the face detection toolkit (Dlib) is used to locate and crop the face positions, retaining only the face image frames. To remove redundant interference such as head movements, this method uses Procurstes to align different faces in the samples to a uniform standard. The calculation formula is shown below.

[0064]

[0065] in These represent the scaling factor, rotation matrix, and translation vector, respectively. , These refer to two different face image matrices, where each row of the matrix represents a feature point. Coordinates. Specifically, represent The first of the matrix OK, represent The first of the matrix OK.

[0066] This method uses orthogonal matrices. Apply it to the matrix Thus, the transformed matrix is ​​obtained. To ensure data consistency, this method normalizes all micro-expression samples and sets the cropped output size to [value missing]. Then, as Figure 2 As shown, histogram equalization is then used to improve the recognition of micro-expression details, and processing is performed using the Euler motion method and TV-L1 optical flow method.

[0067] Step 2: Euler motion amplification is applied to the histogram-equalized micro-expression images. This enhances the amplitude of facial muscle movements in the micro-expression sequence and also alters the brightness values ​​of the moving areas, thereby reducing the difficulty of micro-expression recognition. The input video frame sequence is... ,in Represents spatial coordinates, Representing time. To amplify micro-expression sequences to higher spatial resolution, a spatial transformation of the video sequence is first required. This invention uses a two-dimensional discrete Fourier transform (DFT) to convert it into a frequency domain representation, as follows:

[0068] in, It is a frequency domain representation. and Is it frequency? and Components in direction, Indicates in Take the direction , Indicates in Take the direction The value of . Through the phase and amplitude information in the frequency domain representation, changes in phase information amplify the subtle movements of micro-expressions. Therefore, the phase change of each frequency component is calculated using the following formula.

[0069]

[0070] in It is the phase of the frequency component. Based on the above processing, in order to accurately capture the magnified facial motion, we adopted the TV-L1 optical flow method.

[0071] Step 3: Because micro-expressions involve low facial movement intensity, adjusting the input image to a smaller shape will result in significant information loss. Therefore, as... Figure 1 As shown in the "Multidimensional Preprocessing" module, this invention divides the input micro-expression image processed by optical flow into upper and lower parts to obtain optical flow images around the eyes and mouth, and adjusts the size to a shape of [224, 224, 3] as input for local feature extraction. In addition, the global micro-expression image processed by histogram equalization is also cropped to a shape of [224, 224, 3] as input for global feature extraction.

[0072] Step 4: In the "Multi-granularity Global / Local Feature Extraction" module, this invention utilizes two parts to achieve this goal: multi-granularity global feature extraction (coarse-grained feature extraction) and multi-granularity local feature extraction (fine-grained feature extraction). To reduce the possibility of overfitting on small datasets, the model avoids larger ResNet versions. This invention sends all input images to the same base layer created by the first two ResNet18 layers to extract similar features. For the base extraction layer, the input size is [224, 224, 3], and the convolution kernel size is... Step size is The model gradually reduces the size of the feature map and increases the number of channels. The convolution kernel for the second basic feature extraction is... Step size is Finally, a feature map of [28, 28, 64] is obtained. The global image obtained in step three is also processed through the same base layer to extract the corresponding feature map.

[0073] Step 5: A channel attention mechanism is introduced in the secondary feature extraction of global micro-expression features. Utilizing the learned channel weights, it can focus more attention on channels related to micro-expressions, enabling the model to extract diverse features. The first convolutional kernel for secondary feature extraction around the eyes and mouth is... Step size is The resulting feature map output is [14, 14, 128], which is then processed by a step size of... The size is The convolution kernel's feature extraction yielded a feature map of [7,7,256]. This concludes the secondary feature extraction for the mouth area. Since the eyes are more detailed than the mouth, subsequent feature extraction will utilize... Step size is A 512-dimensional convolution is used to perform a final fine-grained feature extraction on the eye features, resulting in a feature map of [7,7,512].

[0074] Step 6: Perform max pooling on the two feature maps obtained in Step 5 to obtain a 256-dimensional feature vector for each. and The global facial features, after max pooling, also yield a 256-dimensional feature vector. Then, use As weights for two local eigenvectors and Weighted processing is performed to guide the feature extraction process of local features, enabling multi-granularity computation. Higher similarity values ​​mean that local features are more similar to global features, thus receiving higher weights. Finally, the final weighted feature vector is obtained by concatenating the two weighted local feature vectors. The dimension is 512. This process allows global information to be used to guide the representation and learning of local features.

[0075] Step 7: Finally, the weighted feature vector is fed into a linear layer with an input dimension of 512 and an output dimension of 64, and then non-linearly mapped using the ReLU activation function. Next, the ReLU-activated data is fed into a linear layer from 64 dimensions to 5 dimensions, ultimately obtaining a 5-dimensional output feature vector. In forward propagation, the input tensor... First, a linear transformation from 512 dimensions to 64 dimensions is performed, followed by a non-linear mapping using the ReLU activation function. Then, the 64-dimensional data after ReLU activation is fed into a linear layer that transforms the 64-dimensional data into 5-dimensional data, ultimately yielding a 5-dimensional output feature vector. This output feature vector is the final output of the model; the 64-dimensional features in the intermediate layer serve to map the 512-dimensional input to the 5-dimensional output.

[0076] Step 8: Next, backpropagation is performed to calculate the gradient and update all trainable parameters. Throughout the training process, the ultimate goal of this invention is to minimize the sum of the losses of the classification task, ultimately obtaining a classification model for the micro-expression recognition method based on the multi-granularity cognitive enhancement of facial features of this invention. In subsequent experiments or practical applications, this classification model can be called the FEATNet model.

[0077] Example 3 This invention has been implemented in at least part of the project "Research on Human-like Emotional Interaction and Intelligent Adaptation in Virtual Learning Environments", including student micro-expression recognition algorithm experiments and student classroom emotion detection system. This embodiment demonstrates some of the implementation effects of this invention.

[0078] This embodiment proposes a micro-expression recognition system based on the aforementioned micro-expression recognition method with multi-granularity cognitive enhancement of facial features. It analyzes and makes decisions by capturing students' faces in online classrooms through an RGB camera, thereby enhancing facial features from coarse to fine granular and improving the accuracy and robustness of student emotion detection in the classroom.

[0079] This embodiment trains the method according to the schemes described in Embodiments 1 and 2 above, and encapsulates the trained preferred model into an API service. The Flask backend framework is used to build the service to provide an interface. The service is then deployed to a server for use by the system in this embodiment.

[0080] according to Figure 3 As shown, the architecture of a micro-expression recognition system with multi-granularity cognitive enhancement of facial features in this invention specifically includes a teacher terminal, a student terminal, a model call, and a database.

[0081] The student client application consists of four modules: client login module, RGB image acquisition module, micro-expression recognition module, and data storage module.

[0082] Specifically, the client login module is used for student login; the RGB image acquisition module captures student facial images through an RGB camera; the micro-expression recognition module is equipped with a recognition model trained based on the micro-expression recognition method based on multi-granularity cognitive enhancement of facial features in Embodiments 1 and 2, which is used to analyze the student's current state; the data storage module stores the detected results in the database, so that the teacher can view the student's emotional changes throughout the class.

[0083] The teacher client application is divided into three modules: a client login module, a lecture module, and a data viewing module. Specifically, teachers log in through the client login module; teachers conduct classroom teaching through the lecture module and can view students' emotions in real time through the lecture module; the teacher's data viewing module allows teachers to intuitively read the continuous changes in students' classroom emotions.

[0084] Furthermore, the system architecture specifically includes: front-end, back-end, model invocation, and database. ① A web page is used as the client's front-end interface, allowing users to interact with the system through a browser; ② The back-end handles client requests and provides services; ③ The student's back-end loads and invokes a trained model using Python's deep learning library to perform micro-expression detection; ④ MySQL is used as the database to store students' continuous emotional output during class.

[0085] In this embodiment of the micro-expression recognition method and system based on multi-granularity cognitive enhancement of facial features, both the student and teacher systems adopt a front-end and back-end separation architecture, as well as database connection and storage, and deployment and invocation of the micro-expression classification model. Specifically: (1) Front-end and back-end separation architecture: The front-end uses the Vue framework to implement the user interface and interaction logic.

[0086] The backend uses the Spring Boot framework to implement backend services and API interfaces.

[0087] (2) Data storage module selection: We chose to use a MySQL database to store the data.

[0088] (3) Model deployment and invocation module: When students are in class, the optimal micro-expression recognition model is invoked to extract coarse-grained and fine-grained features of micro-expressions. The coarse-grained information is used to weight the fine-grained information, and finally the classification of micro-expressions is output.

[0089] (4) System Implementation Environment Development platform: PyCharm, Python 3.6, with the experimental environment set up using Anaconda.

[0090] Operating System: Windows 10, 64-bit Computer configuration: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz; 64.00GB RAM; NVIDIA GeForce GTX 2080Ti graphics card; This system can monitor and provide feedback on students' emotional changes in class in real time, and can store the classroom status in a database for later review.

[0091] The implementation results of the micro-expression recognition algorithm are shown in Tables 1 and 2.

[0092] Table 1

[0093] As shown in Table 1, the performance comparison results on the 5-class datasets (CASMEII-5 and SAMM-5) demonstrate that, compared to other traditional machine learning and deep learning-based methods, the FEATNet model of this invention achieves better results on these two five-class datasets. Specifically, on the CASMEII-5 dataset, the accuracy is improved by approximately 4%, and the F1-score is the second best. Compared to other methods, FEATNet of this invention primarily employs a multi-granularity cognitive computation approach, utilizing coarse-grained features to guide the extraction of fine-grained features, thus contributing to the model's performance.

[0094] Table 2

[0095] Table 2 shows the performance comparison results on the 3-class classification datasets (CASMEII-3 and SAMM-3). This embodiment conducted the same experiments on the 3-class classification datasets (CASMEII-3 and SAMM-3). On the SAMM-3 dataset, the UF1 and UAR of the FEATNet model of this invention are 0.8277 and 0.8152, respectively, demonstrating that the FEATNet model of this invention can handle blurred image datasets well and can capture subtle changes in facial micro-expressions.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A micro-expression recognition method with enhanced multi-granularity facial feature recognition, characterized in that: The method includes data acquisition, multi-dimensional preprocessing methods, multi-granularity global feature extraction, multi-granularity local feature extraction, multi-granularity cognitive enhancement, and feature classification; Among them, a multi-dimensional preprocessing method is adopted to extract the start frame and vertex frame from the video frame, use Euler video to enlarge the two frames of the image, then use histogram equalization to highlight the texture features, and finally use TV-L1 optical flow method to obtain the preprocessed image. Feature vectors are obtained by using a multi-granularity global and local feature extraction method, and transfer learning is introduced when extracting multiple local features; In the process of multi-granularity cognitive enhancement, the multi-granularity global feature extraction module is used to extract the coarse-grained global feature vector of the face using the channel attention mechanism, and a weighted algorithm is designed to guide the extraction of local fine-grained features of the eyes and mouth. The final feature vector, obtained by concatenating the coarse-grained global feature vector of the face with the feature vectors of the eyes and mouth respectively, is used as the input component of the classification model for model training to obtain the classification model; The formula for highlighting texture features in a face image through histogram equalization during preprocessing is as follows: in, This indicates that the grayscale value of a pixel in the image is... The probability density function, It is the cumulative distribution function. It is the first after histogram equalization A cumulative probability density function, Indicates the total number of pixels. To balance pixel values, It is a gray level. Indicates an integer rounded to the nearest whole number; In the multi-granularity global feature extraction process, the preprocessed optical flow map is processed using an attention mechanism to obtain the global feature matrix. , The average value of each channel is obtained based on the global feature matrix. : in It is a length of The vector, This refers to the number of channels. and These refer to the height and width of the feature map, respectively. Then As input, obtain the two channel weights. , : in, , These are the weights of the two fully connected layers. , These are the bias terms of the two fully connected layers. This represents the activation function. It is a scaling factor used to reduce computational complexity; Finally, the obtained channel weights are multiplied by the input feature map to obtain a feature map recalibrated by the spatial attention mechanism. This map is then fed into the Facial Block for feature extraction and mapped to a coarse-grained global feature vector of the face through a fully connected layer. ; During the multi-granularity local feature extraction process, when extracting fine-grained local features for the eyes and mouth, the optical flow map generated after preprocessing is cropped to extract the optical flow map around the eyes. This is then used as input to the Base Layer for preliminary feature extraction, and finally fed into the Eyes Block for fine-grained eye feature extraction. After passing through a fully connected layer, it is mapped to an eye feature vector. ; During the feature extraction process, when extracting mouth features, the captured optical flow map is fed into the Base Layer, which is learned from the eye-based feature extraction network, for basic extraction. Further fine-grained feature extraction is performed using the Mouth Block, and after passing through a fully connected layer, it is mapped to a mouth feature vector. .

2. The micro-expression recognition method with multi-granularity cognitive enhancement of facial features according to claim 1, characterized in that: When extracting the start frame and vertex frame from the video frames during the multidimensional preprocessing, face alignment is performed in the following manner: in These represent the scaling factor, rotation matrix, and translation vector, respectively. , These refer to two different face image matrices, where each row of the matrix represents a feature point. coordinate, represent The first of the matrix OK, represent The first of the matrix OK.

3. The micro-expression recognition method with multi-granularity facial feature recognition enhancement according to claim 2, characterized in that: When enlarging the image during preprocessing, the following formula is used: The input video frame sequence is: ,in Represents spatial coordinates, Indicates time, It is a frequency domain representation. and Is it frequency? and Components in direction, Indicates in Take the direction The value, Indicates in Take the direction The value; The phase and amplitude information in an image is represented in the frequency domain. Changes in phase information amplify subtle movements in micro-expressions. The formula for calculating the phase change of each frequency component is as follows: in It is the phase of the frequency component.

4. The micro-expression recognition method with multi-granularity cognitive enhancement of facial features according to claim 3, characterized in that: In the process of multi-granularity feature recognition enhancement, As a guiding vector, it is weighted onto two linear layers, namely and And concatenate them together to form a feature vector , As input components for the final classification, where The calculation method is as follows: in Indicates weighted average. and Represents parameters in the network. and This indicates the bias amount.

5. The micro-expression recognition method with multi-granularity cognitive enhancement of facial features according to claim 4, characterized in that: In the feature classification process, The micro-expression categories are classified using the input of the final fully connected layer. The constructed model is then trained, the parameters in the model are updated, and finally, a classification model is obtained.

6. A micro-expression recognition system with multi-granularity facial feature recognition enhancement, characterized in that: It is used to execute the micro-expression recognition method with multi-granularity cognitive enhancement of facial features as described in any one of claims 1-5. The system includes a teacher terminal, a student terminal, a model calling module, and a database. The teacher terminal can view the real-time emotional changes of students and the emotional change trend of the whole class. The student terminal can call the model to monitor the micro-expressions of students in real time and store them in the database. The student client includes a client login module, an RGB image acquisition module, a micro-expression recognition module, and a data storage module. The client login module is used for student login. The RGB image acquisition module captures student facial images through an RGB camera. The micro-expression recognition module is equipped with a classification model trained using a micro-expression recognition method based on multi-granularity cognitive enhancement of facial features. The data storage module stores the detected results in a database. The teacher client includes a client login module, a lecture module, and a data viewing module; teachers log in through the client login module. The teacher module allows teachers to conduct classroom teaching and view students' emotions in real time; the data viewing module allows teachers to read the continuous changes in students' classroom emotions.

Images