Method and system for consciousness evaluation based on micro-expression recognition

Abstract

This invention discloses a consciousness assessment method and system based on micro-expression recognition. The method includes: configuring speech stimulation materials; playing the speech stimulation materials to multiple subjects; collecting facial expression image data of each subject when receiving the speech stimulation; extracting feature data of each subject from the facial expression image data; training an SVM classifier using the feature data of multiple subjects and their corresponding user categories; obtaining feature data of users with unknown categories and inputting it into the trained SVM classifier to obtain the user category corresponding to that user. The consciousness assessment method and system based on micro-expression recognition of this invention can more directly reflect the consciousness level of subjects, improve the accuracy and convenience of consciousness state judgment, and can provide more accurate assessments for patients who cannot undergo MRI or EEG, increasing its universality.

Description

Technical Field

[0001] This invention relates to a method and system for assessing consciousness based on micro-expression recognition. Background Technology

[0002] Assessing the level of consciousness is a key focus and challenge in modern medicine and neuroscience, significantly impacting the diagnosis and prognosis of patients with acute and chronic disorders of consciousness. Traditional assessment methods rely on patient behavior, such as visual and auditory startle responses. Clinically, the Coma Recovery Scale-Revised (CRS-R) is typically used as the gold standard for assessing patient consciousness. However, heterogeneity arising from the type and severity of brain injury, as well as the patient's physiological condition at the time of assessment, can affect the accuracy of the assessor's judgment of the level of consciousness. Therefore, repeated assessments by multiple assessors are often necessary in clinical practice.

[0003] With the rapid development of neuroelectrophysiology and neuroimaging, EEG and MRI techniques are widely used to assist in the assessment of patients' state of consciousness and prognosis. However, these methods still have shortcomings. On the one hand, patients' responses to stimuli during assessment vary in strength, and clinical assessors cannot visually capture some rapid and subtle responses, leading to inaccurate assessments. Furthermore, the assessment of consciousness in patients with acute and chronic disorders of consciousness relies heavily on the clinical assessor's experience, making objectivity and standardization difficult. On the other hand, most patients who have suffered severe brain injury or undergone decompressive craniectomy have severe skull deformities or intracranial metal implants, limiting the signal acquisition capabilities of EEG and MRI techniques and narrowing their target audience. Summary of the Invention

[0004] This invention provides a method and system for consciousness assessment based on micro-expression recognition to solve the aforementioned technical problems, specifically adopting the following technical solution:

[0005] A consciousness assessment method based on micro-expression recognition includes the following steps:

[0006] Configure speech stimulation materials;

[0007] The audio stimulus material was played to multiple subjects from different user categories, which were divided into type I and type II based on the subjects’ state of consciousness.

[0008] Collect facial expression image data of each subject when receiving speech stimulation;

[0009] Extracting feature data for each subject from facial expression image data;

[0010] The SVM classifier was trained using feature data from multiple subjects and their corresponding user categories.

[0011] The feature data of users with unknown categories are obtained and input into a trained SVM classifier to obtain the user category corresponding to that user.

[0012] Furthermore, the specific method for extracting feature data for each subject from facial expression image data is as follows:

[0013] Face tracking using facial expression image data;

[0014] Extract feature points from the tracked face images;

[0015] The extracted feature points are converted into feature points of neutral expressions and mapped onto the reference frame through similarity transformation;

[0016] Mask the environmental image surrounding the facial region of the face image mapped into the reference frame;

[0017] Feature data for each subject is extracted from the aligned face images.

[0018] Furthermore, the specific method for converting the extracted feature points into feature points of neutral expressions through similarity transformation is as follows:

[0019] Select a subset of feature points from the extracted feature points that can reliably reflect facial expressions;

[0020] The selected stable feature points are transformed into feature points of the central expression and mapped onto the reference frame through similarity transformation.

[0021] Furthermore, the feature points are transformed using the least squares overlap method.

[0022] Furthermore, the feature data includes appearance features and geometric features.

[0023] Furthermore, the specific method for extracting feature data for each subject from the aligned face image is as follows:

[0024] Extract HOG feature vectors from aligned face images as appearance features;

[0025] The non-rigid parameters set during face tracking and the positional parameters of the extracted feature points are used as geometric features;

[0026] Individual neutral facial expression correction is performed based on appearance and geometric features.

[0027] Furthermore, the specific method for extracting HOG feature vectors from the aligned face image as appearance features is as follows:

[0028] The gamma correction method is used to normalize the aligned face image and calculate the gradient of each pixel separately;

[0029] Histogram normalization is performed on the gradient of each pixel;

[0030] Divide the face image into several blocks to obtain the HOG feature vector of each face image;

[0031] The appearance features are obtained by dimensionality reduction of the HOG feature vectors.

[0032] Furthermore, the specific method for masking the environmental image surrounding the facial region is as follows:

[0033] The convex hull algorithm is used to enclose the feature points of the aligned face to mask the surrounding environment image.

[0034] Furthermore, after training the SVM classifier using feature data from multiple subjects and their corresponding user categories, the consciousness assessment method based on micro-expression recognition also includes:

[0035] The SVR classifier was trained using feature data from multiple subjects and their corresponding response intensity categories.

[0036] After obtaining the feature data of users of unknown categories, the data is input into a trained SVR classifier to obtain the corresponding reaction intensity classification for that user.

[0037] A consciousness assessment system based on micro-expression recognition, comprising:

[0038] The configuration module is used to configure speech stimulation materials;

[0039] The voice playback module is used to play voice stimulation materials to multiple subjects, who come from different user categories. The user categories are divided into type 1 and type 2 according to the subjects' different states of consciousness.

[0040] The image acquisition module is used to acquire facial expression image data of each subject when receiving speech stimuli.

[0041] The feature extraction module is used to extract feature data for each subject from facial expression image data;

[0042] An SVM classifier is used to train the classifier using feature data from multiple subjects and their corresponding user categories.

[0043] After the SVM classifier is trained, the feature data of users with unknown categories are input into the trained SVM classifier to obtain the user category corresponding to that user.

[0044] Furthermore, the image acquisition module includes:

[0045] The face tracking submodule is used to perform face tracking on facial expression image data;

[0046] The feature point extraction submodule is used to extract feature points from the face image tracked by the face tracking submodule;

[0047] The alignment submodule is used to convert the feature points extracted by the feature point extraction submodule into feature points of neutral expression and map them onto the reference frame through similarity transformation;

[0048] The masking submodule is used to mask the environmental image around the facial region of a face image mapped into the reference frame;

[0049] The feature extraction submodule is used to extract feature data for each subject from the aligned face image.

[0050] Furthermore, the alignment submodule uses similarity transformation to convert the feature points extracted by the feature point extraction submodule into feature points of neutral expression and maps them to the reference frame. The specific method is as follows:

[0051] The alignment submodule selects some feature points that can stably reflect facial expressions from the feature points extracted by the feature point extraction submodule, and then uses similarity transformation to convert the selected stable feature points into feature points of the central expression and map them into the reference frame.

[0052] Furthermore, the feature extraction submodule includes:

[0053] The vector extraction unit is used to extract HOG feature vectors from the aligned face image as appearance features.

[0054] The data acquisition unit is used to acquire the non-rigid parameters set during face tracking and the position parameters of the extracted feature points as geometric features;

[0055] The correction unit is used to correct individual neutral facial expressions based on appearance and geometric features.

[0056] Furthermore, the vector extraction unit includes:

[0057] The gradient calculation subunit is used to normalize the aligned face image using the Gamma correction method and calculate the gradient of each pixel separately.

[0058] The normalization subunit is used to perform histogram normalization on the gradient of each pixel;

[0059] Sub-units are used to divide a face image into several blocks to obtain the HOG feature vector of each face image;

[0060] Dimensionality reduction subunits are used to reduce the dimensionality of the HOG feature vectors obtained after dividing the subunits to obtain appearance features.

[0061] Furthermore, the consciousness assessment system based on micro-expression recognition also includes:

[0062] The SVR classifier is used to train the classifier using feature data from multiple subjects and their corresponding response intensity categories.

[0063] After the SVR classifier is trained, the feature data of users with unknown categories are input into the trained SVR classifier to obtain the corresponding reaction intensity category of the user.

[0064] The advantages of this invention are that the consciousness assessment method and system based on micro-expression recognition provided can more directly reflect the consciousness level of the subject, improve the accuracy and convenience of judging the state of consciousness, and enable patients who cannot undergo MRI or EEG to receive more accurate assessments, thus increasing its universality. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of a consciousness assessment method based on micro-expression recognition according to the present invention. Detailed Implementation

[0066] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0067] like Figure 1 The following is a method for assessing consciousness based on micro-expression recognition, comprising the following steps: S1: Configure speech stimulation materials. S2: Play speech stimulation materials to multiple subjects. The subjects are from different user categories, which are divided into a first type and a second type based on the subjects' states of consciousness. S3: Collect facial expression image data of each subject when receiving speech stimulation. S4: Extract feature data of each subject from the facial expression image data. S5: Train an SVM classifier using the feature data of multiple subjects and their corresponding user categories. S6: Obtain feature data of users with unknown categories and input it into the trained SVM classifier to obtain the user category corresponding to that user. Micro-expression recognition is achieved by detecting the occurrence and intensity of facial action units (AUs). AUs are defined by the Facial Activity Coding System (FACS) and are used to describe different facial muscle movement changes. Through the above steps, the subjects' states of consciousness are determined by capturing and analyzing their facial expressions when stimulated. The following details the above steps.

[0068] For step S1: Configure speech stimulation materials.

[0069] Specifically, relevant speech paradigms required for clinical assessment are adopted. These include startling stimuli, commands, and communication. In this application, the main focus is on observing the subjects' facial expressions under different speech stimuli; therefore, the first step is to prepare speech stimulus materials. These materials can be prepared as follows.

[0070] Frightening and stimulating effect: Gunshot, lasting 1 second, followed by a 10-second interval before the next gunshot is played, repeated 10 times.

[0071] Commands: Look up (down, left, right), open (close) your mouth, etc. Speech rate: 3 seconds per second. After each command finishes, wait 15 seconds before playing the next command. Each command is repeated 5 times, for a total of 6 commands and 30 repetitions.

[0072] Communicative voice prompts: For example, "Are you in the hospital?" If you are in the hospital, look upwards; if you are not in the hospital, look downwards. Each voice prompt lasts 10-11 seconds. After each prompt, there is a 15-second interval before the next prompt is played. Each prompt is repeated 5 times, for a total of 6 prompts and 30 prompts.

[0073] For step S2: Play the speech stimulation material to multiple subjects respectively.

[0074] Specifically, after configuring the speech stimulation materials in step S1, these speech stimulation materials are played to multiple subjects.

[0075] The participants came from various user categories, which were further divided into Type I and Type II based on their state of consciousness. In this application, Type I was defined as vegetative state (VS) and minimally conscious state (MCS), and Type II as healthy state (HC).

[0076] Understandably, in order to identify the different responses of subjects with different levels of consciousness to these speech stimuli, it is first necessary to test different categories of subjects and collect data as a basis for subsequent judgments.

[0077] For step S3: Collect facial expression image data of each subject when receiving speech stimulation.

[0078] During the speech stimulation of the subjects, the micro-expression changes of the subjects were recorded in the whole process through a high-definition camera.

[0079] For step S4: Extract feature data for each subject from facial expression image data.

[0080] In this application, the specific method for extracting feature data for each subject from facial expression image data is as follows:

[0081] Face tracking was performed on facial expression image data. Specifically, a face tracker in the CLNF (Constrained Local Neural Field) model was used to track the patient's face.

[0082] Feature points are extracted from the tracked face image. In this application, 68 feature points of the face are extracted using a feature point detector in a CLNF model.

[0083] The extracted feature points are converted into neutral facial expression feature points and mapped onto the reference frame through a similarity transformation. To better analyze facial texture features and avoid the influence of head movements, the face tracking results need to be mapped to a common reference frame to remove changes caused by scaling and rotation. To map the face tracking results to the same reference frame, a similarity transformation is used to convert the feature points detected by the CLNF model into neutral facial expression feature points. In this application, the feature points are similarly transformed using the least squares overlap method. Specifically, the method for converting the extracted feature points into neutral facial expression feature points through similarity transformation is as follows: A subset of feature points that stably reflect facial expressions are selected from the extracted feature points. These stable feature points are then converted into central facial expression feature points and mapped onto the reference frame through the similarity transformation. It is understood that to reduce the impact of large facial expressions on the similarity transformation, only stable facial feature points are used for the similarity transformation. In this application, 24 feature points from 68 feature points are selected as stable facial feature points: the sides of the cheeks, the lower eyelids, and around the nose.

[0084] The environmental image surrounding the facial region of the face image mapped into the reference frame is masked. Specifically, the method for masking the environmental image around the facial region is to use a convex hull algorithm to enclose the aligned feature points of the face to mask the surrounding environmental image.

[0085] Feature data for each subject is extracted from the aligned face images. In this application, the feature data includes appearance features and geometric features.

[0086] The specific method for extracting feature data for each subject from the aligned face image is as follows:

[0087] Extract HOG (Histograms of Oriented Gradient) feature vectors from aligned face images as appearance features.

[0088] The non-rigid parameters set during face tracking and the positional parameters of the extracted feature points are used as geometric features.

[0089] Individual neutral expression correction is performed on appearance and geometric features, which means subtracting the median feature value of the video from the features of each frame.

[0090] The specific method for extracting HOG feature vectors as appearance features from aligned face images is as follows:

[0091] The gamma correction method is used to normalize the aligned face image and calculate the gradient of each pixel.

[0092] The gradient of each pixel is histogram normalized.

[0093] The face image is divided into several blocks to obtain the HOG feature vector of each face image.

[0094] The appearance features are obtained by dimensionality reduction of the HOG feature vectors.

[0095] Specifically, in this application, the aligned face image is 112*112 pixels. HOG feature extraction first involves normalizing the color space using Gamma correction, calculating the gradient for each pixel, normalizing the gradient histogram, and dividing the image into blocks of 2*2 units (8*8 pixels), resulting in 12*12 blocks. Each block has a 31-dimensional histogram feature, giving each face image a 4464-dimensional feature vector. Then, PCA (Principal Component Analysis) is used to reduce the dimensionality of the HOG feature vector, retaining 95% of the feature information to obtain the face appearance feature vector.

[0096] For step S5: Train the SVM classifier using feature data from multiple subjects and their corresponding user categories.

[0097] For step S6: Obtain the feature data of users with unknown categories and input it into the trained SVM classifier to obtain the user category corresponding to the user.

[0098] Preferably, after training the SVM classifier using feature data from multiple subjects and their corresponding user categories, the consciousness assessment method based on micro-expression recognition further includes:

[0099] The SVR classifier is trained using feature data from multiple subjects and their corresponding response intensity categories. After obtaining feature data from users with unknown categories, this data is then input into the trained SVR classifier to obtain the corresponding response intensity category for that user.

[0100] Based on the results of micro-expression analysis, the changes in the intensity of the Action Unit (AU) before and after the subject was stimulated are compared to assess the subject's facial expressions when stimulated by speech. Preferably, the micro-expression score can be added to the original CRS-R scoring module to form a new behavioral score, which allows for a more detailed classification of the subject's level of consciousness.

[0101] This application also discloses a consciousness assessment system based on micro-expression recognition, comprising: a configuration module, a voice playback module, an image acquisition module, a feature extraction module, and an SVM classifier.

[0102] Specifically, the configuration module is used to configure the speech stimulus material. The speech playback module is used to play the speech stimulus material to multiple subjects. These subjects come from different user categories, which are divided into Type 1 and Type 2 based on their state of consciousness. The image acquisition module is used to acquire facial expression image data of each subject when receiving the speech stimulus. The feature extraction module is used to extract feature data for each subject from the facial expression image data. The SVM classifier is used to train the classifier using the feature data of multiple subjects and their corresponding user categories. After the SVM classifier is trained, the feature data of users with unknown categories is input into the trained SVM classifier to obtain the user category corresponding to that user.

[0103] In a preferred embodiment, the image acquisition module includes: a face tracking submodule, a feature point extraction submodule, an alignment submodule, a masking submodule, and a feature extraction submodule.

[0104] The face tracking submodule performs face tracking on facial expression image data. The feature point extraction submodule extracts feature points from the face image tracked by the face tracking submodule. The alignment submodule converts the feature points extracted by the feature point extraction submodule into neutral expression feature points and maps them onto the reference frame using a similarity transformation. The masking submodule masks the environmental image surrounding the facial region of the face image mapped onto the reference frame. The feature extraction submodule extracts feature data for each subject from the aligned face image.

[0105] As a preferred implementation, the alignment submodule converts the feature points extracted by the feature point extraction submodule into feature points of neutral facial expressions and maps them into the reference frame through similarity transformation. The specific method is as follows: the alignment submodule selects some feature points that can stably reflect facial expressions from the feature points extracted by the feature point extraction submodule, and then converts the selected stable feature points into feature points of the central facial expression and maps them into the reference frame through similarity transformation.

[0106] In a preferred embodiment, the feature extraction submodule includes: a vector extraction unit, a data acquisition unit, and a correction unit.

[0107] The vector extraction unit extracts HOG feature vectors from the aligned face image as appearance features. The data acquisition unit acquires the non-rigid parameters set during face tracking and the positional parameters of the extracted feature points as geometric features. The correction unit performs individual-neutral expression correction on the appearance and geometric features.

[0108] As a preferred implementation, the vector extraction unit includes: a gradient calculation subunit, a normalization subunit, a partitioning subunit, and a dimensionality reduction subunit.

[0109] The gradient calculation subunit normalizes the aligned face image using Gamma correction and calculates the gradient for each pixel. The normalization subunit performs histogram normalization on the gradient of each pixel. The partitioning subunit divides the face image into several blocks to obtain the HOG feature vector for each face image. The dimensionality reduction subunit reduces the dimensionality of the HOG feature vectors obtained from the partitioning subunit to obtain the appearance features.

[0110] As a preferred implementation, the consciousness assessment system based on micro-expression recognition also includes an SVR classifier.

[0111] The SVR classifier is trained using feature data from multiple subjects and their corresponding response intensity categories. After the SVR classifier is trained, feature data from users with unknown categories are input into the trained SVR classifier to obtain the corresponding response intensity category for that user.

[0112] For more specific details on the execution of each module, please refer to the corresponding section in the aforementioned consciousness assessment method based on micro-expression recognition, which will not be repeated here.

[0113] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for assessing consciousness based on micro-expression recognition, characterized in that, Includes the following steps: Configure speech stimulation materials; The speech stimulation material was played to multiple subjects, who came from different user categories, which were divided into a first type and a second type based on the subjects’ states of consciousness. Collect facial expression image data of each subject when receiving speech stimulation; Extract feature data for each subject from the facial expression image data; The SVM classifier was trained using feature data from multiple subjects and their corresponding user categories. The feature data of users with unknown categories are obtained and input into the trained SVM classifier to obtain the user category corresponding to the user. The specific method for extracting feature data of each subject from the facial expression image data is as follows: Face tracking is performed on the facial expression image data; Extract feature points from the tracked face images; The extracted feature points are transformed into feature points of neutral expressions through similarity transformation, so as to map the face tracking results to the same reference frame; Mask the environmental image surrounding the facial region of the face image mapped into the reference frame; Extract feature data for each subject from the aligned face images; The feature data includes appearance features and geometric features; The specific method for extracting feature data of each subject from the aligned face image is as follows: Extract HOG feature vectors from the aligned face image as the appearance features; The non-rigid parameters set during face tracking and the position parameters of the extracted feature points are used as the geometric features; Individual neutral facial expression correction is performed on the appearance features and geometric features.

2. The consciousness assessment method based on micro-expression recognition according to claim 1, characterized in that, The specific method for converting the extracted feature points into feature points of neutral expressions through similarity transformation is as follows: Select a subset of feature points that can reliably reflect facial expressions from the extracted feature points; The selected stable feature points are transformed into feature points of neutral expressions through similarity transformation.

3. The consciousness assessment method based on micro-expression recognition according to claim 2, characterized in that, The feature points are similarly transformed using the least squares overlap method.

4. The consciousness assessment method based on micro-expression recognition according to claim 1, characterized in that, The specific method for masking the environmental image around the facial area is as follows: The convex hull algorithm is used to enclose the feature points of the aligned face to mask the surrounding environment image.

5. The consciousness assessment method based on micro-expression recognition according to claim 1, characterized in that, After training the SVM classifier using feature data from multiple subjects and their corresponding user categories, the consciousness assessment method based on micro-expression recognition further includes: The SVR classifier was trained using feature data from multiple subjects and their corresponding response intensity categories. After obtaining the feature data of users of unknown categories, the data is input into the trained SVR classifier to obtain the corresponding reaction intensity classification for the user.

6. A consciousness assessment system based on micro-expression recognition, characterized in that, Include: The configuration module is used to configure speech stimulation materials; The voice playback module is used to play the voice stimulation material to multiple subjects respectively, wherein the multiple subjects come from different user categories, and the user categories are divided into a first type and a second type according to the different states of consciousness of the subjects; The image acquisition module is used to acquire facial expression image data of each subject when receiving speech stimuli. The feature extraction module is used to extract feature data for each subject from the facial expression image data; An SVM classifier is used to train the classifier using feature data from multiple subjects and their corresponding user categories. After the SVM classifier is trained, the feature data of users with unknown categories are input into the trained SVM classifier to obtain the user category corresponding to that user. The feature extraction module includes: The face tracking submodule is used to perform face tracking on facial expression image data; The feature point extraction submodule is used to extract feature points from the face image tracked by the face tracking submodule; The alignment submodule is used to convert the feature points extracted by the feature point extraction submodule into feature points of neutral expressions through similarity transformation, so as to map the face tracking results to the same reference frame; The masking submodule is used to mask the environmental image around the facial region of a face image mapped into the reference frame; The feature extraction submodule is used to extract feature data for each subject from the aligned face image; The feature extraction submodule includes: The vector extraction unit is used to extract HOG feature vectors from the aligned face image as appearance features. The data acquisition unit is used to acquire the non-rigid parameters set during face tracking and the position parameters of the extracted feature points as geometric features; The correction unit is used to correct individual neutral facial expressions based on appearance and geometric features.

Images