Assistant diagnosis system for disturbance of consciousness based on pain intensity estimation

Through an awareness disorder assisted diagnosis system based on pain intensity estimation, computer vision and deep learning technology are used to analyze facial expressions of DOC patients, solving the problems of high diagnostic misdiagnosis rate and equipment complexity in the existing technology, and realizing the awareness level of DOC patients Accurate evaluation and efficient diagnosis.

CN119069110APending Publication Date: 2024-12-03SOUTH CHINA NORMAL UNIV

Patent Information

Application Number
CN202410982232.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is very high when diagnosing patients with DOC, especially when distinguishing between minimal consciousness state (MCS) and persistent vegetative state (VS), and the existing methods are expensive, complex in operation or are affected by the environment, making it difficult to achieve. Accurate diagnosis.

Method used

Using an assisted diagnosis system for awareness disorders based on pain intensity estimation, computer vision technology analyzes the facial expressions of DOC patients when receiving pain stimulation, uses deep learning network to extract features, combines VAS scale to evaluate pain perception ability, assist clinical diagnosis to improve diagnosis accuracy sex.

Benefits of technology

It provides a simple and easy-to-use efficient tool, which improves the accuracy of diagnosis of DOC patients, simplifies operations, reduces equipment costs, and enhances the ability of medical staff to formulate personalized treatment plans.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119069110A_ABST
    Figure CN119069110A_ABST
Patent Text Reader

Abstract

The invention discloses a disturbance of consciousness auxiliary diagnosis system based on pain intensity estimation. The disturbance of consciousness auxiliary diagnosis system comprises a data acquisition module, a data analysis module, an interactive interface module and a system management module, the data acquisition module is used for acquiring data: a medical worker records a facial expression video of a DOC patient while applying pain stimulation to the left finger of the DOC patient, and after the emotion of the patient is mild, the medical worker applies pain stimulation to the right finger of the DOC patient and records the facial expression video of the DOC patient; the method has the beneficial effects that the method is simple and easy to use, the diagnosis accuracy of the DOC patient is improved, and a more efficient tool is provided for medical personnel; according to the method, the computer vision technology is applied to medical auxiliary diagnosis, so that medical staff can formulate a treatment scheme in time, and more accurate diagnosis and treatment services are provided for patients; the pain perception ability of the patient is quantified by analyzing the facial expression of the DOC patient, so that the consciousness level of the patient is detected, and the method is simpler and more efficient.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, computer vision and clinical medicine, and specifically relates to a consciousness disorder auxiliary diagnosis system based on pain intensity estimation. Background Art

[0002] Disorders of consciousness (DOC) are neuropsychiatric disorders of wakefulness and consciousness caused by trauma, stroke, or hypoxia. In recent years, with the development of emergency and critical care technologies, more and more lives of patients with severe craniocerebral injury have been saved, but this has also led to the emergence of a large number of DOC patients. According to conservative estimates, there are about 70,000 to 150,000 new DOC patients in my country each year, and the total treatment cost is at least 30 to 50 billion yuan per year, which has brought a huge burden to society, the economy, and people's livelihood. According to the level of consciousness, chronic disorders of consciousness can be divided into two levels: minimally conscious state (MCS) and persistent vegetative state (VS), the latter of which is also called unresponsive wakefulness syndrome (UWS); MCS patients can show weak signs of consciousness and behavior in response to external stimuli, while VS patients have no observable consciousness and behavioral response; when treating DOC patients, it is crucial to obtain an accurate diagnosis as soon as possible to generate an individualized treatment plan; MCS patients have higher rehabilitation potential than VS patients, and accurate diagnosis can help doctors determine the level of consciousness and cognitive function of DOC patients, thereby formulating appropriate rehabilitation intervention plans for patients; however, it is challenging for doctors to accurately diagnose patients, and misdiagnosis is easy to occur when distinguishing VS patients from MCS patients.

[0003] At present, the main methods for detecting consciousness of DOC patients are: clinical behavioral method, neuroimaging method and neuroelectrophysiological method. When using clinical behavioral method to evaluate the level of consciousness of patients, if a specifically designed behavioral assessment scale is not used or the assessment is improper, about 40% of DOC patients will be misdiagnosed. The vast majority of DOC patients have impaired motor ability, and it is difficult to accurately reflect their level of consciousness through behavioral scale assessment alone. Neuroimaging is another effective assessment method, which mainly uses PET / fMRI equipment, but these devices are expensive and bulky and cannot be used for bedside testing. In addition to neuroimaging technology, electrophysiological methods are another type of auxiliary diagnostic method that has been widely studied in recent years, among which the most widely used is electroencephalogram (EEG) technology. However, when diagnosing DOC patients, one challenge of using EEG technology is the variability of its results, which may be affected by factors such as the patient's level of awakening, the specific tasks used in the assessment, and the presence of artifacts in the data. In addition, the operation of EEG equipment and data interpretation are complex and require professional training.

[0004] The existing patents disclosed for the cognitive state recognition system and method involving facial expressions include "Cognitive state recognition system and method based on facial micro-expressions" with application number CN112201343A and "Consciousness assessment method and system based on micro-expression recognition" with application number CN202210036474.6. Both methods are based on micro-expression recognition technology to assess the individual's state of consciousness; their core content includes using video data to capture the changes in the subject's facial micro-expressions, and classifying them through feature extraction and machine learning models to determine the subject's level of consciousness or state, but micro-expressions are usually short-lived and small in amplitude, and are easily affected by environmental factors and video acquisition conditions; the existing patents disclosed for the auxiliary diagnosis system and method for DOC patients include Application number CN202310464892.X, "Auxiliary diagnosis system for patients with disorders of consciousness based on P300 EEG and transfer learning", and CN202110431490, "A auxiliary diagnosis system for disorders of consciousness based on analysis of auditory evoked EEG signals", both attempt to use EEG data to evaluate the level of consciousness of DOC patients; however, DOC patients are often unable to actively cooperate with the EEG acquisition process, so it is difficult to collect EEG signals from DOC patients. In addition, high-quality EEG acquisition equipment is expensive and requires professional maintenance; the existing published patent application number CN202311598152.1, entitled "Auxiliary diagnosis method for disorders of consciousness based on facial expressions", uses the quantitative pain intensity index PSPI-r composed of facial action units (AUs) to evaluate the pain perception ability of DOC patients; however, the number of publicly available AU data sets is limited, and there are not many types of pain-related AUs that can be provided, which will affect the accuracy and reliability of this method in practical applications.

[0005] Studies have shown that as the level of consciousness increases, the pain perception ability of DOC patients also increases; facial movements and expression changes are closely related to pain and convey most of the pain-related information. Summary of the invention

[0006] The purpose of the present invention is to provide a consciousness disorder auxiliary diagnosis system based on pain intensity estimation, analyze the facial expressions of DOC patients when they are exposed to pain stimulation through computer vision technology, quantify the pain perception ability of different DOC patients, and develop a simple and easy-to-use DOC patient auxiliary diagnosis system to assist existing clinical methods, improve the accuracy of DOC patient consciousness detection, and provide medical staff with more efficient tools.

[0007] To achieve the above object, the present invention provides the following technical solutions: a consciousness disorder auxiliary diagnosis system based on pain intensity estimation, comprising a data acquisition module, a data analysis module, an interactive interface module, and a system management module;

[0008] The data acquisition module is used to collect data: the medical staff applies pain stimulation to the left finger of the DOC patient and records the facial expression video at an interval of 30 seconds. After the patient calms down, the medical staff applies pain stimulation to the right finger and records the facial expression video. The collected facial video of the DOC patient is input into the data analysis module;

[0009] The data analysis module is used to analyze data: after receiving the patient's facial expression video from the data acquisition module, it is necessary to extract frames, crop, and perform data enhancement preprocessing on them, and then extract features and predict VAS values ​​through a deep learning network to obtain the VAS values ​​of the DOC patients during the period of pain stimulation, and store the experimental information and data in the system;

[0010] The interactive interface module is used for the interaction between the user and the interface: through the front-end rendering, the system is visualized, and the user interacts with the system through the mouse. The user views the name, gender, behavior scale diagnosis category, and VAS value of the DOC patient when receiving pain stimulation on the interface, and the final judgment category of the DOC patient is output on the interface;

[0011] The system management module is used to manage and maintain the auxiliary diagnosis system for patients with impaired consciousness to ensure its stable operation. The system management module includes a data storage module, a system configuration management module, and a log recording and monitoring module; the data storage module is responsible for data storage in the system; the system configuration management module allows the administrator to configure various parameters and settings of the system; the log recording and monitoring module records the system's operation log and operating status.

[0012] As a preferred technical solution of the present invention, the data storage module is responsible for data storage in the system, including database management and file storage.

[0013] As a preferred technical solution of the present invention, the system configuration management module allows the administrator to configure various parameters and settings of the system, including interface language, time format, and default settings; the logging and monitoring module records the system's operation log and operating status, including user login logs, error logs, and system performance monitoring.

[0014] As a preferred technical solution of the present invention, the specific method for collecting data is as follows: a 1080p, 60fps camera device is used to collect facial expression videos of DOC patients when they are subjected to pain stimulation, and a professional doctor applies the same pain stimulation to the nail beds of the patient's left and right hands at different time periods. The facial expression video data of the DOC patient is collected multiple times under the operation of a professional doctor.

[0015] As a preferred technical solution of the present invention, a video of the facial expression of a DOC patient when receiving pain stimulation is collected, the patient's face is located in the center of the video screen, and the video is recorded in a well-lit environment.

[0016] As a preferred technical solution of the present invention, a VAS scale is also included, which represents the pain intensity on a scale of 0 to 10, wherein 0 represents no pain and 10 represents extreme pain.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The auxiliary diagnosis system for patients with impaired consciousness based on pain intensity estimation is simple and easy to use, which improves the accuracy of diagnosis of DOC patients and provides more efficient tools for medical staff;

[0019] This invention is the application of computer vision technology in medical-assisted diagnosis, which helps medical staff to formulate treatment plans in a timely manner, thereby providing patients with more accurate diagnosis and treatment services;

[0020] By analyzing the facial expressions of DOC patients, the patients' pain perception ability can be quantified, thereby detecting their level of consciousness, which is simpler and more efficient;

[0021] The system is easy to operate and medical staff can quickly get started using it without complex training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall structure diagram of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 ,The present invention provides a consciousness disorder auxiliary diagnosis system based on pain intensity estimation, including a data acquisition module, a data analysis module, an interactive interface module, and a system management module;

[0025] The function of the data acquisition module is to collect data: the medical staff applies certain pain stimulation to the left finger of the DOC patient and records the facial expression video at intervals of 30 seconds. After the patient calms down, they apply certain pain stimulation to the right finger and record the facial expression video. The collected facial video of the DOC patient is input into the data analysis module.

[0026] The function of the data analysis module is to analyze data: after receiving the patient's facial expression video from the data acquisition module, it is necessary to perform pre-processing operations such as frame extraction, cropping, and data enhancement. Then, the deep learning network is used to extract features and predict VAS values, and the VAS values ​​of DOC patients during the period of pain stimulation are obtained. The experimental information and data are stored in the system for visualization in the interface.

[0027] The function of the interactive interface module is the interaction between the user and the interface: through the front-end rendering, the system can be visualized; the user interacts with the system through the mouse, and the user can view the DOC patient's name, gender, behavior scale diagnosis category, and VAS value when receiving pain stimulation on the interface; the interface will output the final judgment category of the DOC patient;

[0028] The system management module includes various aspects of the management system, such as: data storage module, system configuration management module and log recording and monitoring module; the data storage module is responsible for data storage in the system, including database management and file storage. Administrators can back up and restore data, manage storage space, and ensure data security and integrity; the system configuration management module allows administrators to configure various parameters and settings of the system, such as interface language, time format, default settings, etc.; the log recording and monitoring module records the system's operation log and operating status, including user login logs, error logs and system performance monitoring; administrators can view log information, identify problems and take timely measures to resolve them. Through the system management module, administrators can effectively manage and maintain the auxiliary diagnosis system for patients with impaired consciousness to ensure its stable operation.

[0029] Data collection:

[0030] A 1080p, 60fps camera device was used to collect videos of the facial expressions of DOC patients when they were subjected to pain stimulation. The patient's face should be in the center of the video screen, and the video should be recorded in a well-lit environment. Professional doctors applied the same pain stimulation to the nail beds of the left and right hands of the patients at different time periods. The facial expression video data of DOC patients were collected multiple times under the operation of professional doctors.

[0031] Data preprocessing:

[0032] To process the video containing the facial expressions of DOC patients, we first need to perform frame extraction. The specific process is as follows: Assuming that the original video is V, which contains N frames, the segmentation process can be expressed as dividing the original video V into several sub-videos V i , remove the first and last frames of each sub-video;

[0033] V i=V[t1+δ,t2-δ]

[0034] Where t1 and t2 are the start and end times of the segment, respectively, and δ is the time length to remove the first and last frames;

[0035] Then, the video is sampled at intervals, and a certain number of picture frames are extracted to construct samples to ensure the representativeness and effectiveness of the samples; i Extract the interval sampling frames, assuming the interval is k frames:

[0036] S i = {V i [t]∣t=t1+nk,n∈Z,t1+nk≤t2}

[0037] S i Indicates that from the sub-video V i The set of sample frames extracted from V i [t] represents sub-video V i The frame corresponding to time t; k represents the frame sampling interval, the unit is the number of frames, and n is an integer; then, the image frame and the marker points are rotated, translated, and scaled using the generalized Procrustes analysis (GPA) to ensure that the facial marker points in different image frames are aligned to similar positions; for each facial marker set {X1, X2, …, X m}, perform rotation, translation and scaling operations to align the marker points; set X i is the marker matrix of the i-th frame, and the GPA process can be expressed as:

[0038]

[0039] X i represents the marker matrix of the i-th frame, indicating the coordinates of all facial markers on the frame; R i Represents the rotation matrix of the i-th frame, used to rotate the marker matrix X i , aligning them to similar positions; T i Represents the translation matrix of the i-th frame, used to translate the marker matrix X i ;s i Indicates the scaling factor of the i-th frame, used to scale the marker matrix X i ; Represents the average position matrix of the marker points, which is the reference point matrix after alignment; Represents the square of the Frobenius norm, which is used to measure the difference between matrices;

[0040] Next, calculate the boundaries of these markers and obtain the minimum and maximum x, y coordinates for subsequent image frame cropping:

[0041]

[0042] X i [j, 0] represents the x coordinate of the jth marker point in the i-th frame; i [j, 1] represents the y coordinate of the jth marker point in the i-th frame; x min represents the minimum x coordinate of all markers in all frames; x max Indicates the maximum x coordinate of all markers in all frames; y min The minimum y coordinate of all markers in all frames; max Indicates the maximum y coordinate of all markers in all frames;

[0043] Then, the image frame is normalized to convert the pixel values ​​from the range of [0, 1] to the range of [-1, 1];

[0044] I′=2I-1

[0045] Among them, I is the original picture frame, and I′ is the standardized picture frame;

[0046] After that, the head is placed in the center of the image through center cropping operation, and the image size is adjusted to 224×224 pixels; assuming that the original cropping area is [x min :x max ]×[y min :y max ], the size of the cropped image I″ is w×h and is resized to 224×224:

[0047] I″=resize(I′[x min :x max ,y min :y max ], 224×224)

[0048] Pain intensity estimation model based on spatiotemporal feature extraction of facial expressions:

[0049] The model is divided into a spatial feature extraction module and a temporal feature extraction module. The spatial feature extraction module uses a multi-scale model to extract the global spatial information of facial expressions and uses a local attention network to extract local spatial information. After weighted fusion of the extracted global spatial information and local spatial information, the information is input into the temporal feature extraction module to capture the pain characteristics that change dynamically with facial expressions.

[0050] After the image frame is preprocessed, it is first input into the feature pre-extractor to obtain the mid-level face features; the feature pre-extractor consists of 1 2D convolution layer and 4 basic blocks. The basic block structure is the basic building block used in ResNet-18; the mid-level face features can usually capture the general features of the face, such as contour, eyes, nose and mouth; compared with the low-level features (such as edges, colors, etc.) and high-level features (such as expressions, identities, etc.), the mid-level features provide a higher-level description of the face, which helps to identify and distinguish different faces;

[0051] After extracting the middle-level face feature map, a dual-branch network is designed to process the extracted feature map, thereby obtaining global and local features at the same time; in the first branch, a multi-scale module is used to learn global multi-scale features; this module takes the entire extracted feature map as input and introduces a symmetric structure to learn multi-scale features. This method can ensure that both the previous and next feature subsets contain richer scale information; first, a feature map X is obtained through a 3×3 convolution operation; then, this feature map X is evenly divided into n feature map subsets along the channel axis, each subset is denoted by Xi, where i belongs to {1, 2, ..., n};

[0052] X i =split(X,axis=channels,n)

[0053] X represents the feature map obtained by the 3×3 convolution operation; split represents segmentation; n represents the number of feature map subsets into which the feature map is split according to the channel axis; axis=channels represents segmentation according to the channel axis;

[0054] The purpose of this is to split the original feature map by channel, so that each subset Xi maintains the same spatial size as the original feature map X, but only contains 1 / n of the original channels; then, a 3×3 convolution operation is performed on each feature map subset Xi; X′ i Represents the feature map after convolution;

[0055] X′ i =Conv 3×3 (X i )

[0056] X i represents the i-th feature map subset; X′ i Represents the feature map of the i-th feature map subset after the 3×3 convolution operation; Conv 3×3 Represents a 3×3 convolution operation

[0057] This step allows each subset to capture information independently, because the 3×3 convolution operation performed on each subset is designed to identify or extract features of different scales; ultimately, all convolution-processed feature map subsets will be reconnected to form a comprehensive multi-scale feature representation X′;

[0058] X′=concat(X′1,X′2,...,X′ n , axis = channels)

[0059] X′1,X′2,...,X′ n Represents all feature map subsets after 3×3 convolution operations; concat means reconnecting all subsets along the channel axis, and axis=channels means connecting along the channel axis to form a comprehensive multi-scale feature representation;

[0060] Since multi-scale convolution considers both deep semantics and shallow geometric features, the learned multi-scale features not only enhance the diversity of global features, but also reduce the sensitivity of deep convolution to occlusion and posture changes; therefore, the network can obtain a more comprehensive global feature representation;

[0061] The second branch is mainly used to obtain local features. First, the extracted feature map is divided into several non-overlapping regional feature maps along the spatial axis, and then several parallel local attention networks are used to learn local salient features. After two 3×3 convolutions, we can get F∈R H×WxC The feature map represented by the input image is then used as the attention network using a convolutional block attention module (CBAM). The CBAM can sequentially infer attention maps along two separate dimensions (channel and spatial), and then multiply the attention maps by the input feature map for adaptive feature refinement.

[0062] CBAM is divided into a channel attention module and a spatial attention module; the channel attention module M c (F) is calculated as:

[0063] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0064] Spatial attention module M s (F) is calculated as:

[0065] M s (F) = σ([Conv 7×7 (AvgPool(F))]+[Conv 7×7 (MaxPool(F))])

[0066] The final feature F″ is expressed as:

[0067] F′=M c (F)·F

[0068] F″=M s (F′)·F′

[0069] F represents the input feature map; M c (F) represents the output of the channel attention module; M s (F) represents the output of the spatial attention module; F′ represents the feature map after the channel attention module; F″ represents the feature map after the spatial attention module; σ represents the activation function (such as sigmoid), MLP represents multi-layer perceptron; AvgPool represents the average pooling operation, MaxPool represents the maximum pooling operation, Conv 7×7 Represents a 7×7 convolution operation.

[0070] The extracted global spatial information and local spatial information are weighted and fused and then input into the temporal feature extraction module; the invention uses a temporal Transformer encoder to extract temporal features to deeply explore the dynamic changes of facial expressions over time; the temporal Transformer uses a self-attention mechanism to capture long-distance dependencies in temporal data, and at the same time improves the representation ability of the model through a multi-head attention mechanism; after the image frame is processed by the spatial feature extraction module, spatial features can be obtained; then, these spatial features are combined with position codes and input into the temporal Transformer encoder; in the encoder, each spatial feature and position code is converted into a query, key and value vector, and multi-head attention calculation is performed; the multi-head attention calculation formula is:

[0071]

[0072] Where Q, K, and V represent query, key, and value vectors, respectively. k is the dimension of the key vector; T is used here to transpose the key vector matrix K so that it can perform dot product calculation with the query vector matrix Q; A softmax operation is performed on each row of the similarity matrix to convert the similarity into a weight. Softmax ensures that the sum of the weights of each row is 1, indicating the relative importance of different positions. The value vector matrix V is then weighted averaged using the softmax weights. The calculated weighted values ​​are used to generate intermediate features. After layer normalization and residual connection, the output of a single Transformer encoder block is obtained. The outputs of multiple encoders are averaged to obtain the final space-time representation. Through this process, the model can effectively capture the temporal dynamic characteristics of facial expression sequences.

[0073] Method for assessing the state of consciousness based on pain intensity estimation:

[0074] The consciousness level of MCS patients is higher than that of VS patients. Studies have shown that compared with MCS patients, VS patients involve narrower brain areas when perceiving the same pain stimulus, and the connection damage between pain-related areas in the brain of VS patients is more serious, which indicates that as consciousness increases, the perception of pain stimuli in DOC patients will also increase. The facial expressions of DOC patients when receiving the same pain stimulus are used to quantify the different perception abilities of different DOC patients to the same pain stimulus. Combined with the currently commonly used clinical behavioral diagnosis method for diagnosing different levels of consciousness of DOC patients but with a high misdiagnosis rate, the level of consciousness of DOC patients is assisted in the diagnosis, so as to distinguish whether the diagnosed DOC patients are MCS patients or VS patients.

[0075] Pain assessment usually relies on subjective self-assessment using different medical scales, among which the Visual Analog Scale (VAS) is one of the most common methods in clinical assessment; the VAS scale represents pain intensity on a scale of 0 to 10, where 0 represents no pain and 10 represents extreme pain; in the video sequence, each facial expression of pain corresponds to a VAS value; studies have shown that MCS patients and VS patients have different levels of consciousness and different pain perception abilities; there are significant differences in the pain expressions and VAS values ​​shown by DOC patients when receiving the same pain stimulus, among which the VAS values ​​of MCS patients are usually higher than those of VS patients; therefore, the VAS value can be used to measure the consciousness level of DOC patients; after many experiments, a VAS threshold value θ was selected as an evaluation indicator for measuring the consciousness level of DOC patients; when a DOC patient receives a pain stimulus, if the VAS value exceeds θ, the DOC patient is judged as an MCS patient, and if the VAS value is lower than θ, the patient is judged as a VS patient.

[0076] Although the embodiments of the present invention have been shown and described, as detailed above, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A consciousness disorder auxiliary diagnosis system based on pain intensity estimation, characterized by: Including data acquisition module, data analysis module, interactive interface module, and system management module; The data acquisition module is used to collect data: the medical staff applies pain stimulation to the left finger of the DOC patient and records the facial expression video at an interval of 30 seconds. After the patient calms down, the medical staff applies pain stimulation to the right finger and records the facial expression video. The collected facial video of the DOC patient is input into the data analysis module; The data analysis module is used to analyze data: after receiving the patient's facial expression video from the data acquisition module, it is necessary to extract frames, crop, and perform data enhancement preprocessing on them, and then extract features and predict VAS values ​​through a deep learning network to obtain the VAS values ​​of the DOC patients during the period of pain stimulation, and store the experimental information and data in the system; The interactive interface module is used for the interaction between the user and the interface: through the front-end rendering, the system is visualized, and the user interacts with the system through the mouse. The user views the name, gender, behavior scale diagnosis category, and VAS value of the DOC patient when receiving pain stimulation on the interface, and the final judgment category of the DOC patient is output on the interface; The system management module is used to manage and maintain the auxiliary diagnosis system for patients with impaired consciousness to ensure its stable operation. The system management module includes a data storage module, a system configuration management module, and a log recording and monitoring module; the data storage module is responsible for data storage in the system; the system configuration management module allows the administrator to configure various parameters and settings of the system; the log recording and monitoring module records the system's operation log and operating status.

2. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 1, characterized in that: The data storage module is responsible for data storage in the system, including database management and file storage.

3. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 1, characterized in that: The system configuration management module allows administrators to configure various parameters and settings of the system, including interface language, time format, and default settings; the logging and monitoring module records the system's operation logs and operating status, including user login logs, error logs, and system performance monitoring.

4. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 1, characterized in that: The specific method for collecting data is as follows: a 1080p, 60fps camera device is used to collect facial expression videos of DOC patients when they are subjected to pain stimulation. Professional doctors apply the same pain stimulation to the nail beds of the patients' left and right hands at different time periods. The facial expression video data of DOC patients are collected multiple times under the operation of professional doctors.

5. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 4, characterized in that: Videos of DOC patients' facial expressions when they were receiving painful stimulation were collected. The patient's face was in the center of the video screen, and the video was recorded in a well-lit environment.

6. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 1, characterized in that: The specific method of the pretreatment is as follows: To process the video containing the facial expressions of DOC patients, we first need to perform frame extraction. The specific process is as follows: Assume that the original video is V, which contains N frames; the segmentation process is represented by dividing the original video V into several sub-videos V i , remove the first and last frames of each sub-video; V i =V[t1+δ,t2-δ] Where t1 and t2 are the start and end times of the segment, respectively, and δ is the time length to remove the first and last frames; Then, the video is sampled at intervals, and picture frames are extracted to construct samples to ensure the representativeness and effectiveness of the samples. i Extract the interval sampling frames, assuming the interval is k frames: S i ={V i [t]∣t=t1+nk,n∈Z,t1+nk≤t2} S i Indicates that from the sub-video V i The set of sample frames extracted from V i [t] represents sub-video V i The frame corresponding to time t; k represents the frame extraction interval, the unit is the number of frames, and n is an integer; Next, we use the generalized Procrustes analysis (GPA) to rotate, translate, and scale the image frames and facial markers to ensure that the facial markers in different image frames are aligned to similar positions. For each facial marker set {X1, X2, …, X m }, perform rotation, translation and scaling operations to align the marker points; set X i is the marker matrix of the i-th frame, and the GPA process can be expressed as: X i represents the marker matrix of the i-th frame, indicating the coordinates of all facial markers on the frame; R i Represents the rotation matrix of the i-th frame, used to rotate the marker matrix X i , so that they are aligned to similar positions; T i Represents the translation matrix of the i-th frame, used to translate the marker matrix X i ;s i Indicates the scaling factor of the i-th frame, used to scale the marker matrix X i ; Represents the average position matrix of the marker points, which is the reference point matrix after alignment; Represents the square of the Frobenius norm, which is used to measure the difference between matrices; Next, calculate the boundaries of these markers and obtain the minimum and maximum x, y coordinates for subsequent image frame cropping: X i [j,0] represents the x coordinate of the jth marker point in the i-th frame; i [j,1] represents the y coordinate of the jth marker point in the i-th frame; min represents the minimum x coordinate of all markers in all frames; x max Indicates the maximum x coordinate of all markers in all frames; y min The minimum y coordinate of all markers in all frames; max Indicates the maximum y coordinate of all markers in all frames; Then, the image frame is normalized to convert the pixel values ​​from the [0,1] range to the [-1,1] range; I'=2I-1 Where, I is the original picture frame, and I' is the standardized picture frame; After that, the head is placed in the center of the image through center cropping operation, and the image size is adjusted to 224×224 pixels; assuming that the original cropping area is [x min :x max ]×[y min :y max ], the size of the cropped image I' is w × h and is resized to 224 × 224: I”=resize(I'[x min :x max ,y min :y max ],224×224) 7. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 1, characterized in that: It also includes a pain intensity estimation model based on the extraction of spatiotemporal features of facial expressions, and the corresponding contents are as follows: After the image frame is preprocessed, it is first input into the feature pre-extractor to obtain the middle-level facial features; The feature pre-extractor consists of 1 2D convolutional layer and 4 basic blocks. The basic block structure is the basic building block used in ResNet-18. The mid-level face features can capture the general features of the face. After extracting the middle-level face feature map, a dual-branch network is designed to process the extracted feature map, thereby obtaining global and local features at the same time; In the first branch, a multi-scale module is used to learn global multi-scale features. This module takes the entire extracted feature map as input and introduces a symmetric structure to learn multi-scale features. First, a 3×3 convolution operation is performed to obtain the feature map X. Then, this feature map X is evenly divided into n feature map subsets along the channel axis. Each subset is represented by Xi, where i belongs to {1, 2, ..., n}; X i =split(X,axis=channels,n) X represents the feature map obtained by the 3×3 convolution operation; split represents segmentation, n represents the number of feature map subsets into which the feature map is segmented along the channel axis; axis=channels represents segmentation along the channel axis; The original feature map is split by channel, so that each subset Xi maintains the same spatial size as the original feature map X, but only contains 1 / n of the original channels; then, a 3×3 convolution operation is performed on each feature map subset Xi, X' i Represents the feature map after convolution, X′ i =Conv 3×3 (X i ) X i represents the i-th feature map subset, X' i Represents the feature map of the i-th feature map subset after the 3×3 convolution operation, Conv 3×3 Represents a 3×3 convolution operation; This step allows each subset to capture information independently. Ultimately, all convolutional feature map subsets will be reconnected to form a comprehensive multi-scale feature representation X'. X′=concat(X′1,X′2,…,X′ n ,axis=channels) X'1,X'2,…,X' n Represents all feature map subsets after 3×3 convolution operations; concat means reconnecting all subsets along the channel axis, and axis=channels means connecting along the channel axis to form a comprehensive multi-scale feature representation; The second branch is used to obtain local features. First, the extracted feature map is divided into several non-overlapping regional feature maps along the spatial axis, and then several parallel local attention networks are used to learn local salient features. After two 3×3 convolutions, we can get F∈R H×WxC The feature map represented by the convolutional block attention module (CBAM) is then used as the attention network; CBAM can sequentially infer the attention map along two separate dimensions, and then multiply the attention map by the input feature map for adaptive feature refinement; The extracted global spatial information and local spatial information are weighted and fused and then input into the temporal feature extraction module; The temporal Transformer encoder is used to extract temporal features to deeply explore the dynamic changes of facial expressions over time; the temporal Transformer uses the self-attention mechanism to capture the long-distance dependencies in the temporal data, and the multi-head attention mechanism is used to improve the representation ability of the model; after the image frame is processed by the spatial feature extraction module, the spatial features can be obtained; These spatial features are then combined with the positional encodings and fed into the temporal Transformer encoder, where each spatial feature and positional encoding is converted into query, key, and value vectors and multi-head attention is calculated; the multi-head attention calculation formula is: Where Q, K, and V represent query, key, and value vectors, respectively. k is the dimension of the key vector, T is used here to transpose the key vector matrix K so that it can be dot-producted with the query vector matrix Q; A softmax operation is performed on each row of the similarity matrix to convert the similarity into a weight; softmax ensures that the sum of the weights of each row is 1, indicating the relative importance of different positions, and then the value vector matrix V is weighted averaged using the softmax weights. The calculated weighted values ​​are used to generate intermediate features. After layer normalization and residual connection, the output of a single Transformer encoder block is obtained; the outputs of multiple encoders are averaged to obtain the final space-time representation.

8. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 7, characterized in that: Individual dimensions include channels and spaces; CBAM is divided into a channel attention module and a spatial attention module. The channel attention module M c (F) is calculated as: M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) Spatial attention module M s (F) is calculated as: M s (F)=σ([Conv 7×7 (AvgPool(F))]+[Conv 7×7 (MaxPool(F))]) The final feature F" is expressed as: F′=M c (F)·F F″=M s (F′)·F′ F represents the input feature map, M c (F) represents the output of the channel attention module, M s (F) represents the output of the spatial attention module, F' represents the feature map after the channel attention module, F" represents the feature map after the spatial attention module, σ represents the activation function, MLP represents the multi-layer perceptron, AvgPool represents the average pooling operation, MaxPool represents the maximum pooling operation, Conv 7×7 Represents a 7×7 convolution operation.

9. The consciousness disorder auxiliary diagnosis system based on pain intensity estimation according to claim 1, characterized in that: Also included is the VAS scale, which indicates pain intensity on a scale of 0 to 10, where 0 indicates no pain and 10 indicates extreme pain.

Citation Information

Patent Citations

  • Cognitive state recognition system and method based on facial micro-expressions

    CN112201343A

  • Auxiliary diagnosis system for disturbance of consciousness based on auditory evoked electroencephalogram signal analysis

    CN113116306A

  • Micro-expression recognition-based consciousness assessment method and system

    CN114565957A

  • Auxiliary diagnosis system for patients with disturbance of consciousness based on P300 electroencephalogram and transfer learning

    CN116491901A

  • Facial expression-based disturbance of consciousness auxiliary diagnosis method

    CN117643453A

Cited By

  • Wearable pain relieving equipment and remote monitoring system thereof

    CN120478838A

  • Closed-loop olfactory stimulation awareness assessment method, equipment and medium

    CN120788582A

  • Facial dimension emotion-based disturbance of consciousness auxiliary diagnosis method and model

    CN121885152A