Device for classifying pain severity using facial expression, and method for classifying pain severity using the same

Abstract

The present invention relates to a pain severity classification device comprising: a data collection unit for collecting facial image data of a patient; a first analysis unit for analyzing first feature information from the facial image data; a second analysis unit for analyzing second feature information from the facial image data; a third analysis unit for analyzing the pain severity of the patient based on the first feature information and the second feature information; and a control unit for controlling the operation of the pain severity classification device of the patient.

Description

Technology Field

[0001] The present invention relates to a pain severity classification device and a pain severity classification method using the same, specifically for classifying pain severity using facial expressions. Background Technology

[0002] Pain occurs when the affected area is touched during patient treatment in the medical field. In such cases, it can be utilized as an auxiliary tool for medical staff, allowing them to distinguish the severity and criticality of symptoms based on the pain's intensity and to recognize injuries the patient may not be aware of.

[0003] Various pain assessment scales have been developed in relation to the diagnosis of such pain, including the NRS (Numeral Rating Scale), VAS (Visual Analog Scale), and FLACC (face, legs, activity, cry, consolability scale). However, since pain is felt directly only by the patient, there is a problem in that it is difficult to objectify pain, such as different scores being obtained on the aforementioned pain assessment scales even if pain occurs in the same part due to the same cause.

[0004] For example, when humans feel pain, various facial expression changes occur, such as grimacing or tensing the orbicularis oris muscle to endure the pain, and research is being conducted to diagnose pain by analyzing these changes. However, in the case of analyzing pain through facial expression changes, there are instances where it is difficult to analyze pain solely based on these changes, such as when the patient is accustomed to enduring pain.

[0005] The background paper of this study (De Sario GD, Haider CR, Maita KC, Torres-Guzman RA, Emam OS, Avila FR, Garcia JP, Borna S, McLeod CJ, Bruce CJ, Carter RE, Forte AJ. Using AI to Detect Pain through Facial Expressions: A Review. Bioengineering (Basel). 2023 May 2;10(5):548) is a paper that demonstrates that pain can be detected using facial expressions. The problem to be solved

[0006] The present invention aims to solve the problems of the aforementioned conventional technology by providing a pain severity classification device and a classification method for objectively evaluating the severity of pain in real time through artificial intelligence.

[0007] However, the technical problems that the embodiments of the present invention aim to solve are not limited to those described above, and other technical problems may exist. means of solving the problem

[0008] As a technical means for achieving the above-mentioned technical task, the first aspect of the present invention relates to a pain severity classification device comprising: a data collection unit for collecting facial image data of a patient; a first analysis unit for analyzing first feature information from the facial image data; a second analysis unit for analyzing second feature information from the facial image data; a third analysis unit for analyzing the pain severity of the patient based on the first feature information and the second feature information; and a control unit for controlling the operation of the pain severity classification device of the patient.

[0009] According to one embodiment of the present invention, the first analysis unit may include, but is not limited to: a landmark extraction unit that extracts face landmark coordinates from the face image data; a pupil detection unit that detects a pupil in the face image data; a pupil feature extraction unit that extracts pupil feature information from the pupil; and a physical feature extraction unit that extracts the first feature information based on the landmark coordinates and the pupil feature information.

[0010] According to one embodiment of the present invention, the face landmark coordinates may include coordinates obtained by converting the face image data into three-dimensional coordinates, but are not limited thereto.

[0011] According to one embodiment of the present invention, the pupil characteristic information may include at least one of pupil constriction speed, pupil dilation speed, pupil size change amount, pupil re-dilation time, pupil response delay time, and pupil average size, but is not limited thereto.

[0012] According to one embodiment of the present invention, the first feature information may include at least one of the patient's eye blinking rate, mouth opening rate, average pupil size, pupil size change amount, interocular distance, corner of the mouth angle, tension of the orbicularis oris muscle, distance between the eyebrows and eyes, and distance between the nose and lips, but is not limited thereto.

[0013] According to one embodiment of the present invention, the second analysis unit may include an emotional feature extraction unit that extracts the second feature information based on the face image data, but is not limited thereto.

[0014] According to one embodiment of the present invention, the emotional feature extraction unit may include at least one artificial intelligence model among MobileNetV2 and MobileNetV3, but is not limited thereto.

[0015] According to one embodiment of the present invention, the third analysis unit may include a comprehensive feature extraction unit that derives third feature information based on the first feature information and the second feature information, and a pain analysis unit that analyzes the severity of the patient's pain based on the third feature information, but is not limited thereto.

[0016] According to one embodiment of the present invention, the first analysis unit and the second analysis unit may include a preprocessing unit for preprocessing the face image data, but are not limited thereto.

[0017] According to one embodiment of the present invention, the pain severity classification device may further include a brainwave collection unit for collecting brainwaves of the patient, and a fourth analysis unit for analyzing the pain severity of the patient based on the brainwaves, the first feature information, and the second feature information, but is not limited thereto.

[0018] In addition, the second aspect of the present invention relates to a method for classifying pain severity using a pain severity classification device, comprising: (a) a step of collecting facial image data of a patient; (b) a step of analyzing first feature information and second feature information from the facial image data; and (c) a step of analyzing the pain severity of a patient based on the first feature information and the second feature information.

[0019] According to one embodiment of the present invention, the step (b) may include, but is not limited to, (b1) a step of extracting physical features of the patient's face from the face image data; and (b2) a step of extracting emotional features of the patient from the face image data.

[0020] According to one embodiment of the present invention, the step (c) may include, but is not limited to, (c1) a step of extracting third feature information based on the physical feature and the emotional feature; and (c2) a step of analyzing the severity of the patient's pain based on the third feature information.

[0021] According to one embodiment of the present invention, step (c2) may analyze the pain severity of the patient by inputting the third feature information into at least one machine learning classification model among Support Vector Machine (SVM), Random Forest, Gradient Boosting Machine (GBM), Multi-Layer Perceptron, CNN, RNN, RF, and XGBoost, but is not limited thereto.

[0022] In addition, the third aspect of the present invention relates to a recording medium in which a pain severity classification method according to the second aspect is implemented and recorded in the form of a program executed on a computer.

[0023] The means for solving the problem described above are merely exemplary and should not be interpreted as intended to limit the present invention. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and the detailed description of the invention. Effects of the invention

[0024] According to the means for solving the problem of the present invention described above, the pain severity classification device and method according to the present invention can classify pain severity more accurately than a pain severity classification method that uses only one of the three pieces of information by combining geometric information of the face, physical information of the pupil, and information on emotional expression in a mutually complementary manner.

[0025] In addition, the above-described pain severity classification device and method analyze pain severity based on image data of a patient's face acquired through a camera, and unlike body-attached biosignal measurement sensors, it facilitates easy analysis of pain severity in an everyday environment and can be extended not only to clinical settings but also to routine monitoring and pre-screening.

[0026] In addition, the above pain severity classification device and method can generate pain-related feature information by using not only facial image data but also brainwave signals together, thereby increasing the reliability and accuracy of the AI ​​pain diagnosis system to the highest level through a multimodal approach.

[0027] In addition, the above pain severity classification device and method can explicitly extract and combine information on interpretable features (such as facial grimace) when pain occurs, thereby allowing the AI ​​model to trace back or explain the basis for diagnosing pain severity, and thus comply with the explainable AI principles required in the medical field.

[0028] In addition, there is a limitation in that subjective pain scales cannot be used for patients who are unable or have difficulty communicating, such as infants, dementia patients, and unconscious patients. However, since the pain severity classification device and method according to the present invention automatically analyze physical facial reactions and physiological reactions (pupils), they can provide objective pain evaluation indicators even for patient groups for whom subjective pain scales cannot be used.

[0029] However, the effects obtainable from this invention are not limited to those described above, and other effects may exist. Brief explanation of the drawing

[0030] FIG. 1 is a schematic diagram of a pain severity classification device according to one embodiment of the present invention. FIG. 2 is a flowchart of a pain severity classification method according to one embodiment of the present invention. FIG. 3 is a schematic diagram of a pain severity classification method according to one embodiment of the present invention. Figure 4 is a schematic diagram of a pain severity classification method according to one embodiment of the present invention. Specific details for implementing the invention

[0031] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0032] However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly illustrate the present invention in the drawings, parts unrelated to the description have been omitted, and similar parts throughout the specification have been given similar reference numerals.

[0033] Throughout this specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other elements interposed between them.

[0034] Throughout the entire specification, when a component is described as being located "on," "on top," "on top," "under," "on bottom," or "on bottom" of another component, this includes not only cases where the component is in contact with the other component but also cases where another component exists between the two components.

[0035] Throughout this specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0036] Hereinafter, a pain severity classification device and a pain severity classification method using the same according to one embodiment and example of the present invention will be described.

[0037] As a technical means for achieving the above-mentioned technical task, the first aspect of the present invention relates to a pain severity classification device comprising: a data collection unit for collecting facial image data of a patient; a first analysis unit for analyzing first feature information from the facial image data; a second analysis unit for analyzing second feature information from the facial image data; a third analysis unit for analyzing the pain severity of the patient based on the first feature information and the second feature information; and a control unit for controlling the operation of the pain severity classification device of the patient.

[0038] Throughout the entire specification, the term "patient" refers to a person experiencing pain and may refer to a person whose possession of a disease has not been confirmed. That is, the pain severity classification device is intended to extract facial image data of a person (patient) from raw image data and classify the pain severity of said person (patient) based thereon.

[0039] The pain severity classification device according to the present invention classifies the pain severity of a patient based on facial image data captured of the patient's face. Specifically, since changes occur in various aspects such as facial expression, pupil size, and the shape of the corners of the mouth when a patient feels pain, the pain severity of the patient can be analyzed and classified by analyzing these elements.

[0040] Furthermore, a patient's facial expressions contain various emotions, and specific emotions can be derived from these expressions using an artificial intelligence model. Through this process, it is possible to extract the patient's emotions to determine whether the patient is in pain or not.

[0041] In this regard, while conventional technology can analyze emotions based on facial expressions, it has limitations, such as reduced reliability due to changes in lighting or the intentional alteration of expressions to conceal emotions. The present invention analyzes pain through changes in facial expressions and emotions, and utilizes pupil responses—which are difficult for the patient to control—to classify and diagnose the pain felt by the patient with maximum accuracy.

[0042] In this case, since the present invention diagnoses and analyzes the severity of pain in a non-face-to-face and non-contact state, it can be used to analyze the severity of a patient's pain even in non-face-to-face medical situations (such as telemedicine). As will be described later, pain severity can be accurately classified by diagnosing and analyzing pain severity through a comprehensive combination of facial expressions, emotional expressions, and pupil movements—rather than simply one of facial expressions or emotional expressions alone—and additionally utilizing brainwaves.

[0043] Throughout the entire specification, the description of 'pain severity' refers to the degree to which a patient feels pain and may be quantified. Based on the numerical value of the pain severity, patients may be classified as 'no pain,' 'mild pain,' 'moderate pain,' 'severe pain,' etc.

[0044] FIG. 1 is a schematic diagram of a pain severity classification device according to one embodiment of the present invention, and FIGS. 2 to 4 are flowcharts of a pain severity classification method according to one embodiment of the present invention. Specifically, FIG. 3 is a more detailed illustration of the process of FIG. 2, and FIG. 4 is a detailed illustration of a part of FIG. 3.

[0045] According to one embodiment of the present invention, the data collection unit may collect facial image data of a patient, but is not limited thereto.

[0046] The above data collection unit may include a camera device, and can collect the patient's face as a still image or video image using a video camera, smartphone, camcorder, etc.

[0047] In this regard, when the data collection unit collects the patient's face image data, if the patient's face is not illuminated by a certain level of lighting or higher, the patient's face may be captured darkly, and noise may occur in the data. Accordingly, the data collection unit may additionally include a sensor that detects ambient illumination, and may re-capture the patient's face image data based on the ambient brightness measured by the illumination detection sensor.

[0048] Face image data may refer to image data in which a human face is identified from the raw image data, but is not limited thereto.

[0049] According to one embodiment of the present invention, the first analysis unit may extract and analyze first feature information from the face image data, but is not limited thereto.

[0050] According to one embodiment of the present invention, the first analysis unit may include, but is not limited to: a landmark extraction unit that extracts face landmark coordinates from the face image data; a pupil detection unit that detects a pupil in the face image data; a pupil feature extraction unit that extracts pupil feature information from the pupil; and a physical feature extraction unit that extracts the first feature information based on the landmark coordinates and the pupil feature information.

[0051] The first analysis unit described above can collect information on the physical features of the face from the face image data. While the second analysis unit described later extracts the patient's emotional data from the face image data to analyze the pain felt by the patient, the first analysis unit can analyze the patient's pain through the physical information of the face from the face image data.

[0052] The above landmark extraction unit automatically extracts landmark coordinates corresponding to key points of the face from the above face image data. The above landmark coordinates may be coordinates indicating the location of eyelids, eyebrows, the tip of the nose, lip contours, jawlines, etc.

[0053] The aforementioned key points of the face may refer to body parts that constitute the face, such as the eyes, nose, mouth, eyebrows, lip contours, and jawline, and the landmark coordinates are the coordinates of the aforementioned key points of the face extracted based on any one of these or a specific point on the face.

[0054] According to one embodiment of the present invention, the face landmark coordinates may include three-dimensional coordinates of the face image data, but are not limited thereto. In this case, the three-dimensional coordinates may include x, y, and z coordinates, and the z coordinate may represent a relative depth from the camera.

[0055] Since the aforementioned facial image data includes depth information, the distance between the camera and a specific point on the face can be included as depth, and the degree to which the specific point protrudes from the face can be determined through the relative coordinates of these depths. In this case, the depth may appear smaller closer to the tip of the nose and larger closer to the eyebrows; however, depending on the criteria, it may appear larger closer to the tip of the nose and smaller closer to the eyebrows.

[0056] According to one embodiment of the present invention, the landmark extraction unit may extract the landmark coordinates using at least one artificial intelligence model among Mediapipe Face Mesh, MediaPipe Face Mesh, Dlib's shape predictor, OpenFace, FaceAlignment library, PRNet, 3DDFA, MTCNN, RetinaFace, InsightFace, OpenCV's Facemark API, and Deep Alignment Network, but is not limited thereto. In this case, the artificial intelligence model for extracting landmark coordinates may be referred to as the first model.

[0057] Generally, an artificial intelligence model comprises multiple neural network layers, and when data is input into the artificial intelligence model, it passes through the multiple neural network layers sequentially and is gradually transformed into abstracted features. For example, when raw image data is input into a first model, the initial neural network layer extracts basic features such as contours, lines, and colors from the input raw image data, the intermediate neural network layer extracts components corresponding to key points of the face, such as the eyes, nose, and mouth, from the raw image data, and the later neural network layer can learn and extract the human facial expression patterns, etc. The operation of such an artificial intelligence model can be applied in the same way to a second model that intends to extract pupil feature information from face image data, and a third model that intends to extract emotion expression information from face image data.

[0058] The above landmark extraction unit can precisely estimate landmark coordinates in real-time for each frame using the above artificial intelligence model (first model), and the more landmark coordinates used in the above artificial intelligence model, the more precisely the patient's face can be simulated.

[0059] The pupil detection unit extracts an eye portion from the face image data or landmark coordinates extracted by the landmark extraction unit, and detects a pupil area in the eye portion. The pupil detection can be performed using the landmark coordinate information or by using a separate eye detection algorithm.

[0060] Generally, pain is known to affect pupillary response; upon the onset of pain, there is a tendency for the initial pupillary diameter to increase, the amplitude of pupillary constriction to decrease, the pupillary response latency to increase, and the pupillary re-dilation time to shorten. Since these pupillary movements reflect physiological responses that are difficult to control by the patient's will, they are difficult for the patient to intentionally conceal, unlike facial expressions. Therefore, pain can be analyzed through the aforementioned pupillary movements or characteristic information.

[0061] According to one embodiment of the present invention, the pupil feature extraction unit may extract pupil feature information using at least one artificial intelligence model among U-Net, Attention U-Net, RITnet, EllSeg, DeepVOG, PupilNet, LSTM, GRU, Temporal Convolutional Network, Transformer, ResNet, EfficientNet, MobileNet, AutoEncoder-based models, and combinations thereof, but is not limited thereto. The artificial intelligence model for pupil feature extraction may be referred to as a second model.

[0062] The pupil feature extraction unit above can estimate the center coordinates and radius of the pupil in the eye region image detected by the pupil detection unit using the artificial intelligence model (second model), and can extract and analyze pupil feature information such as pupil movement from the face image data.

[0063] According to one embodiment of the present invention, the pupil characteristic information may include, but is not limited to, at least one of pupil constriction speed, pupil dilation speed, pupil size change amount, pupil re-dilation time, pupil response delay time, and pupil average size. The pupil characteristic information is a quantification of the change in pupil size over time and is a physiological indicator reflecting the response of the autonomic nervous system.

[0064] The above pupil constriction rate refers to the speed at which the pupil constricts, and the above pupil dilation rate refers to the speed at which the pupil dilates. The above pupil size change amount refers to the change in pupil size over a certain period of time and can be calculated as the standard deviation of pupil size.

[0065] The above pupil re-dilation time refers to the time it takes for the pupil to dilate again after constriction, and the time required to recover to a certain level of its original size upon dilation following constriction. For example, ReDIL25 is a type of pupil re-dilation time, referring to the time required for the pupil to recover to 25% of its original size after constricting due to light stimulation. The above pupil response latency (LAT) refers to the time it takes for the pupil's response to the stimulus to appear. The above pupil average size (INIT) refers to the average diameter of the pupil within the measurement interval.

[0066] The aforementioned pupil feature information can be inferred through facial image data.

[0067] The physical feature extraction unit extracts the first feature information based on physical features including the landmark coordinates and the pupil feature information. The physical feature extraction unit calculates geometric relationships such as distance, ratio, and angle between the landmark coordinates to extract feature values ​​representing local and structural changes in the face.

[0068] According to one embodiment of the present invention, the first feature information may include at least one of the patient's eye blinking rate, mouth opening rate, average pupil size, pupil size change amount, interocular distance, corner of the mouth angle, tension of the orbicularis oris muscle, distance between the eyebrows and eyes, and distance between the nose and lips, but is not limited thereto. At this time, the first feature information may mean a probability for a specific state (e.g., eye blinking probability), and may be a vectorized movement of key points of the face, so it may be called a first feature vector.

[0069] The Eye Aspect Ratio (EAR) mentioned above quantifies the degree of eye opening by calculating the ratio of Euclidean distances between specific eye landmarks, such as the upper and lower eyelid points and the outer corner of the eye. The Eye Aspect Ratio can be calculated into statistical values ​​such as the number of blinks per minute and the average opening time through changes over time.

[0070] The above Mouth Aspect Ratio (MAR) quantifies the degree of mouth opening by calculating the ratio of the distances between the upper and lower lip landmarks and the left and right corners of the mouth landmarks.

[0071] The above interbrow distance is calculated by determining the Euclidean distance between the inner landmarks of both eyebrows to reflect the degree of frowning. The above corner of the mouth angle is calculated by converting the change in the vertical position of the corner of the mouth into an angle relative to the horizontal, and the corner of the mouth angle may increase when pain occurs.

[0072] The distance between the eyebrows and the eyes refers to the distance between the eyebrow landmark and the eyelid landmark, and the distance between the nose and the lips refers to the distance between the nose tip landmark and the lip landmark.

[0073] The first feature information above may include the pupil feature information. That is, the first feature information may include the average pupil size, pupil size change amount, pupil response delay time, pupil re-dilation time, etc., and may be expressed as f_geom, f_pupil, etc.

[0074] Specifically, f_geom is a geometric feature vector of the face calculated based on facial landmark coordinates, and is a set of data that quantifies the distance, angle, ratio, etc., between landmarks. For example, EAR (Eye Aspect Ratio) is the ratio of the vertical length to the horizontal length of the eye, representing a value that quantifies the degree of eye opening and closing (blinking, squinting, etc.); MAR (Mouth Aspect Ratio) is the ratio of the vertical length to the horizontal length of the mouth, representing a value that quantifies the degree of mouth opening or the act of tightly closing the lips; Facial Distances refers to Euclidean distance values ​​between major parts that change during facial expression changes due to pain, such as the distance between the eyebrows, the distance between the eyebrows and the eyes, and the distance between the nose and the lips; and Facial Angles refers to angle values ​​that measure the inclination between landmarks, such as the drooping or pulling of the corners of the mouth.

[0075] In addition, f_pupil is physiological response data extracted from the pupil detection unit and represents a feature vector reflecting the autonomic nervous system response. For example, f_pupil includes Average Pupil Size, Pupil Size Variation / Amplitude, Pupil Response Latency, and Pupil Re-dilation Time.

[0076] The above first feature information can be configured in the form of a vector, which can be referred to as a first feature vector. The above first feature vector is a plurality of calculated scalar feature values ​​arranged in order.

[0077] The first analysis unit can analyze the movement of the main points of the face and the pupil after detecting landmark coordinates corresponding to the main points of the face in the face image data and the pupil area in the face image data. When pain occurs in a patient, changes occur in various parts of the face such as the glabella, eyes, nose, cheeks, mouth, and pupil, so the first analysis unit can diagnose the pain based on physical changes in the main points of the face and the movement of the pupil.

[0078] In addition to the landmark coordinates and pupil feature information, the first analysis unit may further analyze at least one of the following: a three-dimensional rotation angle of the head calculated based on landmarks, or head pose information characterized by an avoidance reaction of lowering or tilting the head back when pain occurs, gaze direction information which is information regarding whether the part the patient is looking at is the painful area or avoiding the eyes using landmarks of the pupil and eyes, and skin color change information estimated based on changes in heart rate or blood flow by detecting minute skin color changes in the face image data, but is not limited thereto.

[0079] Generally, when pain is felt, avoidance reactions such as turning the head, lowering the head, or tilting it backward may occur, and various gaze patterns such as avoiding or diverting the gaze from the painful area may appear. Additionally, when pain occurs, facial flushing or paleness may be induced by autonomic nervous system responses. The first analysis unit described above can further analyze various physical information generated by the aforementioned pain (head pose information, gaze direction information, and skin color change information, etc.).

[0080] According to one embodiment of the present invention, the second analysis unit may include an emotional feature extraction unit that extracts the second feature information based on the face image data, but is not limited thereto.

[0081] The emotional feature extraction unit described above is intended to extract a face portion from the face image data and analyze the emotions expressed on the face. Specifically, it can analyze emotional expressions by finding the landmark coordinates of key points on the face and analyzing the location, angle, and movement of the key points on the face.

[0082] According to one embodiment of the present invention, the emotional feature extraction unit may use at least one artificial intelligence model among MobileNetV2 and MobileNetV3, but is not limited thereto. MobileNetV2 and MobileNetV3 are emotion classification models based on a lightweight CNN architecture.

[0083] In addition, the emotional feature extraction unit may use an artificial intelligence model including at least one of MobileNetV2, MobileNetV3, ShuffleNet, EfficientNet, SqueezeNet, VGG, ResNet, Inception, DenseNet, SE-ResNet, CBAM, Vision Transformer, Swin Transformer, DeiT, EmotiNet, DeXpression, Multi-task CNN, 3D CNN, Two-Stream CNN, LSTM-CNN combined model, and Facial Graph Network, but is not limited thereto. At this time, an artificial intelligence model that analyzes emotional expressions in facial image data, such as at least one artificial intelligence model among MobileNetV2, MobileNetV3, ShuffleNet, EfficientNet, SqueezeNet, VGG, ResNet, Inception, DenseNet, SE-ResNet, CBAM, Vision Transformer, Swin Transformer, DeiT, EmotiNet, DeXpression, Multi-task CNN, 3D CNN, Two-Stream CNN, LSTM-CNN combined model, and Facial Graph Network, can be referred to as a third model.

[0084] The third model described above can recognize parts corresponding to a human face in a video. Specifically, it recognizes shapes resembling eyes, nose, mouth, ears, etc. in the video as a face, and can analyze the facial expressions of the face to analyze emotional expressions. In this case, the emotional expression may be extracted from multiple data sets that have been pre-trained in the third model and have been trained to identify specific emotions based on facial expressions captured in specific situations.

[0085] That is, the emotional feature extraction unit can derive emotional expression information of the patient through the facial image data. In this regard, since the third model expresses the emotions analyzed from the facial image data as probabilities, the second feature information can be configured in the form of a vector containing probability values ​​for each emotional state, and this can be referred to as a second feature vector. For example, the second feature vector may be in the form of [P(sadness), P(pain), P(anger), P(neutral)].

[0086] In relation to the above second feature information, the higher the probability of feeling emotions related to pain (e.g., suffering), the more likely it is to be judged that pain is present or severe, and the higher the probability of feeling emotions unrelated to pain (e.g., joy), the more likely it is judged that pain is not felt.

[0087] The above second feature information can be expressed as P(emo).

[0088] Specifically, P(emo) (Emotion Probability Features) refers to probability vectors for each emotion output by the emotion analysis AI model of the emotion feature extraction unit. At this time, it can be composed of probability values ​​for seven emotional states according to the classification criteria of the image uploaded as face video data. The seven emotional states can be classified as angry, disgust, fear, happy, sad, surprise, and neutral, and it can be inferred that the pain is more severe as the probability of being happy is low and the probability of being angry, disgust, fear, and sadness is high.

[0089] For example, P(emo) can be expressed as [P(angry), P(disgust), P(fear), P(happy), P(sad), P(surprise), P(neutral)] and as a number as [3.51, 0.00, 1.95, 0.10, 18.18, 0.02, 76.23]. The above numbers represent the probability (%) of the corresponding emotion or a normalized value between 0 and 1.

[0090] The first feature information (first feature vector) extracted from the first analysis unit refers to physical changes in key points of the face that can be identified in the face image data, and the second feature information (second feature vector) extracted from the second analysis unit refers to emotional expressions inherent in the face image data using an artificial intelligence model. At this time, since the first feature information includes not only grimacing, the degree of upward tilt of the corners of the mouth, and the degree of opening of the mouth, but also changes in the pupil (contraction, dilation, etc.) caused by pain, pain can be analyzed through the movement of the pupil caused by physiological changes, even if the patient is accustomed to hiding their emotions.

[0091] Furthermore, if the physical feature extraction unit of the first analysis unit intends to analyze pain based on local features of the face, the emotional feature extraction unit of the second analysis unit analyzes pain based on global features of the entire face.

[0092] Specifically, the emotional feature extraction unit quantifies global expression information, such as subtle muscle tremors, complex facial expressions, and changes in skin texture appearing over the entire face, by learning it through a deep learning model, and forms a mutually complementary relationship with the local features of the first analysis unit, which focuses on physical changes in specific areas such as the blinking rate and the mouth opening rate.

[0093] According to one embodiment of the present invention, the first analysis unit and the second analysis unit may include a preprocessing unit for preprocessing the face image data, but are not limited thereto. The preprocessing unit may perform preprocessing such as brightness correction, noise removal, and size adjustment of the face image data.

[0094] Regarding the above-mentioned face image data, the video or photo may contain shadows caused by objects, patterns of trees, or drawings, in addition to people. In such cases, only the face image data of a specific person must be extracted, and to do this, it is necessary to preprocess the face image data and then derive landmark coordinates, pupil feature information, and emotion expression information.

[0095] To do this, it is necessary to check the ratio of skin color within a predetermined distance centered on a key point of the face, and at this time, the ratio of skin color within the predetermined distance may vary depending on the type of key point of the face. That is, the preprocessing unit can extract only face image data from raw image data and then adjust the brightness, noise, size, etc. of the face image data.

[0096] According to one embodiment of the present invention, the third analysis unit may include a comprehensive feature extraction unit that derives third feature information based on the first feature information and the second feature information, and a pain analysis unit that analyzes the severity of the patient's pain based on the third feature information, but is not limited thereto.

[0097] The above-described comprehensive feature extraction unit derives the third feature information by combining the first feature information and the second feature information. Specifically, the third feature information may be obtained by sequentially concatenating related information among the first feature information and the second feature information, or by applying different weights to each of the first feature information and the second feature information and performing a weighted sum.

[0098] For example, if the first feature information includes N feature values ​​and the second feature information includes M feature values, the third feature information may be composed of a vector including (N+M) feature values.

[0099] According to one embodiment of the present invention, when the comprehensive feature extraction unit combines the first feature information and the second feature information, it may apply different weights according to the importance of each feature information, but is not limited thereto. The weights may be determined through prior learning, set manually, or dynamically adjusted through an attention mechanism. The importance may also be determined through prior learning or set manually.

[0100] According to one embodiment of the present invention, the comprehensive feature extraction unit may perform a normalization process before combining the first feature information and the second feature information, but is not limited thereto. The normalization may include at least one of Min-Max normalization, Z-score normalization, or L2 normalization, and thereby the efficiency of feature combination can be improved by adjusting feature values ​​with different scales to the same range.

[0101] According to one embodiment of the present invention, the comprehensive feature extraction unit may include at least one of a fully connected layer, a feature fusion layer, or a 1x1 convolutional layer to non-linearly combine the first feature information and the second feature information, but is not limited thereto.

[0102] The above-mentioned third analysis unit is integrated feature information that includes physical changes in the local structure of the face, pupil response, and global emotional expression information of the face, and can be referred to as an integrated feature vector or a final feature vector. The accuracy of pain classification and diagnosis can be improved by using third feature information that includes both the first feature information and the second feature information, compared to classifying pain using only one of the first feature information and the second feature information.

[0103] The above third feature information can be referred to as a final feature vector and includes all structural information, physiological information, and expressive information of the face. For example, the above final feature vector may be in the form of [f_geom_1, ... , f_geom_N, f_pupil_1, ... , f_pupil_K, P(emo_1), ... , P(emo_M)].

[0104] The above pain analysis unit analyzes the severity of the patient's pain based on the above third feature information.

[0105] According to one embodiment of the present invention, the pain analysis unit may analyze pain severity through at least one machine learning classification model among Support Vector Machine (SVM), Random Forest, Gradient Boosting Machine (GBM), Multi-Layer Perceptron, CNN, RNN, RF, and XGBoost, but is not limited thereto. The machine learning classification model used in the pain analysis unit may be referred to as the fourth model for convenience.

[0106] The above pain analysis unit can classify pain severity, such as 'no pain', 'mild pain', 'moderate pain', and 'severe pain', based on the input third feature information.

[0107] The third feature information (final vector) is constructed by simply concatenating the first feature information (geometric and physiological-based features) and the second feature information (emotion analysis features), which have undergone a normalization process, in the vector dimension direction.

[0108] For example, the first feature information, which combines geometric information (f_geom) such as the distance between facial feature points (landmarks) representing lip pressure, mouth opening, and muscle tension around the mouth, and pupil feature information (f_pupil) reflecting behavioral and physiological characteristics such as pupil dilation and variability, and the second feature information, which consists of emotional characteristics (P(emo)) including probability elements for emotion classes such as 'pain', 'sadness', 'anxiety', and 'anger', are integrated into a single vector (third feature information) for diagnosing pain severity. In other words, the final vector is formed by simply concatenating (or performing element-wise matrix operations) the first and second feature information in the vector dimension direction, and the third feature information is constructed as a single integrated feature representation of behavioral / physiological information and emotional information.

[0109] The above third feature information (final vector) is input into the pain analysis unit of the pain severity diagnosis device. The pain analysis unit can derive pain severity using a pain classification model (MLP, SVM, or time-series-based deep learning model) learned using a preset clinical threshold or pain-related scale test results (e.g., NRS, VAS, FLACC).

[0110] Specifically, if the value of each feature element exceeds a clinically established reference value or shows a specific combination pattern, for example, if high muscle tension around the mouth, prominent pupil dilation, and a high probability of negative emotion are observed, the pain analysis unit may determine that pain exists in the patient and that the severity of the patient's pain is increasing.

[0111] In addition, the pain analysis unit can pre-train a model (4th model) using pain scale test scores (NRS / VAS, PCS) obtained from a patient as correct labels. The 4th model can automatically learn the statistical and non-linear relationship between the third feature information input into the pain analysis unit and the pain scale scores to calculate pain grades consisting of normal, mild, moderate, and severe.

[0112] According to one embodiment of the present invention, the pain severity classification device may additionally include a brainwave collection unit for collecting the patient's brainwaves, and a fourth analysis unit for analyzing the patient's pain severity based on the brainwaves, the first feature information, and the second feature information, but is not limited thereto. In this case, the fourth analysis unit may analyze the patient's pain severity based on the brainwaves and the third feature information.

[0113] The pain severity classification device according to the present invention analyzes the pain severity of a patient in a non-face-to-face and non-contact manner. However, when diagnosing and analyzing a patient's pain in a face-to-face medical setting, there is no need to analyze the pain severity non-face-to-face; therefore, to achieve an accurate analysis of pain severity, the patient's brainwaves can be collected and the pain severity can be analyzed.

[0114] The above-mentioned brainwave collection unit collects the patient's brainwave signals through an EEG sensor. The above-mentioned fourth analysis unit generates a multimodal feature vector by combining the EEG feature vector extracted from the brainwaves with the above-mentioned first feature information and the above-mentioned second feature information, and can analyze the severity of pain based thereon.

[0115] Through this, the reliability and accuracy of pain diagnosis can be improved via a multimodal approach that analyzes precise biosignals, such as brainwaves, together with highly accessible imaging signals.

[0116] In addition to brain waves, the pain severity classification device described above can diagnose and analyze pain severity using various biosignals such as voice, electrocardiogram, electromyogram, skin conductivity, blood pressure, breathing patterns, and blood oxygen saturation.

[0117] In particular, in the case of voice, it can be collected together with facial image data. The pain severity classification device may further include a fifth analysis unit that extracts the patient's voice from facial image data in video format and analyzes the voice to analyze the patient's pain severity.

[0118] The brainwaves analyzed by the fourth analysis unit and the voice analyzed by the fifth analysis unit can be used for pain severity analysis together with the first feature information and the second feature information.

[0119] According to one embodiment of the present invention, the pain severity classification device may additionally include a comparison unit that compares the pain severity (first pain severity) analyzed based on the first feature information and the second feature information with the pain severity (second pain severity) directly evaluated by the patient, but is not limited thereto.

[0120] The above first pain severity is calculated by inputting the patient's facial image data into an artificial intelligence model (models 1 through 3) to derive first feature information and second feature information, and then inputting it into the above fourth model, thereby utilizing an artificial intelligence model. That is, there may be a difference between the above first pain severity and the pain severity (second pain severity) directly evaluated by the patient.

[0121] Since the first pain severity and the second pain severity can be quantified, the pain felt by the patient can be analyzed in more detail based on the difference between the two pain severitys, thereby contributing to the relief of the patient's pain.

[0122] In addition, the second aspect of the present invention relates to a method for classifying pain severity using a pain severity classification device, comprising: (a) a step of collecting facial image data of a patient; (b) a step of analyzing first feature information and second feature information from the facial image data; and (c) a step of analyzing the pain severity of a patient based on the first feature information and the second feature information.

[0123] Regarding the method for classifying pain severity according to the second aspect of this invention, detailed explanations have been omitted for parts that overlap with the first aspect of this invention; however, even if such explanations are omitted, the contents described in the first aspect of this invention may be applied identically to the second aspect of this invention.

[0124] First, (a) collect the patient's facial image data. The collection of facial image data can be performed through a data collection unit.

[0125] At this time, the collection of the face image data can be performed by collecting raw image data including the patient's face image data.

[0126] The above face image data or raw image data may be a still image or a video.

[0127] Next, (b) the first feature information and the second feature information are analyzed from the above face image data.

[0128] According to one embodiment of the present invention, the step (b) may include, but is not limited to, (b1) a step of extracting physical features of the patient's face from the face image data; and (b2) a step of extracting emotional features of the patient from the face image data.

[0129] The above step (b1) can be performed by the first analysis unit, and the above step (b2) can be performed by the second analysis unit.

[0130] Step (b1) above includes the steps of extracting face landmark coordinates from the face image data, detecting the pupil, and extracting pupil features. Step (b1) above extracts physical features including eye blinking ratio, mouth opening ratio, average pupil size, etc. based on the extracted landmark coordinates and pupil features.

[0131] The pupil detection described above is performed from face image data or the eye region among the detected face regions. The pupil feature extraction is performed by applying image processing techniques, such as thresholding and ellipse fitting, to the detected eye region image to estimate the center coordinates and radius of the pupil, and by quantifying the change in pupil size over time.

[0132] The above physical features may include geometric features such as eye blinking rate, mouth opening rate, interocular distance, corner of the mouth angle, tension of the orbicularis oris muscle, distance between eyebrows and eyes, distance between nose and lips, and pupil features such as average pupil size, pupil size change amount, pupil response delay time, pupil re-dilation time.

[0133] Step (b2) above is a step of inputting the facial image data into a pre-trained emotion analysis model to derive emotion expression information and extracting emotional features therefrom. The emotional features can be defined as the probability of the emotion felt by the patient in the patient's facial image data.

[0134] Step (b1) and Step (b2) above may be performed simultaneously or sequentially.

[0135] Next, (c) the severity of the patient's pain is analyzed based on the first feature information and the second feature information.

[0136] According to one embodiment of the present invention, the step (c) may include, but is not limited to, (c1) a step of extracting third feature information based on the physical feature and the emotional feature; and (c2) a step of analyzing the severity of the patient's pain based on the third feature information.

[0137] The above step (c1) involves extracting third feature information by combining the first feature information and the second feature information, wherein combining means combining the two feature information in various ways, such as simple connection or weighted sum.

[0138] The above third feature information refers to feature information that includes local structural changes, physiological responses, and global expression changes of the face.

[0139] The above step (c2) is a step of analyzing the severity of the patient's pain by inputting the above third feature information into a machine learning classification model.

[0140] According to one embodiment of the present invention, step (c2) may analyze the pain severity of the patient by inputting the third feature information into at least one machine learning classification model among Support Vector Machine (SVM), Random Forest, Gradient Boosting Machine (GBM), Multi-Layer Perceptron, CNN, RNN, RF, and XGBoost, but is not limited thereto.

[0141] Additionally, step (a) may further include a step of collecting at least one of various biosignals among the patient's voice, brainwave, electrocardiogram, electromyogram, skin conductivity, blood pressure, breathing pattern, and blood oxygen saturation, and steps (b) and (c) may analyze the biosignals to analyze the patient's pain severity along with first feature information and second feature information.

[0142] The pain severity classification method according to the present invention can overcome the limitations of conventional technology that uses only one scale or type of information by complementarily combining local structural information of the face, physiological pupil information reflecting autonomic nervous system activity, and global expressive emotional information.

[0143] In addition, the third aspect of the present invention relates to a recording medium in which a pain severity classification method according to the second aspect is implemented and recorded in the form of a program executed on a computer.

[0144] The present invention is to be explained in more detail through the following examples, but the following examples are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0145] [Example]

[0146] First, patient facial image data was collected via a camera, while ensuring data quality by detecting ambient brightness using an illuminance sensor.

[0147] Next, the above face image data was input into the first analysis unit and the second analysis unit.

[0148] The first analysis unit extracts the patient's face portion from the face image data, then collects the coordinates of key points on the patient's face, such as the eyes, nose, and mouth, as landmark coordinates, and extracts geometric features such as eye blinking, mouth opening, and frowning by tracking changes in the landmark coordinates. In addition, physiological responses such as pupil rotation / dilation speed and changes in pupil size, which are difficult for the patient to intentionally conceal, are extracted as pupil feature information. The geometric features and pupil feature information based on the landmark coordinates represent physical feature information.

[0149] In addition, the second analysis unit utilized a pre-trained emotion analysis model to extract probability information of anger, disgust, fear, happiness, sadness, surprise, and neutrality as emotional feature information.

[0150] Next, the physical feature information and emotional feature information were combined to generate third feature information. The third feature information was input into a deep learning / machine learning classification model of the pain analysis unit to finally analyze the pain severity.

[0151] At this time, it can be expanded to a multimodal method that enhances the accuracy of pain severity diagnosis by additionally combining brainwave signals or voice data as needed.

[0152] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical concept or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0153] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents thereof should be interpreted as being included within the scope of the present invention.

Claims

Claim 1 A device for classifying the severity of pain comprises: a data collection unit for collecting facial image data of a patient; a first analysis unit for analyzing first feature information from the facial image data; a second analysis unit for analyzing second feature information from the facial image data; a third analysis unit for analyzing the severity of the patient's pain based on the first feature information and the second feature information; and a control unit for controlling the operation of the patient's severity classification device. The first analysis unit comprises: a landmark extraction unit for extracting facial landmark coordinates from the facial image data; a pupil detection unit for detecting a pupil in the facial image data; a pupil feature extraction unit for extracting pupil feature information from the pupil; and a physical feature extraction unit for extracting the first feature information based on the landmark coordinates and the pupil feature information. A pain severity classification device comprising: a face landmark coordinate, wherein the face landmark coordinate comprises a three-dimensional coordinate system of the face image data; and wherein the physical feature extraction unit calculates the geometric relationship between the face landmark coordinates to analyze pain through physical changes based on feature values ​​representing local and structural changes of the face and pupil movement. Claim 2 delete Claim 3 delete Claim 4 A pain severity classification device according to claim 1, wherein the pupil characteristic information includes at least one of pupil constriction speed, pupil dilation speed, pupil size change amount, pupil re-dilation time, pupil response delay time, and pupil average size. Claim 5 A pain severity classification device according to claim 4, wherein the first feature information comprises at least one of the patient's eye blinking rate, mouth opening rate, average pupil size, pupil size change amount, interocular distance, corner of the mouth angle, tension of the orbicularis oris muscle, distance between the eyebrows and eyes, and distance between the nose and lips. Claim 6 A pain severity classification device according to claim 1, wherein the second analysis unit comprises an emotional feature extraction unit that extracts the second feature information based on the face image data. Claim 7 In claim 6, the above emotional feature extraction unit comprises at least one artificial intelligence model among MobileNetV2 and MobileNetV3, a pain severity classification device. Claim 8 A pain severity classification device according to claim 1, wherein the third analysis unit comprises a comprehensive feature extraction unit that derives third feature information based on the first feature information and the second feature information, and a pain analysis unit that analyzes the pain severity of the patient based on the third feature information. Claim 9 A pain severity classification device according to claim 1, wherein the first analysis unit and the second analysis unit include a preprocessing unit for preprocessing the face image data. Claim 10 In claim 1, the pain severity classification device further comprises a brainwave collection unit for collecting the brainwaves of the patient, and a fourth analysis unit for analyzing the pain severity of the patient based on the brainwaves, the first feature information, and the second feature information. Claim 11 A method for classifying pain severity using a pain severity classification device, comprising: (a) collecting facial image data of a patient; (b) analyzing first feature information and second feature information from the facial image data; and (c) analyzing the pain severity of the patient based on the first feature information and the second feature information; wherein step (b) comprises: (b1) extracting physical features of the patient's face from the facial image data; and (b2) extracting emotional features of the patient from the facial image data; and step (b1) comprises: extracting facial landmark coordinates from the facial image data; calculating geometric relationships between the facial landmark coordinates; detecting pupils from the facial image data; extracting pupil feature information; and analyzing pain through physical changes based on the geometric relationships and the pupil feature information. Claim 12 delete Claim 13 A method for classifying pain severity according to claim 11, wherein step (c) comprises: (c1) a step of extracting third feature information based on the physical features and the emotional features; and (c2) a step of analyzing the pain severity of the patient based on the third feature information. Claim 14 In claim 13, the above step (c2) is a method for classifying pain severity, wherein the above third feature information is input into at least one machine learning classification model among Support Vector Machine (SVM), Random Forest, Gradient Boosting Machine (GBM), Multi-Layer Perceptron, CNN, RNN, RF, and XGBoost to analyze the pain severity of the patient. Claim 15 A recording medium in which the pain severity classification method according to Paragraph 11 is implemented and recorded in the form of a program executed on a computer.

Images