Determination of the level and / or source of pain from micro-expressions using machine learning

Abstract

An apparatus includes a facial expression detector device and a computing device configured to receive a facial pattern input from the facial expression detector, identify one or more micro-expressions in the facial pattern input, and predict one or more of a level of pain and a source of pain from the identified one or more micro-expressions. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

Description

[0001] DETERMINATION OF THE LEVEL AND / OR SOURCE OF PAIN FROM MICRO-EXPRESSIONS USING MACHINE LEARNING

[0002] TECHNICAL FIELD

[0003] The aspects of the disclosed embodiments relate generally to contact-free estimation of the level and the source of pain of a person, and in particular, determining the source of pain from micro-expressions a person makes.

[0004] BACKGROUND

[0005] The face of a person is a tangible projector panel of the mechanisms that govern emotional behaviors and health. Different sources of pain can potentially produce different visual symptoms, including facial expressions. Such facial expressions can include, but are not limited to, prominent forehead, eye squeeze, naso-labial furrow, taut tongue, and an angular opening of the mouth. Facial expressions of pain can provide a reliable and accurate source of information regarding an individual’s health condition.

[0006] Traditional methods of facial expression analysis to determine a source of pain can be unreliable because different sources of pain can share similar overall facial expression deformation. Additionally, changes in facial expressions caused by different source of pain can be very subtle, fleeting and non-prototypic.

[0007] While subtle expressions that occur for each specific pain source can be learned, micro-expressions, which last only for a short moment, tend to be of lower intensity and difficult to read. Thus, the accuracy, robustness, and complexity of facial expression analysis are an issue when multiple contact-free modalities are applied for real-world pain intensity assessment.

[0008] It would be advantageous to be able to unobtrusively estimate the level and source of pain of individuals that are incapable of articulating and expressing their pain experiences. Accordingly, it would be desirable to provide a method and apparatus that addresses at least some of the problems described above.

[0009] SUMMARY

[0010] The aspects of the disclosed embodiments are generally directed to visual assessment of the source of pain based on facial micro-expressions. The aspects of the disclosed embodiments use contact free methodologies for determining the source of pain from analysis of facial images together with machine learning and artificial intelligence. According to a first aspect, the above and further objectives and advantages are obtained by an apparatus. In one embodiment, the apparatus includes a facial expression detector device; and a computing device configured to: receive a facial pattern input from the facial expression detector; identify one or more micro-expressions in the facial pattern input; and predict one or more of a level of pain and a source of pain from the identified one or more micro-expressions. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0011] In a possible implementation form the computing device is further configured to extract at least one physiological signal from the identified one or more micro-expressions in the facial pattern input and use the extracted at least one physiological signal in the prediction of the one or more of the level of pain and the source of pain. The visual data can be combined with physiological signals to make the pain inference more reliable.

[0012] In a possible implementation form the computing device is further configured to estimate a pain level from the identified one or more micro-expressions in the facial pattern input and use the estimated pain level in the prediction of the one or more of the level of pain and the source of pain. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0013] In a possible implementation form the extracted at least one physiological signal and the estimated pain level is used in the prediction of the one or more of the level of pain and the source of pain. The visual data can be combined with physiological signals to make the pain inference more reliable.

[0014] In a possible implementation form the facial expression detection device comprises a camera. The aspects of the disclosed embodiments enable unobtrusively estimating the level and source of pain of individuals that are incapable of articulating and expressing their pain experiences.

[0015] In a possible implementation form the facial expression detection device comprises a wearable device. The aspects of the disclosed embodiments enable unobtrusively estimating the level and source of pain of individuals that are incapable of articulating and expressing their pain experiences.

[0016] In a possible implementation form the facial expression detection device comprises a mobile communication device. The aspects of the disclosed embodiments enable unobtrusively estimating the level and source of pain of individuals that are incapable of articulating and expressing their pain experiences. In a possible implementation form the computing device comprises a convolutional neural network. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0017] According to a second aspect, the above and further objectives and advantages are obtained by a computer implemented method. In one embodiment, the computer implemented method includes predicting a level of pain and a source of pain from an image of a face by receiving a facial pattern input; identifying one or more micro-expressions in the facial pattern input; and predicting one or more of a level of pain and a source of pain from the identified one or more micro-expressions. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0018] In a possible implementation form, the method further includes extracting at least one physiological signal from the identified one or more micro-expressions in the facial pattern input and using the extracted at least one physiological signal in the prediction of the one or more of the level of pain and the source of pain. The visual data can be combined with physiological signals to make the pain inference more reliable.

[0019] In a possible implementation form, the method further includes estimating a pain level from the identified one or more micro-expressions in the facial pattern input and using the estimated pain level in the prediction of the one or more of the level of pain and the source of pain. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0020] In a possible implementation form, the method further includes using the extracted at least one physiological signal and the estimated pain level is used in the prediction of the one or more of the level of pain and the source of pain. The visual data can be combined with physiological signals to make the pain inference more reliable.

[0021] In a possible implementation form, the method further includes using a wearable device to provide the facial input image. The aspects of the disclosed embodiments enable unobtrusively estimating the level and source of pain of individuals that are incapable of articulating and expressing their pain experiences.

[0022] In a possible implementation form, the method further includes using a neural network to identify the one or more micro-expressions in the facial pattern input and predict one or more of the level of pain and the source of pain from the identified one or more micro-expressions. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the microexpressions that are caused by pain in a contactless manner. According to a third aspect, the above and further objectives and advantages are obtained by an apparatus. In one embodiment, the apparatus includes a facial detector or device that is configured to receive an input image or images. The facial detector is configured to detect the facial images in the input image. The facial images are then resized, geometrically normalized and aligned. Physiological health measures or physiological signals are extracted from the normalized face images. Pain intensity is estimated by combining the physiological health measures and facial expressions that are detected or otherwise identified by the face detector. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0023] In a possible implementation form, the deep normalized facial features are fed into a 3D CNN network that outputs a confidence value for each source of pain. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0024] According to a fourth aspect, the above and further objectives and advantages are obtained by a computer program product. In one embodiment, the computer program product has computer readable instructions embodied on a non-transitory computer readable medium, which when executed by a computing device, are configured to carry out the method according to any one of the possible implementation forms. The aspects of the disclosed embodiments are directed to using machine learning to characterize the source of pain from the micro-expressions that are caused by pain in a contactless manner.

[0025] These and other aspects, implementation forms, and advantages of the exemplary embodiments will become apparent from the embodiments described herein considered in conjunction with the accompanying drawings. It is to be understood, however, that the description and drawings are designed solely for purposes of illustration and not as a definition of the limits of the disclosed invention, for which reference should be made to the appended claims. Additional aspects and advantages of the invention will be set forth in the description that follows, and in part will be obvious from the description, or may be learned by practice of the invention. Moreover, the aspects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS In the following detailed portion of the present disclosure, the invention will be explained in more detail with reference to the example embodiments shown in the drawings, in which like references indicate like elements and:

[0027] Figure 1 illustrates a schematic block diagram of a system incorporating aspects of the disclosed embodiments.

[0028] Figure 2 illustrates a schematic block diagram of an exemplary system incorporating aspects of the disclosed embodiments.

[0029] Figure 3 illustrates an exemplary architecture of a computational model for calculating physiological signals incorporating aspects of the disclosed embodiments.

[0030] Figure 4 illustrates exemplary architecture of a computational model for pain estimation incorporating aspects of the disclosed embodiments.

[0031] Figure 5 illustrates an exemplary architecture of a computational model for classification of a type of pain incorporating aspects of the disclosed embodiments.

[0032] Figure 6 illustrates an exemplary architecture of a computational model for classification of a source of pain incorporating aspects of the disclosed embodiments.

[0033] DETAILED DESCRIPTION OF THE DISCLOSED EMBODIMENTS

[0034] Figure 1 illustrates a diagram of an exemplary system incorporating aspects of the disclosed embodiments. As shown in Figure 1 the system 100 generally includes a facial expression detector device 104, also referred to herein as a face detector device, and a computing apparatus 106. The aspects of the disclosed embodiments are generally directed to contact-free methodologies for determining the level and source of pain from micro-expressions using machine learning (ML) and artificial intelligence (Al).

[0035] The computing device 106 of Figure 1 is configured to receive a facial pattern input detected by the facial expression detector and identify one or more micro-expressions in the facial pattern. From the identified micro-expressions, the computing apparatus 106 is configured to provide a prediction 108. The prediction 108 in this example is one or more of a level of pain and a source of pain from the identified one or more micro-expressions. Facial expression can be considered as a reflective and spontaneous reaction of painful experiences. A micro-expression is an involuntary facial expression that a person makes when experiencing an emotion, or pain. Micro-expressions generally last only 0.5 to 4.0 seconds. The aspects of the disclosed embodiments are directed to analyzing such micro-expressions in order to estimate the level and sources of pain of a person utilizing advances in machine learning and artificial intelligence. The aspects of the disclosed embodiments find particular application to contactless pain analysis of persons such as neonates and post-surgery patients who may be unable to articulate or express their pain experience.

[0036] As shown in the example of Figure 1, the facial expression detector device 104 is configured to receive as input 102, or otherwise obtain, images or a video of the face of a person. In one embodiment, the input image 102 can comprise a video, recording or other rendering of an individual ’ s face . For example, in one embodiment, the input image 102 is in the form of digital image(s) or recording(s). The input image 102 will allow for facial movements and micro-expressions to be detected, generally referred to herein as patterns.

[0037] In one embodiment, the face detector 104 is configured to detect or find the human face in the input image or video 102. For example, the face detector 104 can be configured to detect the human face in each frame of the input image or video 102. The face detector or device 104 is generally configured to find and analyze the human face in each frame of such video.

[0038] The facial expression detector device 104 generally comprises any suitable device that is configured to capture, or receive as an input, a video of a person’s face. In one embodiment, the facial expression detector device 104 can comprise a camera or video recorder. For example, the facial expression detector device 104 can comprise a camera device of a smart phone, or other similar mobile device.

[0039] In one embodiment, the facial expression detector device 104 is a wearable device. For example, the aspects of the disclosed embodiments are configured to be implemented in wearable devices that include a camera, such as smart glasses. In one embodiment, the facial expression detector device 104 is a wearable device, such as the Huawei™ smart vision glasses. The facial expression detector 104 can also include or comprise a Liquid Crystal Display (LCD) monitor equipped with a simple Red Green Blue (RGB) camera.

[0040] As an example, the smart glasses could be worn by a physician or other observer and used to capture the facial image(s) of the individual. Once the facial image(s) are captured and provided to the computing apparatus, the computing apparatus 106 is configured to estimate the level and source of pain. This can be especially useful with neonates and post-surgery patients that are incapable of articulating and expressing their pain experiences. The wearable device or smart glasses referred to herein can be used as a basis for the development of future extensions of the aspects of the disclosed embodiments. For instance, the wearable device can be equipped with both conventional and nonconventional sensing technologies. Human face images acquired at visible spectrum or using conventional 2D cameras may have inherent restrictions that hinder the inference of some specific details in the face. With the progress of sensing technologies, many extremely hard computer vision problems are becoming much easier to solve. Using near-infrared (NIR) imaging, the effect of illumination variations can be significantly reduced. The low-cost depth sensors allow extracting directly 3D information, together with RGB colour images.

[0041] The computing apparatus 106 is generally configured to analyze the detected images or patterns received from the facial expression detector 104. In one embodiment, the computing apparatus 106 comprises or includes one or more computational models, as will be further described herein. The computing apparatus 106 is configured to provide an estimate or prediction 108 of one or more of the level and source of pain of the person.

[0042] Multimodal analysis has drawn much attention in recent years since it can enhance the overall accuracy and robustness of recognition systems over mono-modal approaches. Recently, deep learning (DL) models or architectures for early, late, and model-based fusion have been found to provide a discriminant feature representation, and outperform conventional fusion techniques by capturing the complex nonlinear interaction in multimodal data. The aspects of the disclosed embodiments are configured to develop and compare specialized deep learning models for dynamic, multimodal, and spatio-temporal information fusion in visual expression recognition, based on multiple contact-free modalities. Thes contact free modalities include, but are not limited to face, facial gestures and physiological signals. Generally, the contact free modalities can be captured in videos, for example.

[0043] Figure 2 illustrates one embodiment of an exemplary system 200 incorporating aspects of the disclosed embodiments. In this example, a wearable device an input image or video 202 is configured to capture or otherwise receive and input image or video. As described above, the input image or video 202 is generally of the face of a person.

[0044] A face detector 206 is configured to detect facial expressions, such as micro-expressions from the input image video. In the example of Figure 2, the output of the face detector 206 is provided to one or more computations models.

[0045] As shown in the example of Figure 2, the output of the face detector 206 is provided to a first computational model 208. The first computational model 208 is configured to calculate physiological signals from the output of the face detector 206. Physiological signals, such as speech, head, eye and body movements, as well as different signals that are closely related to emotional changes (e.g., heart rate, temperature, galvanic skin response, impedance cardiography, and blood pressure) can provide important information pertaining to pain assessment, and can provide additional cues to improve the pain anaylsis. The aspects of the disclosed embodiments are directed to pain assessment from a continuous video stream in natural environmental conditions. The visual data can be combined with physiological signals to make the pain inference more reliable.

[0046] While physiological signals can be measured using biosensors, the aspects of the disclosed embodiments reconstruct the physiological signals from the input images 202, without biosensors, together with the facial expressions for pain assessment. In one embodiment, the facial or face images can be resized, geometrically normalized and aligned for further processing. The physiological health measures can be extracted from the normalized face images. The physiological health measures, also referred to herein as signals, can include, but are not limited to, heart rate, respiratory rate and heart rate variability. In alternate embodiments, any suitable physiological health measures can be extracted other than including heart rate, respiratory rate and heart rate variability.

[0047] The deep normalized facial features are fed into a computational model 208 that outputs a confidence value for each source of pain. The computational model(s) 208 is configured to reconstruct the physiological signals from the input image or video 202 of the face. The obtained physiological signals can be used as additional cues for the pain analysis. In one embodiment, the computational model 208 of the computing apparatus 106 comprises a neural network, such as a 3D Convolutional Neural Network (3D CNN). Although a 3D Convolutional Neural Network is generally referred to herein, the aspects of the disclosed embodiments are not so limited. In alternate embodiments, any suitable machine learning system can be used, other than including a 3D Convolutional Neural Network. For example, the computational model can comprise a neural network or a deep neural network, that is configured to provide a prediction of a source and level of pain based on facial image inputs, as is generally provided for herein.

[0048] Referring again to Figure 2, in one embodiment, an output of the first computation model for calculating physiological signals 208 is provided to one or more of a second computational model for pain estimation 210, a third computational model for classification of a type of pain 212 and a fourth computational model for classification of a source of pain 214. The pain intensity can then be estimated or predicted by combining the extracted physiological signals and facial expressions from the face images of the input image or video 202.

[0049] As illustrated in the example of Figure 2, in one embodiment, the pain intensity can include an indication of the source of pain and an indication as to the chronic or non-chronic nature of the pain. This information can be provided back to the wearable device 204, for example. Alternatively, the information can be provided to any suitable output device, other than including the wearable device 204.

[0050] Figure 3 illustrates one example of an architecture 300 of a computational model for calculating physiological signals, such as the first computation model 208 of Figure 2. In this example, data from the face detector 302 of this embodiment, similar to the other embodiments described herein, is provided to a deep convolutional neural network 304, such as that generally described herein. The output of the neural network 304 in this example includes physiological factors such as a heart rate 306, a respiratory rate 308 and a heart rate variability 310. In alternate embodiments, the output of the neural network 304 could include any suitable physiological factors or signals.

[0051] Figure 4 illustrates an exemplary architecture 400 for a computational model for pain estimation, such as the second computation model 210 of Figure 2. In this example, data from the face detector 402 of this embodiment, similar to the other embodiments described herein, is provided to a deep convolutional neural network 404, such as that generally described herein. The output 406 of the neural network 404 in this example includes an estimation or prediction of the pain level. As an example, the estimation or prediction of the pain level can be on a scale from 0 to 15.

[0052] Figure 5 illustrates an exemplary architecture 500 for a computational model for classification of the type of pain, such as the third computational model 212 of Figure 2. In this example, the physiological data outputs 306, 308, 310 of the architecture 300 of Figure 3, and the pain level output data 406 from the architecture 400 of Figure 4 serve as the inputs to a deep convolutional neural network 504, such as that generally described herein. The outputs 506 and 508 of the neural network 504 in this example includes an estimation of chronic pain and an estimation of non-chronic pain.

[0053] Figure 6 illustrates an exemplary architecture 600 for a computational model for classification of the source of pain, such as the fourth computational model 214 of Figure 2. In this example, the physiological data outputs 306, 308, 310 of the architecture 300 of Figure 3, and the pain level output data 406 from the architecture 400 of Figure 4 serve as the inputs to a deep convolutional neural network 604, such as that generally described herein. The output 606 of the neural network 604 in this example includes an estimation of the source of pain.

[0054] Finding the source of pain can be very challenging. Generally, different sources of pain produce can different subtle visual changes seen at the micro-expression level. Unlike other facial expressions, micro-expressions can be of a lower intensity and difficult to read. The aspects of the disclosed embodiments are configured to learn the expressions that occur for each specific pain source. In this manner, the source of pain for the subtle and micro-expressions caused by pain can be characterized. In one embodiment, this can occur during a training phase of the aspects of the disclosed embodiments. The deep learning models of the disclosed embodiments can be trained on weakly labeled data, and adapted to specific capture conditions using unlabeled data. To reduce the cost and ambiguity of data annotation, weakly supervised learning (WSL) methods have been proposed for training machine learning models using data with reduced supervision, either with incomplete (a subset of data is labelled), inexact (coarse grained labels), or inaccurate supervision (ambiguous labels). The aspects of the disclosed embodiments are not intended to be limited by the particular type of training or training model.

[0055] The aspects of the disclosed embodiments also extend deep Multiple Instance Learning (MIL)-based classification models into novel regression models which are most suitable for video-based recognition of facial expressions linked to pain intensity. Training videos would only require a global annotation, or periodic annotations (for video clips).

[0056] Deep domain adaptation (DA) methods allow for personalizing deep learning models for improved performance in specific operational environments, individuals, sensors, and devices. They learn robust domain-invariant representations from source and target domain samples, usually trained with classification, discrepancy or adversarial loss.

[0057] The aspects of the disclosed embodiments use unsupervised deep domain adaptation for adapting deep learning models using unlabeled videos captured for individuals in an operational environment during a sensor calibration process. Deep domain adaptation strategies, discrepancy-, adversarial-, and reconstruction-based approaches can be used for facial analysis and recognition. For example, the maximum mean discrepancy can be relied upon to measure the difference between source and target domain distributions. Weakly-supervised domain adaptation method for training from weakly-labeled target videos can be used, where labels are provided on a periodic basis.

[0058] The aspects of the disclosed embodiments use computational models to reconstruct physiological signals from a moving face in unconstrained settings. In one embodiment, a Deep Learning model for spatiotemporal fusion is implemented by accounting for temporal coherence over consecutive frames of an input image or video, such as the input image 102 shown in Figure 1, for example. The methods disclosed herein are for weakly-supervised domain adaptation for training from weakly-labeled target videos.

[0059] The deep learning models for spatio-temporal fusion will provide accurate facial expression recognition by accounting for temporal coherence over consecutive frames. The deep learning models of the disclosed embodiments will also provide dynamic context-based gated fusion of information according to operational capture conditions. In one embodiment, the computing device 106 referred to in Figure 2 can comprise a hardware device, such as for example a computer processing device and / or memory and storage. The computer processing device can comprise a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (AUU), a digital signal processor, a microcomputer and a microprocessor, for example. The computer processing device may be configured to carry out program code by performing arithmetical, logical, and input / output operations, according to the program code. Once the program code is loaded into a computer processing device, the computer processing device may be programmed to perform the program code, thereby transforming the computer processing device into a special purpose computer processing device. In a more specific example, when the program code is loaded into a processor, the processor becomes programmed to perform the program code and operations corresponding thereto, thereby transforming the processor into a special purpose processor.

[0060] Thus, while there have been shown, described, and pointed out, fundamental novel features of the invention as applied to the exemplary embodiments thereof, it will be understood that various omissions, substitutions and changes in the form and details of devices and methods illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit and scope of the presently disclosed invention. Further, it is expressly intended that all combinations of those elements, which perform substantially the same function in substantially the same way to achieve the same results, are within the scope of the invention. Moreover, it should be recognized that structures and / or elements shown and / or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising: a facial expression detector device; and a computing device configured to: receive a facial pattern input from the facial expression detector device; identify one or more micro-expressions in the facial pattern input; and predict one or more of a level of pain and a source of pain from the identified one or more micro-expressions.

2. The apparatus according to claim 1, wherein the computing device is further configured to extract at least one physiological signal from the identified one or more micro-expressions in the facial pattern input and use the extracted at least one physiological signal in the prediction of the one or more of the level of pain and the source of pain.

3. The apparatus according to any one of claims 1 or 2, wherein the computing device is further configured to estimate a pain level from the identified one or more micro-expressions in the facial pattern input and use the estimated pain level in the prediction of the one or more of the level of pain and the source of pain.

4. The apparatus according to any one of the preceding claims wherein the extracted at least one physiological signal and the estimated pain level is used in the prediction of the one or more of the level of pain and the source of pain.The apparatus according to any one of the preceding claims wherein the facial expression detection device comprises a camera.The apparatus according to any one of the preceding claims wherein the facial expression detection device comprises a wearable device.The apparatus according to any one of the preceding claims wherein the facial expression detection device comprises a mobile communication device.

8. The apparatus according to any one of the preceding claims wherein the computing device comprises a convolutional neural network.

9. A computer-implemented method for predicting a level of pain and a source of pain from an image of a face, the computer implemented method comprising: receive a facial pattern input; identify one or more micro-expressions in the facial pattern input; and predict one or more of a level of pain and a source of pain from the identified one or more micro-expressions.

10. The computer implemented method according to claim 9, wherein the method further includes extracting at least one physiological signal from the identified one or more micro-expressions in the facial pattern input and using the extracted at least one physiological signal in the prediction of the one or more of the level of pain and the source of pain.

11. The computer implemented method according to any one of claims 9 and 10, wherein the method further includes estimating a pain level from the identified one or more micro-expressions in the facial pattern input and using the estimated pain level in the prediction of the one or more of the level of pain and the source of pain.

12. The computer implemented method according to any one of claims 9 to 11, wherein the method further includes using the extracted at least one physiological signal and the estimated pain level is used in the prediction of the one or more of the level of pain and the source of pain.

13. The computer implemented method according to any one of claims 9 to 12, the method further comprising using a wearable device to provide the facial input image.

14. The computer implemented method according to any one of claims 9-13, further comprising using a neural network to identify the one or more micro-expressions in the facial pattern input and predict one or more of the level of pain and the source of pain from the identified one or more microexpressions.

15. A computer program product comprising computer readable instructions embodied on a non- transitory computer readable medium, which when executed by a computing device, are configured to carry out the method according to any one of claims 9-14.

Images