Psychological disease identification method and system

Abstract

A psychological disease recognition method and system includes a facial change information acquisition unit configured to acquire facial change information indicating a time series change in facial data of a subject when emotion stimulation information is provided to the subject, the emotion stimulation information is used for stimulating any sensation or any combination of sensation of auditory sensation, olfactory sensation, taste sensation, tactile sensation and somatic sensation of the subject to change emotion, and is divided into positive information for increasing comfort and negative information for reducing comfort; a psychological disease determination unit configured to determine a psychological disease state of the subject based on the facial change information; and a psychological state acquisition unit configured to determine a psychological state of the subject based on a contactless psychological technique.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of psychological disease diagnosis, and particularly relates to a psychological disease recognition method and system. BACKGROUND

[0002] With the advent of the big data era, data sharing makes it possible to integrate and analyze multi-modal data. The mental health status of an individual is manifested through various means such as language, body movements, facial expressions, and physiological responses. Different data sources and indicators have their unique advantages, and comprehensive analysis of various data will achieve a more comprehensive and robust evaluation. Multi-modal data analysis methods in the field of computer science have brought innovation to intelligent mental health assessment, and researchers have begun to explore more diverse data collection methods to obtain multi-modal data for fusion modeling, thereby taking advantage of the complementary effects of information. For example, the prior art integrates physiological, speech, facial, and semantic features to construct a prediction model for depression. The above research results show that models containing multi-modal data often achieve the best prediction results.

[0003] In recent years, attempts have been made to estimate human brain activity by using data detected using electroencephalography (EEG), magnetic resonance imaging (fMRI: functional magnetic resonance imaging), and near-infrared spectroscopy (NIRS), however, electroencephalography and near-infrared spectroscopy require preprocessing, such as having the subject abrade electrodes. In addition, magnetic resonance imaging requires measurement in a predetermined MRI room. In short, the above methods have problems such as complex preparation work or limited measurement conditions. In addition, all of the above methods require a huge cost. Therefore, the above methods make it difficult to perform operations such as determining the physical and mental health status of a subject at certain times. SUMMARY

[0004] In view of the above prior art, the purpose of the present application is to provide a more stable psychological disease recognition method and system. It aims to solve the problem that the existing psychological disease diagnosis system is not perfect in collecting the emotional signals of patients when in use, cannot analyze the patient's condition by combining the patient's various physical data, and has poor effect on determining the patient's psychological disease.

[0005] The problem to be solved by the present application is to provide a device and method for determining the mental or physical physiological state of a subject. In particular, the purpose of the present application is to provide a psychological disease determination device for facilitating the determination of the mental disease state of a subject.

[0006] The mental illness determination device according to the first aspect of the present application includes an emotional stimulus information providing unit, a facial change information acquiring unit, a facial change information decomposing unit, a determination component extracting unit, and a mental illness determination unit. The emotional stimulus information providing unit provides emotional stimulus information for stimulating any of the senses or any combination of the senses of hearing, smell, taste, touch, and somatic sensation of the subject to change emotion. The facial change information acquiring unit acquires facial change information indicating a time-series change in facial data of the subject. The facial change information decomposing unit decomposes the facial change information into a plurality of components by using singular value decomposition, principal component analysis, or independent component analysis. The determination component extracting unit extracts a component related to the emotional stimulus information from the plurality of components as a determination component. The mental illness determination unit determines a mental illness state of the subject based on the determination component.

[0007] Non-contact psychological technology refers to the use of advanced technical means to assess and analyze an individual's psychological state, emotional changes, and behavioral characteristics through non-contact means. These technologies are usually based on the cross-fusion of biology, psychology, and computer science, and obtain psychological information by analyzing physiological signals and behavioral manifestations. Here are some common non-contact psychological technologies: 1. Vibraimage Principle: By analyzing the thermodynamic information and head and neck muscle micro-motion image video of individuals in a static balance state, the psychological and physiological state is evaluated. This technology is based on the vestibular-emotional reflex phenomenon, which is that emotional and psychological changes affect the brain's vestibular system, which in turn causes slight vibrations in the facial and neck muscles.

[0008] Application scenarios: mental health screening, medical health field, career and interest orientation test, personality characteristic evaluation, lie detection, etc.

[0009] 2. Psychological state assessment based on multi-modal fusion technology Principle: By collecting multi-modal data such as facial expressions, neck muscle vibrations, and changes in voice tone, and combining machine learning, deep learning, and computer vision technologies, the psychological health state of a person is inferred.

[0010] Application scenarios: smart device development, health management, adolescent mental health screening, psychological evaluation of corporate executives and high-risk groups, driver mental health analysis in the cockpit, etc.

[0011] 3. Non-contact brain function monitoring technology Principle: Advanced sensors and signal processing methods are used to monitor the electrical physiological signals of human brain activity in real time, non-invasively, and non-contact.

[0012] Application scenarios: neuroscience, psychology, biomedical, etc., for studying the relationship between brain function and psychological state.

[0013] 4. Non-contact voice psychological analysis system Principle: Using sound (voice) as a medium, psychological information is collected, tested, and analyzed. By analyzing the voice response of the subject to a wide range of events or specific purposes, the psychological state of the subject is evaluated.

[0014] Application scenarios: mental health assessment, psychological counseling, psychological treatment, etc.

[0015] In the psychological disease determination device according to the first aspect, a determination component is extracted from a plurality of components obtained by singular value decomposition, principal component analysis, or independent component analysis on the face change information. This can help estimate the presence of brain activity of the subject without using electrodes and the like that require preprocessing before attachment. Therefore, the psychological disease state of the subject can be easily determined based on the determined component corresponding to the brain function of the subject.

[0016] The psychological disease determination device according to the second aspect of the present application is the psychological disease determination device according to the first aspect, wherein the emotional stimulus information providing unit provides emotional image information for stimulating the vision of the subject to change the emotion together with the emotional stimulus information.

[0017] In the psychological disease determination device according to the second aspect, stimulating different brain regions enables the determination of various types of psychological diseases. For example, using information for stimulating vision, diseases related to the occipital region of the brain can be diagnosed. Using information for stimulating hearing, diseases related to the temporal and frontal regions of the brain can be diagnosed. In addition, using information about facial images, dementia and autism can be diagnosed. Using voice audio information, anxiety can be diagnosed.

[0018] The psychological disease determination device according to the third aspect of the present application is the psychological disease determination device according to the first aspect or the second aspect, wherein the determination component extraction unit extracts the determination component based on a correlation value between a determination waveform corresponding to the emotional stimulus information and each of the plurality of components. This configuration can specify the determined component corresponding to the brain function of the subject.

[0019] The psychological disease determination device according to the fourth aspect of the present application is the psychological disease determination device according to the third aspect, wherein a correction wave taking into account the physiological response of the human body is used as the determination waveform. When it is found that the determination waveform has a significant correlation with the components obtained from the face change information, the determination waveform shows a high correlation value, so the accuracy of the extraction of the determined component can be enhanced.

[0020] The mental disease determination device according to the fifth aspect of the present application is the mental disease determination device according to the fourth aspect, wherein the determination waveform is shifted after a predetermined time elapses after the emotional stimulus information is provided. A slight delay in phase with respect to the brain response provides an accurate correlation.

[0021] The mental disease determination device according to the sixth aspect of the present application is the mental disease determination device according to the fourth or fifth aspect, wherein the determination waveform is a rectangular wave. The rectangular wave can be adjusted depending on whether the emotional stimulus information is being provided, and thus the determination component can be easily extracted.

[0022] The mental disease determination device according to the seventh aspect of the present application is the mental disease determination device according to the fourth or fifth aspect, wherein the determination waveform is generated from a plurality of waveforms determined by calculating a component in which a correlation is found among the plurality of components and performing the calculation of the component a plurality of times. The determination waveform is optimized using a previous history, and thus shows a high correlation value when a determination waveform is found to have a significant correlation with a component obtained from the facial change information. This can improve the accuracy of extraction of the determination component.

[0023] The mental disease determination device according to the eighth aspect of the present application is the mental disease determination device according to any one or any combination of the first to seventh aspects, wherein the mental disease determination unit determines the state of the mental disease based on any one or any combination of: an amount of change falling within a predetermined range of the determination component with respect to a reference value, a value obtained by performing multivariate regression analysis on the determination component, an area of a region in which the determination waveform is generated, an average value of the determination waveform, and a value of a barycenter of the determination waveform. This enables the mental disease to be determined using an optimal technique according to the computational capacity of the device. Furthermore, an optimal technique can be employed depending on the type of the mental disease.

[0024] The mental disease determination device according to the ninth aspect of the present application is the mental disease determination device according to the first to eighth aspects, further comprising an autonomic nerve activity measurement unit that measures autonomic nerve activity of the subject. The mental disease determination unit determines the type of the state of the mental disease based on the autonomic nerve activity.

[0025] In the mental disease determination device according to the ninth aspect, the ratio of the response to the emotional stimulus originating from the autonomic nervous system and the response to the emotional stimulus originating from the brain activity is determined. This makes it possible to determine the type of disease. Examples of the type of disease that can be determined include "depressive disorder / major depressive disorder, dysthymic disorder, depressive disorder not otherwise specified, depressive- related syndrome", "bipolar disorder / bipolar I disorder, bipolar II disorder", "psychotic disorder", "developmental disorder / pervasive developmental disorder (autism, Asperger's syndrome, Tourette's syndrome), learning disorder LD, attention-deficit hyperactivity disorder ADHD", "psychomotor retardation", "anxiety disorder / generalized anxiety disorder, panic disorder, social anxiety disorder, obsessive-compulsive disorder, dissociative disorder, adjustment disorder", "dementia / dementia of the Alzheimer's type, vascular dementia, Lewy body dementia, frontotemporal dementia", "substance use disorder", "organic mental disease, including symptomatic mental disease", "epilepsy", and "personality disorder"). The mental disease determination device according to the tenth aspect of the present application is the mental disease determination device of the second to ninth aspects, further including a line-of-sight measurement unit that measures the line of sight of the subject. The mental disease determination unit removes a low-reliability determination result based on the line of sight of the subject.

[0026] In the mental disease determination device according to the tenth aspect, by determining whether or not the subject is recognizing the image information, it is possible to remove a low-reliability determination result.

[0027] The mental disease determination device according to the eleventh aspect of the present application includes an emotional stimulus information providing unit, a facial change information acquiring unit, a facial change information decomposing unit, a determination component extracting unit, and a mental disease determination unit. The emotional stimulus information providing unit provides emotional stimulus information for stimulating the senses of the subject to change the emotion. The facial change information acquiring unit acquires facial change information indicating a time-series change in facial data of the subject. The facial change information decomposing unit decomposes the facial change information into a plurality of components by using singular value decomposition, principal component analysis, or independent component analysis. The determination component extracting unit extracts a determination component based on the correlation strength between a determination waveform corresponding to the emotional stimulus information and each of the plurality of components. The mental disease determination unit determines the mental disease state of the subject based on the determination component. The determination waveform is generated from a plurality of waveforms determined by calculating a component found to have a correlation in the plurality of components and performing the calculation of the component a plurality of times.

[0028] In the mental disease determination device according to the eleventh aspect, the determination waveform is optimized using the previous history. By using a determination waveform that shows a high correlation value when a determination waveform is found to have a significant correlation with a component obtained from the facial change information, it is possible to enhance the accuracy of the extraction of the determination component.

[0029] The mental disease determination device according to the twelfth aspect of the present application is the mental disease determination device according to the first to eleventh aspects, in which the mental disease determination unit calculates a correlation value of a determination component for the emotional stimulus information, and determines the state level of the mental disease of the subject on the basis of the calculated correlation value and the determination information. Here, the determination information providing device on the network includes a determination information storage unit that stores a variation amount falling within a predetermined range between the correlation value of the determination component calculated for the emotional stimulus information and a reference correlation value of a reference determination component calculated for the emotional stimulus information in association with the state level of the mental disease as the determination information.

[0030] In the mental disease determination device according to the twelfth aspect, the state level of the mental disease of the subject can be determined by using the determination information providing device on the network. BRIEF DESCRIPTION OF DRAWINGS

[0031] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor.

[0032] Figure 1 The flowchart provided for the present application; Figure 2 The waveform diagram of the model training method provided for the present application; DETAILED DESCRIPTION

[0033] According to the need, detailed embodiments of the present application are disclosed in this paper, but it should be understood that the disclosed embodiments are only to demonstrate the present application, and the present application can be implemented in different and alternative forms. The drawings are not necessarily drawn to scale, and some features can be exaggerated or reduced to show the details of specific components. Therefore, the specific structure and functional details disclosed herein should not be understood as limiting, but only as representative basis to teach those skilled in the art to use the present application differently. All other embodiments obtained by those skilled in the art without creative labor based on the embodiments in the present application are within the scope of protection of the present application.

[0034] Electroencephalogram and near-infrared spectroscopy require preprocessing, such as abrading electrodes on the subject. In addition, magnetic resonance imaging requires measurement in a predetermined MRI room. In short, the above-mentioned methods have problems such as complicated preparation work or limited measurement conditions. In addition, all of the above-mentioned methods require huge costs. Therefore, the above-mentioned methods make it difficult to perform operations such as determining the physical and mental health state of the subject at some times.

[0035] The inventors have considered the possibility of estimating a person's brain activity based on the person's facial skin temperature or based on the condition of facial blood circulation that is considered to be proportional to the facial skin temperature. The person's facial skin temperature can be acquired by using a measuring device such as a thermal imaging device, and the condition of facial blood circulation, i.e., the amount of facial blood circulation, can be estimated from RGB data of a captured facial image obtained by using the imaging device. Thus, the facial skin temperature or the captured facial image can be acquired without attaching a sensor that needs to be handled before attachment, such as an electroencephalogram electrode or a probe.

[0036] On the other hand, it is known that a person's facial skin temperature changes due to various factors such as an ambient temperature and / or autonomic nervous activity. For this reason, if the brain activity is to be estimated based on the facial skin temperature or based on the amount of facial blood circulation that is considered to be proportional to the facial skin temperature, it is considered to be very difficult to determine whether the acquired data reflects only the brain activity.

[0037] As a result of intensive research, the inventors have found that a component reflecting a change in facial skin temperature or a change in the amount of facial blood circulation that reflects brain activity can be identified by detecting facial skin temperature, by using singular value decomposition, principal component analysis, or independent component analysis on time-series facial skin temperature data (which includes detection temperature data and position data (coordinate data) of a detection region) or time-series facial blood circulation amount data obtained from RGB data of a facial image captured from a time series, and analyzing a plurality of components obtained by the decomposition. Then, the inventors have arrived at the present invention in which, by estimating and analyzing a subject's brain activity, the subject's physiological state can be visualized based on the estimated brain activity. Next, a method for acquiring facial skin temperature data and a method for analyzing facial skin temperature data, which the inventors used to obtain the above-mentioned findings, will be described.

[0038] In this test, facial skin temperature data was acquired from six test subjects. Specifically, the test subjects were seated on a chair placed in a climatic chamber in which the room temperature was maintained at 25°C, and the facial skin temperature data was acquired from the entire region of the test subjects' faces by using an infrared thermal imaging device. The infrared thermal imaging device is a device that can detect infrared radiation energy emitted from a target object by using an infrared camera, convert the detected infrared radiation energy into a temperature (here, a temperature expressed in degrees Celsius) on the surface of the target object, and display and store a distribution of the temperature as facial skin temperature data (e.g., image data indicating the temperature distribution).

[0039] Further, in the test, a brain function activation exercise was provided to the test subject during acquisition of the facial skin temperature data. Thus, facial skin temperature data during a brain deactivation time and facial skin temperature data during a brain activation time were acquired. Examples of the brain function activation exercise include a mental task such as causing each test subject to perform a calculation based on a video image displayed on a display device or the like, to recognize a number, a shape, and a color, or to memorize a symbol, a character, or a word. In this test, "BrainHQ" was used as the brain function activation exercise. Each test subject was assigned a task of calculating a written number displayed on a display device and inputting an answer by using a keyboard. In the test, the test subject was presented with the brain function activation exercise for 10 minutes after 5 minutes from the start of acquisition of the facial skin temperature data.

[0040] As analysis of the facial skin temperature data, singular value decomposition was performed on the acquired facial skin temperature data using SVD (singular value decomposition) of MATLAB (registered trademark) as an analysis tool. Singular value decomposition was performed on all of the facial skin temperature data acquired in a time series (30-minute data), in which a factor is time data obtained at 30-second intervals (60 time points within 30 minutes), and a measure is facial skin temperature data (240 x 320 pixels) within that time period (a 30-second time period). By singular value decomposition, the facial skin temperature data X is decomposed into a plurality of components, and a time distribution V and a spatial distribution U of each component and a singular value S indicating the size of each component are calculated.

[0041] The time distribution V and the spatial distribution U of each component determined using singular value decomposition were plotted on a graph, and a component waveform chart and a temperature distribution chart of each component were generated.

[0042] Further, the created component waveform chart and the temperature distribution chart of each component were analyzed to identify a component indicating a change in skin temperature reflecting brain activity.

[0043] The component waveform plot of each component is analyzed to determine the presence of a correlation between the amplitude of the component waveform of the component and each of the brain deactivation time and the brain activation time. Specifically, it is assessed whether there is a correlation between the amplitude shown in the component waveform plot of each component and the brain deactivation period / brain activation period. In this test, a period in which the test subject is not presented with the brain function activation exercise (which is equal to a period of 5 minutes from the start of data acquisition to the passage of 5 minutes and a period of 15 minutes from the passage of 15 minutes after the start of data acquisition to the end of data acquisition) is set as the brain deactivation time for the period in which the facial skin temperature data is acquired. And a period in which the test subject is presented with the brain function activation exercise is set as the brain activation time, which is equal to a period of 10 minutes from the passage of 5 minutes after the start of data acquisition to the passage of 10 minutes. Then, the presence of a correlation between the amplitude shown in the component waveform plot of each component and each of the brain deactivation time and the brain activation time is assessed. The determination of the presence of correlation is made using statistical correlation analysis. When the significance level (A) is 0.05 or less, it is determined that there is a correlation.

[0044] The temperature distribution plot of each component is analyzed to determine whether there is a temperature change in the predetermined surface area. The brain has a mechanism for cooling the brain while maintaining body temperature, called the selective brain cooling system. The selective brain cooling system is known to dissipate heat generated by brain activity through the forehead portion and the area around the paranasal sinuses, including the area around the glabella and nose portion. Therefore, in this test, it is assessed whether a temperature change occurs in the area around the paranasal sinuses and the forehead portion on the temperature distribution plot of each component. The presence of a temperature change is determined by visual inspection or by determining whether the temperature of the area around the paranasal sinuses and the forehead portion differs from the average temperature of the overall measurement data by one standard deviation (SD) or more.

[0045] The determination of the polarity (positive or negative) of the facial skin temperature data X is based on the relationship between the values of the spatial distribution U, the singular value S, and the temporal distribution V. Therefore, at certain times, the polarity appears to be opposite on the component waveform plot and the temperature distribution plot of each component. For this reason, it is assumed that the polarity is excluded from the evaluation of the component waveform plot and the temperature distribution plot.

[0046] In the infrared thermal imaging device, as described above, the infrared radiation energy detected from the target object is converted into temperature, and the distribution of the temperature is used as the facial skin temperature data. When the infrared thermal imaging device is used for a person to acquire the facial skin temperature of the person, temperature changes (so-called noise) unrelated to various brain activities (for example, movement of the face and / or autonomic nervous activity) can also be acquired as the facial skin temperature data. The relative facial skin temperature data is created so that all average values of pieces of temperature data included in the facial skin temperature data acquired at intervals of 30 seconds are set to "0".

[0047] The above describes in detail the method and system for identifying a psychological disease provided by the present application. Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be understood by referring to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be understood by referring to the method part. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0048] It should also be noted that in the present specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A system for identifying a mental illness, characterized by, comprising a face change information acquisition unit configured to acquire face change information indicating a time series change in face data of the subject when emotional stimulus information is provided to the subject, the emotional stimulus information being used to stimulate any of an auditory sense, an olfactory sense, a gustatory sense, a tactile sense, and a somatic sense or any combination of senses of the subject to change an emotion, and being classified into positive information for increasing comfort and negative information for decreasing comfort; a psychological disease determination unit configured to determine a psychological disease state of the subject based on the face change information; and a psychological state acquisition unit configured to determine a psychological state of the subject based on a non-contact psychological technology including a vibration image high technology, a multi-modal fusion technology, a non-contact brain function detection technology, and a non-contact voice psychological analysis.

2. The psychological disease recognition system according to claim 1, characterized in that, Further comprising a determination component extraction unit configured to extract a component related to the emotional stimulus information as a determination component from a plurality of components extracted from the face change information, each of the plurality of components corresponding to one of a plurality of sets of time distribution and spatial distribution.

3. The psychological disease recognition system according to claim 2, characterized in that, The determination component extraction unit is further configured to extract the component related to the emotional stimulus information as the determination component by analyzing a correlation between the emotional stimulus information and each of the plurality of time distributions and then analyzing the spatial distribution.

4. The psychological disease recognition system according to claim 2, characterized in that, The determination component extraction unit is further configured to extract the determination component based on a correlation value between a determination waveform corresponding to the emotional stimulus information and each of the plurality of components.

5. The psychological disease recognition system according to claim 4, characterized in that, A modified wave taking into account a human physiological response is used as the determination waveform.

6. The psychological disease recognition system according to claim 5, characterized in that, The determination waveform is shifted after a predetermined time elapses from when the emotional stimulus information is provided.

7. The psychological disease recognition system according to claim 6, characterized in that, The determination waveform is generated from a plurality of waveforms determined by calculating components found to have a correlation among the plurality of components and performing the calculation of the components a plurality of times.

8. The psychological disease recognition system according to claim 4, characterized in that, The psychological disease determination unit is further configured to determine the state of the psychological disease based on any one or any combination of a change amount falling within a predetermined range of a correlation value of the determination component with respect to a reference value, a value obtained by performing a multivariate regression analysis on the determination component, an area of a region in which the determination waveform is generated, an average value of the determination waveform, and a barycentric value of the determination waveform.

9. The psychological disease recognition system according to claim 1, characterized in that, Further comprising a face change information decomposition unit configured to decompose the face change information into a plurality of components by using singular value decomposition, principal component analysis, or independent component analysis, each of the plurality of components corresponding to one of a plurality of sets of time distribution and spatial distribution.

10. A method of identifying a mental disease for executing the system for identifying a mental disease according to claims 1 to 8, characterized by, When the determination waveform is a bell-shaped Gaussian distribution, it is determined that the psychological state is normal; When the determination waveform appears as a bimodal distribution compared to a normal distribution, it is determined that there is a preliminary sign of a psychological disease; When the determination waveform is shifted to the left side, it is determined that there is a further psychological disease; When the determination waveform is shifted to the right side, it is determined that there is a serious psychological disease.

Images