Information processing system and information processing method

Abstract

To enable estimation of a user's mental condition.SOLUTION: An information processing system comprises: a detection part that detects the degree of an attribute of a user by analyzing a dynamic picture image obtained by shooting the user; a calculation part that calculates first and second variation degrees according to the degree of the attribute of the dynamic picture image; and an estimation part that estimates a user's mental condition on the basis of at least the second variation degree.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system and an information processing method.

Background Art

[0002] There is known a technique for analyzing the emotions received by others in response to the speech of a speaker (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, although the technique of Patent Document 1 can analyze the emotions of the target person, it cannot estimate the specific mental state.

[0005] The present invention has been made in view of such a background, and an object thereof is to provide a technique capable of estimating the mental state of a user.

Means for Solving the Problems

[0006] The main invention of the present invention for solving the above problems is an information processing system, comprising: a detection unit that analyzes a moving image of a user and detects the degree of attributes of the user; a calculation unit that calculates a first variation degree and a second variation degree related to the degree of attributes in the moving image; and an estimation unit that estimates the mental state of the user based at least on the second variation degree.

[0007] The other problems disclosed in the present application and the solutions thereto will be clarified by the embodiments of the invention and the drawings.

Effects of the Invention

[0008] According to the present invention, it is possible to estimate the user's mental state. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of an information processing system. [Figure 2] This figure shows an example of the hardware configuration of management server 2. [Figure 3] This figure shows an example of the software configuration for management server 2. [Figure 4] This diagram illustrates the operation of management server 2. [Figure 5] This figure shows an example of the results of a SHAP analysis. [Modes for carrying out the invention]

[0010] <System Overview> The following describes an information processing system according to one embodiment of the present invention. The information processing system of this embodiment attempts to estimate the user's mental state (especially the degree of depression) from a video of the user.

[0011] In this embodiment, "mental state" refers not merely to temporary physiological states (such as drowsiness or fatigue) or cognitive function states (such as concentration or attention), but to a more sustained emotional and psychological state of health. Specifically, it includes emotional aspects such as the degree of depression, anxiety, and stress, and in psychiatry, it is a concept related to the assessment of the severity of mood-related disorders such as depression and anxiety disorders. Mental state is a concept that captures emotional and mood fluctuations, and is clearly distinguished from cognitive states such as "concentration" and "attention" which result from the allocation of cognitive resources, or physiological states such as "drowsiness" and "fatigue" which mainly result from the level of physical arousal.

[0012] Conventional technologies, such as drowsiness detection systems and concentration monitoring systems, have used attributes such as blink frequency and eye opening degree to detect temporary states. In contrast, the "mental state" estimated in this embodiment is not a temporary cognitive state, but rather an emotional and psychological state that persists for several days to several weeks.

[0013] Furthermore, while self-reporting using questionnaires such as QIDS (Quick Inventory of Depressive Symptomatology), PHQ-9 (Patient Health Questionnaire-9), and MADRS (Montgomery-Asberg Depression Rating Scale) or physician assessments have traditionally been used to evaluate such mental states, the information processing system of this embodiment provides an objective evaluation method that replaces or complements these conventional evaluation methods. In particular, by using a second degree of variability in the degree of attributes (such as the standard deviation of the standard deviation), the stability / instability of emotional expression is quantitatively evaluated, thereby estimating mood-related mental states. This approach enables a deeper evaluation of emotional health, rather than merely detecting temporary cognitive states.

[0014] Figure 1 shows an example of the overall configuration of an information processing system. The information processing system in this embodiment includes a management server 2. The management server 2 is connected to the user terminal 1 via a communication network. The communication network is, for example, the internet and is constructed using public telephone networks, mobile phone networks, wireless communication channels, Ethernet (registered trademark), etc.

[0015] User terminal 1 is a computer operated by the user. User terminal 1 can be, for example, a smartphone, a tablet computer, or a personal computer.

[0016] The management server 2 is a computer that estimates the mental state of a user. The management server 2 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0017] <Management Server> FIG. 2 is a diagram showing an example of the hardware configuration of the management server 2. Note that the illustrated configuration is an example, and it may have other configurations. The management server 2 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, such as a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, such as an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for performing wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 inputs data, such as a keyboard, a mouse, a touch panel, a button, a microphone, etc. The output device 206 outputs data, such as a display, a printer, a speaker, etc. Note that each functional unit of the management server 2 described later is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit of the management server 2 is realized as a part of the storage area provided by the memory 202 and the storage device 203.

[0018] FIG. 3 is a diagram showing an example of the software configuration of the management server 2. The management server 2 includes a learning model storage unit 231, a detection unit 211, a calculation unit 212, an estimation unit 213, and an output unit 214.

[0019] <Storage Unit> The learning model storage unit 231 stores a first learning model for detecting the degree of association with the attributes of the user shown in the moving image (hereinafter referred to as the attribute degree), and a second learning model for estimating the mental state of the user based on the detected degree of the attribute.

[0020] The first and second learning models in the present embodiment are created by machine learning. Machine learning is roughly classified into supervised learning and unsupervised learning. Supervised learning is a method of learning a model using input data and corresponding output data (teacher data), and by adjusting the parameters of the model based on the teacher data, it learns the mapping from the input data to the output data. On the other hand, unsupervised learning is a method of learning the structure and pattern of input data without using teacher data, and it learns the density distribution and feature representation of the input data. As the first and second learning models in the present embodiment, for example, a neural network, a support vector machine, a decision tree, a random forest, etc., which are a type of supervised learning, can be used. Also, a self-organizing map, a k-means method, etc., which are a type of unsupervised learning, can be used. These machine learning algorithms perform processes such as weighting and transformation on the input data in order to learn the relationship between the input data and the output data, and optimize the parameters so that the error with the output data is minimized. Thereby, the mapping from the input data to the output data can be learned.

[0021] The first learning model can be created using machine learning with features and attribute degrees extracted from video as training data. Examples of features that serve as input to the first learning model include images of the face region extracted from each frame of the video, the position and size of organs such as the eyes, nose, and mouth extracted from the face region, and the temporal changes of these organs. On the other hand, examples of attribute degrees that serve as output to the first learning model include the number of blinks, the degree of eye opening, the degree of mouth opening, the position of the eyebrows, the orientation of the face, and the temporal changes of these. Specifically, the first learning model can take features such as images of the face region and the position and size of organs as input and output attribute degrees such as the number of blinks and the degree of eye opening. Possible attributes include the number of blinks, eye offset (angle of the eyes relative to the camera), gaze direction estimated from the eye offset, and facial expression. Facial expressions include anger, disgust, fear, happiness, sadness, surprise, neutrality, and negative / positive emotions. It is possible to infer the average and median frequency and direction of these expressions within a given period (e.g., 1 second), as well as the degree of emotion (facial expression) such as anger.

[0022] The second learning model is a learning model that estimates mental state (degree of depression) when given at least one of the following: the degree (value) of an attribute, a statistical value related to the degree of the attribute (such as the standard deviation), and a statistical value of the said statistical value (such as the standard deviation of the standard deviation). In this embodiment, the features given to the second learning model include at least the statistical value of the statistical value (such as the standard deviation of the standard deviation). The second learning model can be created by machine learning using at least one of the degree of an attribute, a statistical value related to the degree of the attribute, and a statistical value of the said statistical value, along with a mental state judged by an expert, as training data.

[0023] Furthermore, the inventors of this application performed SHAP (SHapley Additive ExPlanations) analysis to visualize the contribution of the features input to the second learning model. Figure 5 shows an example of the results of the SHAP analysis. As shown in Figure 5, features that have the suffix "ss" and represent the standard deviation of the standard deviation (hereinafter referred to as the second degree of variability)—for example, blink_ss, fear_ss, positive_ss—show a higher SHAP value distribution than other mean values ​​or first degree of variability (standard deviation) relative to the model output, confirming that they are extremely important in estimating the degree of depression in the user's mood.

[0024] For example, studies have shown that users with depressive tendencies exhibit significant instability in the fluctuations of certain emotional expressions (especially fear and sadness). Specifically, while the fluctuations (standard deviation) of emotional expression are relatively stable under normal conditions, when mental health deteriorates, the fluctuations themselves tend to become unstable (the standard deviation of the standard deviation increases).

[0025] The degree of second-order variability is obtained by calculating the standard deviation of the attribute degree at predetermined time windows (e.g., 30 seconds) and then recalculating the standard deviation for each of these time windows. This allows us to capture the "fluctuations" of attribute variation itself and quantify long-term instability that is distinct from instantaneous emotional reactions, thereby significantly increasing the sensitivity of detecting the degree of depression (or excitement).

[0026] <Functional Section> The detection unit 211 analyzes the video to detect the degree of the user's attributes. The detection unit 211 can detect the degree of attributes for each interval of a predetermined length. The detection unit 211 can estimate the degree of attributes by providing the video to a first learning model.

[0027] The calculation unit 212 calculates a second degree of variation (second-order variation) of a first degree of variation related to the degree of attributes in the moving image. Specifically, the calculation unit 212 calculates a first degree of variation (for example, the standard deviation of the degree of attributes), which is the degree of variation of the degree of attributes, and further calculates a second degree of variation (for example, the standard deviation of the standard deviation of the degree of attributes), which is the degree of variation of the first degree of variation. In this embodiment, the degree of variation is assumed to be the standard deviation, but it may also be the variance. For example, the calculation unit 212 can calculate a first degree of variation, which is the standard deviation of the degree of anger, and a second degree of variation, which is the standard deviation of the said standard deviation.

[0028] The estimation unit 213 estimates the user's mental state based on at least a second degree of variability. The estimation unit 213 can estimate the mental state based on the second degree of variability and at least one of the first degree of variability or the degree of attributes. The estimation unit 213 can estimate the mental state by providing at least the second degree of variability to a second learning model.

[0029] The output unit 214 outputs the estimated mental state.

[0030] <Operation> Figure 4 is a diagram illustrating the operation of the management server 2.

[0031] The management server 2 acquires video footage captured by the user (S301), estimates the degree of each user attribute at predetermined intervals (e.g., 1 second) based on the acquired video footage and the first learning model (S302), and calculates the degree of variation of the estimated degrees (S303). Here, the degree of variation can be calculated using either the standard deviation or the variance. For example, when calculating the standard deviation of the attribute degrees, the calculation unit 212 calculates the standard deviation from the estimated attribute degrees. On the other hand, when calculating the variance of the attribute degrees, the calculation unit 212 calculates the variance from the estimated attribute degrees. Next, the management server 2 calculates the degree of variation of the calculated degree of variation (S304). For example, when calculating the standard deviation of the standard deviation of the attribute degrees, the calculation unit 212 calculates the standard deviation from the standard deviation of the attribute degrees. On the other hand, when calculating the variance of the variance of the attribute degrees, the calculation unit 212 calculates the variance from the variance of the attribute degrees. Then, the management server 2 provides the second learning model with at least the degree of variation of the degree of variation (and the degree of each attribute and / or the degree of variation of the degrees) to estimate the user's mental state (S305), and outputs the estimated mental state (S306).

[0032] As described above, the information processing system of this embodiment can estimate a user's mental state from video footage of the user, thus enabling easy estimation of the mental state without the need for tests such as QIDS. Furthermore, the information processing system of this embodiment can perform estimation using the standard deviation of the standard deviation of the degree of attributes as a feature. For example, a user whose mental state is deteriorating may overreact to certain topics while showing no interest in other topics. By evaluating the standard deviation of the standard deviation (the degree of variation of the degree of variation), it becomes possible to evaluate how much the degree of variation of facial expressions, etc., varies over time, and an improvement in the accuracy of estimating the mental state can be expected.

[0033] Furthermore, according to the information processing system of this embodiment, by adopting not only the first degree of variability but also the second degree of variability as a feature for the degree of user attributes, it is possible to capture with high accuracy how "unstable" emotional expressions such as facial expressions, gaze, and blinking are over time. As a result, it is possible to estimate mild to moderate mood swings, which tend to be overlooked by conventional mean-centered methods, with high accuracy, thereby improving the reliability of mental health screening.

[0034] Although these embodiments have been described above, they are intended to facilitate understanding of the present invention and are not intended to limit its interpretation. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are also included.

[0035] For example, the processing performed by each functional unit of the management server 2 described above may be executed by any of the functional units. Furthermore, different functional units may be added to perform some of the processing performed by each of the functional units described above. Also, the functional units of the management server 2 may be distributed across multiple computers.

[0036] Furthermore, the information stored in each memory unit of the management server may be stored in any of the memory units. That is, the information stored in the multiple memory units mentioned above may be stored in a single memory unit, or some of the information stored in one memory unit may be stored in another memory unit.

[0037] <Example 1> In the above embodiment, the degree of user attributes was detected from video footage, but it is not limited to this. In addition to video footage, the degree of user attributes may also be detected from audio. For example, the user's voice can be acquired using an audio input device such as a microphone, and the degree of attributes such as the loudness of the user's voice, intonation, and speaking speed can be detected from the acquired audio. For the degree of loudness, for example, the amplitude, sound pressure, or volume of the voice can be used. For the degree of intonation, for example, the amount of change in the fundamental frequency (pitch) of the voice can be used. For the degree of speaking speed, for example, the number of syllables or words per unit time can be used.

[0038] In this way, based on the degree of attributes detected from the audio, a first degree of variability (such as standard deviation and variance) and a second degree of variability (such as the standard deviation of the standard deviation and the variance of the variance) can be calculated, similar to the embodiment described above, and these values ​​can be used to estimate the mental state. For example, the estimation unit 213 can estimate the user's mental state by using the degree of attributes detected from the audio, in addition to the degree of attributes detected from the video, and their degrees of variability. This makes it possible to improve the accuracy of estimating the mental state using information that cannot be obtained from the video alone.

[0039] <Modification 2> In the above embodiment, the degree of one attribute (for example, the degree of smile, the degree of eye opening, etc.) was used as the degree of an attribute, but a value that combines multiple attributes may also be used as the degree of an attribute. For example, the detection unit 211 can detect the degree of smile and the degree of eye opening from the video image and calculate a value that combines these values ​​as the degree of the attribute. As for how to combine the degree of smile and the degree of eye opening, for example, a weighted sum of the degree of smile and the degree of eye opening, or the product of the degree of smile and the degree of eye opening can be used.

[0040] Similar to the embodiment described above, a first degree of variability (such as standard deviation and variance) and a second degree of variability (such as the standard deviation of the standard deviation and the variance of the variance) can be calculated for the degree to which multiple attributes are combined, and these values ​​can be used to estimate the mental state. For example, the estimation unit 213 can estimate the user's mental state by using the degree to which the smile level and eye opening level are combined, and the degree of variability thereto.

[0041] Furthermore, the degree of an attribute may be expressed using a value that combines three or more attributes. For example, a value combining the degree of smile, the degree of eye opening, and the volume of voice can be used as the degree of an attribute. This makes it possible to improve the accuracy of estimating mental state by using complex information that cannot be obtained from a single attribute alone.

[0042] <Variation 3> In the above embodiment, the management server 2 acquired video footage from the user terminal and detected the degree of attributes from the video footage, but it is not limited to this. The user terminal 1 may also detect the degree of attributes from the video footage and send the detected degree of attributes to the management server 2.

[0043] Specifically, the user terminal 1 uses an imaging device such as a camera to photograph the user and detects the degree of the user's attributes from the captured video image, similar to the embodiment described above. The user terminal 1 then transmits the detected degree of attributes to the management server 2. The degree of attributes may be detected, for example, at predetermined time intervals (e.g., every second), and the degree of attributes for each detected time interval may be transmitted to the management server 2.

[0044] Based on the degree of attributes received from the user terminal 1, the management server 2 calculates a first degree of variability of the attribute degree (such as standard deviation and variance) and a second degree of variability of the first degree of variability (such as the standard deviation of the standard deviation and the variance of the variance), similar to the embodiment described above, and uses the calculated degree of variability to estimate the user's mental state.

[0045] Alternatively, instead of providing a management server 2, the user terminal 1 may be equipped with all the functional and memory functions of the management server 2, allowing the user terminal 1 to detect the degree of attributes and also detect the mental state. In this case, the management server 2 may be provided with the functions of managing the learning model and providing input data to the learning model, and the function of sending input data to the management server 2 and generating answers may be delegated to the management server 2.

[0046] <Modification 4> In the above embodiment, the current mental state of the user was estimated based on the degree of the user's current attributes and the degree of their variation, but the embodiment is not limited to this. The future mental state of the user may also be estimated based on the current and past degrees of attributes and their variation.

[0047] Specifically, management server 2 obtains the degree of each attribute and its variability over a predetermined period up to the present (for example, the last week). Then, based on the obtained degree of each attribute and its variability, management server 2 estimates not only the current mental state but also the future mental state.

[0048] One method for estimating future mental state is to predict the degree and variability of future attributes from the current and past degrees and variability of those attributes, and then estimate the future mental state based on the predicted degree and variability of future attributes. For predicting the degree and variability of attributes, time series data analysis techniques (e.g., ARIMA model, RNN, etc.) can be used.

[0049] Alternatively, future mental states may be directly predicted from the current and past trends in mental states. For example, the estimation unit 213 can estimate the trends in mental states over a predetermined period up to the present based on the degree of attributes and the degree of variation over that period, and predict future mental states based on the estimated trends in mental states. Time series data analysis methods can also be used when predicting future mental states from trends in mental states.

[0050] <Modification 5> In the above embodiment, the degree of depression was estimated as a mental state, but it is not limited to this. Other indicators that can be grasped separately from the mental state, such as stress levels or concentration levels, may also be estimated.

[0051] Stress levels can be estimated based on the degree and variability of attributes such as the number and frequency of blinks, the degree to which the eyes are open, the degree to which the mouth is open, the orientation of the face, the volume and intonation of the voice, and the speaking speed. Generally, users with high stress levels tend to blink frequently, have wide-open eyes and mouths, have an unsettled face orientation, have unstable voice volume and intonation, and speak quickly. Therefore, it is possible to estimate stress levels from the degree and variability of these attributes.

[0052] Concentration can be estimated based on the degree and variability of attributes such as the amount and speed of eye movement, the degree to which the gaze is fixed, and the number and frequency of blinks. Generally, users with high concentration tend to have less eye movement and speed, longer periods of fixed gaze, and fewer blinks. Therefore, it is possible to estimate concentration from the degree and variability of these attributes.

[0053] The estimation unit 213 can estimate the user's stress level and concentration level using a learning model that has learned the relationship between the degree and variability of the above-mentioned attributes and the stress level and concentration level.

[0054] <Disclosure Items> Furthermore, this disclosure also includes the following configurations. [Item 1] A detection unit analyzes video footage of the user to detect the degree of the user's attributes, A calculation unit that calculates a second degree of variation of a first degree of variation relating to the degree of the attribute in the aforementioned moving image, An estimation unit that estimates the user's mental state based at least on the second degree of variation, An information processing system characterized by comprising the following features. [Item 2] The information processing system described in item 1, The detection unit detects the degree of the attribute for each predetermined length interval, An information processing system characterized by the following. [Item 3] The information processing system described in item 1, The first and second degrees of variability are expressed by the standard deviation. An information processing system characterized by the following. [Item 4] The information processing system described in item 1, The estimation unit estimates the mental state based on the second degree of variation and at least one of the first degree of variation or the degree of the attribute. An information processing system characterized by the following. [Item 5] The information processing system described in item 1, The estimation unit estimates the mental state by providing at least the second degree of variability to a learning model created by machine learning using at least the second degree of variability and the mental state as training data. An information processing system characterized by the following. [Item 6] The process involves analyzing video footage of the user to detect the degree of the user's attributes, To calculate a second degree of variation of the first degree of variation relating to the degree of the attribute in the aforementioned moving image, At least the mental state of the user is estimated based on the second degree of variation, An information processing method characterized by a computer executing the following. [Explanation of symbols]

[0055] 1 User terminal 2 Management Server

Claims

1. A detection unit analyzes video footage of the user to detect the degree of multiple attributes, including the user's facial expressions. A calculation unit that calculates a second degree of variation of a first degree of variation relating to the degree of the attribute in the aforementioned moving image, An estimation unit that estimates the degree of the user's emotional distress based at least on the second degree of variation, An information processing system characterized by comprising the following features.

2. The information processing system according to claim 1, The detection unit detects the degree of the attribute for each predetermined length interval, An information processing system characterized by the following.

3. The information processing system according to claim 1, The first and second degrees of variability are expressed by the standard deviation. An information processing system characterized by the following.

4. The information processing system according to claim 1, The estimation unit estimates the degree of emotional distress based on the second degree of variation and at least one of the first degree of variation or the degree of the attribute. An information processing system characterized by the following.

5. The information processing system according to claim 1, The estimation unit estimates the degree of emotional distress by providing at least the second degree of variability to a learning model created by machine learning, which uses at least the second degree of variability and the degree of emotional distress as training data. An information processing system characterized by the following.

6. The process involves analyzing video footage of a user to detect the degree of multiple attributes, including the user's facial expressions, To calculate a second degree of variation of the first degree of variation relating to the degree of the attribute in the aforementioned moving image, At least the degree of the user's emotional distress is estimated based on the second degree of variation, An information processing method characterized by a computer executing the following.

Images