Information processing system and information processing method

Abstract

To enable estimation of a user's mental condition.SOLUTION: An information processing system comprises: a detection part that detects the degree of an attribute of a user by analyzing a dynamic picture image obtained by shooting the user; a calculation part that calculates first and second variation degrees according to the degree of the attribute of the dynamic picture image; and an estimation part that estimates a user's mental condition on the basis of at least the second variation degree.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system and an information processing method. [Background technology]

[0002] There is known a technique for analyzing the emotions felt by others in response to a speaker's statement (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-58625 Summary of the Invention [Problem to be solved by the invention]

[0004] However, although the technology of Patent Document 1 can analyze the emotions of a target person, it cannot estimate a specific mental state.

[0005] The present invention has been made in view of the above background, and aims to provide a technique that can estimate the mental state of a user. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an information processing system comprising: a detection unit that analyzes video images of a user to detect the degree of the user's attributes; a calculation unit that calculates a second degree of variation of a first degree of variation related to the degree of the attributes in the video images; and an estimation unit that estimates the mental state of the user based on at least the second degree of variation.

[0007] Other problems and solutions disclosed in this application will be made clear in the section on preferred embodiments of the invention and the drawings. [Effects of the Invention]

[0008] According to the present invention, it is possible to estimate the mental state of a user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of a management server 2. [Figure 3] FIG. 2 illustrates an example of the software configuration of a management server 2. [Figure 4] FIG. 10 is a diagram illustrating the operation of the management server 2. [Figure 5] FIG. 1 shows an example of the results of a SHAP analysis. DETAILED DESCRIPTION OF THE INVENTION

[0010] <System Overview> An information processing system according to an embodiment of the present invention will be described below. The information processing system of this embodiment attempts to estimate a user's mental state (particularly the degree of depression) from a video of the user.

[0011] In this embodiment, "mental state" refers not to a mere temporary physiological state (such as sleepiness or fatigue) or a cognitive function state (such as concentration or attention), but to a more sustained emotional and psychological state of health. Specifically, it includes emotional aspects such as the degree of depression, anxiety, and stress, and in psychiatry, it is a concept related to the severity assessment of mood-related disorders such as depression and anxiety disorders. Mental state is a concept that captures emotional and mood fluctuations, and is clearly distinguished from cognitive states such as "concentration" and "attention" that result from the allocation of cognitive resources, or physiological states such as "drowsiness" and "fatigue" that are primarily caused by the level of physical arousal.

[0012] In conventional technologies, for example, in drowsiness detection systems and concentration monitoring systems, temporary states are detected using attributes such as blinking frequency and eye opening degree. In contrast, the "mental state" estimated in this embodiment is not a temporary cognitive state, but rather an emotional and psychological state that lasts for several days to several weeks.

[0013] Furthermore, while assessment of such mental states has traditionally been based on self-reporting using questionnaires such as the Quick Inventory of Depressive Symptoms (QIDS), Patient Health Questionnaire-9 (PHQ-9), and Montgomery-Asberg Depression Rating Scale (MADRS), or evaluation by a physician, the information processing system of this embodiment provides an objective assessment method that replaces or complements these conventional assessment methods. In particular, by using the second variability of attribute levels (e.g., standard deviation of standard deviations), the stability / instability of emotional expression is quantitatively evaluated, thereby estimating mood-related mental states. This approach enables the assessment of deeper emotional health states, rather than simply detecting temporary cognitive states.

[0014] 1 is a diagram showing an example of the overall configuration of an information processing system. The information processing system of this embodiment is configured to include a management server 2. The management server 2 is communicably connected to a user terminal 1 via a communication network. The communication network is, for example, the Internet, and is constructed using a public telephone line network, a mobile phone line network, a wireless communication path, Ethernet (registered trademark), etc.

[0015] The user terminal 1 is a computer operated by a user, and may be, for example, a smartphone, a tablet computer, or a personal computer.

[0016] The management server 2 is a computer that estimates the user's mental state. The management server 2 may be a general-purpose computer such as a workstation or personal computer, or may be logically realized by cloud computing.

[0017] <Administration Server> FIG. 2 is a diagram illustrating an example of the hardware configuration of the management server 2. Note that the illustrated configuration is an example, and other configurations may also be used. The management server 2 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 205 is used to input data, and is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like. The output device 206 is used to output data, and is, for example, a display, a printer, a speaker, or the like. Each functional unit of the management server 2 described below is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit of the management server 2 is realized as part of the storage area provided by the memory 202 and the storage device 203.

[0018] 3 is a diagram illustrating an example of the software configuration of the management server 2. The management server 2 includes a learning model storage unit 231, a detection unit 211, a calculation unit 212, an estimation unit 213, and an output unit 214.

[0019] <Storage section> The learning model memory unit 231 stores a first learning model for detecting the degree of an attribute of a user appearing in a video image (hereinafter referred to as attribute degree), and a second learning model for estimating the mental state of the user based on the detected attribute degree.

[0020] The first and second learning models in this embodiment are created by machine learning. Machine learning can be broadly divided into supervised learning and unsupervised learning. Supervised learning is a method of training a model using input data and corresponding output data (supervised data), and by adjusting model parameters based on the supervised data, a mapping from input data to output data is learned. On the other hand, unsupervised learning is a method of learning the structure and pattern of input data without using supervised data, and learning the density distribution and feature representation of the input data. As the first and second learning models in this embodiment, for example, neural networks, support vector machines, decision trees, random forests, and the like, which are types of supervised learning, can be used. Self-organizing maps, k-means methods, and the like, which are types of unsupervised learning, can also be used. These machine learning algorithms perform processes such as weighting and conversion on the input data to learn the relationship between the input data and output data, and optimize parameters to minimize the error with the output data. This makes it possible to learn the mapping from input data to output data.

[0021] The first learning model can be created by machine learning using features extracted from video and attribute degrees as training data. Features input to the first learning model include, for example, images of the face region extracted from each frame of the video, the positions and sizes of organs such as the eyes, nose, and mouth extracted from the face region, and temporal changes in these organs. On the other hand, attribute degrees output by the first learning model include, for example, the number of blinks, the degree of eye opening, the degree of mouth opening, the position of eyebrows, the direction of the face, and temporal changes therein. Specifically, the first learning model can input features such as images of the face region and the positions and sizes of organs, and output attribute degrees such as the number of blinks and the degree of eye opening. Attributes can include the number of blinks, eye offset (the angle of the eyes relative to the camera), gaze estimated from the eye offset, and facial expression. Facial expressions include anger, disgust, fear, happiness, sadness, surprise, neutral, negative / positive, etc., and it is possible to infer the number of times these expressions appear in a given period (such as 1 second), the average or median of their direction, and the degree of emotion (facial expression) such as anger.

[0022] The second learning model is a learning model that estimates a mental state (degree of depression) when given at least one of the degree (value) of an attribute, a statistical value (such as a standard deviation) related to the degree of the attribute, and a statistical value of the statistical value (such as a standard deviation of standard deviations). In this embodiment, the feature values ​​given to the second learning model include at least a statistical value of a statistical value (such as a standard deviation of standard deviations). The second learning model can be created by machine learning using at least one of the degree of the attribute, the statistical value related to the degree of the attribute, and the statistical value of the statistical value, as well as a mental state determined by an expert, as training data.

[0023] Furthermore, the inventors conducted SHAP (Shapley Additive Explanations) analysis to visualize the contribution of features input to the second learning model. Figure 5 shows an example of the results of the SHAP analysis. As shown in Figure 5, features representing the standard deviation of standard deviations (hereinafter referred to as second-order variability) with the suffix "ss"—for example, blink_ss, fear_ss, and positive_ss—showed a higher SHAP value distribution for the model output than other average values ​​or first-order variability (standard deviations), and were confirmed to be extremely important in estimating the user's level of depression.

[0024] For example, it has been shown that users with a tendency toward depression show marked instability in the fluctuations of certain emotional expressions (especially fear and sadness). Specifically, it can be said that while the fluctuations (standard deviation) of emotional expressions are relatively stable under normal conditions, when the mental state deteriorates, the fluctuations themselves tend to become unstable (the standard deviation of the standard deviation becomes larger).

[0025] The degree of secondary variation is obtained by calculating the standard deviation of the attribute degree for each predetermined time window (for example, 30 seconds) and then calculating the standard deviation again for the standard deviation values ​​for each time window. This allows us to capture the "fluctuation" of the attribute change itself and quantify long-term instability that is distinct from momentary emotional reactions, thereby significantly improving the detection sensitivity of the degree of depression (or excitement).

[0026] <Functional section> The detection unit 211 analyzes the video to detect the degree of a user's attribute. The detection unit 211 can detect the degree of the attribute for each section of a predetermined length. The detection unit 211 can estimate the degree of the attribute by providing the video to a first learning model.

[0027] The calculation unit 212 calculates a second degree of variation (secondary degree of variation) of a first degree of variation related to the degree of an attribute in a video. Specifically, the calculation unit 212 calculates a first degree of variation (e.g., a standard deviation of the degrees of an attribute) which is a degree of variation in the degrees of an attribute, and further calculates a second degree of variation (e.g., a standard deviation of the standard deviations of the degrees of an attribute) which is a degree of variation of the first degree of variation. In this embodiment, the degree of variation is assumed to be a standard deviation, but it may also be a variance. For example, the calculation unit 212 can calculate a first degree of variation which is a standard deviation of the degree of anger, and a second degree of variation which is a standard deviation of the standard deviation.

[0028] The estimation unit 213 estimates the user's mental state based on at least the second degree of variation. The estimation unit 213 can estimate the mental state based on the second degree of variation and at least one of the first degree of variation and the attribute degree. The estimation unit 213 can estimate the mental state by applying at least the second degree of variation to a second learning model.

[0029] The output unit 214 outputs the estimated mental state.

[0030] <Operation> FIG. 4 is a diagram illustrating the operation of the management server 2.

[0031] The management server 2 acquires video captured by a user (S301), estimates the degree of each attribute of the user for a predetermined period (e.g., one second) based on the acquired video and the first learning model (S302), and calculates the degree of variation of the estimated degrees (S303). Here, standard deviation or variance can be used as the degree of variation. For example, when calculating the standard deviation of the attribute degrees, the calculation unit 212 calculates the standard deviation from the estimated attribute degrees. On the other hand, when calculating the variance of the attribute degrees, the calculation unit 212 calculates the variance from the estimated attribute degrees. Next, the management server 2 calculates the degree of variation of the calculated variance degrees (S304). For example, when calculating the standard deviation of the standard deviation of the attribute degrees, the calculation unit 212 calculates the standard deviation from the standard deviation of the attribute degrees. On the other hand, when calculating the variance of the variance of the attribute degree, the calculation unit 212 calculates the variance from the variance of the attribute degrees. Then, the management server 2 provides at least the degree of variation of the degree of variation (as well as the degree of each attribute and / or the degree of variation of the degree) to a second learning model to estimate the user's mental state (S305), and outputs the estimated mental state (S306).

[0032] As described above, the information processing system of this embodiment can estimate a user's mental state from video images of the user, allowing for easy estimation of the user's mental state without using tests such as QIDS. Furthermore, the information processing system of this embodiment can perform estimation using the standard deviation of the standard deviations of the attribute degrees as a feature. For example, a user whose mental state is deteriorating may overreact to certain topics while showing no interest in other topics. By evaluating the standard deviation of the standard deviations (the degree of variation of the degree of variation), it is possible to evaluate the degree of variation in facial expressions, etc. over time, which is expected to improve the accuracy of mental state estimation.

[0033] Furthermore, according to the information processing system of this embodiment, by adopting not only the first degree of variation in the degree of a user's attributes but also the second degree of variation as feature quantities, it is possible to accurately grasp the degree of "instability" over time of emotional expressions such as facial expressions, gaze, and blinking. As a result, it is possible to estimate with high accuracy even mild to moderate depression, which is often overlooked by conventional mean-centered methods, and it is possible to improve the reliability of mental health screening.

[0034] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0035] For example, the processing by each of the functional units of the management server 2 described above may be performed by any of the functional units. Also, a different functional unit that performs part of the processing by each of the functional units described above may be added. Also, the functional units of the management server 2 may be distributed across multiple computers.

[0036] Furthermore, the information stored in each storage unit of the management server may be stored in any of the storage units. That is, the information stored in the above-mentioned multiple storage units may be stored in one storage unit, or part of the information stored in one of the above-mentioned storage units may be stored in another storage unit.

[0037] <Variation 1> In the above embodiment, the degree of a user's attributes is detected from a moving image, but this is not limiting. In addition to a moving image, the degree of a user's attributes may also be detected from audio. For example, the user's voice can be acquired using an audio input device such as a microphone, and the degree of attributes such as the user's voice volume, voice intonation, and speaking speed can be detected from the acquired audio. For example, the voice amplitude, sound pressure, and volume can be used as the voice volume. For example, the amount of change in the voice fundamental frequency (pitch) can be used as the voice intonation. For example, the speaking speed can be used as the number of syllables or number of words per unit time.

[0038] As in the above embodiment, a first degree of variation (such as a standard deviation or variance) and a second degree of variation (such as a standard deviation of standard deviations or a variance of variances) can be calculated for the attribute degrees based on the detected audio, and these values ​​can be used to estimate the mental state. For example, the estimation unit 213 can estimate the user's mental state by using the attribute degrees detected from the audio, as well as the attribute degrees detected from the video, and their variances. This makes it possible to improve the accuracy of estimating the mental state by using information that cannot be obtained from the video alone.

[0039] <Variation 2> In the above embodiment, the degree of one attribute (for example, the degree of smile or the degree of eye opening) is used as the degree of an attribute, but a value obtained by combining multiple attributes may also be used as the degree of an attribute. For example, the detection unit 211 can detect the degree of smile and the degree of eye opening from a moving image, and calculate a value obtained by combining these values ​​as the degree of an attribute. As a method for combining the degree of smile and the degree of eye opening, for example, a weighted sum of the degree of smile and the degree of eye opening, or a product of the degree of smile and the degree of eye opening, may be used.

[0040] For the degree of combination of the plurality of attributes calculated in this way, a first degree of variation (such as a standard deviation or variance) and a second degree of variation (such as a standard deviation of standard deviations or a variance of variances) can be calculated, and these values ​​can be used to estimate the mental state. For example, the estimation unit 213 can estimate the user's mental state by using the degree of combination of the smile degree and the eye opening degree and the degree of variation thereof.

[0041] Furthermore, a value that combines three or more attributes may be used as the degree of an attribute. For example, a value that combines the degree of smile, eye openness, and voice volume may be used as the degree of an attribute. This makes it possible to improve the accuracy of mental state estimation by using composite information that cannot be obtained from a single attribute alone.

[0042] <Variation 3> In the above embodiment, the management server 2 acquires a video from a user terminal and detects the degree of an attribute from the video, but this is not limited to this. The user terminal 1 may detect the degree of an attribute from the video and transmit the detected degree of the attribute to the management server 2.

[0043] Specifically, the user terminal 1 captures an image of the user using an imaging device such as a camera, and detects the degree of the user's attribute from the captured video image in the same manner as in the above embodiment. The user terminal 1 then transmits the detected degree of the attribute to the management server 2. The degree of the attribute may be detected, for example, at a predetermined time interval (for example, every second), and the degree of the attribute for each detected time interval may be transmitted to the management server 2.

[0044] Based on the attribute degrees received from the user terminal 1, the management server 2 calculates, as in the above embodiment, a first degree of variation (such as a standard deviation or variance) of the attribute degrees and a second degree of variation (such as a standard deviation of standard deviations or a variance of variances) of the first degree of variation, and estimates the user's mental state using the calculated degree of variation.

[0045] Alternatively, the user terminal 1 may be provided with all of the functional units and storage units of the management server 2 without providing a management server 2, and the user terminal 1 may detect the degree of attributes and also detect the mental state. In this case, the management server 2 may be provided with the functions of managing the learning model and providing input data to the learning model, and the function of sending input data to the management server 2 and generating answers may be delegated to it.

[0046] <Variation 4> In the above embodiment, the mental state of the current user is estimated based on the degree of the current user's attributes and the degree of variation thereof, but this is not limited to this. The mental state of the user in the future may also be estimated based on the degree of the current and past attributes and the degree of variation thereof.

[0047] Specifically, the management server 2 acquires the attribute levels and their variability over a predetermined period up to the present (for example, the most recent week), and then estimates not only the current mental state but also the future mental state based on the acquired attribute levels and their variability.

[0048] One method for estimating future mental states is to predict future attribute degrees and their variability from current and past attribute degrees and their variability, and then estimate the future mental state based on the predicted future attribute degrees and their variability. Predicting attribute degrees and their variability can be achieved using time-series data analysis techniques (e.g., ARIMA models, RNNs, etc.).

[0049] Furthermore, future mental states may be predicted directly from changes in current and past mental states. For example, the estimation unit 213 may estimate changes in mental states over a predetermined period up to the present from the degree of an attribute and its variability over that period, and predict future mental states based on the estimated changes in mental states. Time-series data analysis techniques may also be used to predict future mental states from changes in mental states.

[0050] <Variation 5> In the above embodiment, the degree of depression is estimated as the mental state, but this is not limiting. Indicators other than the degree of depression, such as stress level or concentration, which can be grasped separately from the mental state, may also be estimated.

[0051] The stress level can be estimated based on the degree and the degree of variation of attributes such as the number and frequency of a user's blinks, the degree to which the eyes are open, the degree to which the mouth is open, the direction of the face, the volume and intonation of the voice, the speed of speech, etc. Generally, users who are highly stressed tend to blink frequently and frequently, have their eyes and mouths wide open, have an inconsistent direction of the face, have an unstable volume and intonation of the voice, and speak at a fast speed, so it is possible to estimate the stress level from the degree and the degree of variation of these attributes.

[0052] Concentration can be estimated based on the degree and variability of attributes such as the amount and speed of movement of the user's gaze, the degree of fixation of the gaze, the number and frequency of blinks, etc. Generally, users with high concentration tend to have a small amount and speed of movement of the gaze, a long time when the gaze is fixed on one point, and a small number and frequency of blinks, so it is possible to estimate concentration from the degree and variability of these attributes.

[0053] The estimation unit 213 can estimate the user's stress level and concentration level using a learning model that has learned the relationship between the degree of the above-mentioned attributes and the degree of their variance, and the stress level and concentration level.

[0054] <Disclosures> The present disclosure also includes the following configurations. [Item 1] a detection unit that analyzes a moving image of a user and detects a degree of the attribute of the user; a calculation unit that calculates a second degree of variation of a first degree of variation related to the degree of the attribute in the moving image; an estimation unit that estimates a mental state of the user based on at least the second degree of variation; An information processing system comprising: [Item 2] Item 1, an information processing system according to item 1, the detection unit detects the degree of the attribute for each section of a predetermined length; An information processing system characterized by: [Item 3] Item 1, an information processing system according to item 1, the first and second degrees of variation are expressed by standard deviations; An information processing system characterized by: [Item 4] Item 1, an information processing system according to item 1, the estimation unit estimates the mental state based on the second degree of variation and at least one of the first degree of variation and a degree of the attribute; An information processing system characterized by: [Item 5] Item 1, an information processing system according to item 1, the estimation unit estimates the mental state by applying at least the second degree of variation to a learning model created by machine learning using at least the second degree of variation and the mental state as training data; An information processing system characterized by: [Item 6] Analyzing a moving image of a user to detect a degree of the attribute of the user; calculating a second degree of variation of a first degree of variation related to the degree of the attribute in the video; estimating a mental state of the user based on at least the second degree of variation; An information processing method characterized by being executed by a computer. [Explanation of symbols]

[0055] 1. User terminal 2 Management Server

Claims

1. a detection unit that analyzes a moving image of a user and detects a degree of the attribute of the user; a calculation unit that calculates a second degree of variation of a first degree of variation related to the degree of the attribute in the moving image; an estimation unit that estimates a mental state of the user based on at least the second degree of variation; An information processing system comprising:

2. 2. The information processing system according to claim 1, the detection unit detects the degree of the attribute for each section of a predetermined length; An information processing system characterized by:

3. 2. The information processing system according to claim 1, the first and second degrees of variation are expressed by standard deviations; An information processing system characterized by:

4. 2. The information processing system according to claim 1, the estimation unit estimates the mental state based on the second degree of variation and at least one of the first degree of variation and a degree of the attribute; An information processing system characterized by:

5. 2. The information processing system according to claim 1, the estimation unit estimates the mental state by applying at least the second degree of variation to a learning model created by machine learning using at least the second degree of variation and the mental state as training data; An information processing system characterized by:

6. Analyzing a moving image of a user to detect a degree of the attribute of the user; calculating a second degree of variation of a first degree of variation related to the degree of the attribute in the moving image; estimating a mental state of the user based on at least the second degree of variation; An information processing method characterized by being executed by a computer.

Images