Information processing system, information processing device, information processing method, and information processing program

Abstract

In a conventional fatigue / stress examination system, the state of autonomic nerves is measured from a subject's electrocardiographic / pulse data, and the degree of fatigue and stress tendency are expressed in numerical values. However, such numerical value data does not reflect a subjective determination result based on the subject's voice or facial expression during questioning or an interview by a doctor, industrial physician, or health nurse, and therefore does not sufficiently reflect the subject's mood (mental conditions).　The present invention provides an information processing system and the like in which not only quantitative data, such as pulse and heartbeat acquired from a pulse meter, but also the results of a stress check and data such as the subject's voice and facial expression image during counseling using a video call are acquired, and values of stress, brain fatigue, and mood levels are calculated from the data and plotted in a three-dimensional space having an X-axis, a Y-axis, and a Z-axis to visualize the subject's mood and the like. Thus, it is possible to analyze the subject's mood with high accuracy.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, an information processing method, and an information processing program (hereinafter referred to as "information processing system, etc.") for realizing early detection of mental disorders and contributing to solving social problems by analyzing and visualizing various data such as a subject's questionnaire, voice, facial expression, pulse wave, and heartbeat.

[0002] Specifically, a subject (user) uses a questionnaire inspection site on a network such as the Internet to answer a questionnaire about stress (stress check), and then, by using a video chat with a counselor or a pulse wave measuring device, obtains the subject's questionnaire data, facial expression image data, voice data, pulse wave, and heartbeat data, etc. Based on the obtained various data, values of stress level, brain fatigue level, and mood level are calculated, and the calculated values are plotted in a three-dimensional space composed of the X-axis, Y-axis, and Z-axis, thereby visualizing the subject's emotions (mental state), etc., and relating to an information processing system, etc. for assisting diagnosis and treatment.

Background Art

[0003] In recent years, with the spread of information technology (IT), while the simplification and convenience of communication have rapidly evolved, the number of mental disorder patients has been on the increase due to lack of communication, work stress, fatigue, etc. Therefore, a diagnostic system has been proposed that can easily grasp fatigue and stress by the subject himself / herself by quantitatively measuring and objectively evaluating data related to the stress state and fatigue state of a person (subject).

[0004] For example, the fatigue and stress diagnostic system described in Patent Document 1 analyzes electrocardiogram and pulse wave data measured by an electrocardiogram and pulse wave measuring device by using an analysis server on the cloud side, and can grasp the stress state numerically from the balance and strength of the autonomic nerves. The analysis data can also be transmitted to the client terminal side and visually displayed on the client terminal side.

[0005] Furthermore, the health value estimation system described in Patent Document 2 classifies characteristic behaviors (behavioral features) that appear during stress into multiple clusters and quantifies them from behavioral history such as location information, movement information, power on / off status, application launch logs, and the number of phone calls, obtained from various sensors installed in mobile terminals such as smartphones. It then uses machine learning to learn the relationship between these characteristics and the stress state based on previously measured heart rate data, and constructs an estimation model. By comparing the numerical values ​​of newly acquired behavioral features using mobile terminals such as smartphones with the constructed estimation model, it is possible to estimate a health value that indicates the subject's own health status. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2015-054002 [Patent Document 2] Japanese Patent Publication No. 2018-181004 [Non-patent literature]

[0007] [Non-Patent Document 1] Stress Check System Implementation Manual - Ministry of Health, Labour and Welfare (URL https: / / www.mhlw.go.jp / bunya / roudoukijun / anzeneisei12 / pdf / 150709-1.pdf) (Accessed February 15, 2021) [Non-Patent Document 2] Check COVID-19 stress levels with your fingertips: Yamagata University jointly develops device, July 18, 2020, Asahi Shimbun Digital (URL https: / / www.asahi.com / articles / ASN7K6V99N78UZHB00M.html) (Retrieved February 15, 2021) [Non-Patent Document 3] Yuki Aoki (and 3 others), Development of a smartphone application that estimates fatigue levels from voice, E-037 FIT2013 (URL https: / / www.ieice.org / publications / conference-FIT-DVDs / FIT2013 / data / pdf / E-037.pdf) (Retrieved February 15, 2021) [Non-Patent Document 4] Face classification and detection (URL https: / / github.com / oarriaga / face_classification) (Retrieved February 15, 2021) [Non-Patent Document 5] Russell, JA (1 others), Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant, Journal of Personality and Social Psychology, 76(5), 805-819 [Non-Patent Document 6] Sayaka Akiyama (and 1 other), Stress State Estimation Method Using a Pulse Sensor for a QOL Visualization System, Proceedings of the 77th National Convention of the Information Processing Society of Japan, 2U-07, 2015 (URL https: / / ipsj.ixsq.nii.ac.jp / ej / index.php?action=pages_view_main&active_action=repository_action_common_download&item_id=164634&item_no=1&attribute_id=1&file_no=1&page_id=13&block_id=8) (Retrieved February 15, 2021) [Overview of the project] [Problems that the invention aims to solve]

[0008] The fatigue and stress screening system described in Patent Document 1 simultaneously measures the electrocardiogram and pulse wave of a subject, measures the state of the subject's autonomic nervous system from the electrocardiogram and pulse wave data, and centrally manages the fatigue level and stress tendency as fatigue analysis result data, quantifying them in a visual format. However, since this fatigue analysis result data does not reflect the subjective judgment results based on the subject's own voice and facial expressions obtained through interviews and consultations by doctors, industrial physicians, public health nurses, etc., it may not be possible to achieve a highly accurate analysis of the subject's emotions.

[0009] Furthermore, the health value estimation system described in Patent Document 2 can estimate a subject's own health value by constructing an estimation model using accurately quantified data that can serve as more appropriate training data in supervised machine learning. However, this health value estimation system cannot accurately quantify depressed moods if the subject is unaware of their depression, and therefore cannot use subjective judgments based on the subject's voice and facial expressions, which are obtained through interviews and consultations by doctors, industrial physicians, or public health nurses, as training data. As a result, it is not possible to construct an estimation model that sufficiently reflects the subject's emotions (mental state), and therefore the health value estimation system described in Patent Document 2 may not be able to analyze the subject's emotions with high accuracy.

[0010] Therefore, in order to achieve a higher accuracy in analyzing the emotions of subjects compared to the conventional system described above, the present invention acquires not only quantitative data of subjects measured by measuring instruments (pulse waves, heart rate, etc.), but also data such as stress check results based on a stress check questionnaire (Simplified Occupational Stress Questionnaire, Non-Patent Literature 1), and data such as the subject's voice and facial expression images during counseling using communication tools such as video chat (video call). From this data, values ​​for stress level, brain fatigue level, and mood level are calculated, and these calculated values ​​are plotted in a three-dimensional space consisting of the X, Y, and Z axes, thereby providing an information processing system that can visualize the emotions (mental state) of subjects and support diagnosis and treatment. [Means for solving the problem]

[0011] As one embodiment of the information processing device according to the present invention, an information processing device connected to a subject's terminal device for visualizing the subject's emotions includes a data management unit that acquires at least data relating to the subject's voice, facial expression images, and pulse waves, An emotion expression engine unit calculates the degree of brain fatigue based on the frequency of the aforementioned sound, extracts the subject's emotions from the aforementioned facial expression image to calculate the mood level, performs frequency analysis of the pulse wave using Fast Fourier Transform, extracts high-frequency and low-frequency sections to calculate the stress level, A three-axis processing unit that displays a graph in which points are plotted at coordinates corresponding to the brain fatigue level, mood level, and stress level in a three-dimensional space consisting of the X, Y, and Z axes. Includes, The audio data is obtained by recording at least a portion of a video call with the subject via the terminal device. The data relating to the facial expression images is obtained by recording at least a portion of a video call with the subject via the terminal device. The data relating to the pulse wave is acquired from a pulse wave measuring device that measures the pulse wave of the subject, via the terminal device.

[0012] In a preferred embodiment of the information processing device according to the present invention, the data management unit stores the data relating to the subject's voice, facial expression image, and pulse wave, along with the date and time on which the data was acquired, in the storage means of the information processing device. The three-axis processing unit is characterized by displaying a graph in which points are plotted in chronological order in the three-dimensional space at coordinates corresponding to the subject's brain fatigue level, mood level, and stress level for each date and time.

[0013] In a preferred embodiment of the information processing device according to the present invention, the three-dimensional space is divided into a plurality of type-based classification categories, The three-axis processing unit is characterized by notifying a category to which a coordinate point corresponding to the brain fatigue degree, the temperament degree, and the stress degree of the subject belongs among the plurality of type-based classification categories in the three-dimensional space.

[0014] As a preferred embodiment of the information processing apparatus according to the present invention, an improvement plan to be proposed to the subject is determined for each of the plurality of type-based categories. The emotion expression engine unit is characterized by notifying an improvement plan for a category to which a coordinate point corresponding to the brain fatigue degree, the temperament degree, and the stress degree of the subject belongs in the three-dimensional space.

[0015] As a preferred embodiment of the information processing apparatus according to the present invention, the data related to the voice is data obtained by continuously recording, for at least a predetermined recording time, the voice of the subject reading out a predetermined fixed sentence displayed on the terminal device during a video call with the subject via the terminal device.

[0016] As a preferred embodiment of the information processing apparatus according to the present invention, the emotion expression engine unit acquires one or more of the CEM values per subject from the data related to the voice by executing a brain activity index measurement algorithm for measuring a CEM value representing a brain activity index. The brain fatigue degree is characterized by being an average value of one or more of the CEM values.

[0017] As a preferred embodiment of the information processing apparatus according to the present invention, the data related to the pulse wave is data obtained by dividing the pulse wave measured by the pulse wave measuring device for each section with a predetermined time interval as one section.

[0018] As a preferred embodiment of the information processing apparatus according to the present invention, the emotion expression engine unit divides the pulse wave within the one section by a Hamming window for each of the one sections of the pulse wave, and calculates a pulse interval PPI, which is an interval between peaks of one beat of the pulse wave, and a time for the pulse wave within the Hamming window. The emotional expression engine unit generates a time-PPI graph by plotting points at coordinates corresponding to the pulse interval PPI and the time in a two-dimensional space of the horizontal axis time and the vertical axis PPI for each of the one intervals of the pulse wave. The emotional expression engine unit interpolates between discrete values in the time-PPI graph and applies a fast Fourier transform FFT, and integrates the power spectral density PSD of the result of the FFT within the low-frequency interval and the high-frequency interval, respectively, to calculate an LF value corresponding to the low-frequency component, a HF value corresponding to the high-frequency component, and an LF / HF value. The stress level is characterized in that it is based on at least one value among the LF value, the HF value, and the LF / HF value.

[0019] As a preferred embodiment of the information processing apparatus according to the present invention, the low-frequency interval is from 0.04 Hz or more to less than 0.15 Hz. The high-frequency interval is characterized in that it is from 0.15 Hz or more to less than 0.4 Hz.

[0020] As a preferred embodiment of the information processing apparatus according to the present invention, the data regarding the facial expression image of the face is data obtained by continuously recording a moving image of the facial expression of the subject for at least a predetermined recording time during a video call with the subject via the terminal device.

[0021] As a preferred embodiment of the information processing apparatus according to the present invention, the emotional expression engine unit counts each of a plurality of emotional expressions recognized from a moving image of the facial expression of the subject included in the data regarding the facial expression image of the face by executing a facial expression recognition algorithm. The emotional expression engine unit calculates a ratio for each of the plurality of emotional expressions, multiplies a predetermined weight for each of the plurality of emotional expressions by the ratio for each of the plurality of emotional expressions for each emotional expression, and calculates an emotional index for each emotional expression. The mood level is characterized in that it is based on a value obtained by dividing the largest maximum mood index among the mood indexes for each emotional expression by the total value of the mood indexes for each emotional expression.

[0022] A preferred embodiment of the information processing device according to the present invention is characterized in that the plurality of emotion expressions are happy, surprise, neutral, fear, angry, disgust, and sad in Russell's Circle of Emotions model.

[0023] In a preferred embodiment of the information processing device according to the present invention, the data management unit acquires environmental data including at least temperature and humidity, in addition to data relating to the subject's voice, facial expression image, and pulse wave. The emotion expression engine is characterized by adjusting the values ​​of brain fatigue, mood, and stress by multiplying each of these values ​​by a weighting coefficient determined based on a discomfort index calculated from the temperature and humidity included in the environmental data.

[0024] In a preferred embodiment of the information processing device according to the present invention, the data management unit acquires questionnaire data including the subject's voice, facial expression image, pulse wave data, and stress check result score. The emotion expression engine is characterized by adjusting the values ​​of the brain fatigue level, mood level, and stress level by multiplying each of these values ​​by a weighting coefficient determined according to the score included in the questionnaire data.

[0025] As one embodiment of the information processing method according to the present invention, the information processing method is executed on a server that can be connected to the subject's terminal device via a network, The steps include acquiring data from the terminal device relating at least the subject's voice, facial expression image, and pulse wave, The steps include: calculating the degree of brain fatigue based on the frequency of the aforementioned sound; calculating the mood level by extracting the subject's emotions from the aforementioned facial expression image; performing frequency analysis on the pulse wave using Fast Fourier Transform to extract high-frequency and low-frequency sections and calculate the stress level; The step of displaying a graph in which points are plotted at coordinates corresponding to the brain fatigue level, mood level, and stress level in a three-dimensional space consisting of the X, Y, and Z axes. Includes, The audio data is obtained by recording at least a portion of a video call with the subject via the terminal device. The data relating to the facial expression images is obtained by recording at least a portion of a video call with the subject via the terminal device. The data relating to the pulse wave is acquired from a pulse wave measuring device that measures the pulse wave of the subject, via the terminal device.

[0026] As one embodiment of the information processing system according to the present invention, the information processing system is: The aforementioned information processing device, A terminal device that can access the aforementioned information processing device via a network and Includes, The terminal device transmits at least the voice data, the facial expression image data, and the pulse wave data to the information processing device. The information processing device receives data relating to voice, data relating to facial expression images, and data relating to pulse waves, and transmits the brain fatigue level, mood level, and stress level calculated based on the received data to the terminal device, and displays on the terminal device a graph in which points are plotted at coordinates corresponding to the brain fatigue level, mood level, and stress level in a three-dimensional space consisting of the X, Y, and Z axes.

[0027] One embodiment of the information processing program according to the present invention is characterized in that the program is executed by a computer, thereby causing the computer to function as a part of the information processing device.

[0028] Another embodiment of the information processing program according to the present invention is characterized in that the program is executed by a computer, thereby causing the computer to execute each stage of the information processing method. [Effects of the Invention]

[0029] The information processing system according to the present invention acquires not only quantitative data such as pulse waves and heart rate obtained from a pulse wave measuring device, but also data such as the results of stress checks and the subject's voice and facial expression images during counseling using video calls. From this data, it calculates stress levels, brain fatigue levels, and mood levels, and plots them in a three-dimensional space consisting of X, Y, and Z axes. By providing an information processing system that can visualize the subject's emotions and support diagnosis and treatment, it is possible to analyze the subject's emotions with high accuracy, thereby enabling the early detection of mental illness and contributing to the resolution of social issues. [Brief explanation of the drawing]

[0030] [Figure 1] This figure shows an example of the configuration of an information processing system according to one embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the hardware configuration of an information processing device according to one embodiment of the present invention. [Figure 3] This is a block diagram showing the configuration of an information processing device according to one embodiment of the present invention. [Figure 4] Figure 3 shows an example of data stored in the user information database of the information processing device shown in the diagram, presented in tabular format. [Figure 5] This flowchart shows the process of collecting various data from the subject's terminal device. [Figure 6] This figure shows an example of a user interface for conducting stress checks using questionnaires. [Figure 7] This figure shows an example of a user interface that prompts users to register after a stress check. [Figure 8] This figure shows an example of a screen displaying the results of a stress check using a radar chart. [Figure 9] This figure shows an example of a screen displaying comments about the results of a stress check. [Figure 10] This diagram shows how a pulse wave measuring device is used to measure pulse waves and heart rate from a subject's fingertip. [Figure 11] This figure shows an example of the screen display on the subject's terminal device when measuring the subject's pulse wave and heart rate with a pulse wave measuring device. [Figure 12] This figure shows an example of a screen display used to obtain images of a subject's facial expressions from a video call between the counselor and the subject during a counseling session. [Figure 13] This figure shows an example of a screen display used to capture the subject's voice from a video call between the counselor and the subject during a counseling session. [Figure 14] This is a schematic diagram showing the configuration of the emotion expression engine unit, which calculates various indicators representing brain fatigue, mood, and stress from various collected data. [Figure 15] This diagram illustrates the weighting determined based on Russell's Circle of Affect model. [Figure 16] This figure shows an example of how mood levels were calculated from various emotional expressions based on Russell's Circle of Affect model. [Figure 17] This figure shows an example of numerical conversion for plotting the obtained values ​​of various indicators representing brain fatigue, mood, and stress on a three-axis space. [Figure 18] This figure shows an example of a graph displaying a three-dimensional space consisting of X, Y, and Z axes, plotting the changes in a subject's emotions over time, as obtained by an emotion expression engine. [Figure 19] This figure shows an example of a graph displaying a 3D space consisting of X, Y, and Z axes, plotting the emotional changes of another subject obtained by the emotion expression engine over time. [Figure 20] This diagram shows an example of type-based classification categories defined in a three-dimensional space consisting of a tension axis (X-axis), a brain fatigue axis (Y-axis), and a mood axis (Z-axis). [Modes for carrying out the invention]

[0031] Embodiments of the present invention will be described below with reference to the accompanying drawings. The following embodiments are illustrative for explaining the present invention and are not intended to limit the present invention to these embodiments only. Furthermore, the present invention can be modified in various ways without departing from its essence. In addition, the same reference numerals are used for the same components in each drawing whenever possible, and redundant explanations are omitted.

[0032] Figure 1 shows an example of the configuration of an information processing system according to one embodiment of the present invention. The information processing system for visualizing the emotions of a subject includes, as an example, an information processing device 10 and n terminal devices 20-n (where n is any integer value of 1 or more). In the figure, the n terminal devices are shown as terminal devices 20-1, 20-2 through 20-n. However, in the following description, when these n terminal devices are described without distinction, some reference numerals will be omitted and they will simply be referred to as "terminal device 20".

[0033] The information processing device 10 is, for example, a computer that can connect to a network N such as a server. The terminal device 20 is, for example, a personal computer, a laptop computer, a smartphone, a mobile phone, or other terminal that can connect to a network N.

[0034] Network N may be an open network such as the internet, or it may be a closed network such as an intranet connected by a dedicated line. Network N is not limited to these, and a combination of closed and open networks can be used as appropriate, depending on the required level of security.

[0035] The information processing device 10 and the terminal device 20 are connected to a network N and can communicate with each other. The subject (user) can use the terminal device 20 to access the information processing device 10 and send their answers to the stress check questionnaire (medical questionnaire) to the information processing device 10. The stress check questionnaire is, for example, the stress check questionnaire in the Ministry of Health, Labour and Welfare's stress check implementation program (Non-Patent Literature 1).

[0036] Furthermore, subjects can also use the terminal device 20 to have a video call (video chat) with a counselor in order to receive counseling from the counselor. In addition, the terminal device 20 can transmit data on the subject's pulse wave measured using a pulse wave measuring device to the information processing device 10. The pulse wave measuring device can be, for example, a device that measures pulse waves from the subject's fingertip (Non-Patent Literature 2).

[0037] The information processing device 10 can acquire data from the terminal device 20, including at least the subject's voice, facial expression images, and pulse wave data. Based on this data, it can calculate indices representing the subject's emotions (mental state), such as brain fatigue level, mood level, and stress level, which will be described later.

[0038] Figure 2 is a block diagram showing an example of the hardware configuration of an information processing device according to one embodiment of the present invention. In the figure, reference numerals corresponding to the hardware of the information processing device 10 are not enclosed in parentheses. The hardware configuration of the terminal device 20 is the same as that of the information processing device 10, so reference numerals corresponding to the hardware of the terminal device 20 are enclosed in parentheses.

[0039] The information processing device 10 is, for example, a server (computer), and exemplified by comprising a CPU (Central Processing Unit) 11, a memory 12 consisting of ROM (Read Only Memory) and RAM (Random Access Memory), a bus 13, an input / output interface 14, an input unit 15, an output unit 16, a storage unit 17, and a communication unit 18.

[0040] The CPU 11 executes various processes according to the program stored in memory 12, or the program loaded into memory 12 from the storage unit 17. For example, the CPU 11 can execute a program to make the server (computer) function as an information processing device capable of visualizing the emotions of a subject to support diagnosis and treatment. It is also possible to implement at least some of the functions of the information processing device in hardware using application-specific integrated circuits (ASICs), etc.

[0041] Memory 12 also stores data necessary for the CPU 11 to perform various processes. The CPU 11 and memory 12 are interconnected via bus 13. An input / output interface 14 is also connected to this bus 13. An input / output interface 14 is connected to an input unit 15, an output unit 16, a storage unit 17, and a communication unit 18.

[0042] The input unit 15 can be implemented using an input device such as a keyboard or mouse, independent of the main body that houses the other parts of the information processing device 10, and can input various types of information in response to instructions from the user (administrator) of the information processing device 10. The input unit 15 may also consist of various buttons, a touch panel, or a microphone.

[0043] The output unit 16 consists of a display, speakers, etc., and outputs data related to text, still images, videos, audio, etc. The text data, still image data, video data, or audio data output by the output unit 16 is output from the display, speakers, etc., in a way that can be recognized by the user as text, images, videos, or audio.

[0044] The memory unit 17 consists of semiconductor memory such as DRAM (Dynamic Random Access Memory), a solid-state drive (SSD), a hard disk, and other storage devices, and can store various types of data.

[0045] The communication unit 18 enables communication with other devices. For example, the communication unit 18 can communicate with other devices (e.g., terminal devices 20-1, 20-2 to 20-n) via the network N.

[0046] The information processing device 10 may be equipped with a drive as needed, although this is not shown in the diagram. The drive may be fitted with removable media, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. The removable media will store a program that calculates the subject's stress level, brain fatigue level, and mood level, and plots these values ​​in a three-dimensional space consisting of the X, Y, and Z axes to visualize the subject's emotions, as well as various data such as text data and image data. The program and various data read from the removable media by the drive will be installed in the storage unit 17 as needed.

[0047] Next, the hardware configuration of the terminal device 20 will be described. As shown in Figure 2, the terminal device 20 exemplifiedly includes a CPU 21, memory 22, bus 23, input / output interface 24, input unit 25, output unit 26, storage unit 27, and communication unit 28. Each of these units has the same function as the units of the information processing device 10 described above, which have the same name but differ only in their coding. Therefore, redundant explanations will be omitted. Note that when the terminal device 20 is configured as a portable device, the hardware components of the terminal device 20, along with a display and speaker, may be implemented as an integrated device.

[0048] Referring to Figures 2 and 3, the functional configuration of the information processing device 10 included in the information processing system for visualizing the emotions of a subject will be described. Figure 3 is a block diagram showing the configuration of the information processing device according to one embodiment of the present invention. For example, when a server (computer) executes a program to perform processing such as acquiring data on at least the subject's voice, facial expression images, and pulse waves from a terminal device 20, calculating the degree of brain fatigue based on the frequency of the voice, extracting the subject's emotions from the facial expression images to calculate the mood level, performing frequency analysis of the pulse waves using the Fast Fourier Transform to extract high-frequency and low-frequency sections and calculate the stress level, and displaying a graph in which points are plotted at coordinates corresponding to the degree of brain fatigue, mood level, and stress level in a three-dimensional space consisting of the X, Y, and Z axes, the server will function as an information processing device 10, and at least the emotion expression engine unit 111, the three-axis processing unit 112, and the data management unit 113 will function using hardware resources such as the CPU 11 and memory 12.

[0049] Furthermore, by using a portion of the storage area of ​​the storage unit 17, the storage unit 17 can be made to function as a user information database 171. In another embodiment, the user information database 171 can be configured as an external storage device separate from the information processing device 10, and cloud storage, for example, can be used as the external storage device. In these embodiments, the user information database 171 is stored in a single storage device, but it may be stored in two or more separate devices.

[0050] The emotion expression engine unit 111 can calculate brain fatigue level, mood level, and stress level as indicators representing the subject's emotions, based on data related to the subject's voice, facial expression images, and pulse wave acquired from the terminal device 20. For example, the emotion expression engine unit 111 can calculate brain fatigue level based on the frequency of the voice, calculate mood level by extracting the subject's emotions from the facial expression images, and calculate stress level by performing frequency analysis of the pulse wave using Fast Fourier Transform to extract high-frequency and low-frequency sections.

[0051] The 3-axis processing unit 112 can generate a graph plotting points at coordinates corresponding to the brain fatigue level, mood level, and stress level calculated by the emotion expression engine unit 111, and display it on the information processing device 10 or terminal device 20. Furthermore, for each date and time on which data related to the subject's voice, facial expression image, and pulse wave is acquired, the 3-axis processing unit 112 can generate a graph plotting points in a three-dimensional space consisting of the X, Y, and Z axes at coordinates corresponding to the subject's brain fatigue level, mood level, and stress level in chronological order, and display it on the information processing device 10 or terminal device 20.

[0052] The data management unit 113 can acquire data from the terminal device 20, including at least the subject's voice, facial expression image, and pulse wave data, and store it in the information processing device 10's storage means (for example, the user information database 171). The data management unit 113 can also store the subject's voice, facial expression image, and pulse wave data, along with the date and time the data was acquired, in the information processing device's storage means (user information database 171).

[0053] Figure 4 shows an example of data stored in the user information database of the information processing device shown in Figure 3, in tabular format. Table R1 stores user ID, which is information that identifies the subject (user), along with gender (male, female, or other) and age. The user information database 171 can store tables R2 and R3, for example, in association with the user information in table R1.

[0054] Table R2 also stores, in relation to the date and time, the subject's voice data, facial expression image data, pulse wave data, questionnaire data including the subject's responses to stress checks, life log data recording the subject's behavior, and environmental data including temperature and humidity, all received from the terminal device 20. The dates and times included in Table R2 are the date and time when the data related to the subject's voice, facial expression image, and pulse wave was received from the terminal device 20, and the date and time when the subject accessed the information processing device 10 using the terminal device 20.

[0055] Furthermore, the stress level, brain fatigue level, and mood level of the subject, calculated by the emotion expression engine unit 111, are associated with the normalized time of the date and time stored in table R2 and stored as values ​​for the X, Y, and Z axes in a three-dimensional space. The normalization of the date and time can be achieved, for example, by converting the date and time to UNIX® time.

[0056] Figure 5 is a flowchart illustrating the process of collecting various data from a subject's terminal device. This process is performed, for example, by terminal device 20. Terminal device 20 collects questionnaire data, including the subject's answers to a stress check questionnaire, and transmits it to information processing device 10 (step S1). In step S1, along with the questionnaire data, life log data recording the subject's lifestyle, activities, and behavior can also be collected and transmitted to information processing device 10.

[0057] Subsequently, the terminal device 20 displays the medical interview results on its screen (step S2). Figures 6 to 9 show the screen display of the terminal device 20 after the processing from step 1 to step 2 has been completed.

[0058] Figure 6 shows an example of a user interface for conducting a stress check using a questionnaire. It is an example of a stress check questionnaire displayed on the screen of terminal device 20. Below the message "A. We would like to ask you about your work. Please select the one that applies most to you," the question "1. I have to do a lot of work" is displayed. The subject can select from the options "Yes," "Somewhat yes," "Somewhat no," and "No" by clicking or tapping. The same applies to the other questions. The number of questions can be, for example, the 57 items in the Ministry of Health, Labour and Welfare's stress check questionnaire (Non-Patent Literature 1).

[0059] After the subject has answered all items on the stress check questionnaire, the terminal device 20 displays content as shown in Figure 7. Figure 7 shows an example of a user interface that prompts user registration after the stress check. Following the message, "The test is now complete. Please register to check your results," the subject enters, for example, their email address and password, enters other necessary information, and presses the registration button at the bottom of the screen to register their information.

[0060] After registration, content such as that shown in Figure 8 will be displayed on the screen of the terminal device 20. Figure 8 shows an example of a screen displaying the stress check results as a radar chart. The closer to the center of the radar chart, the higher the subject's stress level. Figure 9 shows an example of a screen displaying comments about the stress check results. The terminal device 20 can display comments from a counselor (expert) in the upper half of the screen, such as, "It appears you are currently experiencing a slightly high level of stress..." depending on the subject's questionnaire results.

[0061] Furthermore, the terminal device 20 displays a message in the lower half of the screen such as, "Currently, the number of registered users for Step 2, which allows for chat consultations with experts and stress measurement using measuring devices, has reached its limit. If you wish to use Step 2, please register on the registration page below." By clicking or tapping "Register as a user" at the bottom of the screen, the user can be prompted to register more detailed personal information of the subject. The personal information of the subject transmitted from the terminal device 20 to the information processing device 10 is stored by the data management unit 113, for example, in the user information database.

[0062] Referring again to the flowchart shown in Figure 5, after step S2, the terminal device 20 collects pulse wave data from the pulse wave measuring device that measures the subject's pulse wave (heart rate) and transmits the pulse wave data to the information processing device 10 (step 3). Figure 10 shows how the pulse wave and heart rate are measured from the subject's fingertip using the pulse wave measuring device. The pulse wave measuring device 30 includes a main unit 32, a waveform display unit 34 provided on the main unit 32, and a measurement unit 36. By pressing the subject's fingertip against the measurement unit 36 ​​of the pulse wave measuring device 30, the pulse wave measuring device 30 can measure the subject's pulse wave and heart rate, and a waveform based on the pulse wave and heart rate is displayed on the waveform display unit 34 provided on the main unit 32. The measurement time by the pulse wave measuring device 30 can be, for example, 180 seconds (3 minutes) per measurement.

[0063] The terminal device 20 can communicate with the pulse wave measuring instrument 30 and receive the subject's pulse wave data from the pulse wave measuring instrument 30. Figure 11 shows an example of the screen display of the subject's terminal device when the subject's pulse wave and heart rate are measured with the pulse wave measuring instrument. The terminal device 20 can, for example, display waveforms based on the subject's pulse wave data on the screen, and can also display the pulse interval (Peak-to-Peak Interval: PPI), which is the interval between peaks of one pulse wave, and the values ​​of the low frequency (LF) and high frequency (HF) sections of the pulse wave data on the screen.

[0064] Referring to the flowchart shown in Figure 5, after step S3, the terminal device 20 acquires data related to the subject's facial expression image and audio data by recording at least a portion of the video call (video chat) with the counselor or specialist (step S4). Note that the measurement of the subject's pulse wave using the pulse wave measuring device in step 3 can be continued, and the terminal device 20 can collect pulse wave data even during the video call. After step 4, the terminal device 20 can collect environmental data, including temperature and humidity, and transmit it to the information processing device 10 (step 5). The processing in steps 4 and 5 can be performed in the information processing device 10 instead of the terminal device 20.

[0065] Figure 12 shows an example of a screen display for acquiring images of a subject's facial expressions from a video call between a counselor and a subject during a counseling session. In Figure 12, the subject's facial expressions are displayed at the top of the screen, and the counselor is displayed at the bottom. In order to recognize the subject's facial expressions, the terminal device 20 or information processing device 10 records the video call during the counseling session for at least a predetermined recording time (e.g., 15 minutes). In other words, the data relating to the subject's facial expression images is data obtained by continuously recording the moving images of the subject's facial expressions during a video call with the subject via the terminal device 20 for at least a predetermined recording time.

[0066] Figure 13 shows an example of a screen display for acquiring the subject's voice from a video call between a counselor and a subject during a counseling session. The upper part of the screen shown in Figure 13 displays a pre-written sentence for the subject to read aloud (for example, "Once upon a time, there lived an old man and an old woman. The old man went to the mountains..."), while the lower part of the screen displays the counselor. As the subject reads the pre-written sentence aloud, the terminal device 20 or information processing device 10 records the audio of the subject reading the sentence until at least a predetermined recording time (for example, about 40 seconds) is reached. In other words, the audio data is data recorded continuously for at least a predetermined recording time from the subject reading the pre-written sentence displayed on the terminal device 20 during a video call with the subject via the terminal device 20.

[0067] Figure 14 is a schematic diagram showing the configuration of the emotion expression engine unit, which calculates various indicators representing brain fatigue, mood, and stress from various collected data. Functionally, the emotion expression engine unit 111 can be divided into an emotion expression core unit 111A and an overlapping unit unit 111B. The emotion expression core unit 111A can calculate brain fatigue level, mood level, and stress level, which are quantitative indicators related to brain fatigue, mood, and stress. In addition, the emotion expression core unit 111A can calculate date and time normalization, discomfort index, etc.

[0068] The stacking unit 111B can adjust the brain fatigue level, mood level, and stress level calculated by the emotional expression core unit 111A by multiplying these values ​​by weight coefficients determined based on the discomfort index calculated from the temperature and humidity included in the environmental data. If the discomfort index is DI, the temperature is T, and the humidity is H, then for example, DI can be calculated using the formula DI = 0.81T + 0.01H × (0.99T - 14.3) + 46.3. In addition, although not shown in Figure 14, the stacking unit 111B can acquire questionnaire data including the subject's stress check score and adjust the brain fatigue level, mood level, and stress level calculated by the emotional expression core unit 111A by multiplying these values ​​by weight coefficients determined according to the score included in the questionnaire data.

[0069] The emotion expression engine unit 111 calculates the brain fatigue level in the emotion expression core unit 111A from the subject's voice data acquired as input data. For example, the brain fatigue level can be determined by calculating the CEM (Cerebral Exponent Macro: CEM) value, which represents the brain activity index. The brain activity index measurement algorithm (SiCECA algorithm, Non-Patent Literature 3) developed by the Electronic Navigation Research Institute can calculate the brain activity index (CEM value) from voice. By executing the brain activity index measurement algorithm, the emotion expression engine unit 111 can obtain a CEM value of 1 or more (for example, around 2 to 5) per subject from the subject's voice data. The brain fatigue level corresponds to, for example, the average value of these CEM values ​​of 1 or more.

[0070] Figure 17 shows an example of numerical conversion for plotting the obtained values ​​of various indicators representing brain fatigue, mood, and stress on a three-axis space. In the example shown in Figure 17, four CEM values ​​of 431.08, 360.73, 342.76, and 360.45 are obtained using the brain activity index measurement algorithm, and the brain fatigue level is 373.755, which is the average of these CEM values.

[0071] Referring again to Figure 14, the emotion expression engine unit 111 calculates the mood level in the emotion expression core unit 111A from the facial expression image data of the subject acquired as input data. For example, the mood level is determined according to the count of multiple emotion expressions recognized from the video image of the subject's facial expression based on a facial expression recognition algorithm. As the facial expression recognition algorithm, the algorithm from the open-source software "Face classification and detection" (Non-Patent Literature 4) can be used.

[0072] The emotion expression engine unit 111 can recognize multiple emotion expressions from video footage of a subject's facial expressions included in the data relating to facial expression images by executing a facial expression recognition algorithm. These multiple emotion expressions can be, for example, seven types according to Russell's Circle of Emotion model (Non-Patent Literature 5): happy, surprise, neutral, fear, angry, disgust, and sad.

[0073] The emotion expression engine unit 111 executes a facial expression recognition algorithm (for example, the open-source software "Face classification and detection") to count each of the multiple emotion expressions recognized from the video images of the subject's facial expressions included in the data related to facial expression images. The emotion expression engine unit 111 calculates the proportion of each of the multiple emotion expressions and calculates a mood index for each emotion expression by multiplying the proportion of each emotion expression by a predetermined weight for each emotion expression. The predetermined weight for each of the multiple emotion expressions can be determined based on Russell's cyclic emotion model as shown in Figure 15, for example, as shown in the weighting in Figure 16.

[0074] Figure 15 is a diagram illustrating the weighting determined based on Russell's Circle of Affect model. Figure 16 shows an example of mood scores calculated from various emotional expressions obtained based on Russell's Circle of Affect model. The predetermined weights for each of the multiple emotional expressions can be adjusted in the order of happy, surprise, neutral, fear, angry, disgust, and sad. Referring to Figure 16, for example, the weighting coefficient for happy can be set to 100, surprise to 70, neutral to 50, and so on.

[0075] In the example shown in Figure 16, the proportion of each emotion expression is highest for neutral at 69.91001, followed by happy at 6.299213. By multiplying the proportion of neutral (F) of 69.91001 by a predetermined weighting coefficient (G) of 50, the mood index F×G value of 3495.501 is calculated. Similarly, by multiplying the proportion of happy (F) of 6.299213 by a predetermined weighting coefficient (G) of 100, the mood index F×G value of 629.9213 is calculated. In this way, the emotion expression engine unit 111 calculates the mood index for each emotion expression by multiplying each emotion expression by a predetermined weight and the proportion of each emotion expression.

[0076] The emotion expression engine unit 111 can determine the mood score by dividing the maximum mood index (the highest among the mood indices for each emotion expression) by the sum of the mood indices for each emotion expression. In the example shown in Figure 16, the maximum mood index is 3495.501 for neutral. The mood score is calculated by dividing the maximum mood index (3495.501) by the sum of the mood indices for each emotion expression (4443.33572), resulting in 3495.501 / 4443.33572 = 44.43335. Note that in the example shown in Figure 16, there is no value for disgust, so it is omitted from the display.

[0077] Referring again to Figure 14, the emotion expression engine unit 111 calculates the stress level in the emotion expression core unit 111A from the pulse wave (heart rate) data of the subject acquired as input data. The pulse wave data is data obtained by dividing the pulse wave measured by the pulse wave measuring device 30 into sections, with each section defined as a predetermined time interval (for example, 180 seconds).

[0078] The emotion expression engine unit 111 divides the pulse wave within each section of the pulse wave using a Hamming window, and calculates the pulse interval PPI (Pulse Piper), which is the interval between the peaks of one pulse wave and the next peak, and the time for each section of the pulse wave within the Hamming window. The emotion expression engine unit 111 generates a time-PPI graph for each section of the pulse wave, plotting points at coordinates corresponding to the pulse interval PPI and time in a two-dimensional space with time on the horizontal axis and PPI on the vertical axis. After performing interpolation such as linear interpolation or cubic spline interpolation between discrete values ​​in the time-PPI graph, the emotion expression engine unit 111 applies a Fast Fourier Transform (FFT), and by integrating the Power Spectral Density (PSD) of the FFT result in the low-frequency section and the high-frequency section, it can calculate the LF value corresponding to the low-frequency component, the HF value corresponding to the high-frequency component, and the LF / HF value. As a known method for calculating LF value, HF value, and LF / HF value from pulse interval PPI, for example, there is the stress state estimation method described in Non-Patent Document 5.

[0079] The emotion expression engine unit 111 can use the LF / HF value (sympathetic nervous system index) as the stress level. Alternatively, the stress level can be based on at least one value from the LF value, the HF value, and the LF / HF value. For example, by referring to past data values, the maximum value of LF / HF can be set to 2, and if HF is also used, the maximum value of HF (parasympathetic nervous system index) can be set to 900 for normalization. If there are multiple normalized data points in each window interval, the average value is taken. In the example in Figure 17, 1.22 and 1.44 are calculated as LF / HF values, and their average is taken to get 1.33.

[0080] When using only LF / HF values ​​as the stress level, the values ​​are reversed so that the maximum stress level (MAX) becomes the minimum stress level (MIN), and (MAX2 - standard value) is converted to a single axis. For example, in the example in Figure 17, the stress level axis (tension axis) has a MAX of 2 and a MIN of 0, so it is reversed when converting to an axis. Note that the mood level axis (mood axis) and the brain fatigue level axis (brain fatigue axis) are used as they are without reversing. When using HF values ​​as the stress level, the values ​​are converted to a single axis so that stress MAX is the minimum, then neutral, and finally relaxation MAX is the maximum.

[0081] The low-frequency section can be from 0.04 Hz or higher to less than 0.15 Hz, and the high-frequency section can be from 0.15 Hz or higher to less than 0.4 Hz.

[0082] As shown in Figure 14, the emotion expression engine unit 111 can acquire data such as time (date and time) and environmental data as input data, in addition to data related to the subject's voice, facial expression images, and pulse (heart rate). As described above, the time (date and time) is converted to a normalized time (e.g., UNIX time) by the emotion expression core unit 111A, and the discomfort index is calculated from the temperature and humidity included in the environmental data. The multiplication unit 111B can then multiply the brain fatigue level, mood level, and stress level, respectively, by predetermined weighting coefficients according to the discomfort index.

[0083] Figure 18 shows an example of a graph display in a three-dimensional space consisting of X, Y, and Z axes plotting the emotional changes of a subject obtained by the emotion expression engine over time. Figure 19 shows an example of a graph display in a three-dimensional space consisting of X, Y, and Z axes plotting the emotional changes of another subject obtained by the emotion expression engine over time. The 3-axis processing unit 112 can generate and display a graph in a three-dimensional space consisting of X, Y, and Z axes, plotting points at coordinates corresponding to brain fatigue level, mood level, and stress level. Furthermore, the 3-axis processing unit 112 can display a graph in the three-dimensional space plotting points at coordinates corresponding to the subject's brain fatigue level, mood level, and stress level for each day and time, in chronological order. This makes it possible to visualize the changes in the subject's emotions, thereby enabling highly accurate analysis of the subject's emotions.

[0084] This type of three-dimensional graph display integrates three axes—brain fatigue level, mood level, and stress level—with a time axis, enabling multidimensional analysis and providing a clear and easy-to-understand presentation for subjects, experts, and other users. Conventional stress checks show diagnostic results using radar charts, but they rarely show the correlations between factors. In this invention, as shown in Figures 18 and 19, the correlations between results can be clearly displayed. The axes of the three-dimensional graph display can be rearranged as needed to match the information the user wants to know. In this invention, stress is measured over time, classified into type categories (clusters) based on patterns (trends), and future state predictions become possible.

[0085] Figure 20 shows an example of type-based classification categories defined in a three-dimensional space consisting of a tension axis (X-axis), a brain fatigue axis (Y-axis), and a mood axis (Z-axis). The lower left corner is the starting point, and the closer to the starting point, the higher the stress (X-axis), the higher the brain fatigue (Y-axis), and the more likely the mood (Z-axis) is to decline. The same applies to Figures 18 and 19. In the example of the subject shown in Figure 18, it can be seen that the mood has shifted from the starting point, but in the example of another subject shown in Figure 19, it can be seen that the mood is concentrated near the starting point, and that stress (X-axis) is high, brain fatigue (Y-axis) is high, and the mood (Z-axis) remains low, allowing for proactive consideration of intervention if there is a risk of mental illness.

[0086] The type-based classification categories defined in the three-dimensional space shown in Figure 20 can be defined, for example, as shown in the table below. [Table 1]

[0087] As shown in Figure 20, the three-dimensional space is divided into multiple type-based classification categories, and the three-axis processing unit 112 can notify the subject's terminal device 20 or information processing device 10, etc., of the category to which the coordinate points corresponding to the subject's brain fatigue level, mood level, and stress level belong among the multiple type-based classification categories in the three-dimensional space.

[0088] Furthermore, improvement suggestions are defined for each of several type categories, and the emotion expression engine unit 111 can notify the subject's terminal device 20 or information processing device 10, etc., of the improvement suggestions for the category to which the coordinate points corresponding to the subject's brain fatigue level, mood level, and stress level in three-dimensional space belong.

[0089] Examples of improvement measures are shown in the table below. [Table 2]

[0090] For example, if a subject belongs to type category A, the emotion expression engine unit 111 can notify them of jogging, stretching, trekking, mindfulness, and yoga as improvement measures, as indicated by the circle mark. Similarly, if another subject belongs to type category B, the emotion expression engine unit 111 can notify them of stretching, yoga, and cognitive behavioral therapy as improvement measures, as indicated by the circle mark. In this way, the system can suggest appropriate improvement measures according to the type category to which each subject belongs.

[0091] As described above, the information processing system, etc. according to the present invention acquires not only quantitative data such as pulse waves and heart rate obtained from a pulse wave measuring device, but also data such as the results of stress checks and the subject's voice and facial expression images during counseling using video calls. From this data, it calculates stress levels, brain fatigue levels, and mood levels, and plots them in a three-dimensional space consisting of X, Y, and Z axes. By providing an information processing system, etc. that can visualize the subject's emotions and support diagnosis and treatment, it is possible to analyze the subject's emotions with high accuracy, thereby enabling the early detection of mental illness and contributing to the resolution of social issues. [Industrial applicability]

[0092] The information processing system, etc., according to the present invention can be used for a wide range of applications, such as stress checks in companies, for individuals, and in schools; strengthening athletic mentality; improving concentration during learning; and measuring mental state in job interviews. [Explanation of Symbols]

[0093] 10: Information Processing Device 11, 21: CPU 12, 22: Memory 13, 23: Bus 14, 24: Input / Output Interfaces 15, 25: Input section 16, 26: Output section 17, 27: Storage section 18, 28: Communications Department 20, 20-1, 20-2, 20-n: Terminal device 30: Pulse wave measuring device 32: Main body 34: Waveform display section 36: Measurement Unit 111: Emotion Expression Engine Unit 111A: Core of emotional expression 111B: Stacking Unit Section 112: 3-axis processing unit 113: Data Management Department 171: User Information Database

Claims

1. An information processing device connected to a subject's terminal device for visualizing the subject's emotions, A data management unit that acquires at least the voice, facial expression images, and pulse wave data of the subject, An emotion expression engine unit calculates brain fatigue based on the frequency of the aforementioned sound, extracts the subject's emotions from the aforementioned facial expression image to calculate mood, performs frequency analysis of the pulse wave using Fast Fourier Transform, extracts high-frequency and low-frequency sections to calculate stress level, A three-axis processing unit that displays a graph in which points are plotted at coordinates corresponding to the brain fatigue level, mood level, and stress level in a three-dimensional space consisting of the X, Y, and Z axes. Includes, The audio data is obtained by recording at least a portion of a video call with the subject via the terminal device. The data relating to the facial expression images is obtained by recording at least a portion of a video call with the subject via the terminal device. The information processing device is characterized in that the data relating to the pulse wave is acquired from a pulse wave measuring instrument that measures the pulse wave of the subject via the terminal device.

2. The data management unit stores the data relating to the subject's voice, facial expression image, and pulse wave, along with the date and time on which the data was acquired, in the storage means of the information processing device. The information processing apparatus according to claim 1, characterized in that the three-axis processing unit displays a graph in which points are plotted in chronological order at coordinates corresponding to the subject's brain fatigue level, mood level, and stress level in the three-dimensional space for each date and time.

3. The aforementioned three-dimensional space is divided into multiple type-based classification categories, The information processing device according to claim 1 or 2, characterized in that the three-axis processing unit notifies the category to which the coordinate points corresponding to the subject's brain fatigue level, mood level, and stress level belong among the plurality of type classification categories in the three-dimensional space.

4. For each of the aforementioned multiple type classification categories, improvement plans to be proposed to the subjects are determined. The information processing device according to claim 3, characterized in that the emotion expression engine unit notifies the subject of the suggested improvement for the category to which the coordinate points corresponding to the subject's brain fatigue level, mood level, and stress level in the three-dimensional space belong.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the audio data is data obtained by continuously recording, for at least a predetermined recording time, the audio of the subject reading a predetermined standard phrase displayed on the terminal device during a video call with the subject via the terminal device.

6. The emotion expression engine unit obtains one or more CEM values ​​per subject from the voice data by executing a brain activity index measurement algorithm that measures CEM values ​​representing the brain activity index, The information processing apparatus according to claim 5, characterized in that the brain fatigue level is the average value of 1 or more CEM values.

7. The information processing device according to any one of claims 1 to 6, characterized in that the data relating to the pulse wave is data obtained by dividing the pulse wave measured by the pulse wave measuring instrument into a predetermined time interval, with each of the predetermined time intervals being defined as one section.

8. The emotion expression engine unit divides the pulse wave within each section of the pulse wave using a humming window, and for the pulse wave within the humming window, calculates the pulse interval PPI, which is the interval between the peaks of one pulse wave and the next peak, and the time. The emotion expression engine unit generates a time-PPI graph for each section of the pulse wave, plotting points at coordinates corresponding to the pulse interval PPI and the time in a two-dimensional space with time on the horizontal axis and PPI on the vertical axis. The emotion expression engine unit applies a Fast Fourier Transform (FFT) by interpolating between discrete values ​​in the time-PPI graph, and calculates the LF value corresponding to the low-frequency component, the HF value corresponding to the high-frequency component, and the LF / HF value by integrating the power spectral density (PSD) of the FFT result within the low-frequency and high-frequency sections, respectively. The information processing device according to claim 7, characterized in that the stress level is based on at least one value among the LF value, the HF value, and the LF / HF value.

9. The aforementioned low-frequency section is from 0.04 Hz or higher to less than 0.15 Hz. The information processing apparatus according to any one of claims 1 to 8, characterized in that the high-frequency interval is from 0.15 Hz or higher to less than 0.4 Hz.

10. The information processing apparatus according to any one of claims 1 to 9, characterized in that the data relating to the facial expression image is data obtained by continuously recording the moving image of the subject's facial expression for at least a predetermined recording time during a video call with the subject via the terminal device.

11. The emotion expression engine unit executes a facial expression recognition algorithm to count each of the multiple emotion expressions recognized from the video images of the subject's facial expressions included in the data relating to the facial expression images. The emotion expression engine unit calculates the proportion of each of the multiple emotion expressions, multiplies each of the multiple emotion expressions by a predetermined weight, and calculates a mood index for each emotion expression. The information processing device according to claim 10, characterized in that the mood level is based on the value obtained by dividing the maximum mood index, which is the largest among the mood indices for each emotional expression, by the sum of the mood indices for each emotional expression.

12. The information processing device according to claim 11, characterized in that the aforementioned multiple emotional expressions are happy, surprise, neutral, fear, angry, disgust, and sad in Russell's circle of emotions model.

13. The data management unit acquires environmental data, including at least temperature and humidity, in addition to data on the subject's voice, facial expression images, and pulse waves. The information processing apparatus according to any one of claims 1 to 12, characterized in that the emotion expression engine unit adjusts the values ​​of brain fatigue, mood, and stress by multiplying each of the brain fatigue level, mood level, and stress level by a weighting coefficient determined based on a discomfort index calculated from the temperature and humidity included in the environmental data.

14. The data management unit acquires, in addition to the subject's voice, facial expression images, and pulse wave data, questionnaire data including the subject's stress check score. The information processing device according to any one of claims 1 to 12, characterized in that the emotion expression engine unit adjusts the values ​​of the brain fatigue level, the mood level, and the stress level by multiplying each of the brain fatigue level, the mood level, and the stress level by a weight coefficient determined according to the score included in the medical questionnaire data.

15. An information processing method performed on a server that can be connected to a subject's terminal device via a network, The steps include acquiring data from the terminal device relating at least the subject's voice, facial expression image, and pulse wave, The steps include: calculating the degree of brain fatigue based on the frequency of the aforementioned sound; calculating the mood level by extracting the subject's emotions from the aforementioned facial expression image; performing frequency analysis on the pulse wave using a fast Fourier transform to extract high-frequency and low-frequency sections and calculate the stress level; The steps include displaying a graph in which points are plotted at coordinates corresponding to the brain fatigue level, mood level, and stress level in a three-dimensional space consisting of the X-axis (axis indicating stress level), the Y-axis (axis indicating brain fatigue level), and the Z-axis (axis indicating mood level), and Includes, The audio data is obtained by recording at least a portion of a video call with the subject via the terminal device. The data relating to the facial expression images is obtained by recording at least a portion of a video call with the subject via the terminal device. An information processing method characterized in that the data relating to the pulse wave is acquired from a pulse wave measuring instrument that measures the pulse wave of the subject via the terminal device.

16. The step of acquiring data relating to the subject's voice, facial expression image, and pulse wave includes storing the data relating to the subject's voice, facial expression image, and pulse wave, along with the date and time on which the data was acquired, in the storage means of the server. The information processing method according to claim 15, characterized in that the step of displaying the graph includes the step of displaying a graph in which points are plotted in chronological order in the three-dimensional space at coordinates corresponding to the subject's brain fatigue level, mood level, and stress level for each date and time.

17. The aforementioned three-dimensional space is divided into multiple type-based classification categories, The information processing method according to claim 15 or 16, characterized in that the step of displaying the graph includes a step of notifying the category to which the coordinate points corresponding to the subject's brain fatigue level, mood level, and stress level belong among the plurality of type classification categories in the three-dimensional space.

18. For each of the aforementioned multiple type classification categories, improvement plans to be proposed to the subjects are determined. The information processing method according to claim 17, characterized in that the step of calculating the brain fatigue level, mood level, and stress level includes a step of notifying the subject of the improvement plan for the category to which the coordinate points corresponding to the brain fatigue level, mood level, and stress level of the subject in the three-dimensional space belong.

19. The information processing method according to any one of claims 15 to 18, characterized in that the audio data is data obtained by continuously recording, for at least a predetermined recording time, the audio of the subject reading a predetermined standard phrase displayed on the terminal device during a video call with the subject via the terminal device.

20. The step of calculating the brain fatigue level, mood level, and stress level includes the step of obtaining one or more CEM values ​​per subject from the voice data by executing a brain activity index measurement algorithm that measures CEM values ​​representing the brain activity index, The information processing method according to claim 19, characterized in that the brain fatigue level is the average value of 1 or more CEM values.

21. The information processing method according to any one of claims 15 to 20, characterized in that the data relating to the pulse wave is data obtained by dividing the pulse wave measured by the pulse wave measuring instrument into a predetermined time interval, with each of the predetermined time intervals being defined as one section.

22. The steps for calculating the aforementioned brain fatigue level, mood level, and stress level are: For each section of the pulse wave, the pulse wave within that section is divided by a Hamming window, and for the pulse wave within the Hamming window, the pulse interval PPI, which is the interval between the peaks of one pulse wave and the next peak, and the time are calculated. For each section of the pulse wave, a time-PPI graph is generated by plotting points at coordinates corresponding to the pulse interval PPI and the time in a two-dimensional space with time on the horizontal axis and PPI on the vertical axis. The steps include: applying a Fast Fourier Transform (FFT) by interpolating the discrete values ​​in the time-PPI graph, and integrating the resulting power spectral density (PSD) within the low-frequency and high-frequency sections to calculate the LF value corresponding to the low-frequency component, the HF value corresponding to the high-frequency component, and the LF / HF value; Includes, The information processing method according to claim 21, characterized in that the stress level is based on at least one value among the LF value, the HF value, and the LF / HF value.

23. The aforementioned low-frequency section is from 0.04 Hz or higher to less than 0.15 Hz. The information processing method according to any one of claims 15 to 22, characterized in that the high-frequency interval is from 0.15 Hz or more to less than 0.4 Hz.

24. The information processing method according to any one of claims 15 to 23, characterized in that the data relating to the facial expression image is data obtained by continuously recording the moving image of the subject's facial expression for at least a predetermined recording time during a video call with the subject via the terminal device.

25. The steps for calculating the aforementioned brain fatigue level, mood level, and stress level are: The process involves executing a facial expression recognition algorithm to count each of the multiple emotional expressions recognized from the video footage of the subject's facial expression included in the data relating to the facial expression images, The steps include: calculating the proportion of each of the aforementioned multiple emotional expressions, multiplying each of the aforementioned multiple emotional expressions by a predetermined weight, and calculating a mood index for each emotional expression. Includes, The information processing method according to claim 24, characterized in that the mood level is based on the value obtained by dividing the maximum mood index, which is the largest among the mood indices for each emotional expression, by the sum of the mood indices for each emotional expression.

26. The information processing method according to claim 25, characterized in that the aforementioned multiple emotional expressions are happy, surprise, neutral, fear, angry, disgust, and sad in Russell's Circle of Emotions model.

27. The step of acquiring and storing data on the subject's voice, facial expression images, and pulse waves includes, in addition to the data on the subject's voice, facial expression images, and pulse waves, a step of acquiring environmental data including at least temperature and humidity. The information processing method according to any one of claims 15 to 26, characterized in that the step of calculating the brain fatigue level, mood level, and stress level includes a step of adjusting the values ​​of the brain fatigue level, mood level, and stress level by multiplying each of the brain fatigue level, mood level, and stress level by a predetermined weighting coefficient determined based on a discomfort index calculated from the temperature and humidity included in the environmental data.

28. The step of acquiring and storing data on the subject's voice, facial expression images, and pulse waves includes, in addition to the data on the subject's voice, facial expression images, and pulse waves, the step of acquiring questionnaire data including the subject's stress check result score. The information processing method according to any one of claims 15 to 26, characterized in that the step of calculating the brain fatigue level, mood level, and stress level includes a step of adjusting the values ​​of the brain fatigue level, mood level, and stress level by multiplying each of the brain fatigue level, mood level, and stress level by a weighting coefficient determined according to the score included in the questionnaire data.

29. An information processing device according to any one of claims 1 to 14, A terminal device that can access the aforementioned information processing device via a network and Includes, The terminal device transmits at least the voice data, the facial expression image data, and the pulse wave data to the information processing device. The information processing device receives data relating to voice, data relating to facial expression images, and data relating to pulse waves, and transmits the brain fatigue level, mood level, and stress level calculated based on the received data to the terminal device, and displays on the terminal device a graph in which points are plotted at coordinates corresponding to the brain fatigue level, mood level, and stress level in a three-dimensional space consisting of the X, Y, and Z axes.

30. An information processing program characterized by being executed by a computer, thereby causing the computer to function as a component of the information management device described in any one of claims 1 to 14.

31. An information processing program characterized by being executed by a computer, thereby causing the computer to perform each stage of the information processing method described in any one of claims 15 to 28.

Images