Information processing device, information processing system, and program

Abstract

To objectively measure a stress level of a user and identify stress factors of the user.SOLUTION: An acquisition unit 111 acquires moving image data showing a moving image obtained by photographing a face of a user from a camera 16. A measurement unit 113 measures vital signs of the user from multiple image data included in the moving image data. A selection unit 114 compares the vital signs and a stress level measured by the measurement unit 113 with predetermined thresholds, and determines whether or not a stress reaction to a message presented to the user immediately before measuring the vital signs, etc., has been detected. The selection unit 114 then selects a message to be displayed next according to a result of the determination. A determination unit 115 determines timing to display the next message according to the result of the determination of whether or not the stress reaction has been detected. A presenting unit 116 instructs a display unit 15 or a speaker 17 to present the selected message at the determined timing.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, and a program. [Background technology]

[0002] Compared to illness or injury, the stress a person is experiencing is difficult for others to notice, and even the person themselves may not be aware of it. If people remain unaware and neglect to take measures, stress can become chronic, causing physical and mental problems and even leading to serious situations such as accidents or illness. Therefore, it is important to check for stress on a daily basis before such situations occur.

[0003] One way to assess stress is through self-reporting using a stress questionnaire. This stress questionnaire is made up of multiple questions, and the respondent's responses to these questions are compiled to assess the respondent's stress. Various stress questionnaires have been developed, and a specific example is the 57-item Occupational Stress Questionnaire recommended by the Ministry of Health, Labor and Welfare.

[0004] On the other hand, it has been pointed out that responses to stress questionnaires can be controlled by the respondent's own will, and therefore lack objectivity and reliability. Therefore, a non-contact heart rate variability measurement system using a camera has been developed as an objective means of evaluating stress.

[0005] For example, Patent Document 1 discloses an image processing method for stress monitoring that acquires face image data using an RGB camera, creates hemoglobin image data from the face image data using pigment component separation, creates heart rate data based on the hemoglobin image data, creates heart rate interval data based on the heart rate data, and creates heart rate variability spectrogram data by performing frequency conversion on the heart rate interval data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2017-29318 Summary of the Invention [Problem to be solved by the invention]

[0007] There are a variety of external stimuli (also called stressors) that cause stress. Stressors include environmental factors such as weather and noise, physical factors such as illness and lack of sleep, psychological factors such as anxiety and worry, and social factors such as poor interpersonal relationships and being busy at work.

[0008] The technology described in Patent Document 1 can determine whether a person being measured is in a relaxed state or a stressed state based on an image of the person's face, but it cannot identify the stress factors that are causing the stressed state.

[0009] One of the objects of the present invention is to objectively measure a user's stress level and identify the user's stress factors. [Means for solving the problem]

[0010] In one aspect, the present invention provides an information processing device that presents a message and measures a stress level of a user based on an image of the user's face captured after the message is presented.

[0011] In a preferred embodiment, a message to be presented after one message is selected according to a stress level measured based on an image of the user's face taken after the presentation of the message.

[0012] In a preferred embodiment, the timing for presenting a message following a message is determined according to a stress level measured based on an image of the user's face taken after the message has been presented.

[0013] In a preferred embodiment, the user's normal stress level is measured based on an image of the user's face before a message is presented, the user's stress level in response to the message is measured based on an image of the user's face after the message is presented, and when the user's stress level measured based on an image of the user's face after the presentation has ended approaches a level determined as the normal stress level, the next message after the message is presented.

[0014] In a preferred embodiment, the presentation is made according to a predetermined schedule.

[0015] In a preferred embodiment, the user's stress tendency is diagnosed from the measured change in the user's stress level.

[0016] In a preferred embodiment, a determination is made as to whether the user's line of sight was looking at the presentation based on an image of the user's face taken after the presentation, and if it is determined that the user's line of sight was looking at the presentation, the user's stress level is measured based on the image.

[0017] In another aspect, the present invention provides an information processing system having a server device and a terminal, wherein the terminal presents a message and takes a picture of the user's face after the presentation, and the terminal or the server device measures the user's stress level based on the taken image.

[0018] In another aspect, the present invention provides a program that causes a computer to execute the steps of presenting a message, taking an image of the user's face after the presentation, and measuring the user's stress level based on the image. [Effects of the Invention]

[0019] According to the present invention, it is possible to objectively measure the stress level of a user and identify the stress factors of the user. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing device 1. [Figure 2] FIG. 2 is a diagram showing an example of a user DB 121. [Figure 3] FIG. 10 is a diagram showing an example of a video DB 122. [Figure 4] FIG. 1 is a diagram for explaining an example of the concept of video data. [Figure 5] FIG. 12 is a diagram showing an example of a message DB 123. [Figure 6] FIG. 12 is a diagram showing an example of a vital sign DB 124. [Figure 7] FIG. 10 is a diagram showing an example of a schedule DB 125. [Figure 8] 1 is a diagram showing an example of the appearance of an information processing device 1. FIG. [Figure 9] FIG. 4 is a diagram showing an example of a display area R0. [Figure 10] FIG. 2 is a diagram showing an example of the functional configuration of the information processing device 1. [Figure 11] FIG. 2 is a diagram showing an example of the flow of operations of the information processing device 1. [Figure 12] FIG. 10 is a diagram showing an example of a history display in a browsing mode. [Figure 13] FIG. 10 is a diagram showing an example of a graph display in a viewing mode. [Figure 14] FIG. 10 is a diagram showing an example of the overall configuration of an information processing system 9 according to a modified example. [Figure 15] FIG. 10 is a diagram showing an example of the configuration of a server device 2 according to a modified example. [Figure 16] FIG. 10 is a diagram showing an example of the functional configuration of an information processing system 9 according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0021] <Embodiment><Configuration of information processing device> Fig. 1 is a diagram showing an example of the configuration of an information processing device 1. The information processing device 1 is a device that measures a stress level of a user based on an image of the user's face (hereinafter also referred to as a "face image"). The information processing device 1 shown in Fig. 1 is a mobile terminal, for example, a smartphone.

[0022] 1, the information processing device 1 includes a processor 11, a memory 12, a communication unit 13, an operation unit 14, a display unit 15, a camera 16, and a speaker 17. These are connected by a bus.

[0023] The processor 11 reads and executes a program stored in the memory 12 to control each part of the information processing device 1. The processor 11 is, for example, a CPU (Central Processing Unit).

[0024] The communication unit 13 is a communication circuit that communicatively connects the information processing device 1 to other devices via a wired or wireless connection. For example, the communication unit 13 may have a circuit that complies with a wireless communication system standard such as IMT-2000, IMT-Advanced, or IMT-2020. Furthermore, for example, the communication unit 13 may have a circuit that complies with a wireless LAN standard such as IEEE802.11.

[0025] Furthermore, for example, the communication unit 13 may include a module that realizes near field communication (NFC). Examples of NFC standards include ISO / IEC18092 (NFCIP-1), ISO / IEC14443, ISO / IEC15693, and IEEE802.15.

[0026] The operation unit 14 is equipped with operation buttons, a touch panel, and other operators for issuing various instructions, and receives an operation and sends a signal corresponding to the operation content to the processor 11. This operation is, for example, pressing a button, making a gesture on the touch panel, or the like.

[0027] The operation unit 14 may have a microphone for collecting voice. In this case, the processor 11 may perform voice recognition processing on the voice data representing the user's voice collected by the microphone, and accept the recognition result as the user's operation.

[0028] The display unit 15 has a display screen such as a liquid crystal display, and displays images under the control of the processor 11. A transparent touch panel of the operation unit 14 may be placed on top of the display screen.

[0029] Camera 16 is an imaging unit, such as a digital still camera, that captures an image of the surroundings of information processing device 1. Camera 16 includes an optical system such as a lens, as well as an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The camera 16 shown in Fig. 1 has a common configuration with, for example, the "RGB camera" of Patent Document 1 mentioned above. That is, this camera 16 is a device that acquires image data by arranging a plurality of detection elements, each of which has sensitivity to light intensity in a wavelength range corresponding to each of the colors R (red), G (green), and B (blue), as one pixel unit. The wavelength ranges corresponding to each color are preferably, for example, 580 nm to 680 nm for R (red), 500 nm to 630 nm for G (green), and 410 nm to 530 nm for B (blue).

[0031] The camera 16 may have a polarizing plate as part of its optical system. This polarizing plate is placed in the direction in which the camera 16 photographs the object. This allows the camera 16 to remove reflected components from the surface of the object.

[0032] Furthermore, the camera 16 may detect light in a wavelength range other than the above-mentioned ranges. For example, the camera 16 may be a five-band camera that adds two sensitivities, cyan (C) and orange (O), to a normal RGB camera.

[0033] The memory 12 is a storage means for storing an operating system, various programs, data, etc. that are loaded into the processor 11. The memory 12 includes a RAM (Random Access Memory) and a ROM (Read Only Memory).

[0034] The memory 12 may include a solid state drive, a hard disk drive, etc. The memory 12 also stores a user DB 121, a video DB 122, a message DB 123, a vital sign DB 124, and a schedule DB 125.

[0035] <User DB configuration> 2 is a diagram showing an example of the user DB 121. The user DB 121 is a database that stores information on a plurality of users who use the information processing device 1. The user DB 121 shown in FIG.

[0036] The institution list 1211 is a list listing information about the institutions to which the user belongs. The institution list 1211 stores an institution ID, which is identification information for identifying the institution to which the user belongs, and an institution name, which is the name of the institution. Each institution ID is associated with one user table 1212.

[0037] The user table 1212 is a table that stores information about each user who belongs to an institution identified by a corresponding institution ID. The user table 1212 has columns for user ID, user name, authentication information, gender, and date of birth.

[0038] The user ID in the user DB 121 is identification information that uniquely identifies each user.

[0039] The user name in the user DB 121 is information such as a character string indicating the user's name, etc. Note that the user DB 121 may have columns indicating the user's nickname, email address, phone number, etc. in addition to the column for the user name.

[0040] The authentication information in the user DB 121 is information used to authenticate a user, such as a password or a PIN code.

[0041] The gender in the user DB 121 is one piece of information indicating the attributes of a user, and indicates the gender of the user. For example, in the user DB 121 shown in Fig. 2, the user ID "U1" is stored in association with the gender "male."

[0042] The date of birth in the user DB 121 is one piece of information indicating the attributes of the user, and indicates the date of birth of the user.

[0043] The user DB 121 may have a column for entering information indicating the user's attributes in addition to the gender and date of birth. For example, the user DB 121 may store information on medical history, medication, and work history as information indicating the user's attributes. The user DB 121 may also store data representing the user's facial features (also referred to as facial feature data) as information indicating the user's attributes. The facial feature data may be stored in the authentication information column.

[0044] In the embodiment, the information processing device 1 is a terminal that can be used by multiple users, but it may also be a terminal owned and used by a single user. In this case, the user DB 121 may store a user ID indicating a single user who owns and uses the information processing device 1, in association with information about the user, such as the organization to which the user belongs, the name, and attributes.

[0045] <Video DB configuration> Fig. 3 is a diagram showing an example of the video DB 122. The video DB 122 is a database that stores data of a video (also called video data) of a user's face for each user. The video DB 122 shown in Fig. 3 has a user ID list 1221 and a video data table 1222.

[0046] The user ID list 1221 in the video DB 122 is a list of user IDs. These user IDs are information common to the user IDs stored in the user DB 121.

[0047] The video data table 1222 in the video DB 122 has a start date / time column, a length column, a video ID column, and a video data column. In this video data table 1222, the video data column stores the video data itself. The start date / time column stores information about the date and time when filming of the corresponding video data started (i.e., the start date / time). The length column stores the length of the corresponding video data. This length is measured in units of time such as minutes or seconds. The video ID column stores a video ID, which is identification information that uniquely identifies the corresponding video data.

[0048] FIG. 4 is a diagram illustrating an example of the concept of video data. Video data is made up of multiple still images captured at regular intervals. One frame of these multiple still images is called a frame. The number of still images captured per unit time is called the frame rate. As shown in FIG. 4, video data is made up of multiple ordered frames each associated with image data indicating the still image captured at each instant. Note that video data may be compressed as long as the image data for each frame can be extracted and acquired.

[0049] <Message DB configuration> Fig. 5 is a diagram showing an example of the message DB 123. The message DB 123 is a database that stores messages that the information processing device 1 presents to the user in association with the order in which the messages are presented. The message DB 123 shown in Fig. 5 includes a category ID list 1231 and a message table 1232.

[0050] The category ID list 1231 in the message DB 123 is a list of pairs of category IDs and category names. The category IDs are identification information that uniquely identify the categories to which messages displayed on the information processing device 1 belong. The category names are character strings that indicate the names of the categories identified by the category IDs.

[0051] Each category ID is associated with one message table 1232. The message table 1232 has a message ID column, a message column, a stress factor column, a detected time column, and a non-detected time column. In the message table 1232, the message ID column stores a message ID, which is identification information for identifying a message. The message column stores a character string or the like indicating the content of the message.

[0052] When the stress level measured from the user's face image exceeds a threshold, the information processing device 1 determines that a stress response to the message has been detected. When a stress response to a presented message is detected, the stress factor field stores the type of stress factor causing the stress response.

[0053] For example, if the user's stress level measured after being presented with the message "Are you worried about your relationships with others?" shown in Figure 5 exceeds a threshold, the information processing device 1 determines that the user is experiencing a stress response and identifies the stress factor causing the stress response as "interpersonal relationships at work."

[0054] The information processing device 1 may have multiple thresholds to compare with the measured stress level. In this case, the stress level is evaluated depending on which of the levels defined by the multiple thresholds the measured stress level falls under. The information processing device 1 may tally up a score according to the level to which the measured stress level falls as a score for the corresponding stress factor.

[0055] Furthermore, the information processing device 1 may tally up the measured stress level values ​​themselves as scores for the corresponding stress factors. That is, the information processing device 1 may present a plurality of messages, tally up the user's stress levels measured after the presentation for each message, and identify the stress factor with the highest total score as the stress factor for the user.

[0056] In the message table 1232, the column for when detection occurs stores a message ID indicating the next message to be presented when a user's stress reaction is detected after the corresponding message has been presented.

[0057] The non-detection column stores a message ID indicating the next message to be presented when the user's stress reaction is not detected after the corresponding message has been presented.

[0058] <Vital DB configuration> Fig. 6 is a diagram showing an example of the vital DB 124. The vital DB 124 is a database that stores vital signs (hereinafter also referred to as vitals) measured for each user and indicating the life signs of that user. The vital DB 124 shown in Fig. 6 includes a user ID list 1241 and a vital table 1242.

[0059] The user ID list 1241 in the vital DB 124 is a list of user IDs. The user IDs are information common to the user IDs stored in the user DB 121.

[0060] The vital sign table 1242 in the vital sign DB 124 has a column for measurement date and time, a column for message ID, a column for video ID, and one or more columns for vital signs. The vital sign columns shown in FIG. 6 have four items: pulse rate, respiratory rate, LF, and HF. The measurement date and time column stores the date and time when the vital sign was measured. The message ID column stores a message ID that identifies a message presented before measuring the vital sign. The video ID column stores a video ID that indicates video data including image data used when measuring the vital sign.

[0061] The vital signs column stores each vital sign measured based on a facial image of the user taken after the message identified by the corresponding message ID was presented. Pulse is the number of times the blood vessels in the user's face beat per minute. Pulse is synonymous with heart rate, as it is equal to the number of times the heart beats per minute. Respiratory rate is the number of breaths per minute measured from the facial image of the user.

[0062] The LF and HF columns both store values ​​calculated as a result of power spectrum analysis of heartbeat interval fluctuations. For example, the LF column stores the total value (integral value) of the intensities of low-frequency components, and the HF column stores the total value (integral value) of the intensities of high-frequency components. It is said that low-frequency components correspond to blood pressure fluctuations, and high-frequency components correspond to respiratory fluctuations.

[0063] The stress level is calculated using at least one of these vital signs, and is, for example, LF / HF.

[0064] <Schedule DB configuration> 7 is a diagram showing an example of the schedule DB 125. The schedule DB 125 is a database that stores the schedules of each user described above. The schedule DB 125 shown in FIG.

[0065] The user ID list 1251 in the schedule DB 125 is a list of user IDs. These user IDs are information common to the user IDs stored in the user DB 121.

[0066] The schedule table 1252 in the schedule DB 125 has a date column, a start time column, The database has a column for the date, the column for the end time, and the column for the requirement. The column for the date stores the date on which the requirement is scheduled. The column for the start time stores the time when the requirement is scheduled to start. The column for the end time stores the time when the requirement is scheduled to end. The column for the requirement stores a character string or the like that indicates the content of the requirement.

[0067] The information processing device 1 refers to the schedule table 1252 in the schedule DB and determines the time to present the message according to a predetermined rule. For example, the information processing device 1 presents the message for 10 minutes immediately after arriving at work, and does not present the message when taking a vacation or on a business trip.

[0068] Therefore, this information processing device 1 is an example of an information processing device that presents messages according to a predetermined schedule.

[0069] <Display configuration> Fig. 8 is a diagram showing an example of the appearance of information processing device 1. Information processing device 1 shown in Fig. 8 is a smartphone, and has camera 16 on the same surface as display unit 15. Display unit 15 shown in Fig. 8 is a liquid crystal display on which a transparent touch panel of operation unit 14 is superimposed. This display unit 15 has a display area R0 indicated by diagonal lines.

[0070] Fig. 9 is a diagram showing an example of a display area R0. The information processing device 1 executes an application program (hereinafter also referred to as an app) that measures a stress level. This app is read from the memory 12. When this app is executed, the processor 11 transitions between two modes: a measurement mode and a viewing mode. The display area R0 shown in Fig. 9 is an example of a display when the app is executing the measurement mode.

[0071] The measurement mode is a mode in which the user's face is continuously photographed and the user's stress level is measured based on image data included in the obtained video data. Because camera 16 is provided on the same surface as display unit 15, when the user looks at display area R0 of display unit 15, the photographing range is also directed toward the user. In measurement mode, display area R0 displays a facial image of the user looking at display area R0, photographed by camera 16. Then, this display area R0 displays text, images, etc. in end instruction area R1, explanation area R2, measurement area R3, message area R4, and status notification area R5, superimposed on the facial image.

[0072] The end instruction area R1 is an area for receiving an instruction to end the measurement mode. This end instruction area R1 displays, for example, an icon with two diagonally intersecting lines as shown in Fig. 9. When the user touches this end instruction area R1 with a finger or the like, the information processing device 1 executing the app transitions from the measurement mode to the viewing mode.

[0073] The explanation area R2 is an area that displays a written explanation of how to photograph a face in measurement mode. For example, as shown in Figure 9, the explanation area R2 displays an explanation such as "Keep your face facing forward and keep it still."

[0074] The measurement area R3 is an area that indicates the range within the display area R0 in which the face to be measured should fit. The measurement area R3 is depicted as a circle, for example, as shown in Fig. 9. The area outside the measurement area R3 is processed, for example, by adding hatching or reducing saturation, to let the user know that it is not a measurement target.

[0075] Message area R4 is an area for displaying a message selected by processor 11. For example, message area R4 displays a message such as "Did you have time to enjoy your hobbies?" as shown in FIG. 9. This message is read from message DB 123.

[0076] The status notification area R5 is an area for notifying the status of measurement. When starting to measure vital signs from a face image, this status notification area R5 displays, for example, the character string "measurement started" as shown in FIG.

[0077] <Functional configuration of information processing device> Fig. 10 is a diagram showing an example of the functional configuration of the information processing device 1. In Fig. 10, the communication unit 13 of the information processing device 1 is omitted.

[0078] The processor 11 of the information processing device 1 executes the above-mentioned application, thereby functioning as an acquisition unit 111, an authentication unit 112, a measurement unit 113, a selection unit 114, a determination unit 115, and a presentation unit .

[0079] The acquisition unit 111 acquires control information indicating the content of the user's operation from the operation unit 14. The acquisition unit 111 also acquires video data indicating a video of the user's face captured from the camera 16, and stores the video data in the video DB 122 of the memory 12. This video data includes image data indicating an image of the user's face.

[0080] Note that the acquisition unit 111 may acquire information indicating the user's schedule from the schedule DB 125. If the acquired schedule satisfies a predetermined condition, the acquisition unit 111 activates the camera 16 and causes the camera 16 to start capturing an image of the user's face.

[0081] The authentication unit 112 authenticates the user. When the acquisition unit 111 acquires a user ID and corresponding authentication information in response to a user operation, the authentication unit 112 authenticates the user by comparing the user ID and authentication information with the user DB 121. If the authentication is successful, the authentication unit 112 notifies the processor 11 of this fact, and the acquisition unit 111 activates the camera 16 and causes the camera 16 to start capturing an image of the user's face.

[0082] Note that the acquisition unit 111 may activate the camera 16 and cause the camera 16 to start capturing an image of the user's face before the authentication by the authentication unit 112. For example, if the above-mentioned facial feature data is stored in the user DB 121 as authentication information, the authentication unit 112 may authenticate the user by comparing the facial features of the user included in the captured image with the features indicated by the facial feature data.

[0083] Furthermore, if the information processing device 1 is owned and used by only one user, the processor 11 does not need to function as the authentication unit 112. In this case, the memory 12 does not need to store authentication information in the user DB 121.

[0084] The measuring unit 113 reads out video data from the video DB 122 stored in the memory 12, and measures the user's vital signs from a plurality of image data included in this video data.

[0085] The measurement unit 113 identifies the part of the image data of the multiple frames contained in the video data where the user's face is photographed, for example, based on the group of pixels contained in the measurement area R3 described above, and performs pigment component separation on the identified part.

[0086] The "pigment component separation" performed by the measurement unit 113 is a process of separating specific pigment components from the image data acquired by the camera 16 and generating image data consisting of the pigment components (referred to as pigment component image data). The separated pigment components are pigments present in the user's skin, and include at least a red pigment component derived from hemoglobin. Note that in addition to this red pigment component, the measurement unit 113 may also separate pigment components derived from, for example, melanin. Skin color is determined by light that is repeatedly scattered within the dermis and emitted outside the skin, so hemoglobin contributes greatly to the color.

[0087] After separating the pigment components, the measurement unit 113 generates pulse data indicating the pulse rate based on the time-dependent changes in the pigment component image data thus generated. Here, the pulse data is generated, for example, based on the time-dependent changes in the pixel average value of the pigment component image data composed of the red pigment component derived from hemoglobin. The pulse data may be subjected to various types of filtering.

[0088] After generating the pulse data, the measurement unit 113 generates pulse interval data indicating pulse intervals from the pulse data.The measurement unit 113 then generates pulse fluctuation spectrogram data by performing a frequency transform on the time-varying pulse interval indicated by the pulse interval data.This frequency transform refers to a transform from a time series to a frequency domain.This frequency transform is, for example, a Fourier power spectrum transform.

[0089] The measurement unit 113, which has generated the pulse fluctuation spectrogram data, calculates a stress level indicating the stress state (degree of stress) based on the generated data. For example, the measurement unit 113 calculates LF / HF, which is the ratio of the above-mentioned LF and HF, as the stress level. HF is known as an index of the parasympathetic nervous system, and LF / HF is known as an index of the sympathetic nervous system. The measurement unit 113 stores the measured vital signs in the vital sign DB 124.

[0090] The selection unit 114 compares the vital signs and stress level measured by the measurement unit 113 with predetermined thresholds, and determines whether or not a stress reaction to a message presented to the user immediately before measuring the vital signs, etc. Then, the selection unit 114 selects a message to be displayed next according to the result of the determination.

[0091] For example, the selection unit 114 refers to the message table 1232 of the message DB 123, and if it determines that a stress reaction to the message most recently presented to the user has been detected, it identifies the message ID stored in the detection time column; if it determines that a stress reaction has not been detected, it identifies the message ID stored in the non-detection time column, and prepares the message identified by that message ID as the next message.

[0092] Therefore, the information processing device 1 having the processor 11 that realizes this selection unit 114 is an example of an information processing device that selects the message to be presented after one message in accordance with the stress level measured based on an image of the user's face taken after the presentation of this message.

[0093] The determination unit 115 determines the timing of displaying the next message according to the result of the determination of whether or not a stress response has been detected. Therefore, the information processing device 1 having the processor 11 that realizes the determination unit 115 is an example of an information processing device that determines the timing of displaying a message following a message according to the stress level measured based on an image of the user's face captured after the presentation of the message.

[0094] This timing is determined based on the time (also called interval) that has elapsed since the previous message was displayed. This interval is set to a different value depending on, for example, whether or not a stress response is detected.

[0095] When a stress response is detected, presenting the next message while the influence of the previously presented message remains will impair accurate evaluation of the stress state. Therefore, when a stress response is detected, the determination unit 115 sets a sufficient interval so that the above-mentioned influence does not remain.

[0096] On the other hand, if a stress response is not detected, immediately transitioning to the presentation of the next message will not impair the accurate evaluation of the stress state because there is no influence from the previously presented message. Therefore, if a stress response is not detected, the determination unit 115 sets an interval that is at least shorter than when a stress response is detected.

[0097] When the selection unit 114 selects the next message and the determination unit 115 determines the timing at which to present the message, the presentation unit 116 instructs the display unit 15 or the speaker 17 to present the selected message at the determined timing. In response to this instruction, the display unit 15 displays the message at the timing. In addition, in response to this instruction, the speaker 17 outputs (emits) a sound or the like indicating the message at the timing.

[0098] Furthermore, the measurement unit 113 measures the user's vital signs after the next message is presented, based on the timing determined by the determination unit 115.

[0099] <Operation of information processing device> 11 is a diagram showing an example of the flow of operations of the information processing device 1. When the processor 11 of the information processing device 1 starts executing the above-mentioned application, the processor 11 transitions to a measurement mode, activates the camera 16, and starts capturing a video (step S101).

[0100] Next, processor 11 analyzes the facial image included in the captured video to measure the user's vital signs. At this time, processor 11 has not yet presented a message, so the user is not affected by the presentation of the message, that is, the user is in a normal state. Therefore, at this time, processor 11 measures the user's vital signs in a normal state, and further measures the stress level based on these vital signs (step S102).

[0101] Next, processor 11 selects a message to be presented initially (step S103). Processor 11 may select this message randomly. Alternatively, processor 11 may select a message to be presented initially based on the normal vital signs measured in step S102 described above.

[0102] Processor 11 causes display unit 15 or speaker 17 to present the selected message (step S104). Processor 11 then measures the user's vital signs based on an image of the user's face taken after the message is presented, and further measures the stress level based on these vital signs. This measurement is performed after the message is presented, and therefore is a measurement of the reaction to the message presentation (step S105). Therefore, processor 11 is an example of an information processing device that presents a message and measures the user's stress level based on an image of the user's face taken after the presentation.

[0103] After storing the measured vital signs in vital sign DB 124 of memory 12 (step S106), processor 11 determines whether or not a condition for ending the measurement mode has been met (step S107). The condition for ending the measurement mode is, for example, when a user operates an end instruction area R1 shown in Fig. 9 to end the measurement mode, when all messages to be presented have been presented, etc.

[0104] If it is determined that the conditions for terminating the measurement mode are not met (step S107; NO), the processor 11 compares the measured vital signs or stress level with a threshold value to determine whether or not the user's stress response to the presented message has been detected, and selects the next message to be presented based on the determination result (step S108).

[0105] Then, processor 11 determines the timing for next presenting the selected message (step S109). Processor 11 may determine this timing depending on the determination result of whether or not a stress reaction of the user to the presented message has been detected.

[0106] For example, the information processing device 1 sets the first interval applied when the above-described stress response is detected to be longer than the second interval applied when the stress response is not detected. In this case, when the processor 11 detects the user's stress response to the presented message, the processor 11 applies the first interval longer than the second interval. As a result, the processor 11 waits until the first interval has elapsed since the presentation of the above-described message.

[0107] Note that processor 11 may determine the timing for presenting the next message using other methods. For example, processor 11 stores the normal stress level measured in step S102 in memory 12. Then, in step S109, processor 11 continues measuring the user's stress level after the message is presented, and compares the measured stress level (i.e., the current stress level) with the normal stress level stored in memory 12. Processor 11 may then determine that the timing for presenting the next message described above is the timing when the current stress level approaches a level determined for the normal stress level.

[0108] In this case, this information processing device 1 is an example of an information processing device that measures the user's normal stress level based on an image of the user's face taken before a message is presented, measures the user's stress level regarding the message based on an image of the user's face taken after the message is presented, and presents the next message after the first message when the user's stress level measured based on an image of the user's face taken after the presentation has ended approaches a level determined as the normal stress level.

[0109] After determining the timing, processor 11 determines whether the determined timing has arrived (step S110).

[0110] While it is determined that the determined timing has not yet arrived (step S110; NO), processor 11 continues the determination of step S110.

[0111] On the other hand, if it is determined that the determined timing has arrived (step S110; YES), Then, the processor 11 returns the process to step S104 and causes the display unit 15 or the speaker 17 to display the message selected in step S108.

[0112] In step S107, when it is determined that the condition for ending the measurement mode is met (step S107; YES), processor 11 ends video capture (step S111). Then, processor 11 aggregates the measurement results at the time of reaction, extracts messages in which a stress reaction was detected, identifies the stress factors associated with each of those messages (step S112), and ends the measurement mode.

[0113] Through the operations described above, the information processing device 1 can objectively measure the stress level of the user and identify the stress factors of the user.

[0114] When the measurement mode is ended, the processor 11 transitions to the viewing mode. Fig. 12 is a diagram showing an example of the history display in the viewing mode. In Fig. 12, the display area R0 has a title area R61 and a menu icon area R62. The display area R0 shown in Fig. 12 also has a measurement mode start button R71, a history display button R72, and a graph display button R73.

[0115] When the measurement mode is ended, the processor 11 transitions to a viewing mode and displays the measurement history, which is a history of various pieces of information when the user's stress level was measured in the past.

[0116] 12 includes a measurement date area R81, a suggested message area R82, and a measurement result area R83. The measurement date area R81 is an area that displays information indicating the date and time when the information processing device 1 performed a measurement. The suggested message area R82 is an area that displays a character string indicating a message that the information processing device 1 presented to the user immediately before performing a measurement. The measurement result area R83 is an area that displays a character string indicating the measurement result.

[0117] Each of the measurement date area R81, the suggested message area R82, and the measurement result area R83 constitutes one set of history. When viewing the history in the viewing mode, the user can scroll through the history, which is represented by multiple sets of character strings.

[0118] 12, when the user is allowed to view the history in the view mode, the title area R61 displays the character string "History." The menu icon area R62 is an area for viewing and editing various settings that are referenced when executing the app.

[0119] 12, when the user is allowed to view the history in the viewing mode, the information processing device 1 inverts the color of only the history display button R72 among the measurement mode start button R71, history display button R72, and graph display button R73, and does not invert the colors of the other areas. By this inverted display, the information processing device 1 informs the user that the history display button R72 has been selected among the measurement mode start button R71, history display button R72, and graph display button R73.

[0120] 12, when the user presses the measurement mode start button R71, the information processing device 1 ends the browsing mode and starts the measurement mode. 9 is displayed in the display area R0 of the display unit 15.

[0121] On the other hand, in Fig. 12, when the user presses the graph display button R73, the information processing device 1 displays the measurement history as a graph in the viewing mode, allowing the user to view it. Fig. 13 is a diagram showing an example of a graph display in the viewing mode. In Fig. 13, the display area R0 also has a title area R61, a menu icon area R62, a measurement mode start button R71, a history display button R72, and a graph display button R73.

[0122] For example, the display area R0 shown in Fig. 13 includes a measurement value name area R91 and a graph area R92. Each of the measurement value name area R91 and the graph area R92 constitutes a set of measurement value graphs.

[0123] The measurement value name area R91 is an area that displays the name of a measurement value measured by the information processing device 1. The graph area R92 is an area that displays a graph that shows the change over time of the measurement value whose name is displayed in the corresponding measurement value name area R91. When the user is allowed to view a graph in the viewing mode, the user can scroll through the graph that shows the change over time of the measurement value.

[0124] As shown in FIG. 13, when the user is allowed to view a graph in the view mode, the title region R61 displays the character string "Graph."

[0125] 13, when the user is allowed to view a graph in the viewing mode, the information processing device 1 inverts the color of only the graph display button R73 among the measurement mode start button R71, history display button R72, and graph display button R73, and does not invert the colors of the other areas. By this inverted display, the information processing device 1 informs the user that the graph display button R73 has been selected among the measurement mode start button R71, history display button R72, and graph display button R73.

[0126] <Modification> The above is a description of the embodiment, but the contents of this embodiment can be modified as follows. In addition, the following modifications can be combined with each other.

[0127] In the above-described embodiment, the information processing device 1 has the processor 11 configured as a CPU, but the control means for controlling the information processing device 1 may have other configurations.

[0128] That is, the information processing device 1 may have various types of processors as the processor 11, such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), programmable logic device, etc., in addition to a CPU.

[0129] The operations of the processors in the above-described embodiments may not only be performed by a single processor, but may also be performed by a plurality of processors located at physically separate locations working together.

[0130] Furthermore, the order of the operations of the processor is not limited to the order described in the above embodiment, and may be changed as appropriate.

[0131] In the above-described embodiment, the information processing device 1 is a single mobile terminal, but some of the functions may be realized by an external device.

[0132] 14 is a diagram showing an example of the overall configuration of an information processing system 9 according to a modified example. The information processing system 9 shown in FIG.

[0133] The information processing device 1 is a mobile terminal, such as a smartphone, and the server device 2 is a computer connected to one or more information processing devices 1 so as to be able to communicate with each other.

[0134] The communication line 3 is a line that communicatively connects the information processing device 1 and the server device 2. The communication line 3 may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a combination thereof.

[0135] The number of information processing devices 1 and the number of server devices 2 in the information processing system 9 are not limited to those shown in Fig. 1. For example, the server device 2 may be configured as a cluster system in which multiple devices share functions.

[0136] Fig. 15 is a diagram showing an example of the configuration of a server device 2 according to a modified example. As shown in Fig. 15, the server device 2 has a processor 21, a memory 22, and a communication unit 23. These are connected by a bus.

[0137] The processor 21 reads and executes a program stored in the memory 22 to control each part of the information processing device 1. The processor 21 is, for example, a CPU.

[0138] The communication unit 23 is a communication circuit that connects the server device 2 to other devices via the communication line 3 in a wired or wireless manner so that communication is possible.

[0139] The memory 22 is a storage means for storing an operating system, various programs, data, etc. that are read into the processor 21. The memory 22 includes a RAM and a ROM.

[0140] The memory 22 may include a solid state drive, a hard disk drive, etc. The memory 22 also stores a user DB 221, a message DB 223, a vital sign DB 224, and a schedule DB 225. These are databases common to the user DB 121, the message DB 123, the vital sign DB 124, and the schedule DB 125 stored in the memory 12 of the information processing device 1 according to the embodiment.

[0141] Fig. 16 is a diagram showing an example of the functional configuration of an information processing system 9 according to a modified example. In Fig. 16, the communication unit 13 of the information processing device 1, the communication unit 23 of the server device 2, and the communication line 3 are omitted.

[0142] In the information processing system 9, the processor 11 of the information processing device 1 executes a program stored in the memory 12, thereby functioning as the acquisition unit 111, measurement unit 113, and presentation unit 116 shown in the embodiment, and also as a transmission unit 117.

[0143] In addition, in the information processing system 9, the processor 21 of the server device 2 executes the programs stored in the memory 22 to function as an acquisition unit 211, an authentication unit 212, a selection unit 214, a decision unit 215, and an instruction unit 217.

[0144] The acquisition unit 111 acquires video data showing a video of the user's face captured by the camera 16, and supplies the video data to the measurement unit 113. In this modified example, the information processing device 1 For example, the acquired video data is stored in the RAM of the memory 12, but the video data may be deleted after being used by the measurement unit 113. Furthermore, the acquisition unit 111 acquires a user ID and corresponding authentication information from the operation unit 14 in response to a user operation.

[0145] The measurement unit 113 measures the vital signs of the user from a plurality of image data included in the supplied video data.

[0146] The transmitting unit 117 transmits data indicating the user's vital signs (also referred to as vital data) measured by the measuring unit 113 to the server device 2. The transmitting unit 117 also transmits the user ID acquired by the acquiring unit 111 and the authentication information corresponding thereto to the server device 2.

[0147] In the server device 2, an acquisition unit 211 realized by the processor 21 acquires a user ID and authentication information from the information processing device 1 and supplies them to an authentication unit 212. The authentication unit 212 authenticates the user by comparing the user ID and authentication information with a user DB 221. If the authentication is successful, the authentication unit 212 notifies the processor 21 of this fact, and the acquisition unit 211 acquires the vital data received from the information processing device 1 and stores it in a vital DB 224 of the memory 22.

[0148] The selection unit 214 selects a message to be presented by the information processing device 1 from among the messages stored in the message DB 223, according to the stress level calculated from the vital signs indicated by the acquired vital data.

[0149] The determination unit 215 determines the timing at which the information processing device 1 presents the selected message, depending on the above-mentioned stress level.

[0150] The instruction unit 217 instructs the information processing device 1 to present the message selected by the selection unit 214 at the timing determined by the determination unit 215.

[0151] When the information processing device 1 receives an instruction to present a message from the server device 2, the presentation unit 116 realized by the processor 11 instructs the display unit 15 or the speaker 17 to present the selected message at a determined timing in response to this instruction.

[0152] Even after the message is presented by the display unit 15 or the speaker 17, the acquisition unit 111 acquires video data from the camera 16, and the measurement unit 113 measures the user's vital signs from multiple image data included in the video data. The transmission unit 117 transmits vital sign data indicating the vital signs measured after the presentation to the server device 2. The processor 21 of the server device 2 then associates the received vital sign data with the message presented immediately before the vital sign indicated by the vital sign data was measured, and detects the user's stress response to the message. Based on the stress response detection result, the selection unit 214 selects the next message to present, and the determination unit 215 determines the timing of presenting the next message.

[0153] In other words, this information processing system 9 is a system in which the processor 11 of the information processing device 1 causes the processor 21 of the server device 2 to realize the functions of the authentication unit 112, measurement unit 113, selection unit 114, and decision unit 115 shown in the embodiment.

[0154] As described above, the calculation of the user's stress level may be performed by processor 21, or may be performed by processor 11. In this case, transmission unit 117 may transmit the measured stress level to server device 2. Server device 2 may store the user's stress level in memory 22, regardless of whether it is measured by processor 11 or processor 21.

[0155] Therefore, this information processing system 9 is an example of an information processing system that has a server device and a terminal, where the terminal presents a message and takes a picture of the user's face after the presentation, and the terminal or the server device measures the user's stress level based on the taken image.

[0156] The above-described transmitting unit 117 may transmit the video data acquired by the acquiring unit 111 to the server device 2. In this case, the acquiring unit 211 acquires this video data. Then, the processor 21 of the server device 2 may measure the user's vital signs from a plurality of image data included in the acquired video data. That is, in this case, the processor 21 may realize a function equivalent to the measuring unit 113.

[0157] In this case, the processor 21 may store the acquired video data in a video DB 222 in the memory 22, which is indicated by the dashed line in FIG.

[0158] The information processing device 1 may store the measured stress level of the user in the memory 12. In this case, the processor 11 may diagnose the stress trend of the user from changes in the stored stress level. The information processing device 1 is an example of an information processing device that diagnoses the stress trend of the user from changes in the measured stress level of the user.

[0159] The processor 11 of the information processing device 1 may analyze the user's face from an image captured by the camera 16 after presenting a message on the display unit 15, and identify the user's line of sight. In this case, the processor 11 may determine whether the user's line of sight is looking at the message presentation (display), and may measure the user's vital signs and stress level if it is determined that the user is looking at the message presentation. Furthermore, the processor 11 may issue a warning to the user if it is determined that the user is not looking at the message presentation. This warning may be easily conveyed to a user who is not looking at the display unit 15, for example, by outputting audio from the speaker 17.

[0160] This allows the information processing device 1 to detect the stress response of a user who has viewed a displayed message. This information processing device 1 is an example of an information processing device that determines whether the user's line of sight is looking at the message based on an image of the user's face captured after the message is presented, and measures the user's stress level based on the image if it is determined that the user's line of sight is looking at the message.

[0161] The above-mentioned app is an example of a program that causes a computer to execute the steps of presenting a message, taking an image of the user's face after the message is presented, and measuring the user's stress level based on the image.

[0162] The program for executing each of the above steps may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, a semiconductor memory, etc. This program may also be downloaded via a communication line such as the Internet. [Explanation of symbols]

[0163] 1...information processing device, 11...processor, 111...acquisition unit, 112...authentication unit, 113...measurement unit, 114...selection unit, 115...decision unit, 116...presentation unit, 117...transmission unit, 12...memory, 121...user DB, 1211...affiliated institution list, 1212...user table, 122...video DB, 1221...user ID list, 1222...video data table, 123...message DB , 1231...category ID list, 1232...message table, 124...vital DB, 1241...user ID list, 1242...vital table, 125...schedule DB, 1251...user ID list, 1252...schedule table, 13...communication unit, 14...operation unit, 15...display unit, 16...camera, 17...speaker, 2...server device, 21...processor, 211...acquisition unit, 212...authentication unit, 214...selection unit, 215...decision unit, 217...instruction unit, 22...memory, 221...user DB, 222...video DB, 223...message DB, 224...Vital DB, 225...Schedule DB, 23...Communication unit, 3...Communication line, 9...Information processing system, R0...Display area, R1...Exit instruction area, R2...Explanation area, R3...Measurement area, R4...Message area, R5...Status notification area, R61...Title area, R62...Menu icon area, R71...Measurement mode start button, R72...History display button, R73...Graph display button, R81...Measurement date area, R82...Suggestion message area, R83...Measurement result area, R91...Measurement value name area, R92...Graph area.

Claims

[Claim 1] A storage means for storing a first message, a second message to be presented when stress is detected in response to the first message, and a third message to be presented when stress is not detected in response to the first message, in association with each other; a measuring means for measuring a stress level of a user based on an image of the user's face; a control means for referring to the storage means, presenting a first message, determining whether a stress response of the user has been detected based on the stress level measured by the measurement means after the first message has been presented, and selecting and presenting either the second message or the third message in accordance with the determination result; An information processing device having the above.

Images