Information Processing Apparatus, Information Processing System, and Program

Abstract

An acquisition unit 111 acquires video data showing video obtained by capturing the face of a user with a camera 16. A measurement unit 113 measures a vital sign of the user from multiple image data included in the video data. A selection unit 114 compares a stress level and the vital sign measured by the measurement unit 113 with a predetermined threshold value and determines whether a stress response against a message that is presented for the user immediately before measuring the vital sign, etc. has been detected or not. The selection unit 114 then selects a message to be displayed next according to the determination result. A decision unit 115 decides the timing at which the next message is displayed according to the determination result pertaining to whether the stress response has been detected or not. A presentation unit 116 instructs a display unit 15 or a speaker 17 to present the selected message at the decided timing.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and a program.

Background Art

[0002] Stress experienced by a person is less obvious to those around compared to illness or injury, and in some cases, even the person themselves may not be aware of it. If countermeasures are neglected while remaining unaware, stress can become chronic, causing physical and mental discomfort, and may even lead to serious situations such as accidents or illnesses. Therefore, it is required to check stress on a daily basis before such situations occur.

[0003] As a means of evaluating stress, there is a self-report method using a stress questionnaire. This stress questionnaire is composed of multiple questions, and by aggregating the answers of respondents to these questions, the stress of the respondents is evaluated. Various stress questionnaires have been developed, and specific examples include the 57-item Occupational Stress Simplified Questionnaire recommended by the Ministry of Health, Labour and Welfare.

[0004] On the other hand, it has been pointed out that the answers to stress questionnaires can be controlled by the intention of the respondents themselves, so they lack objectivity and credibility. Therefore, as an objective means of evaluating stress, a non-contact heart rate fluctuation measurement system using a camera has been developed.

[0005] For example, Patent Document 1 discloses an image processing method for stress monitoring that acquires face image data using an RGB camera, creates hemoglobin image data by using pigment component separation on the face image data, creates heart rate data based on the hemoglobin image data, creates heart rate interval data based on the heart rate data, and creates heart rate fluctuation spectrogram data by performing frequency conversion on the heart rate interval data.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-29318 [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] By the way, external stimuli that cause stress (also referred to as stress factors) are various. Stress factors include, for example, environmental factors such as weather and noise, physical factors such as illness and lack of sleep, psychological factors such as anxiety and worry, social factors such as poor human relationships and busy work, and the like.

[0008] The technology described in Patent Document 1 can determine whether the measurement subject is in a relaxed state or a stressed state based on an image obtained by photographing the face of the measurement subject, but cannot identify the stress factors that cause the stressed state.

[0009] One object of the present invention is to objectively measure the stress level of a user and identify the stress factors of the user. [Means for Solving the Problems]

[0010] In one aspect, the present invention provides an information processing apparatus that presents a message and measures the stress level of the user based on an image obtained by photographing the face of the user after the presentation.

[0011] In a preferred aspect, according to the stress level measured based on the image obtained by photographing the face of the user after the presentation of one message, a message to be presented next to the one message is selected.

[0012] In a preferred aspect, according to the stress level measured based on the image obtained by photographing the face of the user after the presentation of one message, the timing of presenting the next message after the one message is determined.

[0013] In a preferred embodiment, the normal stress level of the user is measured based on an image of the user's face taken before presenting a message, the stress level of the user with respect to the one message is measured based on an image of the user's face taken after the presentation, and when the stress level of the user measured based on an image of the user's face taken after the presentation ends approaches a level determined for the normal stress level, the next message after the one message is presented.

[0014] In a preferred embodiment, the presentation is made according to a predetermined schedule.

[0015] In a preferred embodiment, the stress tendency of the user is diagnosed from the measured change in the stress level of the user.

[0016] In a preferred embodiment, based on an image of the user's face taken after the presentation, it is determined whether the user's line of sight was looking at the presentation, and when it is determined that the line of sight was looking at the presentation, the stress level of the user is measured based on the image.

[0017] In another aspect, the present invention provides an information processing system having a server device and a terminal, the terminal presenting a message, taking an image of the user's face after the presentation, and the terminal or the server device measuring the stress level of the user based on the taken image.

[0018] In another aspect, the present invention provides a program for causing a computer to execute steps of presenting a message, taking an image of the user's face after the presentation, and measuring the stress level of the user based on the image.

Advantages of the Invention

[0019] According to the present invention, the stress level of a user can be objectively measured, and the stress factors of the user can be identified.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0021] <Embodiment><Configuration of Information Processing Apparatus> FIG. 1 is a diagram showing an example of the configuration of the information processing apparatus 1. The information processing apparatus 1 is an apparatus that measures the stress level of a user based on an image (hereinafter also referred to as a "face image") obtained by photographing the user's face. The information processing apparatus 1 shown in FIG. 1 is a portable terminal, for example, a smartphone.

[0022] As shown in FIG. 1, this information processing apparatus 1 includes a processor 11, a memory 12, a communication unit 13, an operation unit 14, a display unit 15, a camera 16, and a speaker 17. These are connected by a bus.

[0023] The processor 11 controls each part of the information processing apparatus 1 by reading and executing a program stored in the memory 12. The processor 11 is, for example, a CPU (Central Processing Unit).

[0024] The communication unit 13 is a communication circuit that connects the information processing apparatus 1 to other devices communicably, either by wire or wirelessly. For example, the communication unit 13 may have a circuit compliant with the standards of wireless communication systems such as IMT-2000, IMT-Advanced, IMT-2020, etc. Also, for example, the communication unit 13 may have a circuit compliant with the standards of wireless LANs such as IEEE802.11.

[0025] Further, for example, the communication unit 13 may have a module or the like that realizes Near Field Communication (NFC). Examples of the NFC standard include ISO / IEC18092 (NFCIP-1), ISO / IEC14443, ISO / IEC15693, or IEEE802.15.

[0026] The operation unit 14 includes operation elements such as operation buttons and touch panels for giving various instructions, receives an operation, and sends a signal corresponding to the operation content to the processor 11. This operation is, for example, pressing a button, a gesture on a touch panel, etc.

[0027] Note that the operation unit 14 may have a microphone that collects sound. In this case, the processor 11 may perform speech recognition processing on speech data indicating the user's speech collected by this microphone and accept the recognition result as the user's operation.

[0028] The display unit 15 has a display screen such as a liquid crystal display, and displays an image under the control of the processor 11. A transparent touch panel of the operation unit 14 may be disposed on the display screen.

[0029] The camera 16 is an imaging means for photographing the surroundings of the information processing apparatus 1 and generating an image, and is, for example, a digital still camera. The camera 16 includes, in addition to an optical system such as a lens, an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The camera 16 shown in FIG. 1 has, for example, the same configuration as the “RGB camera” of Patent Document 1 described above. That is, this camera 16 is an apparatus that acquires image data by arranging a plurality of detection elements each having sensitivity to the light intensity in the wavelength region corresponding to each of R (red), G (green), and B (blue) as one pixel unit. The wavelength region corresponding to each color is preferably, for example, R (red) of 580 nm or more and 680 nm or less, G (green) of 500 nm or more and 630 nm or less, and B (blue) of 410 nm or more and 530 nm or less.

[0031] Note that the camera 16 may have a polarizing plate as an optical system. This polarizing plate is disposed in the direction in which the camera 16 photographs an object. Thereby, the camera 16 can remove the reflection component from the surface of the photographing object.

[0032] Further, the camera 16 may detect light other than the wavelength region described above. For example, the camera 16 may be a 5-band camera in which two sensitivities of cyan (C) and orange (O) are added to a normal RGB camera.

[0033] The memory 12 is a storage means for storing an operating system, various programs, data, etc. that are read by the processor 11. The memory 12 includes a RAM (Random Access Memory) and a ROM (Read Only Memory).

[0034] Note that the memory 12 may include a solid state drive, a hard disk drive, or the like. Further, the memory 12 stores a user DB 121, a video DB 122, a message DB 123, a vital DB 124, and a schedule DB 125.

[0035] <Configuration of User DB> FIG. 2 is a diagram showing an example of the user DB 121. The user DB 121 is a database that stores information of a plurality of users who use the information processing apparatus 1. The user DB 121 shown in FIG. 2 has an affiliated organization list 1211 and a user table 1212.

[0036] The affiliated organization list 1211 is a list that enumerates information on the affiliated organizations of the users. The affiliated organization list 1211 stores an affiliated organization ID, which is identification information for identifying the affiliated organization of the user, and an affiliated organization name, which is the name thereof. One user table 1212 is associated with each of the affiliated organization IDs.

[0037] The user table 1212 is a table that stores information on each user belonging to the affiliated organization identified by the corresponding affiliated organization ID. The user table 1212 has columns for a user ID, a user name, authentication information, gender, and date of birth.

[0038] The user ID in the user DB 121 is identification information that uniquely identifies each user.

[0039] The user name in the user DB 121 is information such as a character string indicating the user's name. Note that, in addition to the column for the user name, the user DB 121 may have columns indicating the user's nickname, email address, telephone number, and the like.

[0040] The authentication information in the user DB 121 is information used for user authentication, such as a password, a PIN code, or the like.

[0041] The gender in the user DB 121 is one of the pieces of information indicating the attributes of the user, and indicates the gender of the user. For example, in the user DB 121 shown in FIG. 2, the gender "male" is stored in association with the user ID "U1".

[0042] The date of birth in the user DB 121 is one of the pieces of information indicating the attributes of the user, and indicates the date of birth of the user.

[0043] Note that in addition to gender and date of birth, the user DB 121 may have a column for storing information indicating the attributes of the user. For example, the user DB 121 may store information such as past medical history information, medication information, work history information, etc. as information indicating the attributes of the user. Further, the user DB 121 may store data representing the facial features of the user (also referred to as facial feature data) as information indicating the attributes of the user. The facial feature data may be stored in the authentication information column.

[0044] Also, in the embodiment, the information processing apparatus 1 is a terminal that can be used by a plurality of users, but may be a terminal owned and used by a single user. In this case, the user DB 121 may store in association a user ID indicating a single user who owns and uses the information processing apparatus 1 and information such as the affiliated organization, name, and attributes of that user.

[0045] <Configuration of Video DB> FIG. 3 is a diagram showing an example of the video DB 122. The video DB 122 is a database that stores data of videos (also referred to as video data) in which the face of each user is photographed. The video DB 122 shown in FIG. 3 has a user ID list 1221 and a video data table 1222.

[0046] The user ID list 1221 in the video DB 122 is a list that enumerates user IDs. This user ID is information common to the user IDs stored in the user DB 121.

[0047] The video data table 1222 in the video DB 122 has columns for start date and time, length, video ID, and video data. In this video data table 1222, the column for video data stores the video data itself. The column for start date and time stores information on the date and time when shooting of the corresponding video data started (i.e., the start date and time). The column for length stores the length of the corresponding video data. The unit of this length is, for example, a time unit such as minutes and seconds. The column for video ID stores the video ID, which is identification information that uniquely identifies the corresponding video data.

[0048] Figure 4 is a diagram for explaining an example of the concept of video data. Video data is composed of a plurality of still images taken at regular intervals. One frame of this plurality of still images is called a frame. Also, the number of still images taken per unit time is called the frame rate. As shown in Figure 4, in video data, image data indicating the still image taken at each moment is associated with each of a plurality of ordered frames. Note that the video data may be compressed as long as the image data for each frame can be obtained by expanding them respectively.

[0049] <Configuration of Message DB> Figure 5 is a diagram showing an example of the message DB 123. The message DB 123 is a database that stores the messages presented by the information processing apparatus 1 to the user in association with the order of presentation. The message DB 123 shown in Figure 5 has a category ID list 1231 and a message table 1232.

[0050] The category ID list 1231 in the message DB 123 is a list that enumerates pairs of category IDs and category names. This category ID is identification information that uniquely identifies each category to which the message displayed on the information processing apparatus 1 belongs. The category name is a character string indicating the name of the category identified by the category ID.

[0051] To each category ID, one message table 1232 is associated. The message table 1232 has a column for message ID, a column for message, a column for stress factor, a column for at detection, and a column for at non-detection. In the message table 1232, the column for message ID stores the message ID which is identification information for identifying the message. The column for message stores a character string or the like indicating the content of the message.

[0052] When the stress level measured from the user's face image exceeds the threshold, the information processing apparatus 1 determines that a stress reaction to the message has been detected. The column for stress factor stores the type of stress factor that causes the stress reaction when a stress reaction to the presented message is detected.

[0053] For example, when the stress level of the user measured after presenting the message "Are you worried about human relationships?" shown in FIG. 5 exceeds the threshold, the information processing apparatus 1 determines that the user has a stress reaction, and identifies that the stress factor causing the stress reaction is "interpersonal relationships at work".

[0054] Note that the information processing apparatus 1 may have a plurality of thresholds for comparison with the measured stress level. In this case, the stress level is evaluated according to which level divided by these plurality of thresholds it belongs to. The information processing apparatus 1 may total the scores corresponding to the levels to which the measured stress level belongs as the scores of the corresponding stress factors.

[0055] Also, the information processing apparatus 1 may total the measured stress level value itself as the score of the corresponding stress factor. That is, the information processing apparatus 1 may present a plurality of messages, and total the stress levels of the user measured after the presentation for each message, and identify the stress factor with the highest total score as the stress factor of the user.

[0056] In the message table 1232, the column for detection stores the message ID that indicates the next message to be presented when the user's stress response is detected after presenting the corresponding message.

[0057] The column for non-detection stores the message ID that indicates the next message to be presented when the user's stress response is not detected after presenting the corresponding message.

[0058] <Configuration of Vital DB> Figure 6 is a diagram showing an example of the Vital DB 124. The Vital DB 124 is a database that stores vital signs (hereinafter also referred to as vitals) indicating the vital signs of the user measured for each user. The Vital DB 124 shown in Figure 6 has a user ID list 1241 and a vital table 1242.

[0059] The user ID list 1241 in the Vital DB 124 is a list that enumerates user IDs. This user ID is information common to the user IDs stored in the user DB 121.

[0060] The vital table 1242 in the Vital DB 124 has columns for measurement date and time, message ID, video ID, and one or more vitals. The vital columns shown in Figure 6 are four items: pulse, respiratory rate, LF, and HF. The column for measurement date and time stores the date and time when the vitals were measured. The column for message ID stores the message ID that identifies the message presented before measuring the vitals. The column for video ID stores the video ID indicating the video data including the image data used when measuring the vitals.

[0061] The vital columns store each of the vitals measured based on the face image of the user taken after the message identified by the corresponding message ID is presented. Pulse is the number of times the blood vessels on the user's face pulsate per minute. Since the pulse is equal to the number of times the heart beats per minute, it is synonymous with the heart rate. Respiratory rate is the number of breaths per minute measured from the user's face image.

[0062] The LF and HF columns both store the calculated values as a result of performing a power spectrum analysis on the heartbeat interval variations. For example, the LF column stores the value (integrated value) obtained by summing the intensities of the components in the low frequency (LF) region, and the HF column stores the value (integrated value) obtained by summing the intensities of the components in the high frequency (HF) region. It is said that the low frequency components correspond to blood pressure variations and the high frequency components correspond to respiratory variations.

[0063] The stress level is calculated using at least one of these vital signs. The stress level is, for example, LF / HF.

[0064] <Configuration of the Schedule DB> FIG. 7 is a diagram showing an example of the schedule DB 125. The schedule DB 125 is a database that stores the schedules of each user described above. The schedule DB 125 shown in FIG. 7 has a user ID list 1251 and a schedule table 1252.

[0065] The user ID list 1251 in the schedule DB 125 is a list that enumerates user IDs. This user ID is information common to the user IDs stored in the user DB 121.

[0066] The schedule table 1252 in the schedule DB 125 has columns for date, start time, end time, and requirements. The date column stores the date on which the requirement is scheduled. The start time column stores the time at which the requirement is scheduled to start. The end time column stores the time at which the requirement is scheduled to end. The requirements column stores a character string or the like indicating the content of the requirement.

[0067] The information processing apparatus 1 determines the time to present a message according to a determined rule by referring to the schedule table 1252 of the schedule DB. For example, the information processing apparatus 1 presents a message within 10 minutes immediately after arrival at work, and does not present a message when taking a vacation or on a business trip.

[0068] Therefore, this information processing apparatus 1 is an example of an information processing apparatus that presents messages in accordance with a predetermined schedule.

[0069] <Configuration of the display unit> FIG. 8 is a diagram showing an example of the information processing apparatus 1. The information processing apparatus 1 shown in FIG. 8 is a smartphone and has a camera 16 on the same surface as the display unit 15. The display unit 15 shown in FIG. 8 is a liquid crystal display overlaid with a transparent touch panel included in the operation unit 14. This display unit 15 has a display area R0 indicated by diagonal lines.

[0070] FIG. 9 is a diagram showing an example of the display area R0. The information processing apparatus 1 executes an application program (hereinafter also referred to as an app) that measures the stress level. This app is read from the memory 12. When this app is executed, the processor 11 transitions between two modes: a measurement mode and a browsing mode. The display area R0 shown in FIG. 9 is an example of the display when the app is executing the measurement mode.

[0071] The measurement mode is a mode that continuously captures the user's face and measures the user's stress level based on the image data included in the obtained video data. Since the camera 16 is provided on the same surface as the display unit 15, when the user is looking at the display area R0 of the display unit 15, the shooting range is also directed at the user himself / herself. In the measurement mode, the display area R0 displays the face image of the user looking at the display area R0, which is captured by the camera 16. Then, in this display area R0, characters, images, etc. are respectively displayed in an end instruction area R1, an explanation area R2, a measurement area R3, a message area R4, and a status notification area R5 overlaid on the face image.

[0072] The end instruction area R1 is an area that accepts an instruction to end the measurement mode. This end instruction area R1 displays, for example, an icon in which two lines intersecting diagonally as shown in FIG. 9 are drawn. When the user touches this end instruction area R1 with a finger or the like, the information processing apparatus 1 that executes the app transitions from the measurement mode to the browsing mode.

[0073] The explanation area R2 is an area that shows an explanation of the face photographing method in the measurement mode in text. For example, as shown in FIG. 9, the explanation area R2 displays an explanatory text such as "Please keep the face fixed without moving while facing forward."

[0074] The measurement area R3 is an area that indicates the range within the display area R0 where the face to be measured should fit. For example, as shown in FIG. 9, the measurement area R3 is drawn as a circle. And outside this measurement area R3, in order to make the user recognize that it is not a measurement target, for example, processing such as overlapping hatching or reducing the saturation is performed.

[0075] The message area R4 is an area that displays the message selected by the processor 11. For example, as shown in FIG. 9, the message area R4 displays a message such as "Did you have time to enjoy your hobby?" This message is read from the message DB123.

[0076] The status notification area R5 is an area that notifies the measurement status. When starting the measurement of vital signs from the face image, for example, as shown in FIG. 9, the status notification area R5 displays a string such as "Measurement start".

[0077] <Functional Configuration of the Information Processing Apparatus> FIG. 10 is a diagram showing an example of the functional configuration of the information processing apparatus 1. In FIG. 10, the communication unit 13 of the information processing apparatus 1 is omitted.

[0078] By executing the above-described application, the processor 11 of the information processing apparatus 1 functions as an acquisition unit 111, an authentication unit 112, a measurement unit 113, a selection unit 114, a determination unit 115, and a presentation unit 116.

[0079] The acquisition unit 111 acquires control information indicating the content of the user's operation from the operation unit 14. Further, the acquisition unit 111 acquires video data indicating a video obtained by photographing the user's face from the camera 16 and stores it in the video DB 122 of the memory 12. This video data includes image data indicating the user's face image.

[0080] Note that the acquisition unit 111 may acquire information indicating the user's schedule from the schedule DB 125. When the acquired schedule satisfies a predetermined condition, the acquisition unit 111 activates the camera 16 and starts photographing the user's face with this camera 16.

[0081] The authentication unit 112 authenticates the user. When the acquisition unit 111 acquires a user ID and corresponding authentication information in response to the user's operation, the authentication unit 112 collates this user ID and authentication information with the user DB 121 to authenticate the user. When the authentication is successful, the authentication unit 112 notifies the processor 11 to that effect, and the acquisition unit 111 activates the camera 16 and starts photographing the user's face with this camera 16.

[0082] Note that the acquisition unit 111 may activate the camera 16 and start photographing the user's face with this camera 16 before the authentication by the authentication unit 112. For example, when the above-described face feature data is stored as authentication information in the user DB 121, the authentication unit 112 may authenticate the user by comparing the features of the user's face included in the photographed image with the features indicated by this face feature data.

[0083] Further, when the information processing apparatus 1 is owned and used by only one user, the processor 11 does not necessarily need to function as the authentication unit 112. In this case, the memory 12 does not necessarily need to store authentication information in the user DB 121.

[0084] The measurement unit 113 reads video data from the video DB 122 stored in the memory 12 and measures the user's vital signs from a plurality of pieces of image data included in this video data.

[0085] The measurement unit 113 identifies, for example, the portion where the user's face is captured based on the pixel group within the measurement region R3 described above among the image data of a plurality of frames included in the video data, and performs pigment component separation on the identified portion.

[0086] "Pigment component separation" performed by the measurement unit 113 is a process of separating a specific pigment component from the image data acquired by the camera 16 and generating image data (referred to as pigment component image data) composed of the pigment component. The pigment component to be separated is a pigment present in the user's skin and includes at least a red pigment component derived from hemoglobin. In addition to this red pigment component, the measurement unit 113 may also separate pigment components derived from, for example, melanin. Since the color of the skin is light that is repeatedly scattered inside the dermis and emitted outside the skin, the contribution of hemoglobin is large.

[0087] The measurement unit 113 that has performed pigment component separation generates pulse data indicating the pulse based on the temporal change of the pigment component image data generated thereby. Here, the pulse data is generated, for example, by the temporal change of the pixel average value of the pigment component image data composed of the red pigment component derived from hemoglobin. The pulse data may be subjected to various filtering processes.

[0088] The measurement unit 113 that has generated the film stop data generates pulse interval data indicating the pulse interval from this pulse data. Then, the measurement unit 113 generates pulse fluctuation spectrogram data by performing frequency conversion on the temporal change of the pulse interval indicated by the pulse interval data. This frequency conversion refers to the conversion from the time series to the frequency domain. This frequency conversion is, for example, Fourier power spectrum conversion.

[0089] The measuring unit 113 that generates the fluctuation spectrogram data of the pulse rate calculates the stress level indicating the stress state (degree of stress) based on the generated data. For example, this measuring unit 113 calculates LF / HF, which is the ratio of LF and HF described above, as the stress level. Note that HF is known as an index of the parasympathetic nerve, and LF / HF is known as an index of the sympathetic nerve. The measuring unit 113 stores the measured vital signs in the vital sign DB 124.

[0090] The selection unit 114 compares the vital signs measured by the measuring unit 113 and the stress level with a predetermined threshold value, and determines whether a stress reaction to the message presented to the user immediately before measuring the vital signs or the like is detected. Then, the selection unit 114 selects the next message to be displayed according to the result of the determination.

[0091] For example, when it is determined that a stress reaction to the message presented to the user immediately before is detected, this selection unit 114 refers to the message table 1232 of the message DB 123, and identifies the message ID stored in the column at the time of detection. When it is determined that no stress reaction is detected, the message ID stored in the column at the time of non-detection is identified, and the message identified by the message ID is prepared as the next message.

[0092] Therefore, the information processing apparatus 1 having the processor 11 that realizes this selection unit 114 is an example of an information processing apparatus that selects the next message to be presented after one message according to the stress level measured based on the image obtained by photographing the user's face after the presentation of one message.

[0093] The determination unit 115 determines the timing for next message display according to the result of the above-described determination as to whether a stress reaction has been detected. Thus, the information processing apparatus 1 having the processor 11 that realizes this determination unit 115 is an example of an information processing apparatus that determines the timing for presenting a message next to a certain message according to the degree of stress measured based on an image obtained by photographing the user's face after presenting the certain message.

[0094] This timing is determined based on the elapsed time (also referred to as an interval) from the timing when the previous message was displayed. This interval has, for example, different values determined according to the presence or absence of detection of a stress reaction.

[0095] When a stress reaction is detected, presenting the next message while the influence of the previously presented message remains will impair the accurate evaluation of the stress state. Therefore, when a stress reaction is detected, the determination unit 115 provides a sufficient interval such that the above-described influence does not remain.

[0096] On the other hand, when no stress reaction is detected, even if immediately shifting to the presentation of the next message, since there is no influence of the previously presented message, the accurate evaluation of the stress state will not be impaired. Thus, when no stress reaction is detected, the determination unit 115 provides at least an interval shorter than that when detected.

[0097] When the selection unit 114 selects the next message and the determination unit 115 determines the timing for presenting that message, the presentation unit 116 instructs the display unit 15 or the speaker 17 to present the selected message at the determined timing accordingly. The display unit 15 displays the above-described message at the above-described timing in response to this instruction. Also, the speaker 17 outputs (plays) a voice or the like indicating the above-described message at the above-described timing in response to this instruction.

[0098] Further, the measurement unit 113 measures the user's vital signs after the next message is presented based on the timing determined by the determination unit 115.

[0099] <Operation of the information processing device> FIG. 11 is a diagram showing an example of the operation flow of the information processing device 1. When the processor 11 of the information processing device 1 starts the execution of the above-described application, it transitions to the measurement mode, activates the camera 16, and starts shooting a video (step S101).

[0100] Next, the processor 11 analyzes the face image included in the captured video to measure the user's vital signs. At this time, since the processor 11 has not yet presented a message, the user is in a state where they are not affected by the presentation of the message, that is, in a normal state. Therefore, at this time, the processor 11 measures the user's normal vital signs, and further measures the stress level based on this vital sign (step S102).

[0101] Next, the processor 11 selects the message to be presented for the first time (step S103). The processor 11 may select this message randomly. Also, the processor 11 may select the message to be presented for the first time based on the normal vital signs measured in step S102 described above.

[0102] The processor 11 causes the selected message to be presented on the display unit 15 or the speaker 17 (step S104). Then, the processor 11 measures the user's vital signs based on the image of the user's face captured after the presentation of this message, and further measures the stress level based on these vital signs. Since this measurement is performed after the presentation of the message, it is a measurement at the time of reaction to the presentation of the message (step S105). Therefore, this processor 11 is an example of an information processing device that presents a message and measures the stress level of this user based on the image of the user's face captured after this presentation.

[0103] When the processor 11 stores the measured vital signs in the vital sign DB 124 of the memory 12 (step S106), it determines whether the end condition of the measurement mode is satisfied (step S107). This end condition is, for example, when a user operation to end the measurement mode is performed on the end instruction area R1 shown in FIG. 9, or when all the messages to be presented have been presented.

[0104] When it is determined that the end condition of the measurement mode is not satisfied (step S107; NO), the processor 11 compares the measured vital signs or the stress level with a threshold value to determine whether a user stress reaction to the presented message has been detected, and selects the message to be presented next according to the determination result (step S108).

[0105] Then, the processor 11 determines the timing to present the selected message next (step S109). The processor 11 may determine this timing according to the determination result of whether a user stress reaction to the presented message has been detected.

[0106] For example, the information processing apparatus 1 sets the first interval applied when the above-described detection of the stress reaction is present to be longer than the second interval applied when the detection of the stress reaction is not present. In this case, when a user stress reaction to the presented message is detected, the processor 11 applies the first interval that is longer than the second interval. As a result, the processor 11 waits until this first interval has elapsed since the presentation of the above-described message.

[0107] Note that the processor 11 may determine the timing for presenting the next message in other ways. For example, the processor 11 stores the normal stress level measured in step S102 in the memory 12. Then, in step S109, the processor 11 continues to measure the stress level of the user after the presentation of the message, and compares the measured stress level (i.e., the current stress level) with the normal stress level stored in the memory 12. Then, the processor 11 may determine the timing when the current stress level approaches the level determined for the normal stress level as the timing for presenting the next message described above.

[0108] In this case, this information processing apparatus 1 measures the normal stress level of this user based on an image of the user's face taken before the presentation of one message, measures the stress level of this user with respect to one message based on an image of the user's face taken after this presentation, and when the stress level of this user measured based on an image of the user's face taken after the end of this presentation approaches the level determined for the normal stress level, it is an example of an information processing apparatus that presents the next message of one message.

[0109] When determining the above-described timing, the processor 11 determines whether or not the determined timing has arrived (step S110).

[0110] While it is determined that the determined timing has not arrived (step S110; NO), the processor 11 continues the determination in this step S110.

[0111] On the other hand, when it is determined that the determined timing has arrived (step S110; YES), the processor 11 returns the process to step S104 and causes the message selected in step S108 to be presented on the display unit 15 or the speaker 17.

[0112] In step S107, when it is determined that the end condition of the measurement mode is satisfied (step S107; YES), the processor 11 ends the video shooting (step S111). Then, the processor 11 aggregates the measurement results during the reaction, extracts the messages in which a stress reaction is detected, identifies the stress factors associated with each of those messages (step S112), and ends the measurement mode.

[0113] Through the operations described above, this information processing apparatus 1 can objectively measure the stress level of the user and identify the stress factors of that user.

[0114] Note that when the measurement mode ends, the processor 11 shifts to the browsing mode. FIG. 12 is a diagram showing an example of the display of the history in the browsing mode. In FIG. 12, the display area R0 has a title area R61 and a menu icon area R62. Further, the display area R0 shown in FIG. 12 has a measurement mode start button R71, a history display button R72, and a graph display button R73.

[0115] When the measurement mode ends, the processor 11 shifts to the browsing mode and displays the measurement history. The measurement history is the history of various information when the stress level of the user was measured in the past.

[0116] For example, the display area R0 shown in FIG. 12 includes a measurement date area R81, a prompt message area R82, and a measurement result area R83. The measurement date area R81 is an area for displaying information indicating the date and time when the information processing apparatus 1 performed the measurement. The prompt message area R82 is an area for displaying a character string indicating the message presented to the user immediately before the information processing apparatus 1 performed the measurement. The measurement result area R83 is an area for displaying a character string indicating the measurement result.

[0117] The measurement date area R81, the prompt message area R82, and the measurement result area R83 each constitute a set of history. When the user views the history in the browsing mode, the user can scroll and check the history represented by a plurality of sets of character strings.

[0118] As shown in FIG. 12, when allowing the user to view the history in the browsing mode, in the title area R61, a character string of the title "History" is displayed. The menu icon area R62 is an area for viewing and editing various settings referred to when executing the application and giving an instruction to that effect.

[0119] Also, as shown in FIG. 12, when allowing the user to view the history in the browsing mode, the information processing apparatus 1 only reverses the color of the history display button R72 among the measurement mode start button R71, the history display button R72, and the graph display button R73, and does not reverse the colors of the other areas. By this reversed display, the information processing apparatus 1 notifies the user that the history display button R72 is selected among the measurement mode start button R71, the history display button R72, and the graph display button R73.

[0120] In FIG. 12, when the measurement mode start button R71 is pressed by the user, the information processing apparatus 1 ends the browsing mode and starts the measurement mode. Thereby, the information processing apparatus 1 displays the image shown in FIG. 9 in the display area R0 of the display unit 15.

[0121] On the other hand, in FIG. 12, when the graph display button R73 is pressed by the user, the information processing apparatus 1 displays the measurement history in a graph in the browsing mode and allows the user to view it. FIG. 13 is a diagram showing an example of the graph display in the browsing mode. Also in FIG. 13, the display area R0 has a title area R61, a menu icon area R62, a measurement mode start button R71, a history display button R72, and a graph display button R73.

[0122] For example, the display area R0 shown in FIG. 13 includes a measured value name area R91 and a graph area R92. The measured value name area R91 and the graph area R92 each constitute a set of measured value graphs.

[0123] The measurement value name area R91 is an area for displaying the names of the measurement values measured by the information processing apparatus 1. The graph area R92 is an area for displaying a graph showing the change over time of the measurement values corresponding to the names displayed in the corresponding measurement value name area R91. When the user is to view the graph in the browsing mode, the user can scroll and check the graph representing the change over time of the measurement values.

[0124] As shown in FIG. 13, when the user is to view the graph in the browsing mode, the title area R61 displays a character string of the title "Graph".

[0125] Also, as shown in FIG. 13, when the user is to view the graph in the browsing mode, the information processing apparatus 1 reverses only the color of the graph display button R73 among the measurement mode start button R71, the history display button R72, and the graph display button R73, and does not reverse the colors of the other areas. By this reverse display, the information processing apparatus 1 conveys to the user that the graph display button R73 among the measurement mode start button R71, the history display button R72, and the graph display button R73 is selected.

[0126] <Modification Example> The above is the description of the embodiment, but the content of this embodiment can be modified as follows. Also, the following modification examples may be combined with each other.

[0127] In the above-described embodiment, the information processing apparatus 1 had the processor 11 configured by a CPU, but the control means for controlling the information processing apparatus 1 may have other configurations.

[0128] That is, in addition to the CPU, the information processing apparatus 1 may have various processors such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a programmable logic device, etc. as the processor 11.

[0129] The operations of the processor in the above-described embodiments may be performed not only by a single processor but also by a plurality of physically separated processors working in cooperation.

[0130] Also, the order of each operation of the processor is not limited to the order described in the above-described embodiments and may be changed as appropriate.

[0131] In the above-described embodiments, the information processing apparatus 1 is a single mobile terminal, but part of its functions may be realized by an external apparatus.

[0132] FIG. 14 is a diagram showing an example of the overall configuration of an information processing system 9 according to a modified example. The information processing system 9 shown in FIG. 14 includes an information processing apparatus 1, a server apparatus 2, and a communication line 3.

[0133] The information processing apparatus 1 is a mobile terminal, for example, a smartphone. The server apparatus 2 is a computer communicably connected to one or more information processing apparatuses 1.

[0134] The communication line 3 is a line that communicably connects the information processing apparatus 1 and the server apparatus 2. The communication line 3 may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a combination thereof.

[0135] Note that the number of each of the information processing apparatus 1 and the server apparatus 2 in the information processing system 9 is not limited to that shown in FIG. 1. For example, the server apparatus 2 may be configured by a cluster system in which a plurality of apparatuses share functions.

[0136] FIG. 15 is a diagram showing an example of the configuration of the server apparatus 2 according to a modified example. As shown in FIG. 15, the server apparatus 2 includes a processor 21, a memory 22, and a communication unit 23. These are connected by a bus.

[0137] The processor 21 controls each part of the information processing apparatus 1 by reading and executing the programs stored in the memory 22. The processor 21 is, for example, a CPU.

[0138] The communication unit 23 is a communication circuit that connects the server device 2 to other devices via the communication line 3 in a wired or wireless manner so as to enable communication.

[0139] The memory 22 is a storage means for storing an operating system, various programs, data, etc. that are read by the processor 21. The memory 22 includes a RAM and a ROM.

[0140] Note that the memory 22 may include a solid state drive, a hard disk drive, etc. Also, the memory 22 stores a user DB 221, a message DB 223, a vital DB 224, and a schedule DB 225. These are databases common to the user DB 121, the message DB 123, the vital DB 124, and the schedule DB 125 stored in the memory 12 of the information processing apparatus 1 shown in the embodiment.

[0141] FIG. 16 is a diagram showing an example of the functional configuration of the information processing system 9 according to a modification. In FIG. 16, the communication unit 13 of the information processing apparatus 1, the communication unit 23 of the server device 2, and the communication line 3 are omitted.

[0142] In the information processing system 9, the processor 11 of the information processing apparatus 1 functions as the acquisition unit 111, the measurement unit 113, and the presentation unit 116 shown in the embodiment, and also functions as the transmission unit 117, by executing the programs stored in the memory 12.

[0143] Also, in the information processing system 9, the processor 21 of the server device 2 functions as the acquisition unit 211, the authentication unit 212, the selection unit 214, the determination unit 215, and the instruction unit 217, by executing the programs stored in the memory 22.

[0144] The acquisition unit 111 acquires video data showing a video of the user's face from the camera 16 and supplies this video data to the measurement unit 113. In this modification, the information processing apparatus 1 stores, for example, the acquired video data in the RAM of the memory 12, but the video data may be deleted after being used by the measurement unit 113. Further, the acquisition unit 111 acquires a user ID and authentication information corresponding thereto in response to a user operation from the operation unit 14.

[0145] The measurement unit 113 measures the user's vital signs from a plurality of image data included in the supplied video data.

[0146] The transmission unit 117 transmits data indicating the user's vital signs measured by the measurement unit 113 (also referred to as vital sign data) to the server apparatus 2. Further, the transmission unit 117 transmits the user ID acquired by the acquisition unit 111 and the authentication information corresponding thereto to the server apparatus 2.

[0147] In the server apparatus 2, the acquisition unit 211 realized by the processor 21 acquires the user ID and the authentication information from the information processing apparatus 1 and supplies these to the authentication unit 212. The authentication unit 212 collates the user ID and the authentication information with the user DB 221 to authenticate the user. When the authentication is successful, the authentication unit 212 notifies the processor 21 to that effect, and the acquisition unit 211 acquires the vital sign data received from the information processing apparatus 1 and stores this in the vital sign DB 224 of the memory 22.

[0148] The selection unit 214 selects a message to be presented to the information processing apparatus 1 from among the messages stored in the message DB 223 according to the stress level calculated from the vital signs indicated by the acquired vital sign data.

[0149] The determination unit 215 determines the timing at which the selected message is to be presented to the information processing apparatus 1 according to the above-described stress level.

[0150] The instruction unit 217 instructs the information processing apparatus 1 to present the message selected by the selection unit 214 at the timing determined by the determination unit 215.

[0151] When the information processing apparatus 1 receives an instruction to present a message from the server apparatus 2, the presentation unit 116 realized by the processor 11 instructs the display unit 15 or the speaker 17 to present the selected message at the timing determined according to this instruction.

[0152] Even after the message is presented by the display unit 15 or the speaker 17, the acquisition unit 111 acquires video data from the camera 16, and the measurement unit 113 measures the user's vital signs from a plurality of image data included in this video data. Also, the transmission unit 117 transmits vital data indicating the vital signs measured after the presentation to the server apparatus 2. Thereby, the processor 21 of the server apparatus 2 associates the received vital data with the message presented immediately before the vital signs indicated by the vital data are measured, and detects the user's stress reaction to that message. Then, according to the detection result of this stress reaction, the selection unit 214 selects the next message to be presented, and the determination unit 215 determines the timing to present the next message.

[0153] That is, this information processing system 9 is a system in which the processor 11 of the information processing apparatus 1 causes the processor 21 of the server apparatus 2 to realize the functions of the authentication unit 112, the measurement unit 113, the selection unit 114, and the determination unit 115 shown in the embodiment.

[0154] As described above, the calculation of the user's stress level may be performed by the processor 21, but may also be performed by the processor 11. In this case, the transmission unit 117 may transmit the measured stress level to the server apparatus 2. The server apparatus 2 may store this in the memory 22 regardless of whether the user's stress level is measured by the processor 11 or the processor 21.

[0155] Therefore, this information processing system 9 has a server device and terminals. The terminals present a message, and after this presentation, photograph the user's face. The terminal or the server device is an example of an information processing system that measures the stress level of the user based on the photographed image.

[0156] Note that the above-described transmission unit 117 may transmit the video data acquired by the acquisition unit 111 to the server device 2. In this case, the acquisition unit 211 acquires this video data. Then, the processor 21 of the server device 2 may measure the user's vital signs from the plurality of image data included in the acquired video data. That is, in this case, the processor 21 may realize a function corresponding to the measurement unit 113.

[0157] Also, in this case, the processor 21 may store the acquired video data in the video DB 222 indicated by the broken line in FIG. 15 in the memory 22.

[0158] The information processing device 1 may store the measured stress level of the user in the memory 12. In this case, the processor 11 may diagnose the stress tendency of this user from the change in the stored stress level. This information processing device 1 is an example of an information processing device that diagnoses the stress tendency of this user from the change in the measured stress level of the user.

[0159] The processor 11 of the information processing device 1 may analyze the user's face from the image photographed by the camera 16 after presenting the message by the display of the display unit 15 and specify the user's line of sight. In this case, the processor 11 may determine whether or not the user's line of sight is looking at the presentation (display) of the message, and if it is determined that the user is looking, measure the user's vital signs and stress level. Further, when the processor 11 determines that the user's line of sight is not looking at the presentation of the message, the processor 11 may issue a warning to the user. This warning can be transmitted to the user who is not looking at the display unit 15, for example, by outputting sound from the speaker 17.

[0160] As a result, the information processing apparatus 1 can detect the stress reaction of the user who has seen the displayed message. This information processing apparatus 1 is an example of an information processing apparatus that determines whether or not the user's line of sight was viewing this presentation based on an image of the user's face taken after the presentation of the message, and measures the stress level of the user based on the image when it is determined that the line of sight was viewing the presentation.

[0161] The above-described application is an example of a program that causes a computer to execute a step of presenting a message, a step of taking an image of the user's face after this presentation, and a step of measuring the stress level of the user based on this image.

[0162] The program for executing each of the above steps can be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium such as a magnetic tape and a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, and a semiconductor memory. Further, this program may be downloaded via a communication line such as the Internet.

Explanation of Signs

[0163] 1… Information processing device, 11… Processor, 111… Acquisition unit, 112… Authentication unit, 113… Measurement unit, 114… Selection unit, 115… Decision unit, 116… Presentation unit, 117… Transmission unit, 12… Memory, 121… User DB, 1211… Affiliated organization list, 1212… User table, 122… Video DB, 1221… User ID list, 1222… Video data table, 123… Message DB, 1231… Category ID list, 1232… Message table, 124… Vital DB, 1241… User ID list, 1242… Vital table, 125… Schedule DB, 1251… User ID list, 1252… Schedule table, 13… Communication unit, 14… Operation unit, 15… Display unit, 16… Camera, 17… Speaker, 2… Server device, 21… Processor, 211… Acquisition unit, 212… Authentication unit, 214… Selection unit, 215… Decision unit, 217… Instruction unit, 22… Memory, 221… User DB, 222… Video DB, 223… Message DB, 224… Vital DB, 225… Schedule DB, 23… Communication unit, 3… Communication line, 9… Information processing system, R0… Display area, R1… End instruction area, R2… Explanation area, R3… Measurement area, R4… Message area, R5… Status notification area, R61… Title area, R62… Menu icon area, R71… Measurement mode start button, R72… History display button, R73… Graph display button, R81… Measurement date area, R82… Presentation message area, R83… Measurement result area, R91… Measurement value name area, R92… Graph area.

Claims

1. Measuring means for measuring the stress level of the user based on an image obtained by photographing the user's face; First presenting means for presenting a first message; Second presenting means for selecting and presenting a second message according to the first message and the stress level measured after the presentation of the first message; Identifying means for identifying the stress factors of the user based on the stress level measured after the presentation of the second message An information processing apparatus having the above.

2. The identifying means further identifies the stress factors of the user based on the stress level measured after the presentation of the first message The information processing apparatus according to Claim 1.

3. The second presenting means When the stress level measured after the presentation of the first message exceeds a threshold value, the time interval from the end of the presentation of the first message to the start of the presentation of the second message is longer than when the stress level is below the threshold value, and adjusts the presentation timing of the second message accordingly The information processing apparatus according to Claim 1.

4. The measuring means measures the normal stress level of the user based on an image obtained by photographing the user's face before the presentation of the first message, The second presenting means presents the second message at a timing when the stress level of the user measured based on an image obtained by photographing the user's face after the end of the presentation of the first message approaches a level determined for the normal stress level. The information processing apparatus according to Claim 2.

5. Perform the above presentation according to a predetermined schedule The information processing apparatus according to Claim 1.

6. Diagnose the stress tendency of the user from the change in the measured stress level of the user The information processing apparatus according to Claim 1.

7. Based on an image obtained by photographing the user's face after the above presentation, determine whether the user's line of sight was looking at the presentation, and if it is determined that the line of sight was looking at the presentation, measure the stress level of the user based on the image The information processing apparatus according to Claim 1.

8. Having a server device and a terminal, The terminal Measuring means for measuring the stress level of the user based on an image obtained by photographing the user's face; First presenting means for presenting a first message; Second presentation means for selecting and presenting a second message according to the first message and the stress level measured after the presentation of the first message; Specifying means for specifying the stress factor of the user based on the stress level measured after the presentation of the second message having an information processing system.

9. A program for causing a computer to measure the stress level of a user based on an image obtained by photographing the user's face; present a first message; select and present a second message according to the first message and the stress level measured after the presentation of the first message; specify the stress factor of the user based on the stress level measured after the presentation of the second message and execute.

Images