Program and Information Processing Device

Abstract

To easily determine an emotion of a user.SOLUTION: A program of the present invention allows a computer to perform: a step of generating information related to a pulse wave of a user based on image data obtained by photographing skin of the user, so as to determine an arousing degree of the user based on the generated information; a step of determining a stress degree of the user; and a step of determining an emotion of the user based on a position corresponding to the determined arousing degree and stress degree on a two-dimensional plane being the two-dimensional plane which is defined by an axis indicating the arousing degree and an axis indicating the stress degree and where respective emotions are associated with respective positions.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to a program and an information processing apparatus.

Background Art

[0002] Measurement devices such as electroencephalographs, electrocardiographs, and plethysmographs have been attached to the human body to identify emotions based on electroencephalograms and plethysmograms. For example, Patent Document 1 describes an emotion estimation device that measures the electroencephalogram of a user who is a subject using an electroencephalograph and estimates the emotion of the user based on the measured electroencephalogram data of the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology described in Patent Document 1, in order to estimate the emotion of a user, it is necessary to arrange a plurality of electrodes on the scalp of the user and detect an electroencephalogram signal using an electroencephalograph, resulting in a large-scale device configuration. An object of the present invention is to be able to easily determine the emotion of a user.

Means for Solving the Problems

[0005] In one embodiment, the present invention provides a program for a computer to perform the following steps: generate information about a user's pulse wave based on image data obtained by photographing the user's skin; determine the user's level of arousal based on the generated information; determine the user's level of stress; and determine the user's emotion based on the position on the two-dimensional plane corresponding to the determined level of arousal and stress, on a two-dimensional plane defined by an axis indicating the level of arousal and an axis indicating the level of stress, where each emotion is associated with a position. According to the present invention, a user's emotions can be easily determined without using measuring devices for measuring brain waves or pulse waves.

[0006] In a preferred embodiment, the correspondence between position on the two-dimensional plane and emotion may be based on Russell's annular model.

[0007] In a preferred embodiment, the step of determining the stress level may involve determining the stress level based on the pulse rate or heart rate determined from the image data.

[0008] In a preferred embodiment, the step of determining the stress level may involve receiving input of the stress level from the user.

[0009] In a preferred embodiment, the step of determining the level of alertness may be based on at least one of the high-frequency components in the pulse wave fluctuation spectrum that are higher than a first threshold frequency, and the low-frequency components in the pulse wave fluctuation spectrum that are lower than a second threshold frequency.

[0010] In a preferred embodiment, the degree of arousal relative to the reference value may be calculated by dividing the high-frequency component by the low-frequency component, while the degree of sedation relative to the reference value may be determined based on the high-frequency component.

[0011] In a preferred embodiment, the process may further include a step of determining a reference value for the user's level of alertness for each time period based on a plurality of image data obtained by photographing the user's skin at multiple time periods, and in the step of determining the level of alertness, the level of alertness may be corrected using the reference value.

[0012] In a preferred embodiment, the process may further include a step of determining a baseline value for the user's stress level for each time period based on the stress level acquired during that time period, and in the step of determining the level of arousal, the stress level may be corrected using the baseline value.

[0013] In a preferred embodiment, in the step of determining the stress level, the stress level may be input from the user, while if there is no input from the user, the stress level may be determined based on the pulse rate or heart rate determined from the image data.

[0014] In a preferred embodiment, the step of determining the user's emotions may involve generating an index that shows the change in the user's emotions over a predetermined period.

[0015] Furthermore, the present invention provides an information processing device comprising: an imaging means for imaging a user's skin; a means for generating information about the user's pulse wave based on the image captured and determining the user's level of arousal based on the generated information; a means for acquiring the user's stress level; and a means for determining the user's emotion based on the position on the two-dimensional plane corresponding to the identified stress level and arousal level, where each emotion is defined by an axis indicating the level of arousal and an axis indicating the level of stress, and each emotion is associated with a position on the two-dimensional plane. According to the present invention, a user's emotions can be easily determined without using measuring devices for measuring brain waves or pulse waves. [Brief explanation of the drawing]

[0016] [Figure 1]A diagram showing the hardware configuration of the information processing apparatus according to the embodiment. [Figure 2] A diagram showing the user DB according to the embodiment. [Figure 3] A diagram showing the video DB according to the embodiment. [Figure 4] A diagram for explaining the concept of video data according to the embodiment. [Figure 5] A diagram showing the measurement result DB according to the embodiment. [Figure 6] A diagram showing an example of the display unit and camera of the information processing apparatus according to the embodiment. [Figure 7] A diagram showing the display area of the display unit of the information processing apparatus according to the embodiment. [Figure 8] A diagram showing the functional configuration of the information processing apparatus according to the embodiment. [Figure 9] A diagram showing the operation flow of the information processing apparatus according to the embodiment. [Figure 10] A diagram showing an example of the display of the reference value according to the embodiment. [Figure 11] A diagram showing the operation flow of the information processing apparatus according to the embodiment. [Figure 12] A diagram for explaining the determination of emotion according to the embodiment.

Mode for Carrying Out the Invention

[0017] [Embodiment] [Configuration of Information Processing Apparatus] FIG. 1 is a diagram showing an example of the configuration of the information processing apparatus 1. The information processing apparatus 1 is a device that determines the emotion of a user based on an image obtained by photographing the user's skin. In this embodiment, the image obtained by photographing the user's skin is an image of the face (hereinafter also referred to as a "face image"). The information processing apparatus 1 shown in FIG. 1 is a mobile terminal, for example, a smartphone.

[0018] As shown in FIG. 1, this information processing apparatus 1 is a computer having a processor 11, a memory 12, a communication unit 13, an operation unit 14, a display unit 15, and a camera 16. These are connected by a bus.

[0019] The processor 11 controls each part of the information processing device 1 by reading and executing programs stored in the memory 12. The processor 11 is, for example, a CPU (Central Processing Unit).

[0020] The communication unit 13 is a communication circuit that connects the information processing device 1 to other devices via wired or wireless means. For example, the communication unit 13 may have a circuit that conforms to wireless communication system standards such as IMT-2000, IMT-Advanced, or IMT-2020. Alternatively, for example, the communication unit 13 may have a circuit that conforms to wireless LAN standards such as IEEE802.11.

[0021] Furthermore, for example, the communication unit 13 may have a module that enables Near Field Communication (NFC). Examples of NFC standards include ISO / IEC 18092 (NFCIP-1), ISO / IEC 14443, ISO / IEC 15693, or IEEE 802.15.

[0022] The control unit 14 is equipped with various control elements such as control buttons and a touch panel for issuing various instructions, and receives operations and sends signals corresponding to the content of those operations to the processor 11. These operations include, for example, pressing buttons or making gestures on the touch panel.

[0023] The operation unit 14 may also have a microphone for collecting voice. In this case, the processor 11 may perform voice recognition processing on the voice data representing the user's voice collected by this microphone and accept the recognition result as the user's operation.

[0024] The display unit 15 has a display screen such as a liquid crystal display or an organic EL display, and displays images under the control of the processor 11. A transparent touch panel of the operation unit 14 may be placed on top of the display screen. Furthermore, the information processing device 1 may be equipped with a speaker that outputs audio for notifying the user, or a vibrator that vibrates the main body of the information processing device 1.

[0025] Camera 16 is an imaging means that captures images of the area around the information processing device 1 and generates images; for example, it is a digital still camera. Camera 16 includes an optical system such as a lens, as well as an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Camera 16 is a device that acquires image data by arranging multiple detection elements, each of which is a single pixel, that are sensitive to the light intensity in the wavelength ranges corresponding to each of the colors R (red), G (green), and B (blue). Preferably, the wavelength ranges corresponding to each color are, for example, 580 nm to 680 nm for R (red), 500 nm to 630 nm for G (green), and 410 nm to 530 nm for B (blue).

[0027] Furthermore, the camera 16 may have a polarizing plate as part of its optical system. This polarizing plate is positioned in the direction in which the camera 16 photographs the object. This allows the camera 16 to remove reflective components from the surface of the object being photographed.

[0028] Furthermore, camera 16 may detect light outside the wavelength range described above. For example, camera 16 may be a 5-band camera that adds two sensitivities, cyan (C) and orange (O), to a standard RGB camera.

[0029] Memory 12 is a storage means for storing the operating system, various programs, data, etc., which are loaded into the processor 11. Memory 12 includes RAM (Random Access Memory) and ROM (Read Only Memory).

[0030] The memory 12 may include a solid-state drive, a hard disk drive, etc. The memory 12 also stores the user DB 121, the video DB 122, and the measurement results DB 123.

[0031] <User Database Configuration> Figure 2 shows an example of User DB 121. User DB 121 is a database that stores information of multiple users who use the Information Processing Device 1.

[0032] User DB121 is a collection of data records that store information about each user, and it has fields for User ID, User Name, Authentication Information, and Criteria Value. The User ID in User DB121 is an identifier that uniquely identifies each user.

[0033] In User DB121, the username is information such as a string indicating the user's name. In addition to the username field, User DB121 may also have fields for the user's nickname, email address, telephone number, etc. Authentication information in User DB121 is information used to authenticate users, such as passwords and PIN codes.

[0034] The reference values ​​in User DB121 are the reference values ​​for alertness and stress levels, and each reference value used to determine alertness and stress levels from the measurement results is stored. Two reference values ​​for alertness are stored: an LF / HF reference value and an HF reference value. Each reference value is calculated for each time period, and in this embodiment, the LF / HF reference values ​​for alertness, the HF reference value for alertness, and stress levels are all stored for three time periods.

[0035] User DB121 may also have fields for recording information indicating user attributes other than those mentioned above. For example, User DB121 may store information indicating user attributes such as gender, date of birth, medical history, medication information, and work history. User DB121 may also store data representing the user's facial features (also called facial feature data) as information indicating user attributes. Facial feature data may be stored in the authentication information field.

[0036] Furthermore, in this embodiment, the information processing device 1 is a terminal that can be used by multiple users, but it may also be a terminal owned or used by a single user. In this case, the user DB 121 may store a user ID that indicates a single user who owns and uses the information processing device 1, and information such as the user's name and reference values ​​in association with that user.

[0037] <Video Database Structure> Figure 3 shows an example of the video database 122. The video database 122 is a database that stores video data (also called video data) of each user's face. The video database 122 shown in Figure 3 has a user ID list 1221 and a video data table 1222. The user ID list 1221 in video DB122 is a list of user IDs. These user IDs are the same information as the user IDs stored in user DB121.

[0038] The video data table 1222 in the video DB 122 is a collection of data records having columns for start date and time, length, video ID, and video data. In the video data table 1222, the video data column stores the video data itself. The start date and time column stores information about the date and time (i.e., start date and time) when the shooting of the corresponding video data began. The length column stores the length of the corresponding video data. The unit of this length is, for example, a time unit such as minutes or seconds. The video ID column stores the video ID, which is identification information that uniquely identifies the corresponding video data.

[0039] Figure 4 is a diagram illustrating an example of the concept of video data. Video data consists of multiple still images taken at regular intervals. Each of these still images is called a frame. The number of still images taken per unit time is called the frame rate. As shown in Figure 4, in video data, each of the multiple ordered frames is associated with image data representing the still image taken at each moment. Note that video data may be compressed, as long as the image data for each frame can be extracted and obtained individually.

[0040] <Configuration of the measurement results database> Figure 5 shows an example of the measurement results DB123. The measurement results DB123 is a database that stores vital signs (hereinafter also referred to as vitals) that indicate the vital signs of each user, as well as the level of alertness, stress level, and emotions determined from the vital signs. The measurement results DB123 shown in Figure 5 has a user ID list 1231 and a measurement data table 1232.

[0041] The user ID list 1231 in the measurement results DB123 is a list of user IDs. These user IDs are the same information as the user IDs stored in user DB121. The measurement data table 1232 in the measurement results DB123 is a collection of data records that have columns for measurement date and time, measurement type, video ID, input values ​​entered by the user (indicators related to the user's sensations such as pleasant or unpleasant, or indicators showing the degree of stress the user is feeling), vital signs such as pulse rate, respiratory rate, LF / HF, and HF, and determined arousal level, stress level, and emotion.

[0042] The "Measurement Date and Time" field stores the date and time when vital signs were measured. The "Measurement Type" field stores information to distinguish whether the measurement performed was to determine the user's baseline values ​​or to determine the user's emotions.

[0043] The Video ID field stores the Video ID, which indicates the video data containing the image data used when measuring vital signs. The input value field stores a numerical value representing the user's feelings at the time of measurement, ranging from unpleasant to pleasant, based on user input.

[0044] The columns for vital signs—pulse rate, respiratory rate, LF / HF, and HF—store the respective vital signs measured based on the captured facial image of the user. Pulse rate is the number of times the blood vessels in the user's face beat per minute. Since pulse rate is equal to the number of times the heart beats per minute, it is synonymous with heart rate. Respiratory rate is the number of breaths per minute measured from the user's facial image.

[0045] The LF / HF and HF columns both store values ​​calculated from power spectral analysis of heart rate interval variability. For example, LF is the sum of the intensities of components in the low-frequency range, which is below a predetermined frequency threshold on the low-frequency side (integral value), and HF is the sum of the intensities of components in the high-frequency range, which is above a predetermined frequency threshold on the high-frequency side (integral value). LF / HF is the value obtained by dividing LF by HF. It is said that the low-frequency components correspond to blood pressure variability, and the high-frequency components correspond to respiratory variability.

[0046] The "Arousal Level" field stores the arousal level value calculated based on LF / HF, HF, and the baseline arousal level value stored in User DB121. The "Stress Level" field stores the stress level value calculated based on the above input values ​​or LF / HF and the baseline stress level value stored in User DB121. The "Emotion" field stores a word indicating emotion, determined based on the calculated arousal and stress level values.

[0047] <Configuration of the display unit of the information processing device> Figure 6 shows the external appearance of the information processing device 1. The information processing device 1 shown in Figure 8 is a smartphone and has a camera 16 on the same surface as the display unit 15. The display unit 15 is a liquid crystal display (or organic EL display) with a transparent touch panel of the operation unit 14 superimposed on it. The display unit 15 has a display area R0 indicated by diagonal lines.

[0048] Figure 7 shows an example of the display in display area R0. The information processing device 1 runs an application program (hereinafter also referred to as "app") that measures vital signs and determines the level of alertness, stress level, and emotion. This app is read from memory 12. When this app is executed, the processor 11 can display the information in display area R0 as shown in Figure 7 (hereinafter also referred to as the measurement screen).

[0049] While the measurement screen is displayed, the user's face is continuously captured, and the user's vital signs are measured based on the image data contained in the obtained video data. In addition, the user's stress level is obtained through user input.

[0050] Since the camera 16 is mounted on the same surface as the display unit 15, when the user is looking at the display area R0 of the display unit 15, its shooting range is also directed towards the user. In the measurement screen, the display area R0 displays the user's face image, captured by the camera 16, as the user is looking at the display area R0. Then, overlaid on the face image, the display area R0 displays text, images, etc., in the termination instruction area R1, explanation area R2, input area R3, measurement area R4, and status notification area R5, respectively.

[0051] The termination instruction area R1 is the area that receives instructions to terminate the measurement screen. For example, as shown in Figure 9, the termination instruction area R1 displays an icon with two diagonally intersecting lines. When the user touches the termination instruction area R1 with their finger or the like, the information processing device 1, which is running the application, switches from the measurement screen to another screen.

[0052] Explanation area R2 is an area that displays a textual explanation of how to photograph the face in measurement mode. For example, as shown in Figure 7, explanation area R2 displays explanatory text such as, "Hold the smartphone directly in front of your face, align your face within the circle, and press the 'Start Measurement' button."

[0053] Input area R3 is where the user inputs their current feelings. A scale is displayed showing a range from unpleasant to pleasant. The user can input their current feelings by touching the pointer P0 displayed on the scale with their finger and sliding along the scale horizontally on the screen.

[0054] The measurement area R4 is the area within the display area R0 that should contain the face to be measured. The measurement area R4 is drawn as a circle, for example, as shown in Figure 7. The area outside the measurement area R4 is processed in a way that makes the user aware that it is not the area to be measured, such as by overlaying hatching or reducing the saturation.

[0055] The status notification area R5 is an area that notifies the measurement status. When vital sign measurement is started from a facial image, the status notification area R5 displays the text "Measurement Started," for example, as shown in Figure 7.

[0056] <Functional Configuration of Information Processing Devices> Figure 8 shows an example of the functional configuration of the information processing device 1. In Figure 8, the communication unit 13 of the information processing device 1 is omitted.

[0057] The processor 11 of the information processing device 1 functions as an acquisition unit 111, an authentication unit 112, an arousal level determination unit 113, a stress level determination unit 114, a reference value determination unit 115, an emotion determination unit 116, and a display control unit 117 by executing the application described above.

[0058] The acquisition unit 111 acquires information from the operation unit 14 that indicates the content of the user's operations. For example, it acquires information indicating the user's "current feelings" based on the user's operations in the input area R3 in Figure 7, quantifies it, and stores it in the measurement result DB 123 of the memory 12. The acquisition unit 111 also acquires video data showing a video of the user's face from the camera 16 and stores it in the video DB 122 of the memory 12. This video data includes image data showing the user's face image.

[0059] The authentication unit 112 authenticates the user. When the acquisition unit 111 acquires the user ID and the corresponding authentication information in response to the user's operation, the authentication unit 112 compares the user ID and authentication information with the user DB 121 to authenticate the user. If authentication is successful, the acquisition unit 111 activates the camera 16 and causes the camera 16 to start taking pictures of the user's face.

[0060] The acquisition unit 111 may activate the camera 16 before the authentication unit 112 performs the authentication, causing the camera 16 to start capturing the user's face. For example, if the above-mentioned face feature data is stored as authentication information in the user DB 121, the authentication unit 112 may authenticate the user by comparing the features of the user's face included in the captured image with the features indicated by this face feature data.

[0061] Furthermore, if the information processing device 1 is owned and used by only one user, the processor 11 does not need to function as the authentication unit 112. In this case, the memory 12 does not need to store authentication information in the user database 121.

[0062] The alertness level determination unit 113 reads video data from the video DB 122 stored in the memory 12 and measures the user's vital signs, including pulse waves, from multiple image data contained in the video data. For example, the alertness level determination unit 113 identifies the portion of the image data of multiple frames contained in the video data that contains the user's face, based on, for example, the pixel group that falls within the measurement area R4 described above, and performs pigment component separation on the identified portion.

[0063] The "pigment component separation" performed by the arousal level determination unit 113 is a process that separates specific pigment components from image data acquired by the camera 16 and generates image data composed of those pigment components (referred to as pigment component image data). The separated pigment components are pigments present in the user's skin and include at least a red pigment component derived from hemoglobin. In addition to this red pigment component, the arousal level determination unit 113 may also separate pigment components derived from melanin, for example. Skin color is light that is repeatedly scattered within the dermis and emitted outside the skin, so hemoglobin contributes significantly to this.

[0064] The alertness determination unit 113 generates pulse rate data based on the time-dependent changes in pigment component image data generated by the separation of pigment components. Here, the pulse rate data is generated, for example, by the time-dependent changes in the pixel average value of pigment component image data composed of red pigment components derived from hemoglobin. The pulse rate data may be subjected to various filtering processes.

[0065] The alertness level determination unit 113 generates pulse interval data, which shows the interval between pulses, from the generated pulse data. Then, the alertness level determination unit 113 generates pulse variation spectrogram data by performing a frequency transformation on the temporal change in pulse intervals shown in the pulse interval data. This frequency transformation refers to a transformation from the time series to the frequency domain. This frequency transformation is, for example, a Fourier power spectral transform.

[0066] The alertness level determination unit 113 calculates LF and HF based on the generated pulse rate variability spectrogram data. It then calculates LF / HF, which is the ratio of LF to HF. Note that HF ​​is known as an indicator of the parasympathetic nervous system, and LF / HF is known as an indicator of the sympathetic nervous system. The alertness level determination unit 113 stores the measured vital signs (pulse rate, LF / HF, HF) in the measurement results DB 123.

[0067] The alertness determination unit 113 then reads the target user's alertness reference values ​​(LF / HF reference values ​​and HF reference values ​​for each time period) from the user DB 121. Based on the calculated LF / HF and HF, it calculates the alertness level. The alertness level is determined by correcting the calculated alertness level using the alertness reference values ​​(LF / HF reference values ​​and HF reference values), and the determined alertness level is stored in the measurement results DB 123.

[0068] The stress level determination unit 114 calculates a stress level indicating the stress state (degree of stress). The stress level is calculated based on the value stored in the input value field of the measurement result DB 123 (the value entered by the user from the input area R3 in Figure 7). The stress level determination unit 114 reads the input value of the target user from the measurement result DB 123 and also reads the target user's stress level reference value from the user DB 121. Then, it determines the stress level by correcting the stress level based on the input value using the stress level reference value. The determined stress level is stored in the measurement result DB 123.

[0069] Note that data may not be stored in the input value field of the measurement results DB123. In the screen shown in Figure 7, if the user does not perform an input operation in the input area R3 when starting the measurement, no data will be stored in the input value field of the measurement results DB123.

[0070] If no data is stored in the input value column of the measurement result DB 123, the stress level determination unit 114 reads the LF / HF value from the LF / HF column of the measurement result DB 123 and calculates the stress level based on the LF / HF value. For example, if the input value is to be shown as a number in the range of -4 to +4 according to the user's input operation, the minimum and maximum values ​​of the LF / HF value are set, and the LF / HF value in the range of minimum to maximum value is converted to a value in the range of -4 to +4. In this way, the LF / HF value can be made to a number within the same range as the input value. Then, the stress level is determined by correcting the converted LF / HF value using the stress level standard value. The determined stress level is stored in the measurement result DB 123.

[0071] The reference value determination unit 115 determines a reference value for alertness used to determine the level of alertness, and a reference value for stress used to determine the level of stress. The reference value determination unit 115 calculates pulse rate, LF / HF, and HF from multiple image data contained in the video data by performing the same processing as the alertness determination unit 113 described above. The calculated pulse rate, LF / HF, and HF are then stored in the measurement result DB 123.

[0072] The reference value determination unit 115 then classifies the LF / HF and HF values ​​by time period based on the shooting date and time (start date and time) of the video data, and calculates the average values ​​of LF / HF and HF for each time period. The average values ​​of LF / HF and HF for each time period are then determined as the reference values ​​for the level of alertness. In other words, the reference values ​​for LF / HF and HF for each time period are determined as the reference values ​​for the level of alertness. The reference value determination unit 115 stores the determined reference values ​​for the level of alertness in the "Reference Values ​​for Level of Alertness" column of the user DB 121.

[0073] Furthermore, the reference value determination unit 115 reads the values ​​stored in the input value column of the measurement result DB 123. For data records where no value is stored in the input value column, it reads the LF / HF value from the LF / HF column of the measurement result DB 123. Then, similar to what was done in the stress level determination unit 114 described above, it converts the LF / HF value so that it is within the same range as the numerical values ​​that the input value can take, and sets the LF / HF value to a numerical value within the same range as the input value.

[0074] The reference value determination unit 115 then classifies the input values ​​or converted LF / HF values ​​by time period based on the measurement date and time, and calculates the average value of the input values ​​or converted LF / HF values ​​for each time period. The average value of the input values ​​for each time period is then determined as the reference value for stress level. The reference value determination unit 115 stores the determined reference value for stress level in the reference value column of the user DB 121.

[0075] As described above, the reference value determination unit 115 stores the vital signs (pulse rate, LF / HF, HF) and user input values ​​measured and calculated for the target user at multiple time points in the measurement results DB 123. At this time, the "reference value" is stored in the measurement type column of each data record. The reference value determination unit 115 then reads the LF / HF, HF, and input values ​​stored in the multiple records in the measurement results DB 123 where the "reference value" is stored in the measurement type column, calculates the average value for each time period, and determines the reference value for alertness and stress levels for each time period. The determined reference values ​​are then stored in the reference value column of the user DB 121.

[0076] The emotion determination unit 116 determines the user's emotion based on the level of arousal determined by the arousal level determination unit 113, the level of stress determined by the stress level determination unit 114, and the reference values ​​for arousal and stress determined by the reference value determination unit 115. Specifically, the emotion is determined based on the position on a two-dimensional plane corresponding to the level of arousal and stress, where each emotion is associated with a specific position, defined by axes representing arousal and axes representing stress. More specifically, the association between the position on the two-dimensional plane and the emotion is performed based on Russell's annular model. That is, the emotion determination unit 116 reads the target user's level of arousal and stress from the measurement results DB 123, determines the emotion at the time of measurement based on the position on the two-dimensional plane based on Russell's annular model, and stores the determined emotion in the measurement results DB.

[0077] The display control unit 117 performs control for displaying information on the display unit 15. For example, as shown in Figure 7, it performs control to display characters or images in the display area R0 of the display unit 15. It also performs control to display information to the user, such as the reference value determined by the reference value determination unit 115, the arousal level determined by the arousal level determination unit 113, the stress level determined by the stress level determination unit, and the emotion determined by the emotion determination unit.

[0078] <Operation of the Information Processing Device> Figure 9 is a diagram illustrating an example of the operation flow of the information processing device 1, and shows an example of the operation flow for determining the baseline value of arousal level and the baseline value of stress level. When the processor 11 of the information processing device 1 starts executing the application program described above, first the acquisition unit 111 determines whether or not the measurement timing has arrived (step S201).

[0079] To determine baseline values ​​for alertness and stress levels, it is necessary to record video and accept user input at multiple times throughout the day. For example, measurements could be taken every hour or every 30 minutes during the period from 7:00 to 22:00, which is assumed to be when users are awake, in multiple time slots (e.g., 7:00-10:00, 13:00-16:00, and 19:00-22:00). The measurement timing would be at the hourly or 30-minute intervals.

[0080] If the measurement timing has not yet arrived (step S201: NO), the acquisition unit 111 repeats the determination process in step S201. If the measurement timing has arrived (step S201: YES), it outputs a notification to the user (step S202). The notification to the user may be, for example, by the display control unit 117 displaying a message on the display unit 15 and outputting sound from a speaker (not shown), or by vibrating the main body of the information processing device 1 with a vibrator (not shown).

[0081] When the system detects that the user has touched a portion of the message displayed on the display unit 15 with their finger or the like, the display control unit 117 controls the display unit 15 to display the screen shown in Figure 7. The acquisition unit 111 then controls the camera 16 to start recording video (step S203). Subsequently, the acquisition unit 111 accepts the user's input operation in the input area R3 shown in Figure 7 (step S204). If the user does not perform any operation during a predetermined period (for example, during video recording), the system terminates the acceptance process, treating it as if no input was received.

[0082] Next, the acquisition unit 111 controls the camera 16 to end video recording and acquires the recorded video data from the camera 16. The acquisition unit 111 then assigns an ID (video ID) to the video data and stores it in the video DB 122 along with the start date and time and length. If input is received in step S204, the input value is stored in the measurement result DB 123 along with the measurement date and time and the aforementioned video ID (step S205). In this case, the measurement type column is set to "reference value".

[0083] Next, the reference value determination unit 115 reads the video data stored in the video DB 122 and calculates LH / HF and HF based on that video data as described above (step S206). Then, the reference value determination unit 115 stores the calculated LH / HF and HF values, as well as the input values ​​obtained from the user's input operations in the input area R3, in the measurement result DB 123 (step S207). In this case, the data is stored in the same data record as the data record that stores the input values ​​and video ID described above.

[0084] Next, the acquisition unit 111 determines whether a predetermined number of measurements have been completed (step S208). The processes described in steps S203 to S207 are performed periodically over one day or multiple days at multiple time periods, but a sufficient number of processing steps to calculate the reference value is set in advance. If the acquisition unit 111 has not reached the set number of steps (step S209: NO), it returns to step S201 and repeats the process.

[0085] When the set number of times is reached (step S208: YES), the reference value determination unit 115 calculates and stores the reference values ​​for alertness and stress (step S209). The reference value determination unit 115 reads data records from the measurement data table 1232 of the target user from the measurement results DB 123 where the measurement type column is "reference value," and calculates and determines the reference values ​​for alertness (reference values ​​for LF / HF and HF respectively) and stress for each time period based on the LF / HF, HF, and input values ​​stored in the data records, as described above. The reference value determination unit 115 then stores the determined reference values ​​in the reference value column of the data record of the target user's user ID in the user DB 121.

[0086] Figure 10 shows an example of the display unit 15 showing the confirmation screen for the determined level of alertness. In the screen of Figure 10, "average value" is displayed instead of "reference value" for easier understanding by the user, and the average values ​​of LF / HF, HF, pulse rate, and respiratory rate for each time period are displayed.

[0087] Figure 11 is a diagram illustrating an example of the operation flow of the information processing device 1, and shows an example of the operation flow for determining emotions. The following operations are performed when a user starts, for example, driving a car or watching a movie, and when measuring changes in emotions during driving, watching a movie, etc.

[0088] The processor 11 of the information processing device 1 starts executing the application program described above, and when it receives a measurement start operation from the user, it first reads the target user's threshold values ​​for alertness and stress level from the user DB 121 (step S301).

[0089] Next, the display control unit 117 controls the display unit 15 to display the screen shown in Figure 7. Then, the acquisition unit 111 controls the camera 16 to start recording video (step S302). Subsequently, the acquisition unit 111 accepts user input operations in the input area R3 shown in Figure 7 (step S303). If the user does not perform any operations during a predetermined period (for example, during video recording), the acceptance process is terminated as if no input had been received.

[0090] Next, the acquisition unit 111 controls the camera 16 to end video recording and acquires the recorded video data from the camera 16. The acquisition unit 111 then assigns an ID (video ID) to the video data and stores it in the video DB 122 along with the start date and time and length. If input is received in step S204, the input value is stored in the measurement result DB 123 along with the measurement date and time and the aforementioned video ID (step S304). In this case, the measurement type column is set to "emotion".

[0091] Next, the alertness level determination unit 113 reads the video data stored in the video DB 122, calculates LF / HF and HF based on the video data as described above, and then determines the alertness level using the reference value for alertness level (step S305). The method for determining the alertness level will be described later.

[0092] Next, the stress level determination unit 114 reads the input values ​​stored in the measurement result DB 123 and determines the stress level based on those input values ​​or the LF / HF calculated when determining the arousal level (step S306). The method for determining the stress level will be described later. Next, the emotion determination unit 116 determines the user's emotion based on the determined arousal level and stress level (step S307). The method for determining emotions will be described later.

[0093] The arousal level determination unit 113, the stress level determination unit 114, and the emotion determination unit 116 store the arousal level, stress level, and emotion determined as described above, along with the measurement date and time, and the calculated LF / HF and HF, as a new data record in the measurement results DB 123.

[0094] Next, the acquisition unit 111 determines whether or not to terminate the measurement and the determination process of arousal level, stress level, and emotion (step S309). For example, if the user touches the termination instruction area R1 on the display screen of the display unit 15 in Figure 7 with their finger, the acquisition unit 111 determines that the measurement has ended. A user might terminate the measurement if, for example, they are driving a car and interrupt or end driving. Or, if they are watching a movie, they might terminate the movie screening.

[0095] If the acquisition unit 111 determines that the measurement is complete (step S309: YES), the display control unit 117 performs output processing such as displaying the measurement results on the display unit 15 (step S310), and the process for determining the user's emotions is completed. If the acquisition unit 111 determines that it does not want to end the measurement (step S309: NO), it determines whether or not the measurement timing has arrived (step S311).

[0096] To measure the user's emotional changes, it is necessary to record multiple videos and accept user input over time. For example, during driving or watching a movie, measurements could be taken at predetermined intervals (e.g., every few minutes or every ten minutes or so). The measurement timing is determined at these predetermined intervals.

[0097] If the measurement timing has not yet arrived (step S311: NO), the acquisition unit 111 repeats the decision process in step S311. If the measurement timing has arrived (step S311: YES), it outputs a notification to the user (step S312). The notification to the user may be, for example, by the display control unit 117 displaying a message on the display unit 15 and outputting sound through a speaker (not shown), or by vibrating the main body of the information processing device 1 with a vibrator (not shown). When it is detected that the user has touched the part of the message displayed on the display unit 15 with a finger or the like, the process returns to step S302 and the above process is repeated.

[0098] Figure 12 is a diagram illustrating how emotions are determined. Figure 12(A) shows Russell's annular model. Russell's annular model is a model in which, if the vertical axis is taken as arousal (arousal-calmness) and the horizontal axis is taken as an emotion value of pleasant-unpleasant, the "emotions" corresponding to each value on the vertical axis and horizontal axis are arranged in a circular pattern on the coordinate plane.

[0099] Figure 12(B) is a diagram showing a two-dimensional plane for determining emotions in this embodiment, with the vertical axis representing arousal level and the horizontal axis representing stress level. In Figure 12(B), the points where the vertical and horizontal axes intersect indicate the positions of the reference values ​​for arousal level and stress level determined by the reference value determination unit 115.

[0100] First, the method for determining the level of alertness on the vertical axis of Figure 12(B) in the alertness determination unit 113 will be explained. The upper axis on the alert side of the vertical axis shows the ratio of the LF / HF value to the reference value (the reference value of LF / HF for alertness). The lower axis on the calming side shows the ratio of the HF value to the reference value (the reference value of HF for alertness).

[0101] First, it is determined whether the level of alertness at a given measurement is on the alert side or the sedated side. This determination may be based on the LF / HF value at the time of measurement, on the HF value at the time of measurement, or on both the LF / HF and HF values. Alternatively, it may be determined solely by the LF value.

[0102] For example, when making a decision based on the LF / HF value, compare LF / HF with a reference value for LF / HF. In this case, the reference value used is the reference value for the time period corresponding to the measurement date and time. If LF / HF is greater than the reference value for LF / HF, it is determined to be on the awakened side; if LF / HF is less than the reference value for LF / HF, it is determined to be on the sedated side.

[0103] Furthermore, when making a decision based on the HF value, compare HF with the reference value for HF. If HF is greater than the reference value for HF, it is determined to be on the sedation side; if HF is less than the reference value for HF, it is determined to be on the awakening side.

[0104] Furthermore, when determining based on both LF / HF and HF values, if both LF / HF and HF are greater than their respective reference values, the patient is determined to be awake; if both LF / HF and HF are less than their respective reference values, the patient is determined to be sedated. If LF / HF is greater than the reference value for LF / HF and HF is less than the reference value for HF, the patient is determined to be awake. If LF / HF is less than the reference value for LF / HF and HF is greater than the reference value for HF, the patient is determined to be sedated.

[0105] If the alertness determination unit 113 determines that the patient is alert, it calculates the ratio of the LF / HF value to the reference value of LF / HF. If it determines that the patient is calm, it calculates the ratio of the HF value to the reference value of HF. These ratios are determined as the alertness level. These ratios are expressed in %, and the value must be 100% or greater. The maximum value (%) in the vertical axis direction is 140% in Figure 12(B), but this can be set as appropriate. In this manner, the alertness determination unit 113 determines the alertness level.

[0106] Next, the method for determining the stress level on the horizontal axis of Figure 12(B) in the stress level determination unit 114 will be explained. As described above, the stress level determination unit 114 determines the stress level based on the input values ​​stored in the measurement result DB 123, or the LF / HF calculated when determining the arousal level.

[0107] As described above, if the input value is expressed as a number in the range of -4 to +4, the LF / HF value is converted to a number in the range of -4 to +4. If the input value is stored in the data record read from the measurement results DB123, the stored input value is used as the stress level. If the input value is not stored (if no input operation was performed by the user in step S303 of Figure 11), the converted LF / HF value is used as the stress level.

[0108] The stress level determination unit 114 corrects the stress level obtained as described above using a stress level reference value to determine the stress level. In this case, the reference value used is the reference value for the time period corresponding to the measurement date and time.

[0109] The stress level correction is performed by converting the stress level value (in the range of -4 to +4) to a value where the reference value is the central value (i.e., 0). For example, if the stress level is +2 and the reference value is -1, the stress level value is corrected to the negative side to +1. The stress level determination unit 114 determines the stress level by making the correction as described above.

[0110] Next, the method for determining emotions will be explained. The emotion determination unit 116 determines emotions using the two-dimensional plane coordinates shown in Figure 11(B) based on the arousal level and stress level determined as described above. The arousal level is a position on the vertical axis in either the arousal or calming direction, while the stress level, if negative, is a position in the unpleasant direction, and if positive, is a position in the pleasant direction.

[0111] In other words, the position along the vertical axis is determined based on the arousal level value at a given measurement date and time, and the position along the horizontal axis is determined based on the stress level value. The intersection of the vertical and horizontal axis positions represents the user's "emotions."

[0112] The emotion determination unit 116 determines the emotion to be "calm" if, for example, the intersection of the arousal level and stress level is at the position of pointer P1 in Figure 12(B). Since the position of pointer P1 is closest to the "calm" position in Russell's emotion model, the emotion is determined to be "calm". If the intersection of pointer P2 is at the position of pointer P2, since the position of pointer P2 is closest to the "drowsy" position in Russell's emotion model, the emotion is determined to be "drowsy". If the intersection of pointer P3 is at the position of pointer P3, since the position of pointer P3 is closest to the "vigilant" position in Russell's emotion model, the emotion is determined to be "vigilant".

[0113] After one or more measurements are taken and the level of arousal and stress levels are determined, the display unit 15 may display a two-dimensional plane as shown in Figure 12(B), and a pointer indicating the position of emotions on the two-dimensional plane may be displayed.

[0114] <Effects of the Embodiment> (1) According to the above-described embodiment, the following effects can be obtained. Using a mobile device equipped with a camera, such as a widely available smartphone, it is possible to determine the emotions at that moment, eliminating the need to separately prepare measuring instruments such as pulse wave meters or electroencephalographs.

[0115] (2) Measurement is possible by photographing the user's face, so there is no need to attach any components to the user's body. Therefore, measurements can be easily performed without restrictions on location or time. (3) As long as blood flow can be measured, it is sufficient to photograph a small area of ​​skin (typically around the eyes). Therefore, even if the user is wearing a mask and their mouth and nose are not exposed, for example, the user's emotions can be determined.

[0116] Because it can determine the user's emotions, it can be used to prevent drowsiness while driving. In this case, the mobile device should be fixed in a position where the user's face can be captured, such as on the dashboard. Furthermore, the information on the driver's emotions while driving and their location can be used to identify dangerous areas on the road. Alternatively, by fixing a mobile device in a position where the user's face can be captured, changes in the user's emotions while they are watching movies, listening to music, or other content can be recorded, making it useful for marketing research in movie and music production. For example, it is possible to research how emotions changed during different scenes or parts of the content.

[0117] <Variation> The above describes the embodiments, but the contents of the embodiments described above can be modified as follows. Furthermore, the following modifications may be combined with each other.

[0118] (1) In the embodiments described above, the information processing device 1 is assumed to have a processor 11 composed of a CPU, but the control means for controlling the information processing device 1 may have other configurations. That is, the information processing device 1 may have various processors other than a CPU, such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), programmable logic device, etc., as the processor 11.

[0119] (2) The operation of the processor in the above-described embodiment may not be performed by a single processor, but may also be performed by multiple processors located in physically separate locations working together. Furthermore, the order of the processor's operations is not limited to the order described in the embodiments described above, but may be changed as appropriate.

[0120] (3) In the above-described embodiment, in order to determine the stress level, pleasant / unpleasant information is obtained by user input, and if the user does not (or cannot) perform an input operation, pleasant / unpleasant information is obtained based on the captured image. However, pleasant / unpleasant information may be obtained by only one of these means.

[0121] (4) In the embodiments described above, emotions are determined by determining and plotting the positions on a two-dimensional plane corresponding to the determined arousal level and stress level. However, the determination of emotions is not limited to determining each plotted position. An index showing the change in the user's emotions over a predetermined period (such as the time spent driving or the period from the start to the end of watching a movie or other content) may be generated.

[0122] For example, an index may be generated based on the shape of the movement trajectory of a plotted position on a two-dimensional plane over time, as shown in Figure 12(B), or based on the average position of multiple plotted positions within a predetermined time (e.g., the centroid position).

[0123] In the above embodiment, the determined positions on a two-dimensional plane corresponding to the determined level of arousal (calmness) and stress level were determined and presented to the user by finally displaying the determined result (emotion) on the screen. However, it is not necessary to present the determined emotion as a position on a two-dimensional plane. For example, as the final information output, a two-dimensional plane display may not be performed, and only text information indicating the emotion may be displayed, or only the text may be read aloud using synthesized speech. Furthermore, the indicators on the vertical axis of the two-dimensional plane described above, arousal and calmness, are interrelated and can typically be considered as opposing concepts; therefore, these indicators may be expressed using only one of these terms. Alternatively, the indicators can be understood as representing both arousal and calmness. In short, in the above-described embodiment, the indicators should be determined based on vital data related to blood flow obtained from the images. Furthermore, the stress level (pleasure / displeasure level), which is the horizontal axis indicator in the two-dimensional plane described above, only needs to be different from the vertical axis indicator, and is typically related to the user's psychology or sensations, and may be expressed by words other than those mentioned above, such as "relaxation level." Furthermore, there is no need to introduce the concept of a two-dimensional plane. Each emotion can be defined by a combination of two indicators: arousal / calmness and stress level. Emotions can then be determined based on these two indicators obtained from measurements (and user input). In short, the present invention only requires that the following steps be performed: generate information about the user's pulse wave based on image data obtained by photographing the user's skin; determine the user's level of alertness based on the generated information; determine the user's stress level; and determine the user's emotions based on the determined level of alertness and stress level. [Explanation of Symbols]

[0124] 1... Information processing device, 11... Processor, 111... Acquisition unit, 112... Authentication unit, 113... Arousal level determination unit, 114... Stress level determination unit, 115... Reference value determination unit, 116... Emotion determination unit, 117... Display control unit, 12... Memory, 121... User DB, 122... Video DB, 1221... User ID list, 1222... Video data table, 123... Measurement result DB, 1231... User ID list, 1232... Measurement data table, 13... Communication unit, 14... Operation unit, 15... Display unit, 16... Camera, R0... Display area, R1... Termination instruction area, R2... Explanation area, R3... Input area, R4... Measurement area, R5... Status notification area, P0~P3... Pointers.

Claims

1. On the computer, The steps include generating information about the user's pulse wave based on image data obtained by photographing the user's skin, and determining the user's level of alertness based on the generated information, The steps include determining the user's stress level, The steps include determining the user's emotion based on the position on the two-dimensional plane corresponding to the determined arousal level and stress level, where each emotion is associated with a specific position on the two-dimensional plane defined by the axis representing the arousal level and the axis representing the stress level, and A program to execute, In the step of determining the level of arousal, The level of arousal is determined based on at least one of the high-frequency components in the pulse wave fluctuation spectrum that are higher than the first threshold frequency, and the low-frequency components in the pulse wave fluctuation spectrum that are lower than the second threshold frequency. The degree of arousal relative to the reference value is calculated by dividing the low-frequency component by the high-frequency component, while the degree of sedation relative to the reference value is determined based on the high-frequency component. program.

2. A computer, The steps include generating information about the user's pulse wave based on image data obtained by photographing the user's skin, and determining the user's level of alertness based on the generated information, The steps include determining the user's stress level, The steps include determining the user's emotion based on the position on the two-dimensional plane corresponding to the determined arousal level and stress level, where each emotion is associated with a specific position on the two-dimensional plane defined by the axis representing the arousal level and the axis representing the stress level, and A program to execute, The steps include determining a first reference value related to the user's level of alertness for each time period based on multiple image data obtained by photographing the user's skin at multiple time periods, The steps include determining a second reference value for the user's stress level for each time period based on the stress level obtained at multiple time periods, and Let's execute this further, In the step of determining the level of arousal, the level of arousal is corrected using the first reference value, and in the step of determining the level of stress, the level of stress is corrected using the second reference value. program.

3. The correspondence between position on the aforementioned two-dimensional plane and emotion is based on Russell's annular model. The program according to claim 1 or 2.

4. In the step of determining the stress level, the stress level is determined based on the pulse rate or heart rate determined from the image data. The program according to claim 1.

5. In the step of determining the stress level, input of the stress level is received from the user. The program according to claim 1 or 2.

6. In the step of determining the stress level, input of the stress level is received from the user, while if no input is received from the user, the stress level is determined based on the pulse rate or heart rate determined from the image data. The program according to claim 1 or 2.

7. In the step of determining the user's emotions, an index is generated that shows the change in the user's emotions within a predetermined period. The program according to claim 1 or 2.

8. A means for photographing the user's skin, A means for determining the level of alertness of the user, which generates information about the user's pulse wave based on the captured image and determines the user's level of alertness based on the generated information. A means for obtaining the user's stress level, A means for determining the user's emotion based on the position on the two-dimensional plane corresponding to the identified stress level and arousal level, where each emotion is associated with a specific position on the two-dimensional plane defined by the axis indicating the level of arousal and the axis indicating the level of stress, and It has, The aforementioned means for determining the level of arousal is, The level of arousal is determined based on at least one of the high-frequency components in the pulse wave fluctuation spectrum that are higher than the first threshold frequency, and the low-frequency components in the pulse wave fluctuation spectrum that are lower than the second threshold frequency. The degree of arousal relative to the reference value is calculated by dividing the low-frequency component by the high-frequency component, while the degree of sedation relative to the reference value is determined based on the high-frequency component. Information processing device.

9. A means for photographing the user's skin, A means for determining the level of alertness of the user, which generates information about the user's pulse wave based on the captured image and determines the user's level of alertness based on the generated information. A stress level determination means for determining the user's stress level, A means for determining the user's emotion based on the position on a two-dimensional plane corresponding to the identified stress level and arousal level, where each emotion is associated with a specific position on the two-dimensional plane defined by the axis representing the level of arousal and the axis representing the level of stress, A means for determining a first reference value relating to the user's level of alertness for each time period, based on a plurality of image data obtained by photographing the user's skin at multiple time periods, A means for determining a second reference value relating to the user's stress level for each time period, based on stress levels acquired at multiple time periods. It has, The arousal level determination means corrects the arousal level using the first reference value, The stress level determination means corrects the stress level using the second reference value. Information processing device.

Images