Information processing systems, information processing methods, and programs

Abstract

To provide a technique for more easily and accurately estimating a state of a person.SOLUTION: The state estimating unit 84 acquires information concerning an estimation result of the heart rate estimated by the heart-rate acquiring unit 121 in a predetermined period. The state estimation unit 84 acquires information on the value of the index acquired by the index acquisition unit 142 in a predetermined period. The state estimation unit 84 standardizes each of the acquired information on the estimation result of the heart rate and the acquired information on the value of the index in accordance with a predetermined standard. The state estimation unit 84 determines whether the value obtained by standardizing each of the information about the estimation result of the heart rate and the information about the value of the index satisfies the condition of the stress state. The state estimation unit 84 determines whether or not a value obtained by standardizing each of the information regarding the estimation result of the heart rate and the information regarding the value of the index satisfies the condition of the flow state.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This disclosure relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Conventionally, in people's lives, a person's physical and mental state is not constant but changes over time. For example, during work, various states such as stress, concentration, and drowsiness transition, and knowing one's own physical state is important for improving work efficiency, maintaining health, and preventing accidents. Also, for example, when driving a car, it is important for the driver to always be awake and concentrated. In this regard, attempts have been made to detect a person's drowsiness based on the heart rate for the purpose of preventing drowsiness during driving (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique described in Patent Document 1 above can only detect the presence or absence of drowsiness based on the heart rate, which is insufficient. This is the same for other conventional techniques.

[0005] This disclosure has been made in view of such a situation, and provides a technique for more simply and accurately estimating a person's state.

Means for Solving the Problems

[0006] To achieve the above object, an information processing system according to one aspect of this disclosure is an information processing system that can be used to acquire information about a target person, Image information acquisition means for acquiring image information relating to an image including the skin of the subject, A means for acquiring brightness information in the skin of the subject included in the image, A first estimation means for estimating the heart rate of the subject based on the brightness information, A second estimation means for estimating the value of an index related to the subject's heart rate based on the brightness information, It is an information processing system equipped with [the following features].

[0007] An information processing method and program according to one aspect of this disclosure are also provided as an information processing method or program corresponding to an information processing system according to one aspect of this disclosure. [Effects of the Invention]

[0008] This disclosure provides a technology for estimating a person's state more simply and accurately. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of an information processing system according to one embodiment of the present disclosure. [Figure 2] This figure shows an example of the hardware configuration of an analysis terminal constituting an information processing system according to one embodiment of the present disclosure. [Figure 3] This figure shows an example of the functional configuration of an analysis terminal and server of an information processing system according to one embodiment of the present disclosure. [Figure 4] This figure shows an example of the functional configuration of the PPG processing unit, in particular, among the functional configurations of the analysis terminal of an information processing system according to one embodiment of the present disclosure. [Figure 5] This figure shows an example of the functional configuration of the heart rate estimation unit, in particular, among the functional configurations of the analysis terminal of an information processing system according to one embodiment of the present disclosure. [Figure 6] This figure shows an example of the functional configuration of the indicator estimation unit, in particular, among the functional configurations of the analysis terminal of an information processing system according to one embodiment of the present disclosure. [Figure 7]This figure shows a typical waveform example of RGB information acquired by an information processing system according to one embodiment of the present disclosure. [Figure 8] This figure shows a typical waveform example of PPG information acquired by an information processing system according to one embodiment of the present disclosure. [Figure 9] This figure shows a typical waveform example of PSD information acquired by an information processing system according to one embodiment of the present disclosure. [Figure 10] This figure shows an example of an image of the progression of heart rate estimated by an information processing system according to one embodiment of the present disclosure. [Figure 11] This figure shows an example of how an information processing system according to one embodiment of the present disclosure detects the peak position of each waveform of a PPG signal from PPG information. [Figure 12] This figure shows a typical example of PPI information acquired by an information processing system according to one embodiment of the present disclosure. [Figure 13] This figure shows an example of the trend of RMSSD values, which are an index of heart rate variability, among the index information acquired by the information processing system according to one embodiment of this disclosure. [Figure 14] This figure illustrates an example of a specific method for detecting the presence or absence of drowsiness in an analysis terminal constituting an information processing system according to one embodiment of the present disclosure. [Figure 15] This figure illustrates an example of a specific method for detecting the presence or absence of continuous drowsiness by an analysis terminal constituting an information processing system according to one embodiment of the present disclosure. [Figure 16] This figure shows an example of an image displayed on an analysis terminal that constitutes an information processing system according to one embodiment of the present disclosure. [Figure 17] This figure shows an example of the flow of drowsiness estimation processing performed by an analysis terminal that constitutes an information processing system according to one embodiment of the present disclosure. [Figure 18] This figure shows an example of the flow of state estimation processing performed by an analysis terminal that constitutes an information processing system according to one embodiment of the present disclosure. [Figure 19]It is a diagram showing an example of an image displayed on an analysis terminal constituting an information processing system according to an embodiment of the present disclosure, and shows an example different from the example of FIG. 16. [Figure 20] It is a diagram for explaining an example of a method of using an estimation result by an information processing system according to an embodiment of the present disclosure. [Figure 21] It is a diagram for explaining an example of a method of using an estimation result by an information processing system according to an embodiment of the present disclosure, and shows an example different from the example of FIG. 20. [Figure 22] It is a diagram for explaining an example of a method of using an estimation result by an information processing system according to an embodiment of the present disclosure, and shows an example different from the examples of FIGS. 20 and 21.

Mode for Carrying Out the Invention

[0010] [Embodiment] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] <Explanation of Outline> First, prior to the specific explanation using FIG. 1, the first service and the second service to which an information processing system (hereinafter referred to as "this system") according to an embodiment of the present disclosure is applicable will be described. In the first service, this system estimates the presence or absence of drowsiness of the subject, the state of concentration of the subject, and the presence or absence of stress based on various biological information (heart rate, RMSSD, etc.) obtained from image information including the skin of the subject. In the second service, this system presents to the subject comprehensive evaluations regarding the drowsiness, concentration state, stress state, etc. of the subject during that period, time-series information of various indicators, etc. based on various biological information of the subject aggregated over a certain period. By using such services in combination, the subject can easily grasp and manage the presence or absence of their own drowsiness and various states of themselves.

[0012] Here, the definitions of each index that can be included in each biological information will be explained. Heart rate is the number of times the heart beats per minute. In this embodiment, the subject's heart rate and heart rate variability index are estimated using rPPG (remoto Photo Plethysmography). rPPG is an optical technique that evaluates changes over time in the reflectance or transmittance of light in a region or volume of interest. In other words, rPPG is a simple and low-cost technique for acquiring biological information that can be correlated with physiological phenomena in the body by detecting changes in blood flow (volume) within the capillaries of cell tissue. The method for obtaining rPPG used by this system is optional, but please refer to the method described in the following document, for example. [Non-Patent Literature 1] Wenjin Wang, Bert den Brinker, Sander Stuijk, and Gerard de Haan, "Algorithmic Principles of Remote-PPG," IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING (July, 2017) Furthermore, in the following explanation, PPG signal (or rPPG signal) refers to various signal information that can be used to detect a person's state, obtained using methods such as PPG (or rPPG).

[0013] RMSSD (Root Mean Square of Successive Differences) is typically defined as the square root of the average of the squared differences between adjacent RRIs (RR Intervals), and can be specifically expressed by the following formula. RMSSD is thought to have a strong correlation with parasympathetic nervous system activity and is used as an indicator of heart rate variability. However, for PPG or rPPG signals, PPI (peak-to-peak interval), which corresponds to RRI, may also be used.

number

[0014] Figure 1 is a block diagram showing the configuration of an information processing system according to one embodiment of the present disclosure. As shown in Figure 1, this system comprises an analysis device 1 and a server 2. Here, as shown in Figure 1, the analysis device 1 comprises an analysis terminal 11 and a camera 12. The analysis terminal 11 analyzes the subject image information described later and performs various processes to estimate the subject's heart rate, concentration level, or stress level. Camera 12 acquires reflected light reflected through a part of the subject's body (particularly the skin, hereinafter referred to as the "measurement site"), and generates image information (hereinafter referred to as "subject image information") based on the acquired reflected light. Furthermore, the analysis terminal 11 of the analysis device 1 and the server 2 are connected via a predetermined network N, such as a LAN (Local Area Network) or the Internet. Note that the network N does not necessarily have to be a LAN or the Internet; communication may be performed by any method, such as Bluetooth®.

[0015] Figure 2 shows an example of the hardware configuration of the analysis terminal 11 in the information processing system shown in Figure 1. The analysis terminal 11 is composed of a general-purpose PC (Personal Computer) or the like. As shown in Figure 2, the analysis terminal 11 includes a control unit 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a bus 24, an input / output interface 25, an output unit 26, an input unit 27, a storage unit 28, a communication unit 29, and a drive 30.

[0016] The control unit 21 is composed of a microcomputer including a CPU, GPU, and semiconductor memory, and executes various processes according to the program recorded in the ROM 22 or the program loaded from the storage unit 28 into the RAM 23. RAM23 also stores information necessary for the control unit 21 to perform various processes.

[0017] The control unit 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input / output interface 25 is also connected to this bus 24. An output unit 26, an input unit 27, a storage unit 28, a communication unit 29, and a drive 30 are connected to the input / output interface 25.

[0018] The output unit 26 consists of various liquid crystal displays and outputs various types of information.

[0019] The input unit 27 consists of various hardware components and inputs various types of information.

[0020] The storage unit 28 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various types of data. For example, it stores various programs as well as various types of data including databases.

[0021] The communication unit 29 controls communication with other devices via the network N, including the Internet.

[0022] A drive 30 is provided as needed. A removable media 41, consisting of a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately mounted on the drive 30. Programs read from the removable media 41 by the drive 30 are installed in the storage unit 28 as needed. The removable media 41 can also store various data stored in the storage unit 28, just like the storage unit 28.

[0023] Here, we will describe an example of the hardware configuration of a camera 12 that can be connected to the analysis terminal 11. Camera 12 consists of, for example, a general-purpose webcam or a 3D depth camera. Camera 12 is used to capture images including the subject's skin. The hardware configuration of server 2 can be basically the same as that of analysis terminal 11 described above, so we will omit the explanation here.

[0024] Figure 3 shows an example of the functional configuration of the analysis terminal 11 and server 2 of an information processing system according to one embodiment of the present disclosure. As shown in Figure 3, the control unit 21 of the analysis terminal 11 functions as an image acquisition unit 80, a PPG processing unit 81, a heart rate estimation unit 82, an index estimation unit 83, a state estimation unit 84, a drowsiness estimation unit 85, a data transmission / reception unit 86, an overall evaluation unit 87, and a result presentation unit 88 by executing various programs.

[0025] The image acquisition unit 80 acquires image information of the subject captured by the camera 12, i.e., subject image information. Here, it is desirable that the subject image information includes image information of the subject's skin. Note that the skin image referred to here does not have to be from any part of the body; for example, it is sufficient if it includes images of the skin of any part such as the eyes, nose, or mouth.

[0026] The PPG processing unit 81 performs various processes to acquire a PPG signal based on the subject image acquired by the image acquisition unit 80. Specifically, as shown in Figure 4, the PPG processing unit 81 is equipped with a skin detection unit 100, an RGB acquisition unit 101, a PPG signal acquisition unit 102, and a correction unit 103.

[0027] The skin detection unit 100 identifies areas of the subject's image information where the subject's skin is exposed.

[0028] The RGB acquisition unit 101 acquires information regarding the brightness of each color in each region identified by the skin detection unit 100 (hereinafter referred to as "RGB information"). Specifically, the RGB acquisition unit 101 acquires, for example, the average value of the brightness of each color in the region identified by the skin detection unit 100 as RGB information. Figure 7 shows an example of a typical waveform representing the RGB information acquired by this system. Furthermore, conventional heart rate estimation techniques using image recognition and the like often involve acquiring a facial image, then extracting regions of various parts of the face (eyes, nose, cheeks, etc.), and estimating heart rate based on the characteristics of each region. In contrast, this system does not rely on the characteristics of each part, so estimation is possible even with only a part of the face or body, as described later. Moreover, it does not necessarily require a facial image, allowing for greater flexibility in the circumstances and environment in which images are captured.

[0029] The PPG signal acquisition unit 102 acquires the PPG signal from the time-series information of the RGB information acquired by the RGB acquisition unit 101. Specifically, the PPG signal acquisition unit 102 extracts the PPG signal from the luminance information using an arbitrary algorithm and applies a filter to the PPG signal to acquire time-series information of the PPG signal during the measurement period (hereinafter referred to as "PPG information"). Figure 8 shows an example of a typical waveform representing the PPG information acquired by this system. Furthermore, while this system can employ any method for acquiring PPG signals and the algorithm for doing so, it can, for example, employ the method described in Non-Patent Document 1 mentioned above.

[0030] The correction unit 103 performs RGB information correction processing, for example, by linear interpolation or deletion of sampled data, when some image information is lost due to failure in capturing the subject or a decrease in FPS (Frames Per Second), or when a signal sampled at a specific frequency is resampled.

[0031] The heart rate estimation unit 82 performs various processes to estimate the subject's heart rate based on the brightness information. Specifically, as shown in Figure 5, the heart rate estimation unit 82 is provided with a power spectrum acquisition unit 120 and a heart rate acquisition unit 121.

[0032] The power spectrum acquisition unit 120 calculates and acquires the power spectral density (hereinafter referred to as "PSD information") of the PPG information acquired by the PPG signal acquisition unit 102. Power spectral density is a representation of the strength of each signal, divided into fixed frequency bands, and expressed as a function of frequency. Figure 9 shows an example of a typical waveform representing the PSD information acquired by this system.

[0033] The heart rate acquisition unit 121 estimates the subject's heart rate based on the PSD information acquired by the power spectrum acquisition unit 120. While this system can employ any existing method for calculating the power spectrum and estimating the heart rate, please refer to the method described in the following literature, for example. [Non-patent document 3] Wim Verkruysse, Lars O Svaasand and J Stuart Nelson “Remote plethysmographic imaging using ambient light” OPTICS EXPRESS (December 2008) Figure 10 shows an example of the estimated changes in heart rate calculated by this system.

[0034] The index estimation unit 83 estimates the value of an index related to the subject's heart rate based on brightness information. Specifically, as shown in Figure 5, the index estimation unit 83 is equipped with a peak detection unit 140, a PPI acquisition unit 141, and an index acquisition unit 142.

[0035] The peak detection unit 140 detects the peak position of the waveform in the PPG information acquired by the PPG signal acquisition unit 102. Figure 11 shows an example of how this system detects the peak position of each waveform of the PPG signal from the PPG information.

[0036] The PPI acquisition unit 141 acquires time-series information of the PPI (hereinafter referred to as "PPI information") that indicates the interval between each peak position of the PPG information (PPG signal) detected by the peak detection unit 140. Figure 12 shows a typical image of the PPI information acquired by this system.

[0037] The index acquisition unit 142 calculates values ​​for various heart rate variability indices based on the PPI information in the subject's PPG information acquired by the PPI acquisition unit 141, and acquires this information (hereinafter referred to as "index information"). In this embodiment, the system calculates values ​​such as RMSSD from among the aforementioned heart rate variability indices as heart rate variability indices. Figure 13 is a diagram showing an example of the trend of the RMSSD value, which is a heart rate variability indice among the index information acquired by the system.

[0038] The state estimation unit 84 estimates the subject's state of concentration or stress based on the heart rate estimated by the first estimation means and the index value estimated by the second estimation means. Specifically, the state estimation unit 84 estimates the subject's state of concentration or stress based on the heart rate estimated by the heart rate acquisition unit 121 and the index information (particularly the RMSSD value) acquired by the index acquisition unit 142. The state estimation unit 84 standardizes the heart rate estimated by the heart rate acquisition unit 121 and the index information (especially the RMSSD value) acquired by the index acquisition unit 142 according to predetermined criteria. While the system may use any method to standardize the values ​​of various indicators, for example, it can be standardized using the following mathematical formula. This system can also standardize other values, such as heart rate, using a similar method.

number

[0039] The drowsiness estimation unit 85 estimates the subject's drowsiness state based on the heart rate estimated by the first estimation means. That is, the drowsiness estimation unit 85 estimates whether or not the subject is drowsy based on the subject's heart rate acquired by the heart rate acquisition unit 121.

[0040] Specifically, with reference to Figure 14, an example of the sleepiness estimation method in the sleepiness estimation unit 85 will be explained. Figure 14 is a diagram illustrating an example of a specific method by which the analysis terminal constituting this system detects the presence or absence of sleepiness. In the example in Figure 14, the heart rate estimated from the acquired PPG signal is shown in units of one minute. Note that the value in units of one minute is calculated as the average of the values ​​estimated every second. Specifically, in the example in Figure 14, it is shown that the heart rate changes every minute as follows: "66.3", "65.4", "62.1", "65.8", "64.2", "63.9", "62.5", "66.1", "67.0", "67.3", and "66.8". Looking at the top of each value, you'll see two indicators: "Baseline Interval" and "Prediction Interval." In other words, the "baseline interval" refers to the subject's heart rate prior to drowsiness detection. On the other hand, the "prediction interval" refers to the heart rate at the time of drowsiness detection and the time immediately preceding it. Physiological phenomena are greatly influenced by individual differences, making it difficult to perform accurate detection simply by setting thresholds. Therefore, by performing drowsiness detection based on time-series fluctuations, as shown in Figure 14, more accurate detection becomes possible.

[0041] Specifically, in the example shown in Figure 14, the difference between the mean (or median) heart rate in the baseline interval and the mean (or median) heart rate in the prediction interval is detected. If the mean heart rate in the prediction interval falls below the mean heart rate in the baseline interval by a predetermined threshold (for example, a decrease of 1.0 point or more in heart rate), the system determines that the subject is drowsy. Conversely, if drowsiness has already been detected, the system may determine that the drowsiness has disappeared if the most recent heart rate measurements show a continuous increase.

[0042] Furthermore, this system may estimate the subject's drowsiness using a standardized value, for example, (prediction interval value - baseline interval mean) / baseline interval standard deviation. This allows the system to estimate the presence or absence of drowsiness in the subject with greater accuracy. Furthermore, for example, if there is a time difference between the prediction interval and the target time for estimating drowsiness, that is, if the prediction interval is an interval prior to the target time, the system may determine that the subject is drowsy if, for example, the average value of the prediction interval is lower than a predetermined threshold compared to the average value of the baseline interval, and the heart rate value at the estimated target time is also lower than a predetermined threshold compared to the average value of the baseline interval. This allows the system to eliminate the possibility that the subject may be awake at the estimated target time, thereby enabling a more accurate determination of whether or not the subject is drowsy.

[0043] Furthermore, this system can also determine whether or not a subject is experiencing continuous drowsiness using the method shown in Figure 15. Figure 15 is a diagram illustrating an example of a specific method by which the analysis terminal constituting this system detects the presence or absence of continuous drowsiness. In the example in Figure 15, the minute-by-minute changes are shown for cases where the subject is determined to be drowsy and cases where they are determined not to be drowsy, using the method shown in Figure 14. In the example in Figure 15, t=P is set as the target time for estimation. In the example in Figure 15, at time t=P, drowsiness was detected for 4 consecutive minutes within the 5-minute period (the interval that serves as the judgment interval in the example in Figure 15), which includes 2 minutes before and after t=P. This system may also determine that the subject is drowsy only if drowsiness is detected continuously for a predetermined period of time or longer, by organizing the presence or absence of drowsiness over time in this manner. In this case as well, the system may also check whether the subject is awake or not based on their current heart rate, as in the case of Figure 14.

[0044] The data transmission unit 86 transmits various information and estimation results acquired by the analysis terminal 11 to the server 2.

[0045] The comprehensive evaluation unit 87 obtains the estimated results of the subject's concentration or stress state estimated by the third estimation means and the estimated results of the subject's sleepiness state estimated by the fourth estimation means for a predetermined period, and calculates the subject's comprehensive evaluation index for that period according to the frequency of each estimation result. That is, the comprehensive evaluation unit 87 calculates a comprehensive evaluation of the subject's state based on the estimated results of the subject's concentration or stress state by the state estimation unit 84 and the estimated results of the presence or absence of sleepiness by the sleepiness estimation unit 85 for a predetermined period. Note that this system can employ any method to calculate the comprehensive evaluation, but for example, it can be calculated using the method shown in the following formula.

number

[0046] The result presentation unit 88 presents the subject with the estimated results regarding the subject's state of concentration or stress, estimated by the third estimation means, and the estimated results regarding the subject's state of sleepiness, estimated by the fourth estimation means. Specifically, the result presentation unit 88 performs various processes to present the subject with the estimated results regarding the subject's state of concentration or stress, estimated by the state estimation unit 84, and the estimated results regarding the presence or absence of sleepiness, estimated by the sleepiness estimation unit 85, etc. Here, Figure 16 shows an example of an image displayed on an analysis terminal constituting an information processing system according to one embodiment of the present disclosure. This system can present various estimation results to the subject by displaying an image like the one shown in Figure 16 on the output unit 26, etc. Although not shown in the example of Figure 16, this system may also present comments regarding the measurement environment to the subject, such as "The camera is too far away, please move closer."

[0047] Next, we will describe an example of the functional configuration of Server 2. As shown in Figure 3, the control unit 210 of server 2 functions as a data transmission / reception unit 240 by executing various programs. In addition, an analysis results DB 300 is provided in one area of ​​the storage unit 220 of server 2.

[0048] The data transmission / reception unit 240 acquires various information, such as various data and estimation results, transmitted from each subject's analysis terminal 11 via the communication unit 200. The data transmission / reception unit 240 also stores the acquired various data, such as various data and estimation results, in the analysis results DB 300.

[0049] Figure 17 shows an example of the flow of drowsiness estimation processing performed by an analysis terminal that constitutes an information processing system according to one embodiment of the present disclosure. In step S1, the sleepiness estimation unit 85 acquires information regarding the estimated heart rate calculated by the heart rate acquisition unit 121 over a predetermined period.

[0050] In step S2, the sleepiness estimation unit 85 determines whether the trend of the estimated heart rate of the subject over a predetermined period satisfies the condition of sleepiness. If the trend of the estimated heart rate results meets the condition for drowsiness, step S2 is determined to be YES and the process proceeds to step S4. In contrast, if the trend of the estimated heart rate does not meet the conditions for drowsiness, step S2 is determined to be NO and the process proceeds to step S3.

[0051] In step S3, the sleepiness estimation unit 85 determines that the subject is not sleepy if the trend of the estimated heart rate does not meet the conditions for sleepiness.

[0052] In step S4, the sleepiness estimation unit 85 determines that the subject is sleepy if the trend of the estimated heart rate results satisfies the conditions for sleepiness. This completes the sleepiness estimation process on the analysis terminal 11.

[0053] Figure 18 shows an example of the flow of state estimation processing performed by an analysis terminal constituting an information processing system according to one embodiment of the present disclosure. In step S21, the state estimation unit 84 acquires information regarding the estimated heart rate calculated by the heart rate acquisition unit 121 over a predetermined period.

[0054] In step S22, the state estimation unit 84 acquires information regarding the values ​​of the indicators (particularly the RMSSD values) acquired by the indicator acquisition unit 142 over a predetermined period.

[0055] In step S23, the state estimation unit 84 standardizes the acquired information regarding the estimated heart rate and the index values ​​(especially the RMSSD values) according to predetermined criteria.

[0056] In step S24, the state estimation unit 84 determines whether the standardized values ​​of the information regarding the estimated heart rate and the index values ​​(especially the RMSSD values) satisfy the conditions for a stressed state. Specifically, in step S24, the state estimation unit 84 determines that the subject is in a stressed state if the standardized value of the heart rate is greater than a predetermined threshold (e.g., 1.0) and the standardized value of the index (especially the RMSSD) is less than a predetermined threshold (e.g., -0.5). If the standardized heart rate and the standardized index (especially RMSSD) meet the conditions for a stressed state, step S24 is determined to be YES, and the process proceeds to step S25. In contrast, if the standardized heart rate and the standardized index (especially RMSSD) do not meet the conditions for a stressed state, step S24 is determined to be NO, and the process proceeds to step S26.

[0057] In step S25, the state estimation unit 84 determines that the subject is in a stressed state if the standardized value of the heart rate and the standardized value of the index (especially RMSSD) satisfy the conditions for a stressed state.

[0058] In step S26, the state estimation unit 84 determines whether the standardized values ​​of the information regarding the estimated heart rate and the index values ​​(especially the RMSSD values) satisfy the conditions for a flow state. Specifically, in step S26, the state estimation unit 84 determines that the subject is in a flow state if the standardized heart rate value is within a predetermined threshold range (e.g., -0.3 to 0.3) and the standardized index value (especially the RMSSD) is greater than a predetermined threshold (e.g., 1.15). If the standardized heart rate and the standardized index (especially RMSSD) satisfy the conditions for a flow state, step S26 is determined to be YES, and the process proceeds to step S27. In contrast, if the standardized heart rate and the standardized index (especially RMSSD) do not meet the conditions for a flow state, step S26 is determined to be NO, and the process proceeds to step S28.

[0059] In step S27, the state estimation unit 84 determines that the subject is in a flow state if the standardized value of the heart rate and the standardized value of the index (especially RMSSD) satisfy the conditions for a flow state.

[0060] In step S28, the state estimation unit 84 determines that the subject is in a normal state if the standardized value of the heart rate and the standardized value of the index (especially RMSSD) do not satisfy either the stress state condition or the flow state condition. This completes the state estimation process of the analysis terminal 11.

[0061] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-described embodiment, and any modifications, improvements, etc., to the extent that they can achieve the purpose of the present disclosure are included in the present disclosure.

[0062] Although not explained in the above embodiment, the overall evaluation calculated by this system can be presented to the subject in various formats. Specifically, for example, Figure 19 shows the time-series changes in the overall evaluation for each predetermined day (for example, every week in the example of Figure 19). By understanding these changes in the overall evaluation, the subject can more efficiently control their own stress levels and concentration levels.

[0063] Furthermore, although not explained in the embodiments described above, we will briefly supplement the features and advantages of the system described herein. This system is a technology that can acquire the subject's PPG signal non-contactually from captured video images and estimate the subject's heart rate variability and heart rate variability index (RMSSD, etc.). Furthermore, this system can estimate the subject's drowsiness, stress level, or concentration level from the estimated heart rate variability and heart rate variability index (RMSSD, etc.). Here are some of the features of this system compared to conventional technologies: (1) This system can simultaneously determine multiple states of a subject, such as stress, concentration (flow state), normal state (neither stressed nor flow state), and presence or absence of drowsiness. (2) This system uses two indicators, heart rate and heart rate variability index (at least RMSSD), to estimate the state. (3) This system can calculate an overall evaluation index for a predetermined period of time (for example, every day) based on the results of the estimation of the subject's state over time and the results of sleepiness detection. The features described in (1) to (3) above are not found in many prior art, including the aforementioned Patent Document 1. (4) Furthermore, the threshold values ​​and overall evaluation calculation methods for state determination in this system are derived from numerous experimental results by the inventors of this disclosure.

[0064] Furthermore, although not explained in the above embodiment, an example of the correction (interpolation) method in this system is briefly shown below. This system may, for example, interpolate RGB information by linear interpolation if an error occurs in acquiring the subject's image and that error is corrected within a predetermined time (e.g., 3 seconds). On the other hand, if an error in acquiring the subject's image persists for a predetermined time (e.g., 3 seconds) or longer, the system may reset the program and acquire the subject's image again. By performing such corrections, the system can improve the accuracy of various estimations.

[0065] Furthermore, although not explained in the above embodiment, this system simply determines whether or not there is stress and whether or not the person is focused (in a flow state), but it is not limited to that. This system may also determine the subject's state in stages, for example, as having some stress, stress, high stress, and being somewhat focused.

[0066] Furthermore, although the above-described embodiment has been explained as estimating states related to concentration or stress, the system is not limited to this. The system may, for example, estimate only one of the states of concentration or stress, or it may also estimate other states of the subject.

[0067] Furthermore, although the above-described embodiment explains that this system acquires RMSSD as an indicator of heart rate variability, it is not limited to this. This system may acquire other indicators, such as SDNN, as an indicator of heart rate variability. SDNN represents the standard deviation of the time interval (heartbeat interval, RR interval) between two consecutive heartbeats within a given period. SDNN can be derived by calculating the standard deviation of the RR interval (time interval from one ventricular excitation to another) within a given period.

[0068] Furthermore, this system may acquire other indicators, not limited to heart rate variability, and utilize them for various estimations. Specifically, this system may acquire indicators such as EAR (Eye Aspect Ratio), which shows the size of the eye area, and utilize them for various estimations. EAR is the ratio of the horizontal to vertical length of an eye. For example, it can be calculated by detecting the eye region from an image of the eye and calculating the ratio of its respective lengths.

[0069] Furthermore, including the examples of the embodiments described above, the indicators adopted by this system do not necessarily have to be the same as the various indicators commonly used, and may include some changes or modifications to the derivation method and the meaning of the detailed indicators.

[0070] Furthermore, while the heart rate was estimated using the PPG signal value in the above embodiment, the method is not limited to this. That is, the heart rate may be estimated using various methods, such as other algorithms, PPG signals of different wavelengths, or the average value of those values.

[0071] Furthermore, although the above embodiment describes the system as acquiring a PPG signal from RGB information, it is not limited to this. The system may acquire a PPG signal, etc., from other information related to brightness, not just color information such as RGB information.

[0072] Furthermore, while specific methods for detecting drowsiness have been described in the embodiments described above (particularly in the embodiments shown in Figures 14 and 15), these methods are merely illustrative and not limiting. The system may, for example, use other different thresholds or criteria to detect the presence or absence of drowsiness. Also, the system may, for example, not utilize the standardization method described above, or may use a standardization method different from the formula described above.

[0073] Furthermore, although the above-described embodiment explained that the camera 12 does not include a control unit for processing various programs, etc., it is not limited to this. That is, the camera 12 may be configured to include a control unit, a storage unit, etc., which are not shown, as needed. In this case, for example, all or part of the various processes performed by the analysis terminal 11, etc., may be performed by the camera 12.

[0074] Furthermore, although detailed explanations were omitted in the embodiments described above, this system has other advantages in addition to those mentioned above, such as the following: (1) This system processes PPG signals in parallel from different exposed areas of skin (e.g., forehead, nose, right cheek, left cheek), detects the most stable signal, and estimates heart rate and other parameters based on that signal. In other words, this system can detect stable signals from PPG signals acquired from multiple parts of the body and estimate heart rate, thus enabling even more accurate estimations. In this respect, most conventional drowsiness detection systems require imaging of specific body parts such as the face, and do not perform parallel processing (even when using PPG signals), thus differing from this system. (2) Furthermore, because parallel processing is performed, even if unexpected behavior of the subject occurs while the PPG signal is being detected (for example, rubbing eyes, continuously looking in a different direction, putting on sunglasses, etc.), it is possible to achieve signal processing that eliminates the influence of such behavior to a certain extent. (3) Furthermore, if it is not possible to successfully acquire a signal from the facial surface, a backup function can be provided to acquire PPG signals from the skin of other parts of the body (such as the hands or arms). (4) Furthermore, by performing the parallel processing described above, it is believed that this system can achieve stable signal acquisition and heart rate estimation compared to conventional sleepiness detection systems, even under different lighting conditions (differences due to weather or the position of the sun) or in situations where it is difficult for conventional sleepiness detection systems to estimate heart rate, such as when a woman is applying makeup.

[0075] Furthermore, although not described in the above embodiment, this system can implement additional functions such as the following. (1) Face usage selection function As mentioned above, this system does not necessarily require imaging the subject's face to detect drowsiness or estimate their state. The face selection function allows users to switch between two modes: one that images the subject's face and detects drowsiness or estimates their state if possible, and another that freely images other body parts and detects drowsiness or estimates their state if the face cannot be detected. (2) Face detection frequency adjustment function The face detection frequency adjustment function uses sensors to detect head movements and adjusts the face detection frequency according to the detected head movements. Specifically, for example, the face detection frequency can be increased when the head movement is large, and decreased when the head movement is small. (3) Stable continuation function The stable continuation function is used when the images acquired from the subject are stable. This system can perform various estimations stably by proceeding with processing only when the subject's head movement and face detection are stable.

[0076] Furthermore, while the above-described embodiments (particularly the embodiment in Figure 14) illustrate how the system detects drowsiness in a subject, it is not limited to these embodiments. The method of drowsiness detection described with reference to Figure 14 is illustrative, and the system may detect drowsiness in a subject using any method based on the subject's heart rate changes.

[0077] Here, we will further explain an example of how to utilize the results of various estimations performed by this system. Specifically, referring to Figures 20 to 22, we will explain a method for classifying viewers by type using the estimation results of the system's assessment of the subjects' concentration, stress, and drowsiness states. First, in the example in Figure 20, the estimated state of the subject's concentration (flow state) is displayed as an example of time-series information. In the example in Figure 20, the vertical axis shows the "flow value," which is the estimated state of the subject's concentration, and the horizontal axis shows "time." Here, the flow value refers to the sum of values ​​obtained by quantifying the estimated state of concentration of the subject every second using a binary value of 1 (concentrated) or 0 (not concentrated), over a one-hour period. In other words, the higher the "flow value" of the subject, the longer the period of time they were in a state of concentration. For example, in the example in Figure 20, the hour from 2 PM to 3 PM had the highest "flow value," with the subject's "flow value" being around 200.

[0078] Next, in the example in Figure 21, the estimated results of the subject's stress level are displayed as an example of time-series information. In the example in Figure 21, the vertical axis shows the "stress value," which is the estimated result of the subject's stress level, and the horizontal axis shows "time." Here, the stress level refers to the sum of values ​​obtained by quantifying the estimated stress level of the subject every second using a binary system of 1 (feeling stressed) and 0 (not feeling stressed), calculated over one hour. In other words, the higher the "stress level," the longer the subject was in a stressed state during that time. For example, in the example in Figure 21, the hour from 18:00 to 19:00 had the highest stress level, with the subject's "stress level" being around 800.

[0079] Furthermore, in the example shown in Figure 22, the estimated results regarding the subject's drowsiness state are displayed as an example of time-series information. In the example in Figure 22, the vertical axis shows the "drowsiness value," which is the estimated result regarding the subject's drowsiness state, and the horizontal axis shows "time." Here, the "drowsiness value" refers to the sum of hourly values ​​obtained by quantifying the estimated state of a subject's sleepiness every 60 seconds using two values: 1 (continuously sleepy) and 0 (not continuously sleepy). In other words, the higher the "drowsiness value," the longer the subject felt continuously sleepy during that time period. Note that in the example in Figure 22, the "drowsiness value" is calculated using different units than the "flow value" and "stress value," so the values ​​have been adjusted to use the same units. Also, for example in the example in Figure 22, the hour from 19:00 to 20:00 is the highest, with the subject's "drowsiness value" being around 600.

[0080] Furthermore, although not explained in the above-described embodiments (particularly the embodiments in Figures 20 to 22), this system can display the changes in each value as a curve to make the changes in each value easier to understand (hereinafter referred to as "fitting"). Specifically, this system may perform fitting using any method, such as kernel density estimation. In addition, this system may perform processing such as logarithmization to bring the data closer to a normal distribution when performing fitting. Moreover, this system does not need to perform fitting if, for example, there is a lot of missing data or insufficient data.

[0081] Furthermore, although not explained in the above-described embodiments (particularly the embodiments in Figures 20 to 22), the various values ​​mentioned above (for example, "flow value," "stress value," "drowsiness value," etc.) may be aggregated for predetermined time periods, and values ​​for each time period (for example, "flow value," "stress value," "drowsiness value," etc.) may be calculated. Specifically, this system may, for example, define 9:00 to 14:00 as morning, 14:00 to 18:00 as afternoon, 18:00 to 21:00 as evening, and calculate values ​​for each time period (for example, "flow value," "stress value," "drowsiness value," etc.).

[0082] Furthermore, although not explained in the above-described embodiments (particularly the embodiments shown in Figures 20 to 22), this system may not simply calculate the above-described various values ​​(e.g., "flow value," "stress value," "absence level," etc.) on a daily basis, but may, for example, aggregate them by day of the week and calculate daily values ​​(e.g., "flow value," "stress value," "absence level," etc.).

[0083] Furthermore, although not explained in the above-described embodiments (particularly those shown in Figures 20 to 22), this system may also classify subjects into types using various calculated values ​​(e.g., "flow value," "stress value," "absentmindedness value," etc.). Specifically, this system could classify individuals based on their tendencies regarding "flow values," "stress values," and "absentmindedness values" for each time of day and day of the week, such as those who tend to concentrate in the morning, those who tend to accumulate stress in the afternoon, those who tend to feel sleepy in the evening, those who tend to experience stress on Mondays, and those who tend to concentrate on Fridays.

[0084] Furthermore, although not explained in the above embodiments (particularly the embodiments in Figures 20 to 22), the estimation and calculation results by this system may reflect significant individual differences. Therefore, this system may, for example, derive a standard "flow value," "stress value," "absentmindedness value," etc. (hereinafter, the standard value for each value will be called the "standard value") for each subject, and classify the type of subject based on the change in values ​​from the standard value. This system may simply use the standard values ​​calculated for each individual as they are for classification, or it may derive more general standard values ​​by statistically processing the standard values ​​derived from a large number of subjects, for example, and use those for classification.

[0085] Furthermore, although not explained in the above-described embodiments (particularly the embodiments shown in Figures 20 to 22), this system may also classify the types of subjects in the following manner, for example. (1) This system may, for example, derive the area for each of the following time periods (morning, noon, and evening) for each day of the week after fitting, and compare it with the reference value for each time period. (2) This system may, for example, classify the type of subject based on whether the area of ​​each time period for "flow value," "stress value," and "drowsiness value" is greater than or less than a baseline value. In other words, this system can estimate the type of state of concentration, stress, and drowsiness of a subject based on the levels of each indicator of "flow value," "stress value," and "drowsiness value" for each day of the week and each time period.

[0086] Furthermore, although not explained in the above-described embodiments (particularly the embodiments shown in Figures 20 to 22), this system may also classify subjects into types based on changes in their state over any given period. Specifically, this system could classify individuals who tend to have high "sluggishness" values ​​in the morning and high "flow" values ​​in the evening as "slow starters." Alternatively, it could classify individuals who tend to have high "flow" values ​​in the morning and high "sluggishness" values ​​in the evening as "start-dash types." In this way, the system can classify individuals based on changes in their state throughout the day. Specifically, this system can classify individuals who tend to have high "flow values" in the first half of the week but high "absentmindedness" or "stress values" in the second half of the week as "strained" or "burnout" types. The system can classify individuals based on the changes in their state over the week. By providing advice and various information according to the type of individual classified in this way, the system can efficiently carry out various tasks while taking into account the individual characteristics of each person. Specifically, for example, if an individual is the type whose "absentmindedness" or "stress values" are high at a certain time of day, the system can provide information encouraging them to take a break or change their mood at that time. Conversely, if an individual's "flow value" is high at a certain time of day, the system can suggest that they perform difficult tasks during that time.

[0087] Furthermore, the series of processes described above can be executed by hardware or by software. In other words, the functional configurations shown in Figure 3 are merely examples and are not particularly limiting. In other words, it is sufficient for the information processing system to have a function that can execute the series of processes described above as a whole, and the type of functional block used to realize this function is not limited to the examples shown in Figure 3. Furthermore, the location of the functional block is not limited to the examples shown in Figure 3, and can be arbitrary. Furthermore, a single functional block may consist of hardware alone, software alone, or a combination of both.

[0088] Furthermore, the number and users of the various hardware components that make up this system are arbitrary, and it may also include other hardware components.

[0089] Furthermore, when a series of processes are executed by software, the programs that make up that software are installed on a computer or other device from a network or storage medium.

[0090] Furthermore, the computer may be one that is built into dedicated hardware. Also, the computer may be one that can perform various functions by installing various programs. In other words, for example, any computer, any mobile terminal such as a smartphone may be freely used as the various hardware in the above-described embodiment. Furthermore, any combination of types and contents of various input and output units may be adopted.

[0091] Furthermore, such a recording medium containing a program may consist not only of a removable medium (not shown) provided separately from the main unit of the device to provide the program to the user, but may also consist of a recording medium that is pre-installed in the main unit of the device and provided to the user.

[0092] Furthermore, in this specification, each step in a program stored on a storage medium does not necessarily have to be processed chronologically in the order described in the embodiments above. It may be processed in a different order, and some processes may be omitted in parallel or individually, or only some processes may be executed.

[0093] Furthermore, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc.

[0094] The effects and advantages of this embodiment will also be achieved when these other embodiments are adopted. Furthermore, it is possible to combine this embodiment with other embodiments, and other embodiments with each other, as appropriate.

[0095] In summary, the program to which this disclosure applies can take various forms having the following configurations. In other words, the information processing systems to which this disclosure may apply are: An information processing system that can be used to obtain information about the target person, Image information acquisition means (for example, image acquisition unit 80) that acquires image information relating to an image including the skin of the subject, A luminance information acquisition means (for example, a PPG processing unit 81) that acquires luminance information in the skin of the subject included in the image, A first estimation means (for example, a heart rate estimation unit 82) estimates the heart rate of the subject based on the brightness information, A second estimation means (for example, an index estimation unit 83) estimates the value of an index related to the subject's heart rate based on the brightness information, It can be equipped with.

[0096] Furthermore, the above-described information processing system may further include a third estimation means (for example, a state estimation unit 84) that estimates the state of concentration or stress of the subject based on the heart rate estimated by the first estimation means and the value of the index estimated by the second estimation means.

[0097] Furthermore, the above-described information processing system may further include a fourth estimation means (for example, a sleepiness estimation unit 85) that estimates the sleepiness state of the subject based on the heart rate estimated by the first estimation means.

[0098] Furthermore, the above-described information processing system may further include a presentation means for presenting to the subject the estimated results of the subject's state of concentration or stress estimated by the third estimation means and the estimated results of the subject's state of sleepiness estimated by the fourth estimation means.

[0099] Furthermore, the luminance information acquisition means may acquire RGB information relating to the time-series changes of RGB contained in the image information, and acquire a PPG signal based on the acquired RGB information.

[0100] Furthermore, the third estimation means may estimate the subject's state of concentration or stress based on values ​​obtained by normalizing or standardizing the heart rate estimated by the first estimation means and the index values ​​estimated by the second estimation means using a predetermined method.

[0101] Furthermore, the third estimation means may estimate whether the subject is in a state of concentration, a state of stress, or a state that is neither concentration nor stress.

[0102] Furthermore, the fourth estimation means may also estimate whether or not the subject is in a state of continuously feeling sleepy.

[0103] Furthermore, the above-described information processing system may further include an evaluation calculation means that acquires the estimated results of the subject's state of concentration or stress estimated by the third estimation means and the estimated results of the subject's state of sleepiness estimated by the fourth estimation means during a predetermined period, and calculates an overall evaluation index for the subject during that period according to the frequency of each estimation result.

[0104] Furthermore, the above-described information processing system may also include a correction means for performing correction when acquiring luminance information if the luminance information acquired by the luminance information acquisition means does not meet predetermined conditions.

[0105] Furthermore, the above-described information processing system may further include a first determination means for determining the type of state of concentration of the subject based on the estimation result of the state of concentration of the subject over a predetermined period.

[0106] Furthermore, the above-described information processing system may further include a second determination means for determining the type of stress state of the subject based on the estimation results of the subject's stress state over a predetermined period.

[0107] Furthermore, the above-described information processing system may further include a third determination means for determining the type of sleepiness state of the subject based on the estimation results of the subject's sleepiness state over a predetermined period.

[0108] Furthermore, information processing methods, which are other aspects of this disclosure, are An information processing method performed by an information processing device that can be used to acquire information about a subject, Image information acquisition step: Acquires image information relating to an image including the skin of the subject, A brightness information acquisition step of acquiring brightness information in the skin of the subject included in the aforementioned image, A first estimation step in which the heart rate of the subject is estimated based on the brightness information, A second estimation step involves estimating the value of an index related to the subject's heart rate based on the brightness information, It can include...

[0109] Furthermore, another aspect of this disclosure, a program, Information processing device that can be used to obtain information about the target person, Image information acquisition step: Acquires image information relating to an image including the skin of the subject, A brightness information acquisition step of acquiring brightness information in the skin of the subject included in the aforementioned image, A first estimation step in which the heart rate of the subject is estimated based on the brightness information, A second estimation step involves estimating the value of an index related to the subject's heart rate based on the brightness information, It is possible to execute a process that includes this process. [Explanation of symbols]

[0110] 1 Analysis device 11 Analysis terminal 12 cameras 21 Control Unit 80 Image acquisition unit 81 PPG Processing Unit 100 Skin detection unit 101 RGB acquisition part 102 PPG signal acquisition section 103 Correction Unit 82 Heart rate estimation unit 120 Power spectrum acquisition unit 121 Heart rate acquisition unit 83 Indicator estimation part 140 Peak detection unit 141 PPI acquisition department 142 Indicator acquisition part 84 State Estimation Unit 85 Sleepiness Estimation Unit 86 Data transmission / reception unit 87. General Evaluation Department 88 Results presentation section 2 servers 210 Control Unit 240 Data transmission / reception unit 300 Analysis result DB

Claims

1. An information processing system that can be used to obtain information about the target person, Image information acquisition means for acquiring image information relating to an image including the skin of the subject, A means for acquiring brightness information in the skin of the subject included in the image, A first estimation means for estimating the heart rate of the subject based on the brightness information, A second estimation means for estimating the value of an index related to the subject's heart rate based on the brightness information, A third estimation means estimates the stress state and concentration state of the subject based on the heart rate estimated by the first estimation means and the index value estimated by the second estimation means, A fourth estimation means for estimating the sleepiness state of the subject based on the heart rate estimated by the first estimation means, A comprehensive evaluation means that calculates a comprehensive evaluation index regarding the subject's condition during a predetermined period by weighting the standardized values ​​of the estimation results of the subject's stress state by the third estimation means, the estimation results of the subject's concentration state by the third estimation means, and the estimation results of the subject's sleepiness state by the fourth estimation means during a predetermined period, An information processing system equipped with the following features.

2. The system further comprises a presentation means for presenting to the subject the estimated results of the subject's concentration or stress state estimated by the third estimation means and the estimated results of the subject's drowsiness state estimated by the fourth estimation means. The information processing system according to claim 1.

3. The luminance information acquisition means acquires RGB information relating to the time-series changes of RGB contained in the image information, and acquires a PPG signal based on the acquired RGB information. The information processing system according to claim 1.

4. The third estimation means estimates the concentration or stress state of the subject based on the heart rate estimated by the first estimation means and the values ​​of the indicators estimated by the second estimation means, obtained by normalizing or standardizing them in a predetermined way. The information processing system according to claim 1.

5. The third estimation means estimates whether the subject is in a state of concentration, a state of stress, or a state that is neither concentration nor stress. The information processing system according to claim 4.

6. The fourth estimation means estimates whether the subject is in a state of continuous drowsiness. The information processing system according to claim 1.

7. If the luminance information acquired by the luminance information acquisition means does not meet predetermined conditions, the system further includes a correction means for performing correction when acquiring the luminance information. The information processing system according to claim 1.

8. The system further comprises type determination means for determining at least one of the subject's stress type, concentration type, and sleep type based on at least one of the subject's stress state estimation result, concentration state estimation result, and sleep state estimation result during a predetermined period. The information processing system according to claim 1.

9. The system further comprises a first criterion determination means for determining a first criterion value, which is a standard value for each of the estimated stress state, concentration state, and sleepiness state of the subject during a predetermined period, respectively. The type determination means determines at least one of the subject's stress type, the subject's concentration type, or the subject's drowsiness type based on the results of each estimation of the subject and the change in value from the first reference value. The information processing system according to claim 8.

10. The system further comprises a second criterion determination means for determining a statistical second criterion value based on the estimated stress levels of a large number of subjects, the estimated concentration levels of a large number of subjects, and the estimated sleepiness levels of a large number of subjects. The type determination means determines, based on the results of each estimation of the subject and the change in value from the second reference value, at least one of the subject's stress state, the subject's concentration state, or the subject's drowsiness state. The information processing system according to claim 8.

11. An information processing method performed by an information processing device that can be used to acquire information about a subject, Image information acquisition step: Acquires image information relating to an image including the skin of the subject, A brightness information acquisition step of acquiring brightness information in the skin of the subject included in the aforementioned image, A first estimation step in which the heart rate of the subject is estimated based on the brightness information, A second estimation step involves estimating the value of an index related to the subject's heart rate based on the brightness information, A third estimation step estimates the stress state and concentration state of the subject based on the heart rate estimated in the first estimation step and the values ​​of the indicators estimated in the second estimation step, A fourth estimation step, based on the heart rate estimated in the first estimation step, estimates the sleepiness state of the subject, A comprehensive evaluation step that calculates a comprehensive evaluation index regarding the subject's condition during a predetermined period by weighting the standardized values ​​of the estimated stress state of the subject from the third estimation step, the estimated concentration state from the third estimation step, and the estimated sleepiness state of the subject from the fourth estimation step, respectively, during a predetermined period. Information processing methods including

12. Information processing device that can be used to obtain information about the target person, Image information acquisition step: Acquires image information relating to an image including the skin of the subject, A brightness information acquisition step of acquiring brightness information in the skin of the subject included in the aforementioned image, A first estimation step in which the heart rate of the subject is estimated based on the brightness information, A second estimation step involves estimating the value of an index related to the subject's heart rate based on the brightness information, A third estimation step estimates the stress state and concentration state of the subject based on the heart rate estimated in the first estimation step and the values ​​of the indicators estimated in the second estimation step, A fourth estimation step, based on the heart rate estimated in the first estimation step, estimates the sleepiness state of the subject, A comprehensive evaluation step that calculates a comprehensive evaluation index regarding the subject's condition during a predetermined period by weighting the standardized values ​​of the estimated stress state of the subject from the third estimation step, the estimated concentration state from the third estimation step, and the estimated sleepiness state of the subject from the fourth estimation step, respectively, during a predetermined period. A program that performs a process that includes this.

Images