Information processing system, information processing method, and program

Abstract

To provide a technique for more easily and accurately estimating a state of a person.SOLUTION: The state estimating unit 84 acquires information concerning an estimation result of the heart rate estimated by the heart-rate acquiring unit 121 in a predetermined period. The state estimation unit 84 acquires information on the value of the index acquired by the index acquisition unit 142 in a predetermined period. The state estimation unit 84 standardizes each of the acquired information on the estimation result of the heart rate and the acquired information on the value of the index in accordance with a predetermined standard. The state estimation unit 84 determines whether the value obtained by standardizing each of the information about the estimation result of the heart rate and the information about the value of the index satisfies the condition of the stress state. The state estimation unit 84 determines whether or not a value obtained by standardizing each of the information regarding the estimation result of the heart rate and the information regarding the value of the index satisfies the condition of the flow state.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Conventionally, in people's daily lives, a person's physical and mental state is not constant but changes over time. For example, while at work, a person goes through various states such as stress, concentration, and drowsiness, and knowing one's own physical state is important for improving work efficiency, maintaining health, and preventing accidents. Also, for example, when driving a car, it is important for the driver to always stay awake and focused. In this regard, attempts have been made to detect a person's drowsiness based on heart rate in order to prevent drowsiness while driving (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-86201 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 mentioned above is insufficient because it can only detect the presence or absence of drowsiness based on the heart rate, and this is also true for other conventional technologies.

[0005] The present disclosure has been made in consideration of such circumstances, and provides a technology for more easily and accurately estimating a person's state. [Means for solving the problem]

[0006] In order to achieve the above object, an information processing system according to an embodiment of the present disclosure includes: An information processing system that can be used to obtain information about a subject, image information acquisition means for acquiring image information relating to an image including the subject's skin; a brightness information acquisition means for acquiring brightness information of the subject's skin included in the image; a first estimation means for estimating a heart rate of the subject based on the luminance information; a second estimation means for estimating a value of an index related to the subject's heart rate based on the luminance information; The information processing system includes:

[0007] An information processing method and a program according to an embodiment of the present disclosure are also provided as an information processing method or a program corresponding to an information processing system according to an embodiment of the present disclosure. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a technology for more easily and accurately estimating a person's condition. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an analysis terminal that configures the information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an analysis terminal and a server of the information processing system according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of an analysis terminal of the information processing system according to an embodiment of the present disclosure, particularly the functional configuration of a PPG processing unit. [Figure 5] FIG. 2 is a diagram illustrating an example of the functional configuration of a heart rate estimation unit, in particular, of the functional configuration of an analysis terminal of the information processing system according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating an example of the functional configuration of an analysis terminal of the information processing system according to an embodiment of the present disclosure, in particular the functional configuration of an index estimation unit. [Figure 7]10A and 10B are diagrams illustrating an example of a typical waveform of RGB information acquired by an information processing system according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram illustrating an example of a typical waveform of PPG information acquired by an information processing system according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a diagram showing an example of a typical waveform of PSD information acquired by an information processing system according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of an image of a transition in heart rate estimated by an information processing system according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating an example of an image when an information processing system according to an embodiment of the present disclosure detects peak positions of each waveform of a PPG signal from PPG information. [Figure 12] FIG. 10 is a diagram illustrating an example of a typical image of PPI information acquired by an information processing system according to an embodiment of the present disclosure. [Figure 13] FIG. 10 is a diagram showing an example of an image of the transition of the value of RMSSD, which is an index of heart rate variability, among the index information acquired by an information processing system according to an embodiment of the present disclosure. [Figure 14] 10A and 10B are diagrams for explaining an example of a specific method by which an analysis terminal included in an information processing system according to an embodiment of the present disclosure detects whether or not drowsiness occurs. [Figure 15] 10A and 10B are diagrams for explaining an example of a specific method by which an analysis terminal constituting an information processing system according to an embodiment of the present disclosure detects whether or not there is continuous drowsiness. [Figure 16] FIG. 10 is a diagram illustrating an example of an image displayed on an analysis terminal included in the information processing system according to an embodiment of the present disclosure. [Figure 17] FIG. 10 is a diagram illustrating an example of a flow of a drowsiness estimation process executed by an analysis terminal included in an information processing system according to an embodiment of the present disclosure. [Figure 18] FIG. 10 is a diagram illustrating an example of the flow of a state estimation process executed by an analysis terminal configuring the information processing system according to an embodiment of the present disclosure. [Figure 19]17 is a diagram showing an example of an image displayed on an analysis terminal constituting the information processing system according to an embodiment of the present disclosure, and is a diagram showing an example different from the example of FIG. 16. FIG. [Figure 20] FIG. 10 is a diagram for explaining an example of a method for using an estimation result by an information processing system according to an embodiment of the present disclosure. [Figure 21] FIG. 21 is a diagram for explaining an example of a method for using an estimation result by an information processing system according to an embodiment of the present disclosure, and is a diagram showing an example different from the example of FIG. 20. [Figure 22] 22 is a diagram for explaining an example of a method for using an estimation result by an information processing system according to an embodiment of the present disclosure, and is a diagram showing an example different from the examples of FIGS. 20 and 21. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] [Embodiment] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] <Summary> First, prior to a specific description using FIG. 1, a first service and a second service to which an information processing system according to an embodiment of the present disclosure (hereinafter referred to as "this system") is applied will be described. In the first service, the system estimates whether the subject is drowsy, whether the subject is concentrating, and whether the subject is stressed based on various biometric information (heart rate, RMSSD, etc.) obtained from image information including the subject's skin. In the second service, the system presents the subject with a comprehensive assessment of the subject's sleepiness, concentration level, stress level, etc. over a certain period, as well as time-series information on various indicators, based on various biological information of the subject collected over that period. By using a combination of these services, the subject can easily understand and manage their own sleepiness and various other conditions.

[0012] Here, the definitions of the indices that can be included in each piece of biometric information will be explained. The heart rate is the number of times the heart beats per minute. In this embodiment, the heart rate and heart rate variability index of the subject are estimated using rPPG (remoto photoplethysmography). rPPG is an optical technique that evaluates changes in the optical reflectance or transmittance of a region or volume of interest over time. rPPG is a simple, low-cost biomedical information acquisition technique that can detect changes in capillary blood flow (volume) in tissues and obtain various information that can be correlated with physiological phenomena in vivo. The method of obtaining rPPG adopted by this system is arbitrary, but for example, please refer to the method in the following document. [Non-Patent Document 1] Wenjin Wang, Bert den Brinker, Sander Stuijk, and Gerard de Haan, "Algorithmic Principles of Remote-PPG," IEEE Transactions on Biomedical Engineering (July 2017) In the following description, a PPG signal (or an rPPG signal) refers to various types of signal information that can be used to detect a person's condition and that is obtained by a technique such as PPG (or rPPG).

[0013] RMSSD (Root Mean Square of Successive Differences) is usually defined as the square root of the average of the squares of the differences between adjacent RRIs (RR Intervals), and is specifically expressed, for example, by the following formula. RMSSD is believed to have a strong correlation with parasympathetic nervous activity and is used as an index of heart rate variability. However, for PPG or rPPG signals, PPI (peak to peak interval), which corresponds to RRI, may also be used.

number

[0014] FIG. 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present disclosure. 1, the system includes an analysis device 1 and a server 2. As shown in FIG. 1, the analysis device 1 includes an analysis terminal 11 and a camera 12. The analysis terminal 11 analyzes the subject image information, which will be described later, and executes various processes for estimating the subject's heart rate, the subject's concentration or stress state, and the like. The camera 12 acquires reflected light reflected through a part of a living body such as a subject (particularly the skin, hereinafter referred to as the "measurement part"), and generates information about the image (hereinafter referred to as "subject image information") based on the acquired reflected light. The analysis terminal 11 of the analysis device 1 and the server 2 are connected via a predetermined network N such as a LAN (Local Area Network) or the Internet. Note that the network N does not have to be a LAN or the Internet, and communication may be performed by any method such as Bluetooth (registered trademark).

[0015] FIG. 2 is a diagram showing an example of the hardware configuration of the analysis terminal 11 in the information processing system of FIG. The analysis terminal 11 is configured as a general-purpose PC (Personal Computer) etc. As shown in Fig. 2, the analysis terminal 11 includes a control unit 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a bus 24, an input / output interface 25, an output unit 26, an input unit 27, a storage unit 28, a communication unit 29, and a drive 30.

[0016] The control unit 21 is configured with a microcomputer or the like including a CPU, a GPU, and a semiconductor memory, and executes various processes according to a program recorded in a ROM 22 or a program loaded from a storage unit 28 into a RAM 23. The RAM 23 also stores information necessary for the control unit 21 to execute various processes as needed.

[0017] The control unit 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input / output interface 25 is also connected to this bus 24. An output unit 26, an input unit 27, a storage unit 28, a communication unit 29, and a drive 30 are connected to the input / output interface 25.

[0018] The output unit 26 is composed of various liquid crystal displays and the like, and outputs various information.

[0019] The input unit 27 is configured with various hardware components and inputs various pieces of information.

[0020] The storage unit 28 is configured by a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various data, such as various programs and various data including databases.

[0021] The communication unit 29 controls communications with other devices via a network N including the Internet.

[0022] The drive 30 is provided as needed. Removable media 41, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached to the drive 30. Programs read from the removable media 41 by the drive 30 are installed in the storage unit 28 as needed. The removable media 41 can also store various data stored in the storage unit 28 in the same way as the storage unit 28.

[0023] Here, an example of the hardware configuration of the camera 12 that can be connected to the analysis terminal 11 will be further described. The camera 12 is configured by, for example, a general-purpose web camera, a 3D depth camera, etc. The camera 12 is used to capture an image including the skin of the subject. The hardware configuration of the server 2 can be basically the same as the hardware configuration of the above-mentioned analysis terminal 11, and therefore a description thereof will be omitted here.

[0024] FIG. 3 is a diagram illustrating an example of the functional configuration of the analysis terminal 11 and the server 2 in the information processing system according to an embodiment of the present disclosure. As shown in Figure 3, the control unit 21 of the analysis terminal 11 executes various programs, etc., and thereby functions as an image acquisition unit 80, a PPG processing unit 81, a heart rate estimation unit 82, an index estimation unit 83, a state estimation unit 84, a drowsiness estimation unit 85, a data transmission / reception unit 86, a comprehensive evaluation unit 87, and a result presentation unit 88.

[0025] The image acquisition unit 80 acquires information about an image of the subject captured by the camera 12, i.e., subject image information. Here, the subject image information preferably includes information about an image of the subject's skin. Note that the skin image referred to here does not necessarily refer to any part of the body, and may include an image of the skin of any part, such as the eyes, nose, or mouth.

[0026] The PPG processing unit 81 executes various processes for acquiring a PPG signal based on the subject image acquired by the image acquisition unit 80. Specifically, as shown in FIG. 4, PPG processing unit 81 includes skin detection unit 100, RGB acquisition unit 101, PPG signal acquisition unit 102, and correction unit 103.

[0027] The skin detection unit 100 identifies an area in the subject's image information where the subject's skin is exposed.

[0028] RGB acquisition unit 101 acquires information regarding the luminance of each color in each of the regions identified by skin detection unit 100 (hereinafter referred to as "RGB information"). Specifically, RGB acquisition unit 101 acquires, for example, the average value of the luminance of each color in the regions identified by skin detection unit 100 as the RGB information. Note that Fig. 7 shows an example of a typical waveform indicating the RGB information acquired by this system. Furthermore, many conventional heart rate estimation technologies using image recognition, etc., acquire an image of the face, then extract the areas of each part of the face (eyes, nose, cheeks, etc.) and estimate the heart rate, etc., from the characteristics of each area. In contrast, this system does not rely on the characteristics of each part, so it is possible to make estimations (described later) using only a part of the face or body, and it is not necessary for the image to include a face, allowing for flexibility in the situation and environment in which the image is captured.

[0029] The PPG signal acquisition unit 102 acquires a PPG signal from the time-series information of the RGB information acquired by the RGB acquisition unit 101. Specifically, the PPG signal acquisition unit 102 extracts a PPG signal from luminance information using an arbitrary algorithm and applies a filter to the PPG signal to acquire time-series information of the PPG signal during the measurement period (hereinafter referred to as "PPG information"). Note that Fig. 8 shows an example of a typical waveform indicating the PPG information acquired by this system. Furthermore, the method and algorithm for acquiring the PPG signal in this system can be any method, but the system can employ, for example, the method described in Non-Patent Document 1 mentioned above.

[0030] When part of the image information is lost due to a failure to capture an image of the subject or a decrease in FPS (Frames Per Second), or when a signal sampled at a specific frequency is re-sampled, the correction unit 103 performs a correction process on the RGB information by, for example, linear interpolation or deleting the sampled data.

[0031] The heart rate estimation unit 82 executes various processes for estimating the heart rate of the subject based on the luminance information. Specifically, as shown in FIG. 5, the heart rate estimation unit 82 includes a power spectrum acquisition unit 120 and a heart rate acquisition unit 121.

[0032] The power spectrum acquisition unit 120 calculates and acquires the power spectrum density (hereinafter referred to as "PSD information") of the PPG information acquired by the PPG signal acquisition unit 102. The power spectrum density is obtained by dividing the strength of each signal into certain frequency bands and expressing the strength of each band as a function of frequency. Note that Fig. 9 shows an example of a typical waveform indicating the PSD information acquired by this system.

[0033] The heart rate acquisition unit 121 estimates the heart rate of the subject based on the PSD information acquired by the power spectrum acquisition unit 120. Here, the present system can adopt any existing method for calculating the power spectrum and estimating the heart rate, but for example, see the method in the following document: [Non-patent document 3] Wim Verkruysse, Lars O Svaasand and J Stuart Nelson “Remote plethysmographic imaging using ambient light” OPTICS EXPRESS (December 2008) FIG. 10 is a diagram showing an example of an image of the transition of the heart rate estimated by this system.

[0034] The index estimation unit 83 estimates the value of an index related to the subject's heart rate based on the luminance information. Specifically, as shown in Fig. 5, the index estimation unit 83 includes a peak detection unit 140, a PPI acquisition unit 141, and an index acquisition unit 142.

[0035] The peak detection unit 140 detects the peak positions of the waveform in the PPG information acquired by the PPG signal acquisition unit 102. Note that Fig. 11 is a diagram showing an example of an image when the present system detects the peak positions of each waveform of the PPG signal from the PPG information.

[0036] The PPI acquisition unit 141 acquires PPI time-series information (hereinafter referred to as "PPI information") indicating the interval between peak positions of the PPG information (PPG signal) detected by the peak detection unit 140. Note that Fig. 12 shows an example of a typical image showing the PPI information acquired by this system.

[0037] The index acquiring unit 142 calculates the values ​​of various indices related to heart rate variability based on the PPI information in the PPG information of the subject acquired by the PPI acquiring unit 141, and acquires the information (hereinafter referred to as "index information"). Note that in this embodiment, the system calculates, for example, a value such as RMSSD from among the indices of heart rate variability described above as an index of heart rate variability. Also, Fig. 13 is a diagram showing an example of an image of the transition of the value of RMSSD, which is an index of heart rate variability, from among the index information acquired by the system.

[0038] The state estimation unit 84 estimates the concentration or stress state of the subject based on the heart rate estimated by the first estimation means and the value of the index estimated by the second estimation means. That is, the state estimation unit 84 estimates the concentration or stress state of the subject based on the heart rate estimated by the heart rate acquisition unit 121 and the index information (particularly the RMSSD value) acquired by the index acquisition unit 142. The state estimation unit 84 standardizes the heart rate estimated by the heart rate acquisition unit 121 and the index information (especially the RMSSD value) acquired by the index acquisition unit 142 according to a predetermined standard. Note that the method by which the system standardizes the values ​​of various indices may be any method, but for example, standardization can be performed using the method of the following formula. Note that the system can also standardize other values ​​such as the heart rate using a similar method.

number

[0039] The drowsiness estimation unit 85 estimates the drowsiness state of the subject based on the heart rate estimated by the first estimation means. That is, the drowsiness estimation unit 85 estimates whether the subject is drowsy or not based on the heart rate of the subject acquired by the heart rate acquisition unit 121.

[0040] An example of a method for estimating drowsiness in the drowsiness estimation unit 85 will be described specifically with reference to Fig. 14. Fig. 14 is a diagram for explaining an example of a specific method for the analysis terminal constituting this system to detect the presence or absence of drowsiness. In the example of Fig. 14, the heart rate value estimated from the acquired PPG signal is shown in units of one minute. Note that the value per minute is calculated as the average of the values ​​estimated every second. Specifically, the example of Fig. 14 shows that the heart rate changes every minute as follows: "66.3", "65.4", "62.1", "65.8", "64.2", "63.9", "62.5", "66.1", "67.0", "67.3", "66.8". Now, if you look at the top of each number, you will see two displays: the "baseline interval" and the "prediction interval." That is, the "baseline interval" indicates the subject's heart rate before drowsiness detection. On the other hand, the "prediction interval" indicates the heart rate at the time of drowsiness detection and the timing immediately before. Physiological phenomena are significantly affected by individual differences, making it difficult to perform highly accurate detection by simply setting a threshold value. Therefore, detecting drowsiness based on time-series fluctuations as shown in Figure 14 enables more accurate detection.

[0041] 14, the system detects the difference between the average (or median) heart rate in the baseline interval and the average (or median) heart rate in the prediction interval, and determines that the subject is drowsy if the average heart rate in the prediction interval has fallen below the average heart rate in the baseline interval by a predetermined threshold or more (for example, the heart rate has decreased by 1.0 point or more). Conversely, if drowsiness has already been detected, the system may determine that the drowsiness has disappeared if the most recent measured heart rate value has continuously increased.

[0042] The system may estimate the subject's drowsiness using a standardized value obtained by, for example, (value of prediction interval - average value of baseline interval) / standard deviation of baseline interval, etc. This enables the system to more accurately estimate whether the subject is drowsy. Furthermore, for example, when there is a time difference between the prediction interval and the target time for estimating drowsiness, i.e., when the prediction interval is a period prior to the target time, the system may determine that the subject is drowsy if, for example, the average value of the prediction interval is lower than the average value of the baseline interval by a predetermined threshold and the heart rate value at the target time of estimation is also lower than the average value of the baseline interval by a predetermined threshold. This allows the system to eliminate the possibility that the subject may be awake at the target time of estimation, thereby enabling more accurate determination of whether the subject is drowsy.

[0043] Furthermore, the present system can also determine whether or not the subject is experiencing continuous drowsiness using the method shown in Fig. 15. Fig. 15 is a diagram for explaining an example of a specific method by which the analysis terminal constituting the present system detects whether or not the subject is experiencing continuous drowsiness. The example of FIG. 15 shows minute-by-minute changes in the cases where the subject is determined to be drowsy and not drowsy using the method shown in FIG. 14 or the like. In the example of FIG. 15, t=P is set as the target time for estimation. In the example of FIG. 15, at the time of t=P, drowsiness is detected for four consecutive minutes, which is a five-minute period (the period that is the determination period in the example of FIG. 15) that includes two minutes before and after t=P. The present system may organize the presence or absence of drowsiness in chronological order in this way, and determine that the subject is drowsy only if drowsiness is detected for a predetermined period of time or more. Note that in this case, the present system may also check whether the subject is awake or not based on the current heart rate, as in the case of FIG. 14.

[0044] The data transmission / reception unit 86 transmits various information and estimation results acquired by the analysis terminal 11 to the server 2.

[0045] The comprehensive evaluation unit 87 acquires the estimation results of the subject's state related to concentration or stress estimated by the third estimation means and the estimation results of the subject's state related to drowsiness estimated by the fourth estimation means for a predetermined period, and calculates a comprehensive evaluation index for the subject for that period according to the frequency of each estimation result. That is, the comprehensive evaluation unit 87 calculates a comprehensive evaluation of the subject's state based on the estimation results of the subject's state of concentration or stress estimated by the state estimation unit 84 for the predetermined period and the estimation results of the presence or absence of drowsiness of the subject estimated by the drowsiness estimation unit 85. Note that the system can use any method to calculate the comprehensive evaluation, but for example, it can be calculated using the method shown in the following formula as an example.

number

[0046] The result presenting unit 88 presents to the subject the estimation result of the subject's concentration or stress state estimated by the third estimation means and the estimation result of the subject's drowsiness state estimated by the fourth estimation means. That is, the result presenting unit 88 executes various processes for presenting to the subject the estimation result of the subject's concentration or stress state estimated by the state estimation unit 84 and the estimation result of the presence or absence of drowsiness of the subject estimated by the drowsiness estimation unit 85. 16 is a diagram illustrating an example of an image displayed on an analysis terminal constituting an information processing system according to an embodiment of the present disclosure. The system can present various estimation results and the like to the subject by, for example, displaying an image such as that shown in FIG. 16 on the output unit 26 or the like. Although not shown in the example of FIG. 16, the system may also present the subject with a comment regarding the measurement environment, such as "The camera is far away, please move closer."

[0047] Next, an example of the functional configuration of the server 2 will be described. 3, the control unit 210 of the server 2 executes various programs and the like to function as a data transmission / reception unit 240. An analysis result DB 300 is provided in one area of ​​the storage unit 220 of the server 2.

[0048] The data transmitting / receiving unit 240 acquires various information such as the various information and estimation results transmitted from the analysis terminal 11 of each subject via the communication unit 200. The data transmitting / receiving unit 240 also stores the acquired various information such as the various information and estimation results in the analysis result DB 300.

[0049] FIG. 17 is a diagram illustrating an example of the flow of a drowsiness estimation process executed by an analysis terminal configuring an information processing system according to an embodiment of the present disclosure. In step S1, the drowsiness estimation unit 85 acquires information about the estimation result of the heart rate estimated by the heart rate acquisition unit 121 during a predetermined period.

[0050] In step S2, the drowsiness estimation unit 85 determines whether or not the transition of the estimation result of the subject's heart rate over a predetermined period satisfies the condition of drowsiness. If the transition of the estimated heart rate satisfies the condition of drowsiness, step S2 is determined as YES and the process proceeds to step S4. On the other hand, if the transition of the estimated heart rate does not satisfy the condition of drowsiness, step S2 is determined as NO and the process proceeds to step S3.

[0051] In step S3, the drowsiness estimation unit 85 determines that the subject is not drowsy if the transition of the estimation result of the heart rate does not satisfy the condition for drowsiness.

[0052] In step S4, the drowsiness estimation unit 85 determines that the subject is drowsy if the transition of the estimation result of the heart rate satisfies the condition of drowsiness, and the drowsiness estimation process of the analysis terminal 11 is thereby completed.

[0053] FIG. 18 is a diagram illustrating an example of the flow of a state estimation process executed by an analysis terminal configuring an information processing system according to an embodiment of the present disclosure. In step S21, the state estimation unit 84 acquires information relating to the estimation result of the heart rate estimated by the heart rate acquisition unit 121 for a predetermined period of time.

[0054] In step S22, the state estimation unit 84 acquires information relating to the index values ​​(particularly the RMSSD values) acquired by the index acquisition unit 142 during a predetermined period.

[0055] In step S23, the state estimation unit 84 standardizes the acquired information on the estimation result of the heart rate and the information on the index value (particularly the RMSSD value) according to a predetermined standard.

[0056] In step S24, the state estimation unit 84 determines whether or not the standardized values ​​of the information related to the estimated heart rate and the information related to the index value (particularly the RMSSD value) satisfy the condition for a stress state. Specifically, in step S24, the state estimation unit 84 determines that the subject is in a stress state if the standardized value of the heart rate is greater than a predetermined threshold (e.g., 1.0) and the standardized value of the index (particularly the RMSSD) is less than a predetermined threshold (e.g., -0.5). If the standardized heart rate value and the standardized index value (particularly RMSSD) satisfy the condition of a stressed state, step S24 is determined as YES, and the process proceeds to step S25. On the other hand, if the standardized heart rate value and the standardized index (particularly RMSSD) value do not satisfy the condition of a stressed state, step S24 is determined as NO, and the process proceeds to step S26.

[0057] In step S25, the state estimation unit 84 determines that the subject is in a stress state when the standardized value of the heart rate and the standardized value of the index (particularly RMSSD) satisfy the conditions for a stress state.

[0058] In step S26, the state estimation unit 84 determines whether or not the standardized values ​​of the information related to the estimated heart rate and the information related to the index value (particularly the RMSSD value) satisfy the condition for a flow state. Specifically, in step S26, the state estimation unit 84 determines that the subject is in a flow state if the standardized value of the heart rate is within a predetermined threshold value (e.g., from −0.3 to 0.3) and the standardized value of the index (particularly the RMSSD) is greater than a predetermined threshold value (e.g., 1.15). If the standardized heart rate value and the standardized index value (particularly RMSSD) satisfy the flow state condition, step S26 is determined as YES, and the process proceeds to step S27. On the other hand, if the standardized heart rate value and the standardized index value (particularly RMSSD) do not satisfy the condition of the flow state, step S26 is determined as NO, and the process proceeds to step S28.

[0059] In step S27, the state estimation unit 84 determines that the subject is in a flow state when the standardized value of the heart rate and the standardized value of the index (particularly RMSSD) satisfy the conditions for a flow state.

[0060] In step S28, the state estimation unit 84 determines that the subject is in a normal state if the standardized value of the heart rate and the standardized value of the index (particularly RMSSD) do not satisfy either the condition for a stress state or the condition for a flow state. This ends the state estimation process of the analysis terminal 11.

[0061] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope that can achieve the object of the present disclosure are included in the present disclosure.

[0062] Although not described in the above embodiment, the overall evaluation calculated by the present system can be presented to the subject in various formats. Specifically, for example, FIG. 19 shows the time-series changes in the overall evaluation for each predetermined day (for example, for each week in the example of FIG. 19). By grasping such changes in the overall evaluation, the subject can more efficiently control his or her own state of stress and concentration.

[0063] Although the description has been omitted in the above embodiment, the features and advantages of the system according to the present disclosure will be briefly supplemented below. This system is a technology that can acquire a subject's PPG signal without contact using captured video images and estimate the subject's heart rate variability and heart rate variability index (RMSSD, etc.). Furthermore, this system can estimate the subject's drowsiness, stress, or concentration state from the estimated heart rate variability and heart rate variability index (RMSSD, etc.). Here, the features of this system compared to the prior art include, for example, the following points. (1) This system can simultaneously determine multiple states of a subject, such as stress state, concentration (flow state), normal state (neither stress state nor flow state), and whether or not the subject is drowsy. (2) This system uses two indicators to estimate the state: heart rate and heart rate variability index (at least RMSSD). (3) This system can calculate a comprehensive evaluation index for a predetermined time period (e.g., for each day) based on the results of hourly estimation of the subject's state and the results of drowsiness detection. At least the features described in (1) to (3) above are not found in many prior art technologies, including the aforementioned Patent Document 1. (4) Furthermore, the threshold values ​​for determining the state and the method for calculating the overall evaluation in this system have been derived from the results of numerous experiments conducted by the inventors of the present disclosure.

[0064] Although the explanation has been omitted in the above embodiment, an example of a correction (interpolation) method in this system will be briefly described below. For example, if an error occurs in acquiring an image of a subject and the error is corrected within a predetermined time (e.g., 3 seconds), the system may complement the RGB information by a method such as linearly complementing the RGB information. On the other hand, if an error in acquiring an image of the subject continues for a predetermined time (e.g., 3 seconds) or more, the system may reset the program and acquire an image of the subject again. By performing such correction, the system can improve the accuracy of various estimations.

[0065] Although not described in the above embodiment, the system simply determines whether or not the subject is stressed and whether or not the subject is focused (in a flow state), but this is not limited thereto. The system may also determine the subject's state in stages, such as slightly stressed, stressed, very stressed, or slightly focused.

[0066] In the above embodiment, the present system has been described as estimating a state related to concentration or stress, but is not limited thereto. For example, the present system may estimate only one of the states of concentration or stress, or may further estimate other states of the subject.

[0067] In the above embodiment, the system is described as acquiring RMSSD as an index of heart rate variability, but this is not limited thereto. The system may acquire other indexes, such as SDNN, as an index of heart rate variability. SDNN indicates the standard deviation of the time interval between two consecutive heartbeats (heart beat interval, RR interval) within a certain period. SDNN can be derived by calculating the standard deviation of the RR interval (the time interval between ventricular excitations) within a given period.

[0068] Furthermore, the present system may acquire other indices other than those of heart rate variability and use them for various estimations. Specifically, the present system may acquire an index indicating the size of the eye area, such as EAR (Eye Aspect Ratio), and use it for various estimations. EAR is the ratio of the horizontal and vertical lengths of the eye. For example, it can be calculated by detecting the eye area from an image and calculating the ratio of the respective lengths.

[0069] It should be noted that the indices adopted by this system, including the examples of the above-mentioned embodiments, do not necessarily have to be identical to the various indices that are commonly used, and may include some changes or modifications to the derivation method and the meaning of the detailed indices.

[0070] Furthermore, for example, in the above-described embodiment, the heart rate is estimated using the value of the PPG signal, but this is not particularly limited to this. For example, the heart rate may be estimated using various methods, such as other algorithms, PPG signals with other wavelengths, or the average value of those values.

[0071] Furthermore, for example, in the above-described embodiment, the present system has been described as acquiring a PPG signal from RGB information, but this is not limited thereto. The present system may acquire a PPG signal from other information related to brightness, for example, rather than from color information such as RGB information.

[0072] Furthermore, for example, in the above-described embodiments (particularly the embodiments of FIGS. 14 and 15 ), specific methods for detecting drowsiness have been described, but the above-described methods for detecting drowsiness are merely examples and are not limiting. For example, the present system may employ other thresholds or criteria to detect the presence or absence of drowsiness. Furthermore, for example, the present system may not use the above-described standardization method, or may use a standardization method other than the above-described formula.

[0073] Furthermore, for example, in the above-described embodiment, the camera 12 has been described as not including a control unit that processes various programs, etc., but this is not particularly limited. That is, the camera 12 may be configured to include a control unit, a storage unit, etc. (not shown) as necessary. In this case, for example, all or part of the various processes executed by the analysis terminal 11, etc. may be executed by the camera 12.

[0074] Furthermore, although detailed description has been omitted in the above embodiment, the present system has the following advantages in addition to the various advantages described above. (1) This system processes PPG signals in parallel from different exposed areas of the skin (e.g., forehead, nose, right cheek, left cheek), detects the most stable signal, and estimates heart rate and other parameters based on that signal. In other words, this system can detect stable signals from PPG signals obtained from multiple parts of the body and estimate heart rate, etc., enabling even more accurate estimation. In this respect, many conventional drowsiness detection systems differ from this system, as they require imaging of specific parts of the body, such as the face, and do not perform parallel processing (even when using PPG signals). (2) Furthermore, because parallel processing is performed, even if the subject behaves unexpectedly while detecting the PPG signal (e.g., rubbing their eyes, continuously facing a different direction, wearing sunglasses, etc.), it is possible to achieve signal processing that eliminates the influence of such behavior to some extent. (3) Furthermore, if signals cannot be properly obtained from the surface of the face, a backup function can be provided to obtain PPG signals from the skin of other parts of the body (such as the hands or arms). (4) Furthermore, by performing the above-mentioned parallel processing, it is believed that it will be possible to obtain a more stable signal and estimate the heart rate than conventional drowsiness detection systems, even under different light sources (depending on the weather or the position of the sun) or in cases where it is difficult for conventional drowsiness detection systems to estimate the heart rate, such as when a woman is wearing makeup.

[0075] Furthermore, although the description has been omitted in the above embodiment, the present system can additionally implement the following functions, for example. (1) Face selection function As described above, the present system does not necessarily need to capture an image of the subject's face to detect drowsiness or estimate their state. The face use selection function is a function that can switch between a mode in which the subject's face is captured and used to detect drowsiness or estimate their state when it can be captured, and a mode in which other body parts are freely captured and used to detect drowsiness or estimate their state when the face cannot be detected. (2) Face detection frequency adjustment function The face detection frequency adjustment function is a function that detects head movement using a sensor or the like and adjusts the face detection frequency according to the detected head movement. Specifically, for example, if the head movement is large, the face detection frequency can be increased, and if the head movement is small, the face detection frequency can be decreased. (3) Stable continuity function The stable continuation function is used when the images acquired from the subject are stable. This system can perform various estimations stably by proceeding with processing only when the subject's head movement and face detection are stable.

[0076] Furthermore, in the above-described embodiment (particularly the embodiment in FIG. 14), the method by which the present system detects drowsiness of a subject has been exemplified, but is not limited thereto. The drowsiness detection method by the present system described with reference to FIG. 14 is an example, and the present system may detect drowsiness of a subject by any method based on changes in the subject's heart rate.

[0077] Here, we will further explain an example of how to use the results of various estimations made by this system. Specifically, we will explain a method of classifying viewers by type using the results of estimations made by this system regarding the state of concentration, stress, and drowsiness of the subject, with reference to Figures 20 to 22. First, in the example of Figure 20, the estimation result of the subject's state of concentration (flow state) is displayed as an example of time-series information. In the example of Figure 20, the vertical axis shows the "flow value," which is the estimation result of the subject's state of concentration, and the horizontal axis shows "time." Here, the flow value refers to the hourly total value of the estimated results of the subject's concentration state for each second, quantified into two values, 1 (concentrated) and 0 (not concentrating). In other words, the higher the subject's "flow value," the longer the subject spent in a state of concentration during that time. In the example in Figure 20, for example, the hour from 2:00 PM to 3:00 PM was the highest, with the subject's "flow value" being around 200.

[0078] Next, in the example of Fig. 21, the estimation result of the subject's stress state is displayed as an example of time-series information. In the example of Fig. 21, the vertical axis shows the "stress value," which is the estimation result of the subject's stress state, and the horizontal axis shows "time." Here, the stress value refers to the hourly total value of the estimated results of the subject's stress state for each second, quantified into two values, 1 (feeling stressed) and 0 (not feeling stressed). In other words, the higher the subject's "stress value," the longer the subject spent in a stressful state during that time. In the example of Figure 21, for example, the hour from 6:00 PM to 7:00 PM was the highest, with the subject's "stress value" being around 800.

[0079] In addition, in the example of Fig. 22, the estimation result of the subject's drowsiness state is displayed as an example of time-series information. In the example of Fig. 22, the vertical axis displays the "drowsy value," which is the estimation result of the subject's drowsiness state, and the horizontal axis displays "time." Here, the vagueness value indicates the hourly total value of the estimated results of the subject's sleepiness state every 60 seconds, quantified into two values: 1 (continuously feeling sleepy) and 0 (continuously not feeling sleepy). In other words, the higher the "vagueness value," the longer the subject felt sleepy during that time. In the example of FIG. 22, the "vagueness value" is calculated in different units than the "flow value" and "stress value," so the value is adjusted to use the same units. In addition, in the example of FIG. 22, the hour from 7:00 PM to 8:00 PM is the highest, with the subject's "vagueness value" being around 600.

[0080] Although not described in the above embodiments (particularly the embodiments of FIGS. 20 to 22), the present system can display the change in each value as a curve to make the change in each value easier to understand (hereinafter referred to as "fitting"). Specifically, the present system may perform fitting using any method, such as kernel density estimation. Note that when performing fitting, the present system may perform processing such as logarithmic conversion to bring the data closer to a normal distribution. Furthermore, the present system may not need to perform fitting, for example, when there is a large amount of missing data or when there is insufficient data.

[0081] Although not described in the above embodiments (particularly the embodiments of FIGS. 20 to 22), the various values ​​described above (e.g., "flow value," "stress value," "dazedness value," etc.) may be aggregated for each predetermined time period to calculate a value for each time period (e.g., "flow value," "stress value," "dazedness value," etc.). Specifically, the present system may calculate a value for each time period (e.g., "flow value," "stress value," "dazedness value," etc.) by defining, for example, 9:00 to 14:00 as morning, 14:00 to 18:00 as afternoon, and 18:00 to 21:00 as evening, etc.

[0082] Furthermore, although not explained in the above-described embodiments (particularly the embodiments of Figures 20 to 22), the present system may not simply calculate the various values ​​described above (e.g., "flow value," "stress value," "dazed value," etc.) for each day, but may instead aggregate the values ​​for each day of the week and calculate values ​​for each day of the week (e.g., "flow value," "stress value," "dazed value," etc.).

[0083] Furthermore, although not explained in the above-described embodiments (particularly the embodiments of Figures 20 to 22), the present system may classify subjects by type, for example, using various calculated values ​​(e.g., "flow value," "stress value," "vagueness value," etc.). Specifically, the system may classify subjects into types based on the tendencies of "flow value," "stress value," and "dazed value" for each time period and each day of the week, such as people who tend to concentrate easily in the morning, people who tend to accumulate stress in the afternoon, people who tend to feel sleepy in the evening, people who tend to get stressed easily on Mondays, and people who tend to concentrate easily on Fridays.

[0084] Furthermore, although not described in the above-described embodiments (particularly the embodiments of FIGS. 20 to 22), the results of estimations and calculations by the present system may significantly reflect individual differences. Therefore, the present system may, for example, derive a reference "flow value," "stress value," "absentmindedness value," etc. for each subject (hereinafter, the reference value for each value will be referred to as a "reference value") and classify the subject's type based on the change in value from the reference value. The present system may simply use the reference value calculated for each individual as is for classification, or may, for example, statistically process reference values ​​derived from a large number of subjects to derive a more general reference value for classification.

[0085] Although not described in the above embodiments (particularly the embodiments of FIGS. 20 to 22), the present system may classify the type of subject in the following manner, for example. (1) For example, the system may derive the area for each of the “flow value,” “stress value,” and “dazed value” for each time period (morning, afternoon, and evening) on ​​each day of the week after fitting, and compare it with the reference value for each time period. (2) For example, the system may determine whether the area of ​​each time period for each of the "flow value," "stress value," and "drowsiness value" is greater or smaller than a reference value, and classify the subject's type based on each pattern. In other words, the system can estimate the subject's concentration-related state, stress-related state, and sleepiness-related state type based on the level of each of the "flow value," "stress value," and "drowsiness value" indices for each day of the week and each time.

[0086] Although not described in the above embodiments (particularly the embodiments of FIGS. 20 to 22), the present system may classify the type of the subject based on, for example, changes in the subject's condition over an arbitrary period of time. Specifically, the system may classify, for example, a person who tends to have a high "vagueness value" in the morning and a high "flow value" in the evening as a "slow starter." The system may also classify, for example, a person who tends to have a high "flow value" in the morning and a high "vagueness value" in the evening as a "start dash type." In this way, the system may classify the type of the subject based on changes in the subject's condition throughout the day. Specifically, the system may classify, for example, a person whose "flow value" is high in the first half of the week but whose "absentee value" or "stress value" tends to increase in the second half of the week as a "shortness of breath" or "burnout" type. The system may classify the type of the subject in this way based on changes in the subject's condition over the course of a week. By providing advice and various information according to the type of subject thus classified, the system can enable the subject to efficiently perform various tasks while taking into consideration the subject's individual characteristics. Specifically, for example, if the subject's "absentee value" or "stress value" is high during a certain time period, the system may provide information encouraging the subject to take a break or change of mood at that time. Conversely, if the subject's "flow value" is high during that time period, the system may suggest performing difficult tasks during that time period.

[0087] The above-described series of processes can be executed by hardware or software. In other words, the functional configurations in FIG. 3 and the like are merely examples and are not particularly limited. That is, it is sufficient that the information processing system is provided with a function that can execute the above-described series of processes as a whole, and the type of functional block used to realize this function is not limited to the example shown in Fig. 3. Furthermore, the location of the functional block is not limited to the example shown in Fig. 3, and may be arbitrary. Furthermore, one functional block may be configured by hardware alone, by software alone, or by a combination of these.

[0088] Furthermore, the number of various hardware components constituting this system and the number of users are optional, and the system may also include other hardware components.

[0089] Furthermore, when a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium.

[0090] The computer may also be a computer that is built into dedicated hardware, or a computer that can execute various functions by installing various programs. That is, for example, the various hardware in the above-described embodiments may be freely adopted as any computer, any mobile terminal such as a smartphone, etc. Furthermore, any combination of types and contents of various input units and various output units may be adopted.

[0091] Furthermore, the recording medium containing such a program may not only be constituted by a removable medium (not shown) provided separately from the device main body in order to provide the program to the user, etc., but may also be constituted by a recording medium that is provided to the user in a state where it is pre-installed in the device main body.

[0092] Furthermore, in this specification, each step in a program stored on a storage medium does not necessarily have to be processed chronologically in the order described in the above embodiment, but may be processed in a different order, or some processing may be omitted and only some processing may be executed in parallel or individually.

[0093] In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices or a plurality of means.

[0094] The effects of this embodiment can be achieved even when these other embodiments are adopted. Furthermore, this embodiment can be combined with other embodiments, and other embodiments can be combined with each other as appropriate.

[0095] To summarize the above, a program to which the present disclosure is applied can take various forms having the following configurations. That is, an information processing system to which the present disclosure can be applied is An information processing system that can be used to obtain information about a subject, Image information acquisition means (e.g., image acquisition unit 80) for acquiring image information relating to an image including the subject's skin; A luminance information acquisition means (e.g., a PPG processing unit 81) for acquiring luminance information of the subject's skin included in the image; a first estimation means (e.g., a heart rate estimation unit 82) for estimating a heart rate of the subject based on the luminance information; a second estimation means (e.g., an index estimation unit 83) that estimates a value of an index related to the subject's heart rate based on the luminance information; It can be equipped with:

[0096] In addition, the above-mentioned information processing system may further include a third estimation means (e.g., a state estimation unit 84) that estimates the subject's state of concentration or stress based on the heart rate estimated by the first estimation means and the value of the index estimated by the second estimation means.

[0097] Furthermore, the above-described information processing system may further include a fourth estimation means (for example, a drowsiness estimation unit 85) that estimates a state related to drowsiness of the subject based on the heart rate estimated by the first estimation means.

[0098] In addition, the above-mentioned information processing system may further include a presentation means for presenting to the subject the estimated results of the subject's concentration or stress state estimated by the third estimation means and the estimated results of the subject's drowsiness state estimated by the fourth estimation means.

[0099] The luminance information acquisition means may acquire RGB information relating to time-series changes in RGB included in the image information, and acquire a PPG signal based on the acquired RGB information.

[0100] In addition, the third estimation means may estimate the subject's state of concentration or stress based on values ​​obtained by normalizing or standardizing the values ​​of the heart rate estimated by the first estimation means and the index estimated by the second estimation means using a predetermined method.

[0101] The third estimation means may also estimate whether the subject is in a state of concentration, a state of feeling stress, or a state of neither concentration nor stress.

[0102] The fourth estimation means may estimate whether the subject is in a state where he or she is continuously feeling sleepy.

[0103] In addition, the above-mentioned information processing system may further include an evaluation calculation means for acquiring an estimation result of the subject's concentration or stress state estimated by the third estimation means and an estimation result of the subject's drowsiness state estimated by the fourth estimation means during a predetermined period, and calculating a comprehensive evaluation index of the subject during that period according to the frequency of each estimation result.

[0104] The information processing system may further include a correction unit that performs correction when acquiring the luminance information if the luminance information acquired by the luminance information acquisition unit does not satisfy a predetermined condition.

[0105] The information processing system may further include a first determination means for determining a type of the subject's concentration-related state based on an estimation result of the subject's concentration-related state for a predetermined period of time.

[0106] The information processing system may further include a second determination means for determining a type of stress-related condition of the subject based on an estimation result of the stress-related condition of the subject for a predetermined period of time.

[0107] The information processing system may further include a third determination means for determining a type of the subject's sleepiness state based on an estimation result of the subject's sleepiness state for a predetermined period of time.

[0108] Furthermore, an information processing method according to another aspect of the present disclosure includes: An information processing method executed by an information processing device that can be used to acquire information about a subject, an image information acquisition step of acquiring image information regarding an image including the subject's skin; a brightness information acquisition step of acquiring brightness information of the subject's skin included in the image; a first estimation step of estimating a heart rate of the subject based on the luminance information; a second estimation step of estimating a value of an index related to the subject's heart rate based on the luminance information; may include:

[0109] In addition, a program according to another aspect of the present disclosure includes: An information processing device that can be used to acquire information about a target person, an image information acquisition step of acquiring image information regarding an image including the subject's skin; a brightness information acquisition step of acquiring brightness information of the subject's skin included in the image; a first estimation step of estimating a heart rate of the subject based on the luminance information; a second estimation step of estimating a value of an index related to the subject's heart rate based on the luminance information; It is possible to execute a process including the above. [Explanation of symbols]

[0110] 1 Analysis device 11 Analysis terminal 12 Camera 21 Control section 80 Image acquisition unit 81 PPG processing unit 100 Skin detection unit 101 RGB acquisition part 102 PPG signal acquisition section 103 Correction unit 82 Heart rate estimation unit 120 Power spectrum acquisition unit 121 Heart rate acquisition unit 83 Indicator estimation part 140 Peak detector 141 PPI acquisition department 142 Indicator acquisition part 84 State Estimation Unit 85 Drowsiness estimation unit 86 Data transmission / reception unit 87 General Evaluation Department 88 Results presentation section 2 Server 210 Control Unit 240 Data transmission and reception unit 300 Analysis result DB

Claims

1. An information processing system that can be used to obtain information about a subject, image information acquisition means for acquiring image information relating to an image including the subject's skin; a brightness information acquisition means for acquiring brightness information of the subject's skin included in the image; a first estimation means for estimating a heart rate of the subject based on the luminance information; a second estimation means for estimating a value of an index related to the heart rate of the subject based on the luminance information; An information processing system comprising:

2. and a third estimation means for estimating a state of concentration or stress of the subject based on the heart rate estimated by the first estimation means and the value of the index estimated by the second estimation means. The information processing system according to claim 1 .

3. The device further includes a fourth estimation means for estimating a state of drowsiness of the subject based on the heart rate estimated by the first estimation means. The information processing system according to claim 2 .

4. a presentation means for presenting to the subject an estimation result of the state of concentration or stress of the subject estimated by the third estimation means and an estimation result of the state of drowsiness of the subject estimated by the fourth estimation means, The information processing system according to claim 3 .

5. the luminance information acquisition means acquires RGB information relating to time-series changes in RGB included in the image information, and acquires a PPG signal based on the acquired RGB information; The information processing system according to claim 3 .

6. the third estimation means estimates a state of concentration or stress of the subject based on values ​​obtained by normalizing or standardizing the values ​​of the heart rate estimated by the first estimation means and the index estimated by the second estimation means using a predetermined method; The information processing system according to claim 3 .

7. The third estimation means estimates whether the subject is in a state of concentration, a state of feeling stress, or a state of neither concentration nor stress. The information processing system according to claim 6.

8. The fourth estimation means estimates whether the subject is in a state where he or she is continuously feeling sleepy. The information processing system according to claim 3 .

9. The system further includes an evaluation calculation means for acquiring an estimation result of the subject's state related to concentration or stress estimated by the third estimation means and an estimation result of the subject's state related to drowsiness estimated by the fourth estimation means during a predetermined period, and calculating a comprehensive evaluation index of the subject during the period according to the frequency of each estimation result. The information processing system according to claim 3 .

10. The image processing device further includes a correction unit that corrects the luminance information when the luminance information acquired by the luminance information acquisition unit does not satisfy a predetermined condition. The information processing system according to claim 3 .

11. and a first determination means for determining a type of the state related to concentration of the subject based on an estimation result of the state related to concentration of the subject for a predetermined period of time. The information processing system according to claim 3 .

12. further comprising a second determination means for determining a type of the stress-related state of the subject based on the estimation result of the stress-related state of the subject for a predetermined period of time; The information processing system according to claim 3 .

13. and a third determination means for determining a type of the subject's drowsiness state based on an estimation result of the subject's drowsiness state for a predetermined period of time. The information processing system according to claim 3 .

14. An information processing method executed by an information processing device that can be used to acquire information about a subject, an image information acquisition step of acquiring image information regarding an image including the subject's skin; a brightness information acquisition step of acquiring brightness information of the subject's skin included in the image; a first estimation step of estimating a heart rate of the subject based on the luminance information; a second estimation step of estimating a value of an index related to the subject's heart rate based on the luminance information; An information processing method including:

15. An information processing device that can be used to acquire information about a target person, an image information acquisition step of acquiring image information regarding an image including the subject's skin; a brightness information acquisition step of acquiring brightness information of the subject's skin included in the image; a first estimation step of estimating a heart rate of the subject based on the luminance information; a second estimation step of estimating a value of an index related to the subject's heart rate based on the luminance information; A program that executes processing including

Images