Organism detection system, method and program

Abstract

To provide an organism detection system capable of detecting preferably, an autonomic nerve activity.SOLUTION: An organism detection system comprises: an average value and weighting coefficient calculation part 105 for acquiring moving image data of the face image of a subject, classifying respective frames in the moving image data into 2N range (N is an integer of 2 or greater) on the basis of an addition value of an original color RGB component in respective pixels in the frames, excluding an outlier (the top 25% and the bottom 25%) in the respective ranges, and in the condition, seriously considering ranges which match following three conditions that, (1) the number of pixel data is great, (2) dispersion of a difference component value is small, (3) the average of the difference component value is constant regardless of the addition value of the original color RGB component; a centroid value and matching degree calculation part 106 for calculating a value (centroid value) which is obtained by multiplying a weighting coefficient to the average value of the difference component acquired in each range, and a matching degree obtained by calculating the average value of weighting to the set condition; a smoothing processing part for smoothing the centroid value acquired in each frame time-variantly; and a detection part for detecting a blood flow or an activity based on the blood flow.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a biological detection system, a biological detection method, and a program for detecting a biological state such as the autonomic nerve activity of a subject without contact.

Background Art

[0002] In recent years, the need for emotion estimation has been increasing in order for people to live a more comfortable life, such as measuring the fatigue and concentration of workers in the workplace and striving to improve the workplace environment by incorporating appropriate breaks. In measuring fatigue and concentration, various methods of acquiring biological information such as heart rate, pulse wave, electroencephalogram, eye movement, and facial thermal image and performing emotion estimation have been studied so far. In particular, the method using a facial thermal image can be estimated using a far-infrared camera, which is a non-contact device different from other methods, so that it can be measured without giving stress to the subject (target person) and without restricting the behavior of the subject. Therefore, it is considered a useful method.

[0003] As a biological detection system applied to the detection of such autonomic nerve activity, the inventor of the present application previously made the invention described in Patent Document 1. The invention described in Patent Document 1 acquires a difference component (R - B component) between the red component (R) and the blue component (B) of the nose part, which is easily affected by the sympathetic nerve, from an image obtained by photographing the face, and analyzes the time-series data of the R - B component to detect autonomic nerve activity.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technology described in Patent Document 1, autonomic nerve activity can be detected by analyzing the nose part of an image obtained by photographing a subject's face, and measurement can be performed without stressing the subject and without restricting the subject's behavior. The technology of this Patent Document 1 quantitatively extracts a range in which the average value of the R - B difference component is stable over time series. Thereby, autonomic nerve activity can be detected, but depending on the detection state, the detection accuracy may not be high.

[0006] Specifically, when extracting a range in which the average value of the R - B difference component is stable over time series, even if the extracted range is stable in terms of data, there is a possibility that a range where the data is small and aggregated may be extracted. When the data is small and aggregated, it is conceivable that there is not enough data volume to estimate autonomic nerve activity, and in such a case, there is a risk of leading to a decrease in the detection accuracy of autonomic nerve activity.

[0007] In view of such a point, an object of the present invention is to provide a biological detection system, a biological detection method, and a program capable of detecting autonomic nerve activity better than before.

Means for Solving the Problems

[0008] The biological detection system of the present invention acquires moving image data of a face image obtained by photographing the face of a subject, and for each frame of the acquired moving image data, it divides into a 2N range (N is an integer of 2 or more) based on the sum value of the primary color RGB components of each pixel in the frame, excludes outliers of the difference component values (for example, the upper 25% and the lower 25%) within each range, and then focuses on a range that meets the following three conditions: (1) there are many pixel data within the range, (2) the variance of the difference component values is small, and (3) the average of the difference component values is constant regardless of the sum value of the primary color RGB components. It includes a weight coefficient calculation unit, a center of gravity value and fitness calculation unit that calculates a value (center of gravity value) obtained by multiplying and adding the average value of the difference components of the primary color RGB components obtained in each range by the weight coefficient, and the fitness obtained by calculating the average value of the weights for the set conditions, a smoothing processing unit that temporally smooths the center of gravity values obtained in each frame of the moving image data, and a detection unit that detects blood flow or an activity based on the blood flow. For calculating the weight coefficient of the range where the average of the difference component values in (3) is constant regardless of the sum value of the primary color RGB components, the difference components are further divided into m or more small ranges (m is an integer of 3 or more) within each range divided based on the sum value of the primary color RGB components, and the weight coefficient is calculated based on the F value (or P value) obtained by analyzing the variance within the m or more divided small ranges.

[0009] Further, the biological detection method of the present invention is a biological detection method in which an information processing device performs arithmetic processing based on moving image data of a face image obtained by photographing the face of a subject to detect blood flow of the subject or an activity based on the blood flow. As an arithmetic process in the biological detection method performed by the information processing apparatus, moving image data of a face image obtained by photographing the face of a subject is acquired. For each frame of the acquired moving image data, it is divided into a 2N range (N is an integer of 2 or more) based on the sum value of the primary color RGB components in the frame. After excluding outliers (e.g., the upper 25% and the lower 25%) of the difference component values within each range, a weight coefficient calculation unit that emphasizes a range that meets the following three conditions: (1) a large number of pixel data within the range, (2) a small variance of the difference components, and (3) the average of the difference component values being constant regardless of the sum value of the primary color RGB components; a center-of-gravity value and fitness calculation unit that calculates a value obtained by multiplying and adding the average value of the difference components of the primary color RGB components obtained in each range by the weight coefficient, and the fitness obtained by calculating the average value of the weights for the set conditions; a smoothing processing unit that temporally smooths the center-of-gravity value obtained in each frame of the moving image data; and a detection unit that detects blood flow or an activity based on the blood flow are provided. (3) For calculating the weight coefficient of the range where the average of the difference component values is constant regardless of the sum value of the primary color RGB components, the difference components are further divided into m or more small ranges (m is an integer of 3 or more) within each range divided based on the sum value of the primary color RGB components, and the weight coefficient is calculated based on the F value (or P value) obtained by analyzing the variance within the m or more divided small ranges.

[0010] Also, the program of the present invention causes a computer to execute each process of the biological detection method as a procedure.

Effects of the Invention

[0011] According to the present invention, a process for accurately detecting blood flow in blood vessels at a specific location or an activity based on the blood flow from a moving image of a photographed face is performed. In particular, since there is a high possibility of extracting sufficient data for estimating autonomic nerve activity, it becomes possible to detect autonomic nerve activity with high accuracy.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0013] Hereinafter, an exemplary embodiment of the present invention (hereinafter referred to as "this example") will be described with reference to the accompanying drawings. [System Configuration] FIG. 1 is a functional block diagram showing the configuration of the processing performed by the biological detection system 100 of this example. The biological detection system 100 of this example detects autonomic nerve activity and blood flow. The biological detection system 100 of this example captures the image data captured by the camera 1. The image data is an image of the face of a subject for detecting the blood flow of blood vessels at a specific location or an activity based on the blood flow. The image data captured by the camera 1 is moving image data at a certain frame rate, and the image data of each frame is composed of pixel data of the primary colors of red (R), green (G), and blue (B). The frame rate of the image data is, for example, 30 frames / second, and each pixel in each frame indicates the luminance value of red (R), green (G), or blue (B) at a predetermined gradation (for example, 256 gradations).

[0014] The image data captured by the biological detection system 100 is supplied to the RGB component acquisition unit 101, and the red component, the green component, and the blue component are individually acquired. The red component, the green component, and the blue component obtained by the RGB component acquisition unit 101 are respectively supplied to the differential component processing unit 102.

[0015] The differential component processing unit 102 calculates the differential (R-G, R-B, G-B) components for each pixel in each frame for the acquired moving image data, and both those values and the addition value of the primary color RGB components are respectively supplied to the range division processing unit 103.

[0016] The range division processing unit 103 divides the differential (R-G, R-B, G-B) components calculated for each pixel in each frame for the acquired moving image data into 2N ranges (N is an integer of 2 or more) based on the addition value of the primary color RGB components, and the differential (R-G, R-B, G-B) components divided for each range are supplied to the outlier processing unit 104.

[0017] The outlier processing unit 104 sorts the differential (R-G, R-B, G-B) components within each range for the acquired moving image data for each frame, excludes the outliers (the upper 25% and the lower 25%), and the differential (R-G, R-B, G-B) components from which the outliers have been excluded for each range are supplied to the average value and weight coefficient calculation unit 105.

[0018] For each frame of the acquired moving image data, the average value calculation unit 105 calculates, within each range, the average value of the difference components (R - G, R - B, G - B) excluding outliers, and the weighting coefficient to be multiplied by the average value. The calculated average value and weighting coefficient are supplied to the barycentric value and fitness calculation unit 106. The weighting coefficient within each range is calculated so as to emphasize the range that satisfies the following three conditions: (1) a large number of pixel data, (2) a small variance of the difference component values, and (3) the average of the difference component values being constant regardless of the addition value of the primary color RGB components.

[0019] For each frame of the acquired moving image data, the barycentric value and fitness calculation unit 106 performs a barycentric value and fitness calculation process of multiplying the average value of the difference components (R - G, R - B, G - B) excluding outliers within each range by the weighting coefficient and adding them for all ranges, and calculating the fitness obtained by calculating the average value of the weights for the set conditions. Then, the calculated barycentric value and fitness are supplied to the smoothing processing unit 107.

[0020] The smoothing processing unit 107 performs a process of smoothing the barycentric value and fitness calculated for each frame of the acquired moving image data at regular intervals. It is preferable to select the regular interval for smoothing within the range of about 5 seconds to 1 minute. For example, the smoothing processing unit 107 smooths every 20 seconds or every 1 minute.

[0021] The time - series data of the (R - B) component and its fitness of the nose region smoothed by the smoothing processing unit 107 are supplied to the detection unit 108. The detection unit 108 performs a detection process of the stress state of the subject based on the variation status of the time - series data of the (R - B) component of the nose region and its fitness. The detection result (evaluation result) of the stress state detected by the detection unit 108 is output from the output unit 109.

[0022] [Hardware Configuration Example of the Biological Detection System] FIG. 2 shows a hardware configuration example when the biological detection system 100 is configured by a computer which is an information processing device. The computer (biological detection system 100) shown in FIG. 2 includes a CPU (Central Processing Unit) 100a, a main memory unit 100b, a non-volatile storage 100c, a network interface 100d, an image input unit 100e, an output unit 100f, and an operation unit 100g, each connected to a bus.

[0023] The CPU 100a is an arithmetic processing unit that reads and executes program codes of software that realizes the functions performed by the biological detection system 100 from the main memory unit 100b or the non-volatile storage 100c. When the CPU 100a reads program codes from the main memory unit 100b or the non-volatile storage 100c and executes arithmetic processing in the work area of the main memory unit 100b, various processing functional units are configured in the main memory unit 100b. For example, in the main memory unit 100b, an RGB component acquisition unit 101, an averaging processing unit 102, a similar range extraction unit 103, and a detection unit 104 shown in FIG. 1 are configured.

[0024] For the non-volatile storage 100c, for example, a large-capacity information storage medium such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a memory card is used. In the non-volatile storage 100c, software that realizes the functions of the biological detection system 100 and data obtained by the execution of the program are stored.

[0025] For the network interface 100d, for example, a NIC (Network Interface Card) or the like is used, and data is transmitted and received with other devices. The image input unit 100e performs input processing (image acquisition processing) of image data from the camera 1. A display 2 is connected to the output unit 100f, and data such as an image showing the detection result of autonomic nerve activity such as the stress evaluation result is output from the output unit 100f. Then, these detection results and evaluation results are displayed on the display 2. Note that the detection results and evaluation results may be transmitted to an external device via the network interface 100d. The operation unit 100g accepts operations of devices such as a keyboard and a mouse that are operated by an operator of this computer.

[0026] [Flow of biological detection processing] FIG. 3 is a flowchart showing the flow when the biological detection system 100 in this example performs biological detection processing such as autonomic nerve activity with the configuration shown in FIG. 1. First, the image input unit 100e (FIG. 2) of the biological detection system 100 captures moving image data of the face of the subject photographed by the camera 1 (step S11). Then, the captured moving image data is supplied to the RGB component acquisition unit 101 for each frame, and the primary color component values of red (R), green (G), and blue (B) are individually acquired (step S12).

[0027] The primary color component values of red (R), green (G), and blue (B) acquired by the RGB component acquisition unit 101 are supplied to the differential component processing unit 102 for each frame, and for the nose region which is a specific region, differential (R - G, R - B, G - B) component processing of the entire pixels of the color components of the primary colors is performed for each pixel (step S13).

[0028] The differential (R - G, R - B, G - B) components processed by the differential component processing unit 102 are supplied to the range division processing unit 103 for each frame, and range division processing is performed to divide into 2N ranges (N is an integer of 2 or more) based on the addition value of the primary color RGB components (step S14).

[0029] The differential (R - G, R - B, G - B) components range - divided by the range division processing unit 103 are supplied to the outlier processing unit 104 for each frame, and within each range, the differential (R - G, R - B, G - B) components are sorted, and outlier processing to exclude outliers (upper 25% and lower 25%) is performed (step S15).

[0030] The differential (R-G, R-B, G-B) components that have undergone outlier processing in the outlier processing unit 104 are supplied to the average value and weight coefficient calculation unit 105 for each frame, and within each range, the average value of the differential (R-G, R-B, G-B) components excluding outliers and the weight coefficient to be multiplied by the average value are calculated (step S16). The weight coefficients within each range are calculated by emphasizing the range that meets the following three conditions: (1) a large number of pixel data, (2) a small variance of the differential component values, and (3) the average of the differential component values being constant regardless of the additive values of the primary color RGB components.

[0031] The differential (R-G, R-B, G-B) components for which the average value and weight coefficient have been calculated in the average value and weight coefficient calculation unit 105 are supplied to the centroid value and fitness calculation unit 106 for each frame. The centroid value is calculated by multiplying the average value of the differential (R-G, R-B, G-B) components within each range by the weight coefficient and adding them for all ranges, and the fitness calculation is performed by calculating the average value of the weights for the set conditions (step S17).

[0032] The differential (R-G, R-B, G-B) components for which the centroid value has been calculated in the centroid value calculation unit 106 are supplied to the smoothing processing unit 107 for each frame, and the centroid value and fitness calculated for each frame are smoothed at regular intervals for the acquired moving image data (step S18).

[0033] The differential (R-G, R-B, G-B) components that have been smoothed at regular intervals in the smoothing processing unit 107 are supplied to the detection unit 108, and based on the variation status and fitness of the time-series data of the (R-B) component in the nose region, the detection process of the stress state of the subject is performed (step S19). The regular interval for smoothing is preferably selected between about 20 seconds and 1 minute. For example, the smoothing processing unit 107 smooths every 20 seconds or every 1 minute.

[0034] The time-series data and the degree of fitness of the (R - B) component of the nose region smoothed by the smoothing processing unit 107 are supplied to the detection unit 108. The detection unit 108 performs detection processing of the stress state of the subject based on the variation status and the degree of fitness of the time-series data of the (R - B) component of the nose region. The detection result (evaluation result) of the stress state detected by the detection unit 108 is output from the output unit 109.

[0035] [Flow of weight coefficient calculation processing] FIG. 4 is a flowchart showing the details of the weight coefficient calculation processing (step S16) performed by the average value and weight coefficient calculation unit 105. The weight coefficients within each range are calculated by emphasizing the range that meets the following three conditions: (1) a large number of pixel data, (2) a small variance of the difference component values, and (3) the average of the difference component values being constant regardless of the added value of the primary color RGB components. To calculate the weight of (1), the number of data within the range is acquired (step S211), and the weight W1 is calculated (step S212). At the same time, to calculate the weight of (2), the variance within the range is acquired (step S221), and the weight W2 is calculated (step S222). Further simultaneously, to calculate the weight of (3), the F value = (mean square between divided regions) / (mean square within divided regions) obtained by one-way analysis of variance when the range is divided into m parts is acquired (step S231), and the weight W3 is calculated (step S232). In this operation, when the mean square between the divided regions is small, it means that the average values between the regions are similar, and the range where the average of the difference component values is constant regardless of the added value of the primary color RGB components can be known. Based on the combined weight WA obtained by multiplying the weights calculated above (step S24), the weight coefficient A required for the centroid calculation is calculated (step S25).

[0036] [Explanation of each process for detecting autonomic nerve activity] FIG. 5 is a diagram explaining the difference in the penetration depth of the wavelengths of each primary color into human skin. As shown in Fig. 5, in the order of decreasing penetration depth into the skin, there are the red (R) component, the green (G) component, and the blue (B) component. That is, for the blue component, the light reflected from the epidermis of the skin is captured as an image taken by camera 1. Also, for the green component, the light reflected from the dermis deeper than the epidermis is captured as an image taken by camera 1. Capillaries are arranged at this depth of the dermis. Furthermore, for the red component, the light reflected from a location in the subcutaneous tissue deeper than the dermis is captured as an image taken by camera 1. Arteries and veins are arranged at this location in the subcutaneous tissue.

[0037] Here, in the detection unit 108 of this example, by obtaining the (R - B) component obtained by taking the difference between the red (R) component and the blue (B) component, it is considered that the (R - B) component contains information on the entire blood vessel excluding the luminance component. In addition, when obtaining the (G - B) component obtained by taking the difference between the green (G) component and the blue (B) component, it is considered that the information on the entire capillary excluding the luminance component is included. In the following description, an example of obtaining the (R - B) component to obtain information on the entire blood vessel will be described, but it is also possible to obtain the (G - B) component to obtain information on the entire capillary and perform stress evaluation and the like. However, as described below, it is preferable to obtain the (R - B) component to obtain information on the entire blood vessel.

[0038] Therefore, it is considered that the detection unit 108 can obtain information on the blood flow volume of the entire blood vessel by obtaining the (R - B) component. In the biological detection system 100 of this example, in this way, information on the blood flow volume of the entire blood vessel is obtained, and from the change in this blood flow volume, estimation (detection) of the autonomic nerve activity and the like of the subject (test subject) is performed.

[0039] Regarding the acquisition of blood flow in blood vessels in more detail, light that has penetrated deep into the skin is absorbed in the blood through blood vessels, and the unabsorbed portion is considered to appear on the face surface as reflected light. Therefore, assuming that the values of each color component obtained from the captured actual image are the amounts of reflected light, it is necessary to know how much has been absorbed in the blood. It is considered that the component that absorbs light in the blood is hemoglobin. Based on this idea, it is possible to capture the characteristics of blood that absorbs more light from the absorption rates of hemoglobin for each color of light.

[0040] For example, in the wavelength range of 610 nm to 780 nm, which is the wavelength of red light, deoxygenated hemoglobin (Hb) has a higher absorption rate than oxygenated hemoglobin (HbO2). From this, incident red light is not absorbed much when the blood flow from arteries is high, and the amount of reflected light increases. On the other hand, when the blood flow from veins is high, it is more absorbed and the reflected light decreases.

[0041] Also, in the wavelength range of 500 nm to 570 nm, which is the wavelength of green light, both oxygenated hemoglobin and deoxygenated hemoglobin have approximately the same absorption rate. Regarding capillaries, it is considered that the total blood flow flowing through the capillaries is measured, rather than the ratio of blood flow from arteries and blood flow from veins. When the blood flow in capillaries increases, it is considered that the absorption of incident green light increases and the reflected amount decreases. Conversely, when the blood flow in capillaries decreases, the absorption amount decreases and the reflected amount increases. Utilizing these principles, in the case of this example, the detection unit 108 calculates the (R - B) component to detect the blood flow when looking at the entire blood vessel and estimates (detects) autonomic nerve activity.

[0042] FIG. 6 is a diagram for explaining the process in which the RGB component acquisition unit 101 extracts the nose region, which is a specific region, from the face image. When the RGB component acquisition unit 101 acquires the face image of each frame, based on the data of the face shape prepared in advance, as shown in FIG. 7, a plurality of landmarks a indicating the contour position of the cheek of the face, a landmark b indicating the position of the lips, a landmark c indicating the position of the eyes, a landmark d indicating the position of the eyebrows, and a landmark e indicating the position of the nose are respectively set. Then, the RGB component acquisition unit 101 determines the nose region f from the positions of the set landmarks a to e, and performs a process of extracting the color data of the pixels within this nose region f.

[0043] Thereby, the RGB component acquisition unit 101 extracts a specific color range of the nose region of the face image, and for the extracted nose region, obtains the (R - B) component already described. Also, the RGB component acquisition unit 101 also obtains the value obtained by summing the total component values of the red (R) component, green (G) component, and blue (B) components.

[0044] FIG. 7 is a diagram showing the distribution of each pixel of the image of the nose part obtained in one frame of a single measurement, with the horizontal axis being the component value of (R + G + B) and the vertical axis being the (R - B) component value. As shown in FIG. 7 (when 2N = 8), for each frame, the range division processing unit 103 plots the values of each pixel in the nose region with the horizontal axis being the value of (R + G + B) and the vertical axis being the value of the (R - B) component, and divides it into 8 divisions into ranges x1 to x8 based on the value of (R + G + B).

[0045] Also, as shown in FIG. 7 (when 2N = 8), for each frame, the outlier processing unit 104 performs a process of removing the upper 25% value and the lower 25% value for each of the ranges x1 to x8 of the difference component values of the 8 - divided ranges x1 to x8.

[0046] Furthermore, as shown below FIG. 7 (when 2N = 8), for each frame, the average value and weight coefficient calculation unit calculates the average value and weight coefficient of the difference component values of each of the 8 - divided ranges x1 to x8 for each of the ranges x1 to x8. In the average value and weight coefficient calculation unit 105, the average value and the weight coefficient are calculated for each of the ranges x1 to x8, and the weight coefficient within each range increases for a range that satisfies the following three conditions: (1) the number of pixel data is large, (2) the variance of the difference component values is small, and (3) the average of the difference component values is constant regardless of the addition value of the primary color RGB components. Finally, the barycentric value and the degree of fitness (average of the weight WA) are calculated from the numerical values shown below Fig. 7.

[0047] [Example (Example 1) of detecting autonomic nerve activity from actual shooting data] Fig. 8 shows a state in which the detection unit 108 has detected autonomic nerve activity. That is, Fig. 8 shows a detection example of autonomic nerve activity when the measurement time is 800 seconds and the subject is made to perform mental arithmetic. The schedule for 800 seconds is shown above Fig. 8. In the example above Fig. 8, there is a first evaluation period before the start of the measurement time. When the measurement time starts, a rest period, a second evaluation period, a mental arithmetic execution period, a third evaluation period, and a rest period are provided in this order. Then, 800 seconds have elapsed at the end of the second rest period, and finally a fourth evaluation period is provided.

[0048] The characteristic d11 shown in Fig. 8 is a time-series change when the three processes of the current outlier processing, average value and weight coefficient calculation, and barycentric value and degree of fitness calculation are not performed. This characteristic d11 indicates that the width of the change is small and it is difficult to detect autonomic nerve activity. The characteristic d12 shown in Fig. 8 is a time-series change when only the outlier processing is performed and the other two processes are not performed. In the characteristic d12, changes corresponding to autonomic nerve activity appear more than in the characteristic d11.

[0049] Furthermore, the characteristic d13 shown in Fig. 8 is a time-series change when all the current processes are performed. In the case of the characteristic d13 in Fig. 8, it can be seen that the blood flow change in the nasal region is appropriately captured and the autonomic nerve activity of the subject is appropriately detected. That is, during the period when the subject is performing mental arithmetic, there are multiple locations where the (R-B) component value corresponding to autonomic nerve activity fluctuates, particularly significantly decreases, and the autonomic nerve activity in the state of stress caused by the subject's mental arithmetic execution is appropriately detected. Also, even during the rest period, it has been detected that the (R-B) component value increases, and the autonomic nerve activity during rest, that is, in the state without stress, is appropriately detected.

[0050] [Example of detecting autonomic nerve activity from actual shooting data (Example 2)] FIG. 9 shows the state of another example (Example 2) in which the detection unit 108 detects autonomic nerve activity.

[0051] The characteristic d21 shown in FIG. 9 is the time-series change when the three processes of the current outlier processing, average value and weight coefficient calculation, and centroid value and fitness calculation are not performed. This characteristic d21 indicates that the width of the change is small and it is difficult to detect autonomic nerve activity. And the characteristic d22 shown in FIG. 9 is the time-series change when only the outlier processing is performed and the other two processes are not performed. In the characteristic d22, changes corresponding to autonomic nerve activity appear more than in the characteristic d21.

[0052] Furthermore, the characteristic d23 shown in FIG. 9 is the time-series change when all the current processes are performed. Similar to the example shown in FIG. 8, the characteristic d23 in FIG. 9 appropriately captures the blood flow change in the nasal region and appropriately detects the autonomic nerve activity of the subject. That is, in the example of FIG. 9, the (R-B) component value gradually decreases due to the subject's execution of mental arithmetic, and during the rest period after the mental arithmetic execution, the (R-B) component value gradually increases. From this, it can be seen that the autonomic nerve activity in the state where the subject is under stress during the mental arithmetic period and the state where the stress has disappeared thereafter is appropriately detected.

[0053] As described above, according to the biological detection system 100 of this example, it is possible to accurately detect autonomic nerve activities such as stress applied to the subject. In this case, the detected image is image data obtained from the primary color signals captured by the normal camera 1. Therefore, special equipment such as an infrared camera, which was necessary in the past, becomes unnecessary, and there is an effect that accurate detection of autonomic nerve activities can be performed at low cost with an existing system. Moreover, as a component to be detected, since the (R - B) component, which is a component where blood flow is clearly understood, is used, a relatively wide range where the pixel value change is stable is extracted within that (R - B) component. And by using the (R - B) component, which is a component where blood flow is clearly understood, for detection, it is possible to use a relatively wide area restricted from being affected by external light, shadows, etc., and it becomes possible to stably and accurately detect autonomic nerve activities.

[0054] [Modification Example] In the description so far, as the system of this example, a biological detection system 100 for detecting autonomic nerve activities was used. On the other hand, instead of detecting autonomic nerve activities, it is also possible to use a biological detection system in which the detection unit 108 detects blood flow rate and the output unit 109 outputs the state of the detected blood flow rate (biological state). In the examples of FIGS. 8 and 9, the point of causing the subject (test subject) to perform mental arithmetic to detect autonomic nerve activities such as stress is also an example, and the present invention can detect autonomic nerve activities such as stress at that time by changing various environments and situations for the subject.

[0055] Even in the case of a biological detection system that outputs the state of blood flow rate (biological state), the point of using the (R - B) component, which is a component where blood flow is clearly understood, is the same as that of the biological detection system 100. As a result, there is a special effect that the state of blood flow can be detected and output more clearly than before. Also, the point of using an image of the nose for detecting the state of blood flow is an example, and blood flow may be detected and analyzed by the same process from an image of a biological site other than the nose.

[0056] In the above-described embodiment, the (R - B) component value, which is the difference between the red component and the blue component, is obtained, and the autonomic nerve activity and blood flow are detected from the (R - B) component value. However, the (G - B) component, which is the difference between the green component and the blue component, may be obtained, and the autonomic nerve activity and blood flow may be detected from the (G - B) component value. However, as described with reference to FIG. 5, when detecting from the (G - B) component value, the range of the blood flow that can be captured is closer to the capillary side, and the detection situation and detection accuracy are different from those based on the (R - B) component value.

[0057] In addition, the biological detection system 100 shown in FIG. 1 shows an example configured as a dedicated system for detecting autonomic nerve activity and blood flow. However, the biological detection system 100 of this example can be configured by, for example, the computer shown in FIG. 3. Therefore, by incorporating the program for operating as the biological detection system 100 of this example into an information processing device such as a computer or a smartphone, the process of detecting stress can be performed in parallel with the execution of various processes on the computer or smartphone, and it becomes possible to perform the stress evaluation of the target person such as a worker at any time. The program for operating as the biological detection system 100 of this example can be created by executing each process described in the flowcharts of FIGS. 3 and 4 as procedures. In addition, this program can be stored in a recording medium such as various memories, IC cards, SD cards, and optical disks.

Explanation of Reference Numerals

[0058] 1... Camera, 2... Display, 100... Biological detection system, 100a... CPU, 100b... Main memory unit, 100c... Non - volatile storage, 100d... Network interface, 100e... Image acquisition unit, 100e... Image input unit, 100f... Output unit, 100g... Operation unit, 101... RGB component acquisition unit, 102... Difference component processing unit, 103... Range division processing unit, 104... Outlier processing unit, 105... Average value and weighting coefficient calculation unit, 106... Center of gravity value and fitness calculation unit, 107... Smoothing processing unit, 108... Detection unit, 109... Output unit

Claims

1. Obtain moving image data of a face image obtained by photographing the face of a subject, and for each frame of the obtained moving image data, perform range division processing to divide it into a 2N range (N is an integer of 2 or more) based on the sum value of the primary color RGB components in the frame; An outlier processing unit that excludes upper and lower outlier values within each divided range; A weight coefficient calculation unit that calculates a weight coefficient that emphasizes a range that satisfies three conditions: (1) a large number of pixel data, (2) a variance of the difference component that is small enough, and (3) an average of the difference component values that is constant regardless of the sum value of the primary color RGB components, within each range; A centroid value and fitness calculation unit that calculates a centroid value, which is a value obtained by multiplying and adding the average value of the difference components of the primary color RGB components obtained in each range by the weight coefficient, and a fitness obtained by calculating the average value of the weights for preset conditions; A smoothing processing unit that temporally smooths the centroid value and the fitness obtained in each frame of the moving image data; A detection unit that detects blood flow or an activity based on the blood flow; and Outputs a detection value that best conforms to the set conditions A biological detection system.

2. After processing the outliers, calculate a weight coefficient that emphasizes a range that satisfies the above three conditions: (1) a large number of pixel data, (2) a small variance of the difference component, and (3) an average of the difference component values that is constant regardless of the sum value of the primary color RGB components, within the range. The biological detection system according to Claim 1.

3. Calculate a centroid value, which is a value obtained by multiplying and adding the average value of the difference components obtained in each range by the weight coefficient, and a fitness obtained by calculating the average value of the weights for the set conditions. The biological detection system according to Claim 2.

4. Divide the range in which the differential components are separated based on the sum value of the primary color RGB components into m or more (m is an integer of 3 or more) small ranges, and calculate the weight coefficient based on the F value or P value obtained by performing one-way analysis of variance on the m or more divided small ranges. The biological detection system according to claim 3.

5. The differential component of the color component of the primary color is the differential component between the red component and the blue component, or the difference between the green component and the blue component. The biological detection system according to claim 4.

6. The face image is the area of the nose of the face. The biological detection system according to claim 5.

7. A biological detection method in which an information processing device performs arithmetic processing based on moving image data of a face image obtained by photographing a face of a subject to detect blood flow of the subject or an activity based on the blood flow. As the arithmetic processing performed by the information processing device, Obtain moving image data of a face image obtained by photographing a face of a subject, and for each frame of the obtained moving image data, perform a range division process of dividing it into 2N ranges (N is an integer of 2 or more) based on the sum value of the primary color RGB components in the frame. An outlier process of excluding outliers of a predetermined ratio of upper and lower values within each range. After processing the outliers, a weight coefficient calculation that emphasizes the range that satisfies the three conditions of (1) a large number of pixel data, (2) a small variance of the differential component, and (3) a constant average of the differential component values regardless of the sum value of the primary color RGB components within the range. A centroid value and a fitness calculation process that calculate a centroid value, which is a value obtained by multiplying and adding the average value of the differential components of the primary color RGB components obtained in each range by a weight coefficient, and a fitness obtained by calculating the average value of the weights to the set conditions. Including a smoothing process of temporally smoothing the centroid value and fitness obtained in each frame of the moving image data. Output the detection value that best meets the set conditions. Biological detection method.

8. In a program that causes a computer to execute a process of detecting blood flow of a subject or an activity based on the blood flow, based on image data obtained by photographing the face of the subject, obtain moving image data of a face image obtained by photographing the face of the subject, and for each frame of the obtained moving image data, a range dividing procedure for dividing into a 2N range (N is an integer of 2 or more) based on the added value of the primary color RGB components in the frame; an outlier processing procedure for excluding upper and lower outlier values within each range; after processing the outliers, a weight coefficient calculation procedure that emphasizes a range that satisfies the following three conditions: (1) a large number of pixel data in the range, (2) a small variance of the difference components, and (3) the average of the difference component values is constant regardless of the added value of the primary color RGB components; a centroid value and fitness calculation procedure for calculating a value (centroid value) obtained by multiplying and adding the average value of the difference components of the primary color RGB components obtained in each range by a weight coefficient, and a fitness obtained by calculating the average value of the weights for the set conditions; a smoothing processing procedure for temporally smoothing the centroid value and fitness obtained in each frame of the moving image data, and is a program that causes a computer to execute; causes a computer to execute a process of outputting a detection value that most conforms to the set conditions Program.

Images