Processing device, biological information detection method, and program

Abstract

To detect biological information with high accuracy without restricting the living body of a subject.SOLUTION: A processing device includes: a light source that projects a plurality of dots formed by first light to a target; an image sensor that detects second light from the target that has been subjected to the projection of the plurality of dots and generates one or a plurality of image signals; and a processing circuit that determines whether the target is a living body based on the one or a plurality of image signals.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] The present disclosure relates to a processing device, a biological information detection method, and a program. [Background technology]

[0002] Heart rate, blood flow, blood pressure, blood oxygen saturation, etc. are widely used as basic parameters to judge the health condition of a human being. Such biological information related to blood is usually measured by contact type measuring devices. Contact type measuring devices restrict the subject's body, which causes discomfort to the subject, especially when continuous measurements are performed over a long period of time.

[0003] Various attempts have been made to simply measure basic biological information for determining a person's health condition. For example, Patent Document 1 discloses a method for detecting the heart rate without contact from image information such as a face captured by a camera. Patent Document 2 discloses a method for measuring blood oxygen saturation using a white light source and a laser light source, utilizing the laser Doppler effect of laser light scattered behind the surface of a living body. Patent Document 3 discloses a method for measuring blood oxygen saturation by using a normal color camera to eliminate the effects of ambient light. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-218507 A [Patent Document 2] Special Publication No. 2003-517342 [Patent Document 3] Special Publication No. 2014-527863 [Patent Document 4] Japanese Patent Application Publication No. 6-54836 [Patent Document 5] JP 2008-237244 A [Patent Document 6] JP 2002-200050 A [Non-patent literature]

[0005] [Non-Patent Document 1] Aoki, et al., "Non-contact, unrestrained sleeper respiratory monitoring system using near-infrared spot matrix illumination," IEEJ Transactions on Electronics, Information and Systems, June 1, 2004, 124(6), pp.1251-1258 [Non-Patent Document 2] Kuroda, et al., "Analysis of Facial Color and Facial Skin Temperature Due to Emotional Fluctuations and Their Facial Color Synthesis," Proceedings of the Human Interface Society, February 16, 1999, 1(1), 15-20 Summary of the Invention [Problem to be solved by the invention]

[0006] The above-mentioned conventional techniques have problems with the accuracy or stability of the acquired biometric information, or the ease of acquiring the biometric information. In particular, the instability of the measurement due to the movement of the body and the fluctuation of the environmental light has been a major problem for practical use.

[0007] The present disclosure provides a technique capable of stably detecting biological information with high accuracy without restraining the living body of a subject. [Means for solving the problem]

[0008] A biological information detection device according to an embodiment of the present disclosure includes: a first light source that projects a plurality of first dots formed by a first light onto an object including a biological body; A plurality of first dots for detecting second light from the object onto which the plurality of first dots are projected. an imaging device that includes a light detection cell and generates and outputs a first image signal representing a first image of the object onto which the plurality of first dots are projected; A first arithmetic circuit; A second arithmetic circuit; Equipped with.

[0009] The first image includes a plurality of pixels.

[0010] The first arithmetic circuit detects a first portion of the first image, which corresponds to the living body, using the first image signal.

[0011] The second arithmetic circuit calculates biometric information about the living body using data of pixels within the first portion of the first image, among the plurality of pixels.

[0012] The above general or specific aspects may be realized as an element, an apparatus, a system, a method, an integrated circuit, a computer program, a recording medium, or any combination thereof. Effect of the Invention

[0013] According to one aspect of the present disclosure, it is possible to detect biological information such as the heart rate, blood flow, or blood oxygen saturation of a living subject quickly and stably without restraining the living subject. Furthermore, according to another aspect of the present disclosure, it is also possible to determine the physical condition or emotional state of the living subject from the above-mentioned blood information. [Brief description of the drawings]

[0014] [Figure 1A] FIG. 1 is a diagram for explaining a basic concept of acquiring biometric information according to an embodiment of the present disclosure. [Figure 1B] FIG. 1 is a diagram for explaining characteristics of an image of a living body surface acquired by an imaging device; [Diagram 2] FIG. 1 is a diagram showing a configuration of a biological information detection device according to a first embodiment. [Figure 3A] FIG. 1 shows an example of the configuration of an imaging device according to a first embodiment and examples of output images and biometric information. [Figure 3B] FIG. 1 is a block diagram showing a configuration of a computer 20 according to a first embodiment. [Figure 3C] FIG. 1 is a diagram for explaining a method of respiration sensing in embodiment 1. [Figure 4A] FIG. 1 is a first diagram showing an experimental example of human body detection in the first embodiment; [Figure 4B] FIG. 1 is a first diagram showing an experimental example of human body detection in the first embodiment; [Diagram 5] FIG. 1 is a diagram showing an example of a method for calculating contrast used in human body detection in the first embodiment; [Figure 6] 1 is a flowchart showing the flow of image processing in the first embodiment. [Figure 7A] FIG. 13 is a diagram showing a schematic diagram of a biological information detection device and its processing according to a third embodiment. [Figure 7B] FIG. 13 is a diagram for explaining an algorithm of the monitoring system in the third embodiment. [Figure 7C] 11 is a flowchart showing an algorithm of a monitoring system according to a third embodiment. [Figure 8] FIG. 13 is a diagram showing a configuration of a biological information detection device according to a fourth embodiment. [Figure 9] FIG. 13 is a diagram showing an overview of biological information sensing using two image capturing devices in the fourth embodiment. [Figure 10] FIG. 13 is a diagram showing the transmission characteristics of two bandpass filters in the fourth embodiment. [Figure 11] FIG. 13 shows an example of a pulse wave measured by the method of embodiment 4. [Figure 12] FIG. 1 shows the results of measuring blood oxygen saturation using the method of embodiment 4 and a conventional method. [Figure 13] FIG. 13 is a diagram showing the configuration of a stereo camera type biological information detection device according to a fourth embodiment. [Figure 14] FIG. 13 is a diagram showing the configuration of a stereo lens type biological information detection device according to a fifth embodiment. [Figure 15A] FIG. 13 is a diagram showing the results of stress sensing performed using the biological information detection device of the sixth embodiment. [Figure 15B] FIG. 13 is a diagram showing the nose and cheeks in an image according to the sixth embodiment. [Figure 15C] FIG. 13 shows changes in blood flow rate and blood oxygen saturation obtained using the biological information detection device of embodiment 6. [Figure 16] FIG. 13 is a cross-sectional view illustrating a schematic configuration of a biological information detection device according to a seventh embodiment. [Figure 17A] FIG. 13 is a diagram showing the nose and forehead in an image according to the seventh embodiment. [Figure 17B]FIG. 13 is a diagram showing the time change in total blood flow (oxygenated hemoglobin and deoxygenated hemoglobin) and the time change in the proportion of blood flow of oxygenated hemoglobin (oxygen saturation) when the emotion of laughter is induced in the seventh embodiment. [Figure 18] A graph showing the relationship between emotion and total blood flow and oxygen saturation [Figure 19A] FIG. 23 is a diagram illustrating a configuration of a biological information detection device according to an eighth embodiment. [Figure 19B] FIG. 13 shows a plurality of color filters according to the eighth embodiment. [Figure 19C] FIG. 23 is a diagram showing an example of an image generated in the eighth embodiment. [Figure 20A] FIG. 13 is a diagram showing a configuration of a biological information detection device according to a ninth embodiment. [Figure 20B] FIG. 13 shows a plurality of color filters in the ninth embodiment. [Figure 20C] FIG. 23 shows an example of an image generated in the ninth embodiment. [Figure 20D] A diagram showing an example of the configuration of a multispectral sensor with four types of color filters: R, G, B, and IR. [Figure 21] FIG. 1 is a diagram showing a configuration example (comparison example) of a biometric information sensing system using an imaging device. [Figure 22] A diagram showing the absorption and scattering coefficients of hemoglobin, melanin, and water, which are the main components of living organisms, in the visible to near-infrared wavelength range. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

[0016] Remote biometric sensing using a camera is expected to be used in various applications because it can measure continuously for a long time without any sense of restraint. For example, in medical institutions such as hospitals, it is expected that the condition of patients will be constantly monitored to quickly respond to sudden changes in the condition, and that data obtained by long-term monitoring will be used for diagnosis. In addition to use in medical institutions, it is expected that it will be used for preventing sudden death during sleep at home and monitoring sleep apnea syndrome. Furthermore, it is expected that it will be used for applications such as constantly monitoring changes in physical conditions and using them for health management by acquiring physical information data in daily life that is constantly acquired at home or at work and analyzing the data accumulated on a server via the cloud, and sharing the acquired data with medical institutions to use it for medical care. In order to acquire such constant biometric information, a system that constantly measures biometric information without any sense of restraint and without any particular awareness is required. A system using a camera is considered to be ideal for such applications because it allows non-constraint remote measurement.

[0017] However, when using a camera to sense biometric information in daily life, consideration must be given to privacy. Systems that record high-resolution images captured by the camera that could identify the individual in a storage device pose the risk of image leakage and should be avoided as much as possible. Even if the captured images themselves are not recorded, a system in which the camera (or camera lens) is visible from the measurement system may cause psychological resistance in the subject. For this reason, a system in which the camera is invisible is desirable.

[0018] Due to the strong demand from the medical and healthcare fields as described above, various research institutes have been developing systems to realize remote biometric sensing using cameras, and several There are many products on the market. The biggest challenge of a remote biometric sensing system using a conventional camera is the accuracy and stability of the measurement. When a human body is photographed with a camera, most of the light incident on the camera is reflected from the surface of the skin or near the surface. Since there are no blood vessels and no metabolism in the stratum corneum, the outermost layer of the skin, biometric information cannot be obtained from the surface reflection components. It is necessary to detect the light that penetrates into the skin and is reflected by the epidermis where blood vessels are located. The light reflected from the skin is dominated by the component reflected from the surface of the skin, and the light that penetrates into the skin is rapidly lost due to strong light absorption by the body. For this reason, the proportion of light containing biometric information in the reflected light is low. Furthermore, in a system that does not have a measurement illumination system and acquires images using ambient light, instability due to changes in the acquired image signal as the ambient light changes has been an issue.

[0019] Furthermore, in the case of remote sensing, instability of measurement due to body movement is a major issue. The acquired signal fluctuates due to changes in the measurement area and the orientation (angle) relative to the camera caused by body movement, making stable measurement impossible. As already mentioned, most of the signals acquired by the camera are components resulting from surface reflection on the skin that do not contain biological information, and the signal components containing biological information are weak. Surface reflection changes significantly due to changes in the measurement area and orientation caused by body movement, making it impossible to acquire weak biological information. This is the biggest issue with remote biological information sensing using a camera. For stable measurement, it is necessary to stop body movement and perform measurement in a stationary state, which does not fully utilize the advantage of being unconstrained.

[0020] In remote biosensing using a camera, an image of the subject is obtained, and a method of reducing the effects of body movement using this image has been used in some cases. In such a method, the face part is detected from the image obtained from the camera using a face recognition function, and then the part to be measured is recognized by performing face part recognition, so that the measurement part is always captured on the image even if there is body movement, and bioinformation is detected. For example, if the forehead can be detected by part recognition, information on the forehead can always be obtained even if the forehead moves on the image due to body movement.

[0021] However, there are two problems with the method using image recognition. One is that the computational load is large because facial parts are recognized by extracting features from the entire image. Therefore, it is necessary to perform high-speed image processing using a high-performance and expensive computing device, or to use a lower frame rate so that the next frame can be processed after the processing of one frame is completed. High-speed processing is expensive, leading to larger and more expensive devices. When low-speed processing is adopted, the measurement accuracy decreases. The other problem is that even if the influence of body movement is reduced by performing image recognition, there is a limit to the improvement of detection accuracy due to changes in the orientation of the test part caused by body movement (the angle of the normal direction of the test part surface with respect to the front direction of the camera). Since the reflectance of surface reflected light is angle-dependent, when the orientation of the measurement target part changes due to body movement, the amount of surface reflected light reaching the camera fluctuates, and the detection accuracy decreases.

[0022] Thus, the biggest problem in remote biometric sensing using a camera is the instability of measurements due to body movement. Due to the low reliability caused by this instability, remote biometric sensing using a camera has not been widely used for various applications to date.

[0023] As mentioned above, various attempts have been made to measure basic biological information for determining a person's health condition. For example, Patent Document 1 proposes a method for detecting the heart rate without contact from image information such as a face captured by a camera. The method of Patent Document 1 obtains the heart rate by analyzing the spatial frequency components of the acquired color image. However, this method is unable to perform stable detection because the accuracy is reduced by the influence of disturbance light such as indoor lighting.

[0024] A pulse oximeter is commonly used to measure blood oxygen saturation. A pulse oximeter pinches a finger, irradiates the finger with light of two wavelengths in the red to near-infrared wavelength range, and measures the transmittance. This makes it possible to determine the ratio of the oxygenated hemoglobin concentration to the reduced hemoglobin concentration in the blood. A pulse oximeter can measure blood oxygen saturation with a simple configuration. However, as it is a contact-type device, there is an issue that it feels restrictive.

[0025] An example of a non-contact blood oxygen saturation measuring device is disclosed in Patent Document 2. This device uses a white light source and a laser light source to measure blood oxygen saturation by utilizing the laser Doppler effect caused by the laser light scattered behind the surface of the living body. However, this method has the problem that the device configuration is complicated and the obtained signal is weak.

[0026] Patent Document 3 proposes a method for measuring blood oxygen saturation by eliminating the effects of ambient light using a normal color camera. However, even with this method, it is difficult to measure blood oxygen saturation stably with high accuracy because the effects of reflected light on the surface of the skin are large.

[0027] As described above, conventional non-contact methods for measuring biological information such as heart rate, blood flow, and blood oxygen saturation have problems with accuracy and stability.

[0028] Furthermore, in order to measure biometric information using a camera, it is necessary to identify a measurement area (for example, the forehead area) from the camera image and detect the biometric information using image information within that area. There are two methods for identifying the measurement area: a method of designating the measurement area before measurement and a method of automatically setting the measurement area from the image. In the method of designating the measurement area before measurement, the measurer designates the measurement area from the image of the subject before starting the measurement, and continues to measure the same area during the measurement. This method is simple, but the subject is not allowed to move during the measurement, and the advantage of non-restraint in non-contact measurement is lost. To avoid this, a method of automatically setting the measurement area is sometimes used. In such a method, for example, if the measurement area is the forehead area, the camera can perform face recognition from the acquired image, and further perform facial part recognition to identify the forehead area on the image and measure that part.

[0029] FIG. 21 is a diagram showing a schematic diagram of an example (comparative example) of such a system. The imaging device 2, which is a camera in this system, includes a camera housing 6 having an image sensor 7, and an optical system 5, which is a lens. A calculation device (or a calculation circuit) is built into or connected to the image sensor 7 in the imaging device 2. The calculation device performs face recognition on the acquired image (for example, part (a) of FIG. 21), identifies the forehead, and then extracts image data of the forehead (for example, part (b) of FIG. 21). Then, biometric information such as heart rate fluctuation (for example, part (c) of FIG. 21) is generated from the image data of the forehead. Part (c) of FIG. 21 shows the time fluctuation of the value obtained by averaging the image data of the forehead shown in part (b) of FIG. 21 within the forehead region. The face part recognition algorithm used here places a heavy load on the image processing of a computer. For this reason, the cost of the calculation device increases for high-speed processing. In addition, the method using image recognition has a problem that the recognition rate decreases when the body orientation changes or when part of the face is hidden. In addition, there is also a problem that it is easily affected by ambient light. For this reason, it has been difficult to perform continuous and stable measurement.

[0030] In addition to the above problems, the method using face recognition also has the problem that it is not possible to measure parts of the body other than the face (such as the arms or chest). Furthermore, there is also the problem that consideration must be given to privacy. Subjects are psychologically stressed by having their images constantly captured by a camera. However, for highly accurate image recognition, it is necessary to take images using a high-resolution camera. For this reason, there is a risk that subjects may feel a psychological burden from being constantly photographed by a camera.

[0031] The present inventor has focused on the above-mentioned problem and studied a configuration for solving the above-mentioned problem. As a result, it has been found that the above-mentioned problem can be solved by acquiring an image using a light source that projects a dot pattern by light onto the surface of a living body, detecting a living body region (e.g., a human body region) on the image based on a ratio (e.g., contrast described later) of a component due to direct reflection light (also referred to as "surface reflected light") in the image to a component due to scattered light inside the living body (referred to as "internal scattered light"), and separating the component due to direct reflection light and the component due to scattered light inside the living body in the detected living body region by signal processing. That is, the biological information detection device first detects a region on the image that is estimated to be a living body, and measures biological information within that region. With this method, as will be described in detail later, the amount of calculation for image processing can be significantly reduced, and biological information can be detected quickly and stably.

[0032] A biological information detection device according to an embodiment of the present disclosure includes: a first light source that projects a plurality of first dots formed by a first light onto an object including a living body; an imaging device including a plurality of first light detection cells that detect second light from the object onto which the plurality of first dots are projected, and that generates and outputs a first image signal that indicates a first image of the object onto which the plurality of first dots are projected; A first arithmetic circuit; A second arithmetic circuit; Equipped with.

[0033] The first image includes a plurality of pixels.

[0034] The first arithmetic circuit detects a first portion of the first image, which corresponds to the living body, using the first image signal.

[0035] The second arithmetic circuit calculates biometric information about the living body using data of pixels within the first portion of the first image, among the plurality of pixels.

[0036] The first arithmetic circuit can detect a biological region based on the ratio of pixel signals in an area where a dot pattern is projected to pixel signals in an area surrounding the area. For example, based on the ratio (referred to as "contrast") between the standard deviation and average pixel values ​​of a specific pixel included in the image and a plurality of pixels arranged around the specific pixel, it can determine whether or not a biological body is present at a position corresponding to the specific pixel. The second arithmetic circuit generates and outputs information about the biological body using signals from an area where the dot pattern is not projected, mainly, of the image signals. With this configuration, biological information can be acquired with high accuracy.

[0037] In this specification, "biometric information" refers to various information related to a living body, such as heart rate, blood flow, blood pressure, blood oxygen saturation, respiration, etc. In this specification, information obtained from such information that indicates a person's state, such as a person's concentration level or emotion, is also included in "biometric information."

[0038] (principle) The principle of the biological information detection device that enables highly accurate acquisition of biological information will be described below.

[0039] In the biological information detection device according to the embodiment of the present disclosure, light in a wavelength range of, for example, about 650 nm or more and about 950 nm or less is used. This wavelength range is included in the wavelength range from red to near-infrared. In this specification, the term "light" is used not only for visible light but also for infrared light. The above wavelength range is called the "biological window" and is known to have low absorption rate in the body.

[0040] FIG. 22 is a diagram showing the wavelength dependence of the light absorption coefficients of oxyhemoglobin, deoxyhemoglobin, melanin, and water, and the scattering coefficient of light in the body. In the visible light region of 650 nm or less, absorption by blood (i.e., hemoglobin) is large, and in the wavelength region longer than 950 nm, absorption by water is large. Therefore, light in these wavelength regions is not suitable for acquiring information inside the body. On the other hand, in the wavelength range of about 650 nm to about 950 nm, the absorption coefficients of hemoglobin and water are relatively low, and the scattering coefficient is large. The scattering coefficient is one order of magnitude larger than the absorption coefficient, and in the wavelength region of the "biological window," scattering is dominant in the interaction between the skin and near-infrared light. Therefore, light in this wavelength range is strongly scattered after entering the body and returns to the body surface. Since such optical characteristics are unique to the skin, it is possible to distinguish the human body from other substances using this diffuse reflection characteristic.

[0041] The biological information detection device according to the embodiment of the present disclosure mainly utilizes light in the wavelength range corresponding to this “biological window.” For example, by using a dot array light source, it is possible to detect light directly reflected from the surface of the living body and light scattered inside the body and returned by spatially separating them with high accuracy, thereby making it possible to efficiently obtain biological information.

[0042] 1A is a diagram showing a schematic configuration of a biological information detection device according to an exemplary embodiment of the present disclosure. This device includes a light source 1, which is an array point image light source that projects a plurality of discretely arranged point images (sometimes referred to herein as an "array point image" or a "dot pattern") onto an object including a biological body, and a camera 2. The light source 1 is disposed so as to project a plurality of point images onto a biological body 3. The imaging device 2 has an image sensor (also referred to as an "imaging element"), captures an image of a biological body surface 4, and generates and outputs an image signal.

[0043] 1B is a diagram for explaining the characteristics of an image of a biological surface acquired by an imaging device 2. Light L0 emitted from a light source 1 is reflected by a biological surface 4. The surface reflected light L1 reflected by the biological surface 4 maintains the image of the array of point images formed by the light source 1. In contrast, the internally scattered light L2 that enters the inside of a biological body 3, is scattered inside the biological body, and exits from the biological surface 4 loses the image of the array of point images formed by the light source 1 due to strong scattering within the biological body. By using the light source 1, the surface reflected light L1 and the internally scattered light L2 can be easily spatially separated.

[0044] The living body 3 shown in FIG. 1A is human skin, and includes the epidermis 33, the dermis 34, and the subcutaneous tissue 35. The epidermis 33 does not have blood vessels, but the dermis 34 has capillaries 31 and arterioles 32. Since the epidermis 33 does not have blood vessels, the surface reflected light L1 does not contain information about blood. Since the epidermis 33 contains melanin pigment that strongly absorbs light, the surface reflected light L1 from the epidermis 33 becomes noise in acquiring blood information. Therefore, the surface reflected light L1 is not only useless for acquiring blood information, but also hinders the acquisition of accurate blood information. In order to detect biological information with high accuracy, it is extremely important to suppress the influence of the surface reflected light and efficiently acquire information about the light scattered within the body.

[0045] In order to solve the above problems, the embodiment of the present disclosure has a novel configuration in which a light source that projects an array of point images onto a living body and an imaging device (or imaging system) are used to spatially separate directly reflected light and scattered light within the body, thereby making it possible to measure information within the living body in a non-contact manner with high accuracy.

[0046] Conventionally, in order to separate such directly reflected light from the surface of a living body, a method using polarized illumination, such as that disclosed in Patent Document 6, has been used. In the method using polarized illumination, a polarizer with a polarization transmission axis perpendicular to the polarization direction of the illumination light reflected from the subject to be photographed is used. By capturing an image with a camera through such a polarizer, the effects of surface reflected light can be suppressed. However, when it comes to reflection from an uneven surface such as skin, the degree of polarization of the surface reflected light varies depending on the position, and there is an issue that the directly reflected light cannot be sufficiently separated. According to the method of the present disclosure, the directly reflected light and the scattered light can be spatially separated, so that the influence of the surface reflected light can be more effectively suppressed.

[0047] The present disclosure includes aspects described in the following items, for example.

[0048] [Item 1] A biological information detection device according to item 1 of the present disclosure includes: a first light source that projects a plurality of first dots formed by a first light onto an object including a living body; an imaging device including a plurality of first light detection cells that detect second light from the object onto which the plurality of first dots are projected, and that generates and outputs a first image signal that indicates a first image of the object onto which the plurality of first dots are projected; A first arithmetic circuit; A second arithmetic circuit; Equipped with the first image includes a plurality of pixels; the first arithmetic circuit detects a first portion of the first image corresponding to the living body, using the first image signal; The second arithmetic circuit calculates biometric information about the living body using data of pixels within the first portion of the first image, among the plurality of pixels.

[0049] [Item 2] In the biological information detection device according to item 1, the second light includes a third light from a position on the surface of the object where at least one first dot among the plurality of first dots is projected, and a fourth light from a position on the surface of the object that is different from the position on which the plurality of first dots are projected and surrounds the at least one first dot; the first arithmetic circuit detects the first portion of the first image by using a first image signal corresponding to the third light and a first image signal corresponding to the fourth light among the first image signals; The second arithmetic circuit may calculate biometric information about the living organism by using data of pixels corresponding to the fourth light among pixels in the first portion of the first image.

[0050] [Item 3] In the biological information detection device according to item 2, The first arithmetic circuit may determine a ratio between an intensity of a first image signal corresponding to the third light and an intensity of a first image signal corresponding to the fourth light, and detect the first portion of the first image using the ratio.

[0051] [Item 4] In the biological information detection device according to item 2, The first calculation circuit may detect the first portion of the first image by using a ratio between a standard deviation and an average value of an intensity of a first image signal corresponding to the third light and an intensity of a first image signal corresponding to the fourth light.

[0052] [Item 5] In the biological information detection device according to any one of items 1 to 4, The first light may include light having a wavelength of 650 nm or more and 950 nm or less.

[0053] [Item 6] In the biological information detection device according to any one of items 1 to 5, The biological information may include at least one selected from the group consisting of the heart rate of the living body, the blood pressure of the living body, the blood flow rate of the living body, and the blood oxygen saturation of the living body, the concentration of melanin pigment in the skin of the living body, the presence or absence of blemishes on the skin of the living body, and the presence or absence of bruises on the skin of the living body.

[0054] [Item 7] In the biological information detection device according to any one of items 1 to 6, The imaging device includes: a first bandpass filter that transmits the second light; an image sensor having an imaging surface on which the plurality of first photodetector cells are arranged, and light transmitted through the first bandpass filter is incident on the imaging surface; It may further include.

[0055] [Item 8] In the biological information detection device according to any one of items 1 to 7, The second arithmetic circuit may calculate, as the bioinformation, at least one selected from the group consisting of a heart rate of the living body, a blood pressure of the living body, and a blood flow rate of the living body, using a change over time in a value obtained by applying a low-pass filtering process to at least a portion of the pixel data.

[0056] [Item 9] The biological information detection device according to any one of items 1 to 8, Further, a second light source projects a plurality of second dots by a second light onto the object; the first light includes light having a wavelength of 650 nm or more and 800 nm or less, the second light includes light having a wavelength of 800 nm or more and 950 nm or less, the imaging device further includes a plurality of second light detection cells that detect a fifth light from the object onto which the plurality of second dots are projected; The imaging device may generate and output a second image signal representing a second image of the object onto which the plurality of second dots are projected.

[0057] [Item 10] In the biological information detection device according to item 9, The imaging device includes: an image sensor having an imaging surface divided into a first region in which the plurality of first photodetection cells are arranged and a second region in which the plurality of second photodetection cells are arranged; a first optical system that forms the first image in the first area; a second optical system that forms the second image in the second area; and It may further include.

[0058] [Item 11] Item 11. The biological information detection device according to item 10, The imaging device includes: a first bandpass filter disposed on a path of the second light and transmitting the second light; a second bandpass filter disposed on a path of the fifth light and transmitting the fifth light; It may further include.

[0059] [Item 12] In the biological information detection device according to item 9, The imaging device includes: an imaging surface on which the plurality of first photodetection cells and the plurality of second photodetection cells are arranged; a plurality of first bandpass filters facing the plurality of first photodetector cells and transmitting the second light; an image sensor including a plurality of second band pass filters facing the plurality of second photodetector cells and transmitting the fifth light; an optical system for forming the first image and the second image on the imaging plane; It may further include.

[0060] [Item 13] In the biological information detection device according to item 9, The imaging device includes: an imaging surface on which the first photodetection cells, the second photodetection cells, and the third photodetection cells are arranged; a plurality of first bandpass filters facing the plurality of first photodetector cells and transmitting the second light; a plurality of second bandpass filters facing the plurality of second photodetector cells and transmitting the fifth light; an image sensor including a plurality of third bandpass filters facing the plurality of third photodetector cells and transmitting visible light; an optical system for forming the first image and the second image on the imaging plane; Further comprising: the third bandpass filters include a plurality of color filters having different transmission wavelength ranges, The image sensor may generate and output a color image signal using the plurality of third photodetector cells.

[0061] [Item 14] In the biological information detection device according to any one of items 9 to 13, The second arithmetic circuit may calculate information indicating a blood oxygen saturation level of the living body, using the first image signal and the second image signal.

[0062] [Item 15] In the biological information detection device according to any one of items 9 to 13, The second arithmetic circuit includes: Calculating a blood flow rate and a blood oxygen saturation rate of the living body using the first image signal and the second image signal; The blood flow rate and the blood oxygen saturation level of the living body may be used to generate information indicating at least one selected from the group consisting of a physical condition, an emotion, and a concentration level of the living body.

[0063] [Item 16] In the biological information detection device according to any one of items 9 to 13, When the first image and the second image include at least one selected from the group consisting of the forehead and nose of the living body, the second arithmetic circuit uses the first image signal and the second image signal to calculate a time change in blood flow rate and a time change in blood oxygen saturation in the at least one selected from the group consisting of the forehead and the nose; The physical condition of the living body is determined by using the time change of the blood flow rate and the time change of the blood oxygen saturation. , emotion, and concentration level may be generated.

[0064] [Item 17] In the biological information detection device according to any one of items 9 to 13, When the first image and the second image include the forehead and nose of the living body, the second arithmetic circuit uses the first image signal and the second image signal to calculate a time change in blood flow rate and a time change in blood oxygen saturation rate at the forehead and a time change in blood flow rate and a time change in blood oxygen saturation rate at the nose; Information indicating at least one selected from the group consisting of the physical condition, emotion, and concentration level of the living body may be generated based on a comparison between the time change in the blood flow rate and the time change in the blood oxygen saturation in the forehead and the time change in the blood flow rate and the time change in the blood oxygen saturation in the nose.

[0065] [Item 18] Item 18. In the biological information detection device according to any one of items 1 to 17, The first light source may be a laser light source.

[0066] [Item 19] Item 19. In the biological information detection device according to any one of items 1 to 18, The imaging device includes: an image sensor having an imaging surface on which the plurality of first photodetection cells are arranged; an optical system for forming the first image on the imaging plane; an adjustment mechanism for adjusting the focus of the optical system; Further comprising: The adjustment mechanism may be adapted to adjust the focus to maximise contrast of the first image.

[0067] [Item 20] 19. The biological information detection device according to claim 1, The first arithmetic circuit uses the first image signal to determine whether the first image includes at least one selected from the group consisting of a forehead, a nose, a mouth, eyebrows, and hair of the living body; when it is determined that the first image includes at least one selected from the group consisting of the forehead, nose, mouth, eyebrows, and hair of the living body; The second arithmetic circuit may calculate biological information related to the living body.

[0068] [Item 21] 21. The biological information detection device according to any one of items 1 to 20, The first arithmetic circuit may further calculate other biometric information regarding the living body by using data of pixels in a second portion of the first image, the second portion being different from the first portion, among the plurality of pixels.

[0069] [Item 22] 21. The biological information detection device according to any one of items 1 to 20, The second arithmetic circuit may further determine whether the living body has moved by comparing a position of the first part in the first image at a first time with a position of the first part in the first image at a second time.

[0070] In this disclosure, a whole or part of a circuit, unit, device, member or part, or a block diagram All or part of the functional blocks may be implemented by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, the functional blocks other than the memory element may be integrated into one chip. Here, although it is called an LSI or an IC, the name may be changed depending on the degree of integration, and it may be called a system LSI, a VLSI (very large scale integration), or a ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which can reconfigure the connection relationship inside the LSI or set up the circuit partition inside the LSI, can also be used for the same purpose.

[0071] Furthermore, all or part of the functions or operations of the circuit, unit, device, member, or part can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-transitory recording media in which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0072] Hereinafter, the embodiments of the present disclosure will be described in more detail. The following embodiments relate to a biometric information detection device that measures biometric information in a non-contact manner, mainly using a human face as a biological surface. However, the technology of the present disclosure is not limited to the human face, and can also be applied to other parts of the body or the skin of animals other than humans.

[0073] (Embodiment 1) As a first embodiment, a system in which the technology of the present disclosure is applied to non-contact heart rate measurement will be described. With the rise in awareness of health care, the importance of constant biological information sensing is increasing. Systems capable of constantly measuring biological information non-contact are important not only in hospitals but also for health management in daily life. The system of this embodiment can monitor heart rate and heart rate variability in a non-contact manner.

[0074] FIG. 2 is a diagram showing a schematic configuration of the living body detection system of this embodiment. As shown in FIG. 2, the living body detection system of this embodiment includes a light source 1 that emits light rays in the near-infrared wavelength range located away from a living body 3, an imaging device 2 that is a camera that can record an image of the irradiated living body surface 4, and a computer 20 connected to the light source 1 and the imaging device 2. The computer 20 can separate and measure the component of the surface reflected light L1 and the component of the scattered light L2 in the body from the captured image. The computer 20 can detect whether a living body is included in the image based on the intensity of the surface reflected light L1 and the intensity of the scattered light L2 in the body. In addition, it can calculate and output biological information such as heart rate from a signal in the area of ​​the living body in the image.

[0075] The light source 1 is designed to project a dot pattern onto the biological surface 4. The dot pattern is typically a collection of tiny bright points arranged two-dimensionally. Depending on the application, a dot pattern arranged one-dimensionally may be used. In this embodiment, for example, a random dot pattern laser projector RPP017ES from Osela Inc. of the United States can be used as the light source 1. This laser light source emits near-infrared laser light of 830 nm and projects a laser dot pattern of 57446 points within a viewing angle of 45°×45°.

[0076] 3A is a diagram showing an example of the configuration of an imaging device and generated images and biometric information. The imaging device 2, which is a camera, has an optical system 5 and a camera housing 6. The optical system 5 may be a collection of multiple lenses. Inside the camera housing 6, an image sensor 7 and a bandpass filter 8 that transmits only light with a wavelength of 830 nm±10 nm, which is the wavelength of the light from the light source 1, are mounted.

[0077] When a human subject is used as the subject, the image sensor 7 acquires an image signal including a plurality of point images having brightness according to the infrared reflectance at each position. Part (a) of FIG. 3A shows an example of an image represented by such an image signal. From this image signal, an arithmetic circuit in the computer 20 detects only the region of the human body by signal processing, as shown in part (b) of FIG. 3A. This detection is performed based on the ratio of the signal component due to the surface reflected light L1 to the signal component due to the internally scattered light L2.

[0078] As already mentioned, a living body has a unique optical characteristic called a "biological window" for wavelengths from red to near infrared. The absorption coefficient of human skin is small and the scattering coefficient is large in this wavelength range. Therefore, light that has passed through the surface of the skin, which is the biological surface 4, is repeatedly scattered multiple times inside the body and diffuses, and is emitted from the biological surface 4 over a wide range. For this reason, as shown in the enlarged view of part (c) of FIG. 3A, in the area of ​​the human body in the image, an area caused by the internal scattered light L2 occurs around each bright point caused by the surface reflected light L1. In the above wavelength range, a living body is characterized in that the ratio of scattered light is high relative to direct reflected light. In contrast, in objects other than living bodies, the ratio of surface reflected light is overwhelmingly higher than the scattered light. Therefore, it is possible to detect a biological body area based on the ratio of direct light to scattered light. Furthermore, biological information can be obtained at high speed using signals from multiple pixels included in the biological body area in the obtained image. The detection of the human body area in this embodiment using the optical characteristics of the skin is faster and more accurate than the conventional method using image recognition. High-speed and high-precision biological information sensing is possible by detecting the human body area and detecting the human body using the information.

[0079] FIG. 3B is a block diagram showing the configuration of the computer 20. The computer 20 includes an input interface (IF) 21 electrically connected to the imaging device 2, a first arithmetic circuit 22 that performs signal processing to detect a region of a human body in an image, a second arithmetic circuit 23 that calculates bioinformation (pulsation in this embodiment) using image data in the detected human body region, a memory 25 that records various data, a control circuit 26 that controls the operation of the entire device, an output interface (IF) 24 that outputs data generated by the second arithmetic circuit 23, and a display 27 that displays the processing result. Each of the arithmetic circuits 22 and 23 may be an image processing circuit such as a digital signal processor (DSP). In FIG. 3B, the arithmetic circuits 22 and 23 are represented by different blocks, but both may be realized by one circuit. The control circuit 26 may be an integrated circuit such as a central processing unit (CPU) or a microcomputer. The control circuit 26 controls the light source 1 to turn on, the imaging device 2 to capture an image, and the arithmetic circuits 22 and 23 to perform calculations, for example, by executing a control program recorded in the memory 25. The control circuit 26 and the arithmetic circuits 22 and 23 may be realized by a single integrated circuit. In the example of Fig. 3B, the computer 20 includes a display 27, but the display may be an external device electrically connected by wire or wirelessly. The computer 20 may obtain image information from the imaging device 2 in a remote location via a communication circuit (not shown).

[0080] In the example shown in FIG. 3A, the second arithmetic circuit 23 averages the signal components of the internally scattered light L2 within the human body region detected by the first arithmetic circuit 22. This averaging is performed, for example, for each frame of a moving image. As a result, data on the time fluctuation of the average value of the signal components of the internally scattered light L2 is obtained, as shown in part (d) of FIG. 3A. From this data, the period or frequency can be obtained. By measuring the heart rate, the heart rate (number of beats per unit time) can be calculated.

[0081] Furthermore, a similar system configuration can be used to measure respiration at the same time as measuring the heart rate. Figure 3C is a schematic diagram of a system that performs such respiration sensing. The hardware in this example is exactly the same as that shown in Figure 3A. This configuration allows non-contact respiration monitoring to be achieved by image signal processing.

[0082] The interval between human breaths is approximately 3 to 4 seconds (15 to 20 breaths per minute), and the expansion and contraction of the chest and abdominal wall due to breathing is approximately 5 mm in an adult. If this movement of the chest can be measured with an imaging device, breathing can be monitored.

[0083] A method of monitoring respiration from the amount of movement of the chest using a near-infrared dot array light source is disclosed in, for example, Non-Patent Document 1. The system in Non-Patent Document 1 realizes highly accurate non-contact sensing of respiration by determining an examination area in advance for a stationary subject. The system in Non-Patent Document 1 is intended for non-contact respiration monitoring while sleeping, and monitors only respiration with a large system.

[0084] In contrast, the system of the present embodiment shown in Fig. 3C is small and inexpensive, and enables simultaneous heart rate sensing and respiration sensing. Furthermore, stable measurement is possible by following the subject's body movements.

[0085] Hereinafter, with reference to FIG. 3C, the method of measuring respiration in this embodiment will be described. The first arithmetic circuit 22 performs human body detection from the near-infrared image (part (a) of FIG. 3C) acquired by the image sensor 7 by the above-mentioned method, and estimates the face area from the data of the human body area (part (b) of FIG. 3C). Furthermore, the position of the chest is estimated from the center of the face area. For example, the position shifted downward from the center of the face by about 1 to 1.2 times the vertical length of the face area is estimated to be the chest area. Then, the temporal positional fluctuation of the dot array in this chest area (part (d) of FIG. 3C) is measured. At that time, by averaging the positional fluctuation of a plurality of dots, it is possible to measure the positional fluctuation of less than the pixel pitch. It should be noted here that even if the object (chest) moves in the horizontal direction, the position of the captured dot hardly changes, and the position of the dot moves and is measured only when it changes in the depth direction. In the method using a normal image, both the horizontal and vertical movements of the object are detected as movements on the pixels, and the horizontal movement is detected with higher sensitivity, resulting in low measurement accuracy. In contrast, in this embodiment, by using a dot array light source, the horizontal movement of the subject is not detected, but only the vertical movement is detected. This enables highly accurate respiration monitoring. The amount of change between frames of the dot array pattern in the chest area can be obtained by calculating the autocorrelation of the dot array pattern. This average amount of movement represents the up and down movement of the chest due to respiration. As shown in part (e) of FIG. 3C, respiration can be monitored by plotting the average amount of movement of the dot pattern on the time axis. According to the configuration of this embodiment, even if there is body movement, highly accurate respiration sensing is possible while tracking the chest area.

[0086] In the example shown in Fig. 3C, the method of measuring the heart rate using pixel data in the forehead region in the image is the same as the method shown in Fig. 3A. In the example of Fig. 3C, both the heart rate and respiration are measured, but only respiration may be measured.

[0087] Next, an example of a live body detection method performed using actual data will be described.

[0088] FIG. 4A shows an example of an image captured by a normal imaging device that detects visible light. A human face F is seen in the center. The left diagram of FIG. 4B shows the same scene as FIG. 4A, illuminated by a light source 1 with a wavelength of 830 nm, captured by the imaging device 2 of this embodiment. In this image, it is difficult to recognize face F due to strong reflection from box B placed in the foreground. Therefore, in order to detect a human body, first arithmetic circuit 22 calculates the contrast between direct reflected light and scattered light from the near-infrared image.

[0089] 5 is a diagram showing an example of a pixel region used for calculating contrast. Image data is recorded in memory 25 as two-dimensional intensity data. The data of the i-th pixel in the horizontal (x) direction and the j-th pixel in the vertical (y) direction is denoted as Pij. The contrast Cij of this (i,j) pixel is defined as follows: Cij = Sij / Aij

[0090] Here, Sij and Aij are the standard deviation and average value of pixel data within a 7×7 pixel region centered on pixel (i,j), respectively. The higher the ratio of scattered light to directly reflected light, the smaller the standard deviation value Sij, and therefore the smaller the value of Cij. After repeating this process for all pixels, the first arithmetic circuit 22 extracts only pixels whose Cij values ​​are within a predetermined range. As an example, the right side of FIG. 4B shows a part of the region where 0.2 < Cij < 0.47. In this figure, pixels whose Cij values ​​are within the above range are displayed in white, and other pixels are displayed in black. It can be seen that the living body (i.e., face F) part has been correctly extracted.

[0091] In this manner, the first arithmetic circuit 22 in this embodiment calculates the contrast Cij, which is the ratio of the standard deviation to the average pixel value of a specific pixel included in the image and a number of pixels arranged around the pixel. Based on the value, it is possible to determine whether or not a living body is present at a position corresponding to the specific pixel, and output information indicating the presence or absence.

[0092] According to this embodiment, by utilizing the specific optical characteristics of a living body, it is possible to efficiently detect a living body hidden among many objects. Here, in order to obtain the contrast of an image (that is, the contrast between direct reflected light and scattered light), the average value and standard deviation value within a region of 7×7 pixels were obtained, but this is just an example. The size (i.e., the number of pixels) of the pixel region used for the contrast calculation is appropriately set according to the density of a plurality of point images formed by the light source 1 and the resolution of the imaging device 2. In order to suppress the variation in the calculation results, a plurality (for example, 3 or more) of point images may be included within the pixel region to be calculated. By increasing the number of pixels in the region to be calculated, the accuracy of the calculated value of the contrast is improved, but the resolution of the obtained image of the living body decreases. Therefore, the number of pixels in the region to be calculated is appropriately set according to the configuration of the system and the purpose of use. Furthermore, not only the number of pixels to be calculated, but also the interval between the pixels for which this process is repeated affects the processing speed. In the above process, the calculation was sequentially repeated for all pixels, but by increasing the pixel interval for which the calculation is performed, the resolution decreases but the processing speed can be increased. This pixel interval may also be appropriately set according to the configuration of the system and the purpose of use. Similarly, the predetermined range of the contrast is not limited to 0.2 < Cij < 0.47 and may be appropriately set according to the configuration of the system and the purpose of use.

[0093] By the above method, the first arithmetic circuit 22 detects the region of the human body in the two-dimensional image acquired by the imaging device 2. Next, the second arithmetic circuit 23 acquires biological information. Since the region of the human body in the image has already been specified by the first arithmetic circuit 22, biological information is generated using the pixel data within this region. The second arithmetic circuit 23 generates data on the time variation of the heart rate and respiration as biological information, for example, as shown in FIG. 3C. Thereby, it is possible to monitor the heart rate and heart rate variation non-contact.

[0094] In a conventional biological information sensing system using a camera, a method of averaging the pixel data of the biological region in the image to detect biological information was generally used. In contrast, since the biological information sensing system of this embodiment uses a dot array light source, the two-dimensional image This makes it possible to selectively extract internal scattered light containing biological information by removing unnecessary surface reflected light components from the skin surface. The projected dot array image (surface reflected light) is detected as a dot with a high pixel value, and the components scattered within the body (internal scattered light) are detected as pixel values ​​centered around the dot that are lower than the dot. By setting a threshold for light intensity and averaging the pixel data excluding pixel data with a light intensity above a certain level, internal scattered light can be efficiently extracted. This type of processing makes it possible to obtain biological information with high accuracy.

[0095] 6 is a flowchart showing an example of the operation executed by the first arithmetic circuit 22 and the second arithmetic circuit 23 in this embodiment. Here, the operation of the arithmetic circuits 22 and 23 will be described taking as an example a case where a moving image is acquired by the image sensor 7. Note that the following operation can be realized by one or more processors executing a computer program stored in a memory.

[0096] First, the first arithmetic circuit 22 extracts a human body region from the acquired moving image (step S101). The method of extracting the human body region is as described above. Next, the second arithmetic circuit 23 removes data of the pixel values ​​in the extracted human body region, which are in the center of the dot array and correspond to the direct reflection light component, by using a preset threshold value (step S102). Then, the second arithmetic circuit 23 calculates the average value of the pixel values ​​(corresponding to the internal scattering component) in the human body region (step S103). The above steps S101 to S103 are performed for each frame of the moving image. The second arithmetic circuit 23 calculates the period and amplitude of the time fluctuation of the above average value using the data of the frames for a predetermined period (for example, several seconds to several tens of seconds) (step S104). This makes it possible to obtain information on the blood flow in the living body. Since the arterial blood pumped from the heart moves through the blood vessels with a wave motion called a pulse wave, the absorptivity and reflectivity of near-infrared rays change in conjunction with the pulsation. The heart rate can be obtained from the period of this fluctuation in reflectivity. Furthermore, blood pressure or blood flow can be estimated from the amplitude of the pulsation.

[0097] In the step of removing the direct reflection light component using a preset threshold (step S102), a fixed threshold can be used when the distance between the image capture device and the subject is constant. However, since the distance between the image capture device and the subject usually varies in many cases, it is desirable to be able to handle such cases. Therefore, for example, the average pixel value of the entire biological region to be calculated may be calculated, and the threshold value may be changed according to the average value. In such a form, the higher the average pixel value, the higher the threshold value may be set.

[0098] According to this embodiment, it is possible to obtain heartbeat data such as that shown in part (d) of FIG. 3A. Numerous methods have been proposed for non-contact monitoring of heartbeats using normal visible and near-infrared cameras. In these conventional methods, the separation of surface reflected light components and scattered light components is insufficient, so that the non-contact method is susceptible to the influence of disturbance light, making it difficult to perform stable and highly accurate measurement. In contrast, in this embodiment, stable and highly accurate heartbeat measurement is possible by spatially separating the surface reflected light components and scattered light components in the acquired image signal. For example, in remote heartbeat measurement using a conventional camera, detection becomes unstable due to body movements associated with speaking during conversation, making it impossible to perform highly accurate heartbeat measurement. By using the method of this embodiment, it is now possible to stably measure heartbeats if there is body movement of the order of conversation.

[0099] According to this embodiment, the psychological stress of the subject can also be estimated. It is known that psychological stress can be estimated from the temporal fluctuation of the heart rate. When the autonomic nerves are functioning normally, the interval between heartbeats fluctuates, but it is known that the fluctuation in the interval between heartbeats decreases due to stress. The second arithmetic circuit 23 in this embodiment can also detect the presence or absence or the degree of psychological stress based on changes in the fluctuation in the interval between heartbeats. In order to regularly perform stress sensing in daily life, a non-constraint, non-contact heart rate sensing system such as this embodiment is required. Technology is important.

[0100] As described above, by using the system of this embodiment, it is possible to monitor the heart rate or blood pressure at all times, including during sleep, without being restrained. This makes it possible to build a system that constantly monitors the condition of a patient in a hospital, and issues a warning to medical staff when an abnormality occurs. Even in an ordinary home, it is possible to monitor the heart rate of a patient suffering from sleep apnea syndrome at night. Furthermore, in daily life, since stress sensing can be easily performed as described above, it becomes possible to live a more fulfilling daily life.

[0101] (Embodiment 2) In the first embodiment, a system for detecting a human body region from an image and acquiring biometric information from the human body region in the image has been described. A representative application example using human body detection will be described below as the second embodiment. Human body detection is being developed for the purpose of detecting victims buried under rubble at disaster sites, for example. Finding victims within 72 hours after a disaster determines the survival rate of the victims. For this reason, a simple and stable biometric detection system is required. Biometric detection technology is also used in the security and traffic fields. Biometric detection technology plays an important role in the security field to find intruders and in the traffic field to detect pedestrians. There is an increasing need for a system that can selectively detect a living body (especially a human) from an image containing various structures or objects. A living body can be detected by the operation of the first arithmetic circuit 22 in the first embodiment, but by further sensing biometric information (for example, the presence or absence of pulsation) for the area where the living body is detected, more reliable and more advanced living body detection is possible. Below, the operation of this embodiment will be described for each application. The physical configuration in this embodiment is the same as the physical configuration in the first embodiment.

[0102] (1) Finding victims during disasters When natural disasters such as earthquakes, tsunamis, and mudslides occur, early discovery of victims buried under rubble is particularly important from the viewpoint of saving lives. There is a "72-hour wall" that is said to significantly reduce survival rates after three days, and it is necessary to quickly discover victims in a chaotic situation. By using the system of this embodiment, it is possible to capture images even in a situation where rubble is scattered, and detect victims hidden under rubble in real time. Since the system is small, it can be mounted on, for example, a UAV (Unmanned aerial vehicle, a so-called drone). This makes it possible to acquire images from a remote location by remote control and search for survivors, even in a disaster site where access is difficult due to the risk of secondary disasters.

[0103] Although human body detection can be performed by the living body detection process by the first arithmetic circuit 22 in the first embodiment, in order to improve accuracy, in this embodiment, the processing result of the second arithmetic circuit 23 is further utilized. For the area (referred to as a living body area) estimated by the first arithmetic circuit 22 to be a living body, the second arithmetic circuit 23 senses the living body information (for example, the presence or absence of pulsation). This makes it possible to detect a human body more reliably. By determining the presence or absence of body movement in the living body area, it is possible to reduce false detection and increase reliability. The presence or absence of body movement can be determined, for example, by comparing multiple consecutive frames and based on whether or not there is a temporal fluctuation in the living body area. Furthermore, by determining the presence or absence of a heartbeat using pixel data in the living body area, it is possible to dramatically improve the detection accuracy of the human body. According to this embodiment, high-speed and reliable living body detection is possible by combining the living body detection using the optical characteristics of the skin, the body movement detection due to the movement of the living body area, and the heartbeat measurement calculated from the signal strength in the living body area. Furthermore, since the physical condition of the victim can be known from the living body information, it is possible to determine the priority of rescue based on that data.

[0104] In the application of human body detection described below, more reliable live body detection can be achieved by combining the information of (1) live body detection, (2) body movement detection, and (3) heart rate detection in this embodiment.

[0105] (2) Monitoring use Surveillance cameras have become widespread, contributing to the safety and security of citizens' lives. The more surveillance cameras there are, the more important it becomes who checks the surveillance camera footage and how. Currently, it is difficult for people to check images all the time, so images are often stored and used to understand the situation by checking the images after a problem (incident) occurs. For example, a possible use method is to capture the moment a problem occurs from real-time images and respond to the problem immediately. By using the technology disclosed herein, a system can be built that recognizes when a person enters the surveillance camera screen, issues a warning to a person in charge, and has the person check the image in real time. The person in charge does not necessarily need to wait in front of the surveillance camera monitor, and it is possible to build a system in which a warning appears on the mobile device of the person in charge and an image is displayed when a person is detected. Such a system is suitable for monitoring warehouses or back doors of buildings that are not normally accessible to people, or places where access is restricted. In addition, in places where a large number of cameras are monitored intensively, such as buildings, highlighting camera images that detect a person can be useful for preventing abnormalities from being overlooked, early detection of abnormalities, and countermeasures.

[0106] For surveillance purposes, it is possible to obtain more important information by sensing biological information (presence or absence of pulsation) using the second arithmetic circuit 23 in addition to detecting suspicious individuals. The psychological state of tension of a suspicious individual appearing in a surveillance image can be estimated from the heart rate or heart rate fluctuation. From the estimated state of tension, it becomes possible to estimate the danger level (attention level) of that individual. Development of a security system that detects individuals who are highly likely to commit crimes in crowds at airports, commercial facilities, etc., using camera images is underway. The biological detection / biological sensing system of this embodiment can also be applied to such purposes.

[0107] In surveillance applications, the development of image recognition technology has led to the development of a method in which objects are recognized by computers, moving away from the conventional method of human judgment of surveillance images. In such applications, a method is generally used in which images are sent to a host computer and the host computer performs recognition on the side. However, it is necessary to send image data to a computer, which has caused problems such as an increase in communication volume, a decrease in communication speed, and an increase in the load on the host computer. If a surveillance camera can recognize and judge images primarily, the load on communication, recording, and calculation can be significantly reduced. However, there is a problem that if this recognition is not sufficiently reliable, it can lead to events being overlooked. In the human detection method of this embodiment, humans can be detected with high reliability, so when a human is detected, only a partial image including the human can be selectively sent to the host computer. As a result, efficient operation of the surveillance system is possible.

[0108] Furthermore, with the evolution of image recognition technology, it has become possible to identify individuals with high accuracy by images. Currently, in identifying individuals from images, a method is generally used in which an image is sent to a host computer and the host computer performs recognition on the side, but as described above, the load of communication, recording, and calculation is an issue. During calculation, the work of extracting the face for face recognition places a heavy load. By using the detection method of this embodiment, the face part can be easily extracted from within the image. Therefore, it is possible to send only the face part to the host computer and perform individual identification, and the load of individual identification can be significantly reduced. Furthermore, if the number of people is limited, the characteristics can be registered in advance on the surveillance camera side, and the camera side can instantly identify individuals without going through the host computer.

[0109] (3) Automotive applications By installing the system of this embodiment in a car, pedestrians on the road can be constantly recognized, and safer driving can be achieved. Even if a person is hiding behind an object and visibility is poor, it is possible to detect the person and warn the driver. In autonomous driving, it is expected that a problem will arise in which direction to avoid an accident when it is not possible to stop the car by braking and turning left or right is unavoidable. In such a case, it is effective to use this system to detect a human body and change course in a direction to avoid the person. For such applications, the system of this embodiment is particularly suitable because it is required to detect a human body quickly and with high accuracy.

[0110] (4) Human body detection switch There are a wide variety of applications where a human body is detected to switch a power source on and off. For example, applications include detecting a human in a room to control the switch of an air conditioner or a light, controlling an automatic door with high accuracy, detecting a pedestrian at a crosswalk to control a pedestrian signal, and changing the illuminance of a vending machine. This embodiment is applicable to these applications. Using the system of this embodiment, a highly functional switch that does not react to objects or pets but only to people can be realized. For such applications, a small human body detection sensor unit can be configured by integrating the light source, imaging device, and arithmetic circuit of this system.

[0111] (5) Biometric authentication Biometric authentication such as fingerprint authentication, iris authentication, and vein authentication are widely used as a method of personal authentication. As the use of biometric authentication expands, cases and risks of biometric authentication spoofing are increasing. In authentication using images, image duplication techniques such as copying have been used so far. In recent years, with the expansion of the use of iris authentication and 3D printers, the risk of spoofing using even higher accuracy duplication is expanding. As a countermeasure against such risks, a two-stage authentication system is effective. For example, a method is effective in which the target is confirmed to be a living body by the living body detection system of this embodiment, and then normal biometric authentication is performed. By confirming that the target is a living body using the living body detection system of this system, it is possible to increase the reliability of biometric authentication.

[0112] In the above-described applications (1) to (4), the second arithmetic circuit 23 may generate image data in which an area determined to be a human body is superimposed on a visible image, and display the image data on a display. A single image showing a human body area, or an image in which an infrared image and a near-infrared image showing a human body area are superimposed, is different from a human visual image, and therefore there is a problem in humans recognizing the position of a human body even when a human body is detected. To solve this problem, a visible camera may be added to the system shown in FIG. 3A. The second arithmetic circuit 23 may superimpose a visible image obtained from the visible camera and an infrared image obtained from the image sensor 7, and generate image data in which a human body area is superimposed on the visible image. By highlighting and displaying the human body area on the visible image, visibility can be improved. In applications in which a human makes a judgment after detecting a human body, a system capable of superimposing a visible image and an image of a human body area is more effective.

[0113] Furthermore, when using a system with an additional visible camera, the second arithmetic circuit 23 may extract the image contour from the visible image and remove the portion corresponding to the contour from the area estimated to be the human body area. This is because the infrared reflectance may change significantly at the contour of the object, which may result in the contour area of ​​the object being erroneously detected as a human body. By removing the contour area, a human body area image with less noise can be obtained.

[0114] In order to obtain both visible images and near-infrared images for human body detection with a single camera, for example, the visible cut filter can be removed from the camera, and the illumination light can be switched between visible light and near-infrared light for each frame in conjunction with the frame rate of the camera. The advantage of this method is that since a single camera can capture both a visible image and a near-infrared image, there is no parallax between the cameras, making it easy to overlay the images.

[0115] (Embodiment 3) As the third embodiment, a more specific application example in which human body detection and human body information sensing are used in combination will be described. As described above, according to the system of the present disclosure, a human body can be quickly detected and biological information such as a heart rate can be obtained quickly and with high accuracy from data on the detected human body area. By using this, a monitoring system can be realized in personal spaces such as bathrooms, toilets, and bedrooms. Consideration of privacy is particularly important in personal spaces. In a system in which a subject is constantly photographed with a high-resolution camera and the images are used, there is a concern of privacy violation due to the leakage of images, and a psychological burden occurs when photographing due to the presence of the camera. Such problems are solved by the monitoring system of this embodiment.

[0116] Due to the aging of the population, it is said that 10,000 to 20,000 people die in the bath each year in Japan. This number is far greater than the 4,000 to 5,000 traffic fatalities. Causes of death in the bathroom include accidents (drowning) and illness (attacks due to heart disease and brain disease). Most of the deaths are among the elderly, occur in the winter, and the number of cases is increasing annually as the population ages. There are many cases of death in the bathroom, both accidents and illnesses, where the life could have been saved if the abnormality had been discovered early. Because the bathroom is a closed, private space, there are many cases where the discovery is delayed and the person dies. There is a strong demand for a system that can monitor people in the bathroom while respecting their privacy.

[0117] FIG. 7A is a diagram showing a schematic diagram of a biological information detection device and its processing in this embodiment. The configuration of the biological information detection device of this embodiment is basically the same as the configuration shown in FIG. 3A. However, in this embodiment, in consideration of use in a bathroom, the light source 1 and the imaging device 2, which is a camera, are stored in a housing 15 having a waterproof function. The housing 15 has an opening on the front so as not to block the light from the light source 1 and the light returning from the subject 3. The opening is provided with a filter 16 that cuts visible light and transmits near-infrared light. The near-infrared light emitted from the light source 1 passes through the filter 16 and enters the subject 3. The near-infrared light reflected by the subject 3 passes through the filter 16 again, passes through the optical system 5, which is the lens of the imaging device 2, and the bandpass filter 8, and enters the image sensor 7.

[0118] The biological information detection device shown in Fig. 7A further includes a speaker 18 that issues an alarm (alarm sound), and a control device 17. The control device 17 is connected to the imaging device 2, the light source 1, and the speaker 18 and controls them. The control device 17 is an element corresponding to the computer 20 shown in Fig. 2, and includes a first arithmetic circuit 22, a second arithmetic circuit 23, a memory 25, and a control circuit 26, etc., shown in Fig. 3B. When an abnormality is detected, the control circuit 26 in the control device 17 instructs the speaker 18 to issue an alarm.

[0119] In this embodiment, the system is waterproofed and the imaging device 2 is designed to be invisible to the human eye. This makes it possible to reduce the psychological burden of being photographed in a bathroom. The basic system configuration and signal processing are almost the same as in the first embodiment.

[0120] An actual monitoring algorithm in this embodiment will be described below with reference to Figs. 7B and 7C. In this embodiment, the biological information detection device (also referred to as a "monitoring system") shown in Fig. 7A is installed in a corner of the bathroom, and as shown in part (a) of Fig. 7B, the entire bathroom can be monitored. From near-infrared image data acquired by imaging, human body detection (part (b) of Fig. 7B), body movement detection (part (c) of Fig. 7B), and heart rate abnormality detection are performed. (Part (d) of FIG. 7B) is performed. After a human body is detected, if there is no body movement, a first alarm (alarm 1) is issued, for example to the person taking a bath, to alert them. Furthermore, if an abnormal heart rate is detected, a second alarm (alarm 2) is issued, for example to a person outside the bathroom. The operation of the monitoring system of this embodiment will be described in more detail below with reference to the flowchart of FIG. 7C.

[0121] Fig. 7C is a flowchart showing the operation of the monitoring system of this embodiment. First, the first arithmetic circuit 22 detects a human body based on the acquired near-infrared image data in the same manner as in the first embodiment (step S201). If a human body is detected here, the process proceeds to the next step S202 of body movement detection. At this time, the image data used for human body detection is not recorded in a storage device (for example, memory 25 shown in Fig. 3B), and is overwritten with the image data of the next frame of the video, leaving only the data of the human body area. In this way, privacy is protected because no image data that can identify an individual is left behind.

[0122] Next, the second arithmetic circuit 23 detects body movement by comparing data between multiple consecutive frames for the detected human body region (step S202). For example, if there is no body movement for a certain period of time (e.g., 30 seconds) or more, alarm 1 is issued to the person (step S203). This may be an alarm such as, "Are you awake? It's dangerous in the bath. If you are awake, press the OK button." The purpose of alarm 1 is to alert the person and check their condition. If there is no body movement, the second arithmetic circuit 23 further measures the pulsation (step S204). If there is little pulsation or no pulsation can be detected, alarm 2 is issued (step S205). This is an alarm for people outside the bathroom (family members, caregivers, ambulances, etc.). The alarm may be an audio alarm, a telephone call, or an alarm for the purpose of confirmation and rescue requests via the Internet for targets preset in the system.

[0123] According to this embodiment, with a simple system configuration, it is possible to perform three stages of detection: (1) human body detection, (2) body movement detection, and (3) heart rate measurement. Therefore, highly reliable monitoring can be realized.

[0124] In the above example, the three stages of detection, (1) human body detection, (2) body movement detection, and (3) heart rate measurement, are performed in a stepwise manner, but they do not have to be performed in this order. For example, after human body detection, body movement detection and heart rate measurement may be performed in parallel. This allows the bather's heart rate to be constantly monitored, and appropriate advice can be given to the bather. There have been many cases of drowning due to dizziness (hot water fever) caused by changes in heart rate due to vasoconstriction caused by the temperature difference between the changing room and the bathroom, and a decrease in blood flow to the brain and heart due to an increase in body surface blood flow, which in turn causes orthostatic hypotension. By constantly monitoring the heart rate as in this embodiment, changes in the bather's physical condition can be detected in real time. When a change in physical condition is detected, feedback is given to the bather to prevent accidents such as those described above. For example, if the heart rate increases significantly, a message such as "Be careful of dizziness. Hold on to the handrail when standing up and stand up slowly" can be sent.

[0125] In monitoring systems for private spaces such as bathrooms, toilets, and bedrooms, protecting privacy is particularly important. In this embodiment, the near-infrared images captured by the camera are used only for human body detection, and the image data itself is not recorded in a storage medium, but is constantly overwritten with the next frame of data after the human body detection process. The system of this embodiment is also designed not to have an image data output mechanism, making it impossible to obtain image data from outside. Consideration is given to ensuring that privacy is not violated even in the event of an attack by a malicious hacker or the like. Furthermore, the use of near-infrared light makes the camera invisible from the outside, It is possible to monitor without the sense of being photographed. In a monitoring system in a private space, it is particularly important to ensure privacy in terms of hardware and psychology. This embodiment makes it possible to monitor at home while taking privacy into consideration.

[0126] (Embodiment 4) As the fourth embodiment, a system for measuring blood oxygen saturation in a non-contact manner will be described. The main role of blood is to receive oxygen from the lungs and transport it to the tissues, and to receive carbon dioxide from the tissues and circulate it to the lungs. Approximately 15 g of hemoglobin is present in 100 ml of blood. Hemoglobin bound to oxygen is called oxyhemoglobin (HbO2), Hemoglobin that is not bound to oxygen is called reduced hemoglobin (Hb). As shown in Figure 2, the optical absorption characteristics of oxyhemoglobin and reduced hemoglobin are different. Oxyhemoglobin absorbs infrared light with wavelengths of over 830 nm relatively well, while reduced hemoglobin absorbs only red light. It absorbs light (for example, a wavelength of 660 nm) relatively strongly. There is no difference in the absorption rate of near-infrared light with a wavelength of 830 nm. Therefore, in this embodiment, the transmitted light at these two wavelengths, 660 nm and 830 nm, is measured. The ratio of the transmitted light of these infrared light and red light can be calculated to obtain the ratio of the two types of hemoglobin (oxygen saturation). Oxygen saturation is a value that indicates how much of the hemoglobin in the blood is bound to oxygen. Oxygen saturation is defined by the following formula. Oxygen saturation = C(HbO2) / [C(HbO2)+C(Hb)]×100(%) Here, C(Hb) represents the concentration of reduced hemoglobin, and C(HbO2) represents the concentration of oxygenated hemoglobin.

[0127] In addition to blood, living organisms contain other components that absorb light in the red to near-infrared wavelength range, but the time-dependent variation in light absorption rate is mainly due to hemoglobin in arterial blood. Therefore, blood oxygen saturation can be measured with high accuracy based on the variation in absorption rate. Arterial blood pumped from the heart moves through blood vessels as a pulse wave. On the other hand, venous blood does not have a pulse wave. Light irradiated to a living organism is absorbed by each layer of the organism, such as arteries, veins, and tissues other than blood, and passes through the organism. At this time, the thickness of tissues other than arteries does not vary over time. Therefore, scattered light from within the organism shows a time-dependent change in intensity according to the change in the thickness of the arterial blood layer due to pulsation. This change reflects the change in the thickness of the arterial blood layer and does not include the influence of venous blood and tissue. Therefore, by focusing only on the variable component of scattered light, information on arterial blood can be obtained. The pulse can also be obtained by measuring the period of the component that changes over time.

[0128] 8 is a diagram showing the configuration of the system of this embodiment. This system includes light sources 101 and 102, which are two array point image light sources that respectively emit a near-infrared wavelength light beam (e.g., wavelength 830 nm) and a red wavelength light beam (e.g., wavelength 660 nm) arranged at a position away from the living body 3, imaging devices 201 and 202, which are two cameras that can record the illuminated living body surface, and a computer 20 that measures the direct reflected light intensity at the living body surface and the scattered light intensity inside the body from the acquired image separately and calculates living body information from the direct reflected light intensity and the scattered light intensity. Here, in order to measure blood oxygen saturation, two light sources 101 and 102 with different wavelengths and imaging devices 201 and 202 corresponding to the respective light sources are provided.

[0129] 9 is a diagram showing the configuration of an imaging device. Each of the imaging devices 201 and 202 has an optical system 5, which is a lens, and a camera housing 6. The camera housing 6 is equipped with an image sensor 7 and a bandpass filter 802 that selectively transmits near-infrared light (wavelength 830 nm). The camera housing 6 is equipped with an image sensor 7 and a bandpass filter 801 that selectively transmits red light (wavelength 660 nm).

[0130] The light source 101 is, for example, a random dot pattern laser projector from Osela, USA. For example, a random dot pattern laser projector RPP017ES manufactured by Osela Inc., USA, can be used as the light source 102. This laser light source is a near-infrared laser light source of 830 nm, and projects a laser dot pattern of 57,446 points within a viewing angle of 45 x 45 degrees. For example, a random dot pattern laser projector RPP016ES manufactured by Osela Inc., USA can be used as the light source 102. This laser light source is a red laser light source of 660 nm, and projects a laser dot pattern of 23,880 points within a viewing angle of 35 x 35 degrees.

[0131] The computer 20 controls the two imaging devices 201, 202 and the light sources 101, 102 so that the two imaging devices 201, 202 work together to capture images simultaneously. As a result, images of two lights with different wavelengths are generated from the imaging devices 201, 202, as shown on the right side of FIG.

[0132] 10 is a diagram showing the transmittance characteristics of bandpass filters 801 and 802. Bandpass filter 801 has a transmission characteristic of a central transmission wavelength of 830 nm and a bandwidth of 10 nm. Bandpass filter 802 has a transmission characteristic of a central transmission wavelength of 660 nm and a bandwidth of 10 nm. The transmission central wavelengths of bandpass filters 801 and 802 match the central values ​​of the wavelengths of light sources 101 and 102, respectively. Therefore, imaging device 201 captures an image using light with a wavelength of 830 nm, and imaging device 202 captures an image using light with a wavelength of 660 nm.

[0133] The first arithmetic circuit 22 in the computer 20 first extracts the human body region from the moving image, as in the first embodiment. Data selection is performed on the pixel data in the region using a threshold value to remove the directly reflected light component. The average value of the pixel values ​​in the measurement region is then calculated. The above process is performed for each of the 830 nm and 660 nm imaging devices. The average value calculated in this way indicates the intensity of the diffuse reflected light from the living body.

[0134] Fig. 11 is a diagram showing an example of the temporal change in the obtained diffuse reflected light intensity. For both near-infrared light (wavelength 830 nm) and red light (wavelength 660 nm), the reflected light intensity varies over time. Here, the intensities of light from light sources 101 and 102 at the surface of the living body are Ii(830) and Ii(660), respectively, and the time average values ​​of the fluctuating components of the diffuse reflected light from the living body are ΔI(830) and ΔI(660), respectively. The blood oxygen saturation level SpO2 is calculated by the following formula. SpO2= a+b*(log(ΔI(660) / Ii(660))) / (log(ΔI(830) / Ii(830))) The values ​​of a and b in the above equation can be determined from the relationship with the measurement values ​​of an existing pulse oximeter.

[0135] To confirm the accuracy of the measurement device, we used this system to measure oxygen saturation at the fingertip instead of the forehead. A belt (cuff) used for measuring blood pressure was used to apply a constant pressure (200 mmHg) to the upper arm to stop blood flow and measure oxygen saturation at the fingertip.

[0136] For comparison, a commercially available finger-clasp type pulse oximeter was attached to the index finger, and the oxygen saturation of the middle finger was measured non-contact with this system. The first measurement determined the above a and b, and the second and subsequent measurements measured the blood oxygen saturation (SpO2).

[0137] Figure 12 shows a comparison of the measured values ​​obtained by using a contact-type pulse oximeter and the measured values ​​obtained by this embodiment. The two results are roughly the same, indicating that the measurements are performed with high accuracy. The method of this embodiment can simultaneously measure not only the blood oxygen saturation level but also the pulse rate from the pulse wave shown in Figure 11.

[0138] It is known that stress and fatigue can be measured from the fluctuation or frequency characteristics of the pulse wave. By using the system of the present embodiment, the subject's psychological state such as stress can be measured from the pulse wave in a non-contact manner. It is possible to estimate the condition and physical condition.

[0139] (Embodiment 5) As the fifth embodiment, a method of measuring blood oxygen saturation using one camera will be described. In the fourth embodiment, two cameras were used to acquire signals of different light source wavelengths, respectively. This method has the advantage of being able to use existing cameras, but the system configuration becomes complicated because the two cameras are linked to capture images. The acquired data is also independent video data from two cameras, so there is also the issue that data processing to match the time becomes complicated. To avoid this, in this embodiment, a camera capable of acquiring image data for two wavelengths simultaneously with one camera is realized.

[0140] FIG. 13 is a diagram showing the configuration of the biological information detection device of this embodiment. This device has a structure of a twin-lens stereo camera having two imaging devices 201 and 202. Therefore, in this specification, such a method is called a "stereo camera method." The biological information detection device, which is a camera, has a light source 101 (wavelength 830 nm) which is a first laser point image light source and a light source 102 (wavelength 760 nm) which is a second laser point image light source. The reflected light from the biological body illuminated by the light sources 101 and 102 passes through bandpass filters 801 and 802, respectively, and the traveling direction is bent by 90 degrees by mirrors 901 and 902, and an image is formed on the imaging surface of the image sensors 701 and 702 by the optical systems 501 and 502 which are lenses. The bandpass filters 801 and 802 are narrow-band bandpass filters that transmit only light of wavelengths of 830±15 nm and 760±15 nm corresponding to the wavelengths of the two light sources, respectively.

[0141] When the shutter button 11 is pressed, the two light sources 101 and 102 are turned on, and at the same time, the image sensors 701 and 702 capture images of the living body. The captured images are converted into a stereo image format by an image processor (corresponding to the arithmetic circuit 22 or 23 in FIG. 3B), and are stored in a storage device (corresponding to the memory 24 in FIG. 3B) after undergoing image signal processing. Subsequent processing is the same as in the third or fourth embodiment.

[0142] According to this embodiment, the imaging system is configured as a single stereo camera, making the entire system compact and simplifying the signal processing system from the image signal processing to the oxygen saturation calculation in the later stages. This allows for both easy operation and high speed.

[0143] For example, 760 nm and 830 nm in the near infrared region may be used as the wavelengths of the two light sources. The absorption difference between oxygenated hemoglobin and reduced hemoglobin is larger at 660 nm used in the second and third embodiments than at 760 nm, so that oxygen saturation can be measured with higher accuracy. However, since the wavelength of 660 nm is in the visible region, the use of this wavelength may cause a burden on the subject. Furthermore, since fluorescent lamps and LED lighting contain components with a wavelength of 660 nm, there is a problem that they are easily affected by ambient light. In this embodiment, the wavelength of 760 nm was selected in consideration of this. Since the absorption peak of reduced hemoglobin is at 760 nm, it is effective to use a wavelength of 760-780 nm when the light source on the short wavelength side is set in the near infrared region. The wavelengths used are not limited to those mentioned above, and may be appropriately selected according to the purpose and the environment of use.

[0144] (Embodiment 6) As the sixth embodiment, another method for measuring blood oxygen saturation using one camera will be described. In the fifth embodiment, one camera is configured as a stereo camera including two optical systems and two image sensors. In this embodiment, a system is adopted in which an image is divided using multiple lenses, and two different images corresponding to two wavelengths are obtained by one image sensor. The method of this embodiment is called a "stereo lens method." The stereo lens system will be described with reference to FIG. 14.

[0145] FIG. 14 is a cross-sectional view showing a schematic view of a part of the biological information detection device of this embodiment. Although omitted in FIG. 14, the biological information detection device includes two light sources that project dot patterns using light of two wavelengths, 830 nm and 760 nm, for example, in a camera housing 6. As shown in FIG. 14, the optical system 5 has optical systems 501 and 502, which are two pairs of lenses, inside. The optical systems 501 and 502 are designed to form images in two different regions of the imaging surface of the image sensor 7, respectively. In front of the optical systems 501 and 502, two narrow-band bandpass filters 801 and 802 that transmit light of 830 nm and 760 nm, which correspond to the wavelengths of the two light sources, respectively, are arranged.

[0146] With this configuration, two images using light of two wavelengths taken at the same time can be obtained using one image sensor 7. The second control circuit 23 calculates the blood oxygen saturation from these two images in a manner similar to that of the third to fifth embodiments. According to this embodiment, one image signal contains information on two images taken at the same time corresponding to two different wavelengths, making calculation processing easier.

[0147] The results of stress sensing using this stereo lens system are described below. Patent Documents 4 and 5 disclose a method of detecting, by thermography, a drop in temperature around the nose caused by a person feeling stress (tension) or concentrating. Psychological changes cause a drop in blood flow to the nose, which in turn causes a drop in nose temperature. This is commonly detected by thermography. Changes in facial temperature are caused by changes in blood flow. If changes in blood flow can be measured with high accuracy, stress sensing can be performed with higher accuracy and better responsiveness than by measuring changes in surface temperature caused by changes in blood flow.

[0148] FIG. 15A is a diagram showing the results of stress sensing performed using the biological information detection device of this embodiment. As stress, a cold water load was performed by immersing the right hand in cold water (ice water). For comparison, the temperature changes of the nose and cheeks, which are enclosed by dotted lines in FIG. 15B, were measured using thermography. FIG. 15A shows the measurement results. After the cold water load began, the temperature of the nose gradually decreased over about 3 minutes, and stabilized after decreasing by about 1.2°C. After the load ended, it can be seen that the temperature also returned to the original state over about 3 minutes. On the other hand, it can be seen that the temperature of the cheeks was hardly affected by the cold water load and remained stable.

[0149] FIG. 15C is a diagram showing changes in blood flow and blood oxygen saturation obtained using the biological information detection device of this embodiment employing a stereo lens system. From the data on blood flow and oxygen saturation (SpO2) in the face, data on the nose and cheek regions shown by dotted lines in FIG. 15B was extracted. The solid line indicates the time change in blood flow, and the dotted line indicates the time change in oxygen saturation (ΔSpO2). As shown in FIG. 15C, the blood flow in the nose shows a tendency to decrease immediately after the cold and warm stimulation, indicating high time responsiveness. On the other hand, the blood flow in the cheeks shows almost no change. As for oxygen saturation, a decrease in oxygen saturation was observed in the nose along with a decrease in blood flow, but there was almost no change in the cheeks.

[0150] As can be seen from the results, a lot of data can be obtained by measuring blood flow and oxygen saturation in different parts of the face. Based on these data, it is possible to detect emotions (i.e., feelings), physical condition, or concentration with high accuracy. Changes in blood flow due to the influence of the autonomic nervous system Since blood flow varies depending on the part of the face, it is particularly important to measure changes in blood flow in specific areas using a camera. In this case, the accuracy of the measurement can be improved by simultaneously measuring an area with little change in blood flow as a reference.

[0151] (Embodiment 7) As the seventh embodiment, another method for measuring blood oxygen saturation using one camera will be described.

[0152] Fig. 16 is a cross-sectional view showing a schematic configuration of a biological information detection device in this embodiment. This device includes a stereo adapter 10 that can be attached to an optical system 5 that is a normal lens. The stereo adapter 10 is an attachment that includes four mirrors 151, 152, 153, and 154, and two band-pass filters 801 and 802. By using the stereo adapter 10, two images corresponding to two wavelengths can be formed in two different areas of the imaging surface of the image sensor 7. This method is called the "stereo adapter method."

[0153] In the stereo adapter method, two different images corresponding to two wavelengths can be obtained by one image sensor 7 using two pairs of opposing mirrors. Although omitted in FIG. 16, two light sources that emit light of two wavelengths, 830 nm and 760 nm, respectively, are actually built into the camera housing 6. The stereo adapter 10 is attached to the tip of the optical system 5. Two pairs of mirrors (a pair of mirrors 151 and 152 and a pair of mirrors 153 and 154) bend the optical path twice to introduce it into the optical system 5. Narrowband bandpass filters 801 and 802 that transmit light of wavelengths of 830 nm and 760 nm corresponding to the wavelengths of the light sources are mounted between the optical system 5 and the mirrors 151, 152, 153, and 154.

[0154] This biological information detection device can acquire images of two wavelengths at the same time using one image sensor 7. The basic concept is the same as in embodiment 5. The stereo lens method has the advantage that the lenses can be made small, making it possible to miniaturize the entire system. On the other hand, the stereo adapter method has the advantage that, although the entire system becomes larger, it can use high-performance camera lenses, improving the resolution, and that it can use lenses and zoom lenses with different magnifications. The advantage of the stereo adapter method is that it increases the degree of freedom of the system.

[0155] A study was conducted to detect human emotions using the bioinformation detection device, which is a camera, of this embodiment. As described in the fifth embodiment, it is possible to stably detect human emotions or emotions such as stress based on blood flow. Due to changes in human emotions or emotions, the autonomic nerves are activated and the blood flow rate on the skin surface changes. This change in blood flow changes the complexion. Humans normally detect the emotions or physical condition of a target person from the subtle changes in complexion. It is believed that the reason why a doctor who is called an expert can diagnose the physical condition or cause of a disease just by looking at the face of a patient is because he or she can distinguish physical changes from subtle changes in the patient's complexion. It is also said that subtle changes in complexion play an important role when a person with a keen sense reads the emotions of others, along with subtle changes in facial expression. Furthermore, in the fields of games, animation, and computer graphics, which have made significant progress in recent years, research is being widely conducted on subtle changes in a person's complexion in order to give a natural impression or reality to a scene. As can be seen from these examples, facial color represents a person's emotions and physical condition, and studies are underway to read emotions by measuring facial color (for example, Non-Patent Document 2). However, attempts to measure emotions directly from facial color are difficult to stably measure and are not suitable for practical use. This is because changes in facial color vary from person to person, are subtle, and are strongly affected by ambient light and the camera, making stable measurement difficult. There is a demand for a method of measuring changes in facial color more stably and with high accuracy using a method other than colorimetry.

[0156] It is known that a person's complexion is determined primarily by the amount of melanin pigment contained in the skin surface (dermis) and the concentration of hemoglobin (oxyhemoglobin and deoxyhemoglobin (reduced) hemoglobin) in the blood. Melanin pigment does not change in a short period of time (due to aging, sunburn, etc.) The change in emotion can be stably measured by measuring the blood flow rate. In this embodiment, in order to detect the change in emotion, the blood flow rate of oxygenated hemoglobin and deoxygenated hemoglobin, which change the facial color, is directly measured instead of measuring the facial color. As described in the fifth embodiment, the change in the blood flow rate of the face differs depending on the part. This is because the susceptibility to the autonomic nervous system differs depending on the part of the face. For example, the nose has many arteriovenous anastomoses and is susceptible to the autonomic nervous system, whereas the forehead is less susceptible to the cutaneous vasoconstrictor nerve. The arithmetic circuit 22 in this embodiment calculates the blood flow rates of different parts and compares them, thereby making it possible to detect the change in emotion with high accuracy.

[0157] The measurement of changes in blood flow accompanying changes in emotions will be explained below. A stereo adapter type camera as shown in Figure 16 was used to measure blood flow. The subject sat in front of the camera, and the camera captured an image of the subject's face. From a stable state, the subject watched videos that evoked fear, laughter, surprise, disgust, etc., and a color image of the subject's face was obtained. Emotional fluctuations were read from the changes in facial expressions in the video scenes and color images, and the changes in blood flow at that time were measured. Blood flow was measured at the nose and forehead, as shown by the dotted lines in Figure 17A.

[0158] FIG. 17B shows the time change of total blood flow (oxygenated hemoglobin and deoxygenated hemoglobin) and the time change of the ratio of blood flow of oxygenated hemoglobin (oxygen saturation) when the emotion of laughter is induced. It can be seen that the value of blood oxygen saturation changes significantly along with the total blood flow due to the emotional change of laughter. A similar study was performed for other emotions. The results are shown in FIG. 18. FIG. 18 is a plot of the relationship between blood flow and oxygen saturation that occurred during each emotional change, with oxygen saturation on the horizontal axis and total blood flow on the horizontal axis. The amount of change in total blood flow and blood oxygen saturation when emotions other than laughter, such as sadness, surprise, depression, fear, disgust, anger, concentration, and happiness, were calculated in the same way as above. The same measurements were performed on 12 subjects. FIG. 18 shows the average experimental results of the 12 subjects. Although there were individual differences, the changes in total blood flow and blood oxygen saturation showed similar trends in almost all subjects. These results demonstrated that it is possible to detect changes in emotions using at least one of blood flow and oxygen saturation.

[0159] As shown in FIG. 17B, the relationship between oxygen saturation and blood flow rate differs depending on the part of the face. Therefore, by obtaining the blood flow rate and oxygen saturation rate of multiple parts of the face, it is possible to sense emotions with higher accuracy. In the emotion sensing test performed on this embodiment, measurements were taken at three points: the forehead, cheeks, and nose. The changes in oxygen saturation and blood flow rate due to emotional changes were different between the forehead, cheeks, and nose. Therefore, by creating a table showing the relationship between the oxygen saturation and the change in blood flow rate of each part in advance and calculating the correlation between the actual measured values ​​of oxygen saturation and blood flow rate, it is possible to detect emotional changes with high accuracy.

[0160] (Embodiment 8) As the eighth embodiment, a method of measuring blood oxygen saturation using one camera without splitting an image in an optical system will be described. In the third to seventh embodiments, a method of splitting light from two light sources corresponding to two wavelengths for sensing and calculating biological information such as oxygen saturation has been described. The biological information detection device of this embodiment does not split an image, but obtains two image signals of different wavelengths using an image sensor.

[0161] FIG. 19A is a diagram showing a schematic configuration of a biological information detection device in this embodiment. This device separates two images corresponding to two wavelengths not by an optical system but by an image sensor 703. Although a point image light source is omitted in FIG. 19A, in reality, two light sources that respectively emit light of two wavelengths, 830 nm and 760 nm, are built into the camera housing 6. A band-pass filter that transmits light of wavelengths between 730 nm and 850 nm is provided on the front surface of the optical system 5, which is the camera lens. A bandpass filter 8 is disposed on the optical system 5. The bandpass filter 8 cuts visible light and long-wavelength infrared light. The light transmitted through the bandpass filter 8 is imaged on the imaging surface of the image sensor 703 by the optical system 5. The image sensor 703 used here differs from ordinary image sensors in that it has two types of bandpass filters that transmit near-infrared light.

[0162] FIG. 19B is a diagram showing a plurality of filters facing a plurality of photodetection cells arranged on the imaging surface of the image sensor 703. The image sensor 703 has a filter IR1 that selectively transmits light of 680-800 nm, and a filter IR2 that selectively transmits light of wavelengths of 800 nm or more. The filters IR1 and IR2 are arranged in a checkerboard pattern. The lower diagram in FIG. 19B is a diagram showing an example of the wavelength dependency of the transmittance of the filters IR1 and IR2. The image sensor 703 detects two images based on light of two light source wavelengths of 760 nm and 830 nm using a plurality of photodetection cells (also called pixels).

[0163] The first arithmetic circuit 22 (FIG. 3B) reads out data of a plurality of photodetector cells corresponding to a wavelength of 760 nm and data of a plurality of photodetector cells corresponding to a wavelength of 830 nm from the image sensor 703 individually. A human body area is detected for each image. Then, as shown in FIG. 19C, the second arithmetic circuit 23 (FIG. 3B) generates an image of a wavelength of 760 nm and an image of a wavelength of 830 nm by interpolating the data of the missing pixels in each data. The second arithmetic circuit 23 calculates the blood flow rate and the oxygen saturation from these two images. Since these two images completely overlap, the calculation is simpler than calculating the blood flow rate and the oxygen saturation from two divided images. The problem with this method is that there is a concern that color mixing between light sources may occur because the shielding ability of the filter is low compared to using bandpass filters corresponding to each light source.

[0164] (Embodiment 9) As a ninth embodiment, a biological information detection device capable of acquiring not only two images corresponding to light sources of two wavelengths but also a color image without dividing the image will be described.

[0165] Fig. 20A is a diagram showing the configuration of a biological information detection device in this embodiment. Although the point image light source is also omitted in Fig. 20A, two light sources emitting light of two wavelengths, 830 nm and 760 nm, respectively, are built into the camera housing 6. In this embodiment, a bandpass filter is not provided in front of the optical system 5, which is a lens, in order to acquire a color image. Illumination light of the visible light and laser light source is imaged on the imaging surface of the image sensor 704 by the optical system 5. The image sensor 704 used here is different from a normal image sensor and includes a photodetector cell for acquiring a color image and two types of photodetector cells for acquiring a near-infrared image.

[0166] FIG. 20B is a diagram showing a plurality of bandpass filters (or color filters) arranged on the imaging surface of the image sensor 704. The diagram below in FIG. 20B shows the wavelength dependency of the relative sensitivity of the pixels facing each filter. As shown in FIG. 20B, three types of color filters (R, G, B filters) that transmit blue, green, and red light, respectively, a filter IR-1 that transmits light of 650 nm or more, and a filter IR-2 that transmits light of 800 nm or more are arranged on the imaging surface. The arrangement in which two green filters are arranged diagonally adjacent to each other and red and blue filters are arranged diagonally to the diagonal side is the same as the Bayer arrangement of a normal image sensor. The difference from the conventional image sensor is that two filters IR-1 and IR-2 are arranged next to the basic unit of four filters in the Bayer arrangement.

[0167] The filter IR1 in the eighth embodiment and the filter IR-1 in the present embodiment have different transmission wavelength ranges. The filter IR1 in the eighth embodiment is a relatively narrow-band filter that transmits light in the wavelength range of 650-800 nm, whereas in the present embodiment, A filter that transmits light in the wavelength range of 650 nm or more is used. This is to simplify the manufacturing process of the image sensor 703. However, the present invention is not limited to this, and the filter shown in the eighth embodiment can also be used. The filter IR-1 of this embodiment has sensitivity to both 760 nm and 830 nm. Therefore, the second arithmetic circuit 23 subtracts the signal of the photodetector cell facing the filter IR-2 from the signal of the photodetector cell facing the filter IR-1 to calculate a signal corresponding to 760 nm, and then calculates the blood oxygen saturation. As a result, as shown in FIG. 20C, the image sensor 704 produces images of red, blue, and green (color images) and images of each wavelength of 760 nm and 830 nm.

[0168] This method is more prone to color mixing than the eighth embodiment. However, a color image and information on blood flow rate and blood oxygen saturation can be obtained simultaneously with a simple system using one camera. The advantage of this system is that visible and near-infrared images can be obtained without parallax because a single camera is used to obtain visible and near-infrared images. This is particularly effective in applications where visible and near-infrared images are displayed superimposed on each other.

[0169] Here, a configuration example of a bioinformation sensing camera using a multispectral sensor corresponding to five wavelengths, two wavelengths in the infrared range and three wavelengths in the visible range (red, blue, green), has been described. With the human body detection camera described in the first embodiment, color imaging and human body detection are possible by measuring four wavelengths, one wavelength in the infrared range and three wavelengths in the visible range (red, blue, green). For such applications, for example, a multispectral sensor having four types of color filters corresponding to four wavelengths, as shown in FIG. 20D, can be used. The color filter assigns a near-infrared (IR) pixel to one of two green pixels in a Bayer array that is normally used in image sensors. Here, a camera corresponding to a system that turns on a near-infrared light of 850 nm is assumed, and a filter that selectively transmits 850 nm is selected as the near-infrared filter. By using such a camera, it becomes possible to use a single camera system as both a normal color camera and a bioinformation detection camera. Only one surveillance camera is required, and it is easier to cut out a color image of the part where a human body is detected than to use two cameras. Here, a color filter corresponding to 850 nm was used, but the near-infrared filter can be changed according to the near-infrared light source used.

[0170] (Other embodiments) Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments and various modifications are possible. The processes described for each of the above embodiments may also be applied to other embodiments. Examples of other embodiments will be described below.

[0171] In the above embodiment, a laser light source is used as the array point image light source, but other types of light sources may be used. For example, it is possible to use a cheaper LED light source. However, LED light sources have a lower linearity and tend to spread compared to laser light sources. For this reason, when using an LED light source, a dedicated light collecting optical system may be used, or the distance between the object to be imaged and the camera may be restricted.

[0172] The biological information detection device may include an adjustment mechanism for adjusting the focus of the optical system. Such an adjustment mechanism may be realized, for example, by a motor (not shown) and a control circuit 26 shown in FIG. 3B. Such an adjustment mechanism adjusts the focus of the optical system so as to maximize the contrast of the image of the dot pattern projected onto the object by the light source. This improves the accuracy of the contrast calculation described in the first embodiment.

[0173] The first arithmetic circuit 22 detects a biometric region using an image signal output from the image sensor. In this case, if multiple biometric regions (face and hand regions of different people or the same person) are detected in the image, the first arithmetic circuit 22 may determine the biometric region to be detected based on the size or shape of the detection region.

[0174] The second arithmetic circuit 23 may generate information on the inside of the epidermis including at least one of the concentration of melanin pigment, the presence or absence of spots, and the presence or absence of bruises based on the image signal. As described above, the epidermis contains melanin pigment that strongly absorbs light. Spots and bruises are caused by an increase in melanin pigment. Therefore, the concentration of melanin pigment, spots, and bruises can be detected based on the intensity distribution of light from the surface of the living body. The second arithmetic circuit 23 may, for example, extract a direct reflection light component from the surface of the living body from the image signal, and generate information on the inside of the epidermis including at least one of the concentration of melanin pigment, the presence or absence of spots, and the presence or absence of bruises based on the direct reflection light component. The direct reflection light component can be obtained, for example, by determining whether the contrast in embodiment 1 exceeds a predetermined threshold value or by removing low-frequency components in the image signal.

[0175] In this disclosure, the following have been described: a double camera system using two cameras (FIG. 8), a stereo camera system in which two sets of optical systems and two sets of image sensors are mounted on one camera (FIG. 13), a stereo lens system using two sets of lenses and one image sensor (FIG. 14), a stereo adapter system using one lens and one image sensor using a lens adapter (FIG. 16), and a system that splits an image using an image sensor (FIGS. 19A and 20A). As already mentioned, each system has advantages and disadvantages, so the most suitable system can be selected depending on the application.

[0176] As described above, according to the embodiment of the present disclosure, it is possible to measure not only the heart rate and blood flow rate but also the blood oxygen saturation level without restraining the subject and without contacting the subject with a detection device such as a sensor. The subject's emotions or physical condition can also be estimated from the measured values ​​of blood flow rate and oxygen saturation at different parts of the subject. [Industrial Applicability]

[0177] The biological information detection device of the present disclosure is useful for detecting biological information such as a subject's heart rate, blood flow, blood pressure, blood oxygen saturation, emotion, and physical condition. [Explanation of symbols]

[0178] 1, 101, 102 light source 2, 201, 202 Imaging device 3. Living organisms 4. Biological Surface 5, 501, 502 optical system 6 Camera housing 7, 701, 702, 703, 704 Image Sensors 8, 801, 802 Bandpass Filter 901, 902, 151, 152, 153, 154 Mirror 11 Shutter button 15. Cabinet 16 Filters 20 Computer 21 Input Interface 22 1st calculation circuit 23 Second arithmetic circuit 24 Output Interface 25 Memory 26 Control circuit 27 Display 31 Capillaries 32 Arterioles and Veins 33 Epidermis 34 Dermis 35 Subcutaneous tissue L0 Light from the light source L1 surface reflected light L2 Internally scattered light

Claims

1. a light source that projects a plurality of dots of a first light onto a target; an image sensor that detects a second light from the object onto which the plurality of dots are projected and generates one or more image signals; and a processing circuit for determining whether the object is a living body based on the one or more image signals. Processing unit.

2. The processing circuitry includes: performing a biometric authentication of the subject; performing personal authentication of the target based on a result of determining whether the target is a biometric person and a result of the biometric authentication; The processing device of claim 1 .

3. The processing circuit, in determining whether the object is a living body, determining whether the object is a living body by analyzing an intensity distribution of the second light from the object based on the one or more image signals; The processing device according to claim 1 or 2.

4. The processing circuitry includes: generating biometric information corresponding to a heartbeat of the living body based on the one or more image signals when the target is a living body; The processing device according to claim 1 or 2.

5. In generating the biological information, the processing circuit generating the biometric information based on a plurality of first pixels corresponding to a face of the living body, the first pixels being included in the one or more image signals; The processing circuitry further comprises: generating information on a depth direction of the object based on a plurality of second pixel signals corresponding to positions different from a face of the living body, the second pixel signals being included in the one or more image signals; The processing device according to claim 4 .

6. The biometric authentication includes at least one of fingerprint authentication, iris authentication, and vein authentication. The processing device according to claim 2 .

7. 1. A computer-implemented information processing method, comprising: causing a light source to project a plurality of dots of a first light onto a target; causing an image sensor to detect a second light from the object onto which the plurality of dots are projected to generate one or more image signals; determining whether the object is a living body based on the one or more image signals; Information processing methods.

8. A program for causing a computer to execute the information processing method according to claim 7.

Images