Processing apparatus, processing method, and program

Abstract

It detects biological information with high accuracy without restraining the subject's body. [Solution] The processing device comprises a light source that projects a plurality of dots onto an object using a first light, an image sensor that detects a second light from the object onto which the plurality of dots are projected and generates one or more image signals, and a processing circuit. The processing circuit generates biological information corresponding to the heart rate of the object based on the one or more image signals, and generates information in the depth direction of the object based on the one or more image signals.

Description

Technical Field

[0001] The present disclosure relates to a processing device, a processing method, and a program.

Background Art

[0002] As fundamental parameters for judging a human's health condition, heart rate, blood flow rate, blood pressure, blood oxygen saturation, etc. are widely used. These biological information related to blood is usually measured by a contact-type measuring instrument. The contact-type measuring instrument has caused discomfort to the subject, especially when continuously measuring over a long period of time because it restrains the subject's living body.

[0003] Various attempts have been made to easily measure fundamental biological information for judging a human's health condition. For example, Patent Document 1 discloses a method for non-contact detection of a heart rate from image information such as a face photographed by a camera. Patent Document 2 discloses a method for measuring blood oxygen saturation by utilizing the laser Doppler effect of laser light scattered behind the living body surface using a white light source and a laser light source. Patent Document 3 discloses a method for measuring blood oxygen saturation by excluding the influence of ambient light using an ordinary color camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Non-Patent Documents

[0005] [Non-Patent Document 1] Aoki, et al., "Non-contact, unrestrained respiration monitoring system for sleeping persons using near-infrared bright spot matrix illumination," Transactions of the Institute of Electrical Engineers of Japan, Electronics, Information and Systems Division, June 1, 2004, 124(6), pp. 1251-1258. [Non-Patent Document 2] Kuroda, et al., "Analysis of Facial Color and Facial Skin Temperature Associated with Emotional Fluctuations and Synthesis of Facial Color," Proceedings of the Human Interface Society, February 16, 1999, 1(1), 15-20. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The conventional technologies described above had issues with the accuracy or stability of the acquired biological information, or with the ease of acquiring the biological information. In particular, the instability of measurements due to body movement and fluctuations in ambient light was a major challenge for practical application.

[0007] This disclosure provides a technology that can detect biological information with high accuracy and stability without restraining the subject's body. [Means for solving the problem]

[0008] A biological information detection device according to one aspect of this disclosure includes a first light source that projects a plurality of first dots by first light onto an object including a living organism, The system includes a plurality of first photodetector cells that detect second light from the object onto which the plurality of first dots are projected, and displays a first image of the object onto which the plurality of first dots are projected. An imaging device that generates and outputs a first image signal, First arithmetic circuit and The second arithmetic circuit and It is equipped with.

[0009] The first image described above contains multiple pixels.

[0010] The first arithmetic circuit detects a first part corresponding to the living body in the first image using the first image signal.

[0011] The second arithmetic circuit calculates biological information regarding the living body using data of pixels within the first part of the first image among the plurality of pixels.

[0012] The above general or specific aspects may be implemented by an element, a device, a system, a method, an integrated circuit, a computer program, a recording medium, or any combination thereof.

Advantages of the Invention

[0013] According to one aspect of the present disclosure, it becomes possible to detect biological information such as the heartbeat, blood flow rate, or blood oxygen saturation of a living body stably at high speed without restraining the living body of the subject. Further, according to another aspect of the present disclosure, it is also possible to determine a state such as the physical condition or emotion of the subject from information regarding blood as described above.

Brief Description of the Drawings

[0014] [Figure 1A] A diagram for explaining the basic concept of acquisition of biological information in an embodiment of the present disclosure [Figure 1B] A diagram for explaining the characteristics of an image of the living body surface acquired by an imaging device [Figure 2] A diagram showing the configuration of the biological information detection device of Embodiment 1 [Figure 3A] A diagram showing an example of the configuration of the imaging device, the output image, and biological information in Embodiment 1 [Figure 3B] A block diagram showing the configuration of the computer 20 in Embodiment 1 [Figure 3C] A diagram for explaining the method of respiration sensing in Embodiment 1 [Figure 4A] The first diagram showing an experimental example of human body detection in Embodiment 1 [Figure 4B] The first diagram showing an experimental example of human body detection in Embodiment 1 [Figure 5] This figure shows an example of a method for calculating contrast used for human body detection in Embodiment 1. [Figure 6] Flowchart showing the image processing flow in Embodiment 1 [Figure 7A] A schematic diagram showing the biological information detection device and its processing in Embodiment 3. [Figure 7B] Diagram illustrating the algorithm of the monitoring system in Embodiment 3. [Figure 7C] Flowchart showing the algorithm of the monitoring system in Embodiment 3 [Figure 8] Diagram showing the configuration of the biological information detection device in Embodiment 4. [Figure 9] This figure shows an overview of bio-information sensing using two imaging devices in Embodiment 4. [Figure 10] Figure showing the transmission characteristics of the two bandpass filters in Embodiment 4. [Figure 11] A diagram showing an example of a pulse wave measured by the method of Embodiment 4. [Figure 12] This figure shows the results of measuring blood oxygen saturation using the method of Embodiment 4 and the conventional method. [Figure 13] This figure shows the configuration of the stereo camera type biological information detection device in Embodiment 4. [Figure 14] This figure shows the configuration of the stereo lens type biometric information detection device in Embodiment 5. [Figure 15A] This figure shows the results of stress sensing performed using the biological information detection device of Embodiment 6. [Figure 15B] Figure showing the nose and cheek area in the image of Embodiment 6. [Figure 15C] This figure shows the changes in blood flow rate and blood oxygen saturation obtained using the biological information detection device of Embodiment 6. [Figure 16] A schematic cross-sectional view showing the configuration of the biological information detection device in Embodiment 7. [Figure 17A] Figure showing the nose and forehead in the image of Embodiment 7. [Figure 17B]This figure shows the time-dependent changes in total blood flow (oxyhemoglobin and deoxygenated hemoglobin) and the time-dependent changes in the percentage of blood flow containing oxyhemoglobin (oxygen saturation) when laughter is induced in Embodiment 7. [Figure 18] A diagram showing the relationship between emotions, total blood flow, and oxygen saturation. [Figure 19A] A schematic diagram showing the configuration of the biological information detection device in Embodiment 8. [Figure 19B] Figure showing multiple color filters in Embodiment 8 [Figure 19C] Figure showing an example of an image generated in Embodiment 8. [Figure 20A] Diagram showing the configuration of the biological information detection device in Embodiment 9. [Figure 20B] Figure showing multiple color filters in Embodiment 9 [Figure 20C] A diagram showing an example of an image generated in Embodiment 9. [Figure 20D] This figure shows an example configuration of a multispectral sensor having four types of color filters: R, G, B, and IR. [Figure 21] This figure shows an example configuration (comparative example) of a biological information sensing system using an imaging device. [Figure 22] This diagram shows the absorption and scattering coefficients of hemoglobin, melanin, and water, which are major components of living organisms, in the wavelength range from visible light to near-infrared light. [Modes for carrying out the invention]

[0015] Before describing embodiments of this disclosure, we will explain the knowledge that forms the basis of this disclosure.

[0016] Remote biosensing using cameras is expected to have various applications because it allows for continuous, long-term measurements without any sense of constraint. For example, in medical institutions such as hospitals, it is expected to be used to continuously monitor patients' conditions and respond quickly to sudden changes in their condition, as well as to utilize the data obtained from long-term monitoring for diagnosis. In addition to applications in medical institutions, it is also expected to be used in homes to prevent sudden death during sleep and to monitor sleep apnea syndrome. Furthermore, it is expected to be used to acquire physical information data from daily life that is regularly obtained at home or in the workplace, and to analyze the data stored on a server via the cloud to continuously monitor changes in physical condition and use it for health management, as well as to share the acquired data with medical institutions to use it for medical purposes. For such continuous acquisition of biosensing, a system is required that can measure biosensing continuously without any sense of constraint or particular awareness. A camera-based system is considered ideal for such applications because it enables non-constrained remote measurement.

[0017] However, when using cameras to sense biometric information in daily life, privacy considerations are necessary. Systems that record high-resolution images that could identify an individual onto a storage device should be avoided as much as possible due to the risk of image leakage. Even if the acquired images themselves are not recorded, systems in which the camera (or camera lens) is visible from the measurement system can cause psychological resistance to the subject. For this reason, a system in which the camera is not visible is desirable.

[0018] Due to the strong demand in the medical and healthcare fields as described above, various research institutions are developing systems to realize remote biosensing using cameras, and several products are commercially available. The challenges lie in the accuracy and stability of the measurements. When photographing the human body with a camera, the majority of the light incident on the camera is light reflected from or near the surface of the skin. Since there are no blood vessels and metabolism does not occur in the stratum corneum, the outermost layer of the skin, it is not possible to obtain biological information from the surface reflection component. It is necessary to detect light that has penetrated into the interior of the skin and reflected from the epidermis, where blood vessels are present. In the light reflected from the skin, the component reflected from the skin surface is dominant, and the light that has penetrated into the interior of the skin is rapidly lost due to the strong light absorption of living organisms. For this reason, the proportion of light containing biological information in the reflected light is low. Furthermore, in systems that do not have a measurement illumination system and acquire images using ambient light, instability due to changes in the acquired image signal in response to changes in the surrounding ambient light has been a problem.

[0019] Furthermore, in the case of remote sensing, the instability of measurements due to body movement is a major challenge. Changes in the measurement area and orientation (angle) relative to the camera due to body movement cause fluctuations in the acquired signal, making stable measurements impossible. As already mentioned, the majority of the signal acquired by the camera consists of components resulting from surface reflection on the skin that do not contain biological information, and the signal component containing biological information is weak. Changes in the measurement area and orientation due to body movement cause significant changes in surface reflection, making it impossible to acquire weak biological information. This is the biggest challenge of remote biological information sensing using cameras. In order to perform stable measurements, it is necessary to stop body movement and perform measurements in a stationary state, which prevents the full utilization of the advantage of unconstrained measurement.

[0020] In remote biosensing using cameras, images of the subject are obtained, and methods have been used to reduce the effects of body movement using these images. In such methods, the face is detected from the image obtained from the camera using face recognition, and then facial part recognition is performed to identify the part to be measured, so that even if there is body movement, the measurement area is always captured on the image and biological information is detected. For example, if the forehead can be detected by part recognition, then even if the forehead moves on the image due to body movement, information about the forehead can always be obtained.

[0021] However, there are two challenges to using image recognition. First, the computational load is high because facial features are extracted from the entire image to recognize the face. Therefore, it was necessary to either use high-performance and expensive computing devices to perform high-speed image processing, or to reduce the frame rate so that the next frame could be processed only after the previous frame had been processed. High-speed processing is costly, leading to larger and more expensive equipment. If slow processing is used, the measurement accuracy decreases. The second challenge is that even if the effects of body movement are reduced by using image recognition, there are limits to improving the detection accuracy due to changes in the orientation of the subject (the angle of the normal direction of the surface of the subject relative to the camera's front direction) due to body movement. Since the reflectance of surface reflected light is angle-dependent, when the orientation of the part to be measured changes due to body movement, the amount of surface reflected light reaching the camera changes, and the detection accuracy decreases.

[0022] Thus, the biggest challenge in remote biometric sensing using cameras is the instability of measurements due to body movement. Due to the low reliability resulting from this instability, remote biometric sensing using cameras has not yet become widely used in various applications.

[0023] As mentioned above, various attempts have been made to measure basic biological information for determining a person's health status. For example, Patent Document 1 proposes a method for detecting heart rate non-contact from image information such as a face captured by a camera. The method in Patent Document 1 determines the heart rate by analyzing the spatial frequency components of the acquired color image. However, this method is not stable because its accuracy is reduced by the influence of ambient light such as indoor lighting.

[0024] A pulse oximeter is commonly used to measure blood oxygen saturation. The device works by placing a finger between two wavelengths of light in the red to near-infrared range and measuring the transmittance. This allows for the determination of the ratio of oxyhemoglobin concentration to deoxyhemoglobin concentration in the blood. Pulse oximeters can measure blood oxygen saturation with a simple configuration. However, because they are contact-type devices, they have the drawback of being restrictive.

[0025] An example of a non-contact blood oxygen saturation measuring device is disclosed in Patent Document 2. This device uses a white light source and a laser light source to measure blood oxygen saturation by utilizing the laser Doppler effect caused by laser light scattered behind the biological surface. However, this method has the drawbacks of having a complex device configuration and producing a weak signal.

[0026] Patent Document 3 proposes a method for measuring blood oxygen saturation using a conventional color camera to eliminate the influence of ambient light. However, even with this method, the influence of reflected light from the skin surface is significant, making it difficult to measure blood oxygen saturation with high accuracy and stability.

[0027] Thus, conventional non-contact methods for measuring biological information such as heart rate, blood flow, and blood oxygen saturation have challenges in terms of accuracy and stability.

[0028] Furthermore, in order to measure biological information using a camera, it is necessary to identify the measurement area (e.g., the forehead area) from the camera image and detect biological information using the image information within that area. There are two methods for identifying the measurement area: specifying the measurement area before measurement and automatically setting the measurement area from the image. In the method of specifying the measurement area before measurement, the measurer specifies the measurement area from the subject's image before starting the measurement and continues to measure the same area during the measurement. This method is simple, but the subject is not allowed to move during the measurement, and the advantage of non-contact measurement, which is non-restraint, is lost. To avoid this, a method of automatically setting the measurement area is sometimes used. In such a method, for example, if the measurement area is the forehead area, the camera can perform face recognition from the acquired image, and then perform facial part recognition to identify the forehead area on the image and measure that part.

[0029] Figure 21 is a schematic diagram illustrating an example (comparative example) of such a system. The imaging device 2, which is a camera in this system, comprises a camera housing 6 having an image sensor 7 and an optical system 5, which is a lens. The image sensor 7 in the imaging device 2 has a built-in or connected processing unit (or processing circuit). The processing unit performs face recognition on the acquired image (for example, part (a) in Figure 21), identifies the forehead, and extracts image data of the forehead (for example, part (b) in Figure 21). Then, it generates biometric information such as heart rate variability (for example, part (c) in Figure 21) from the image data of the forehead. Part (c) in Figure 21 shows the time variation of the averaged value of the forehead image data shown in part (b) in Figure 21 within the forehead region. The facial part recognition algorithm used here places a heavy load on the computer's image processing. Therefore, the cost of the processing unit is high for high-speed processing. In addition, the method using image recognition has the problem that the recognition rate decreases when the body orientation changes or when part of the face is hidden. Furthermore, it also has the problem of being easily affected by ambient light. For these reasons, it was difficult to measure continuously and stably.

[0030] In addition to the issues mentioned above, methods using facial recognition also have the problem of not being able to measure parts of the body other than the face (such as the arms or chest). Furthermore, there is the issue of needing to consider privacy. Subjects experience psychological stress from constantly having their images captured by a camera. However, high-resolution cameras are necessary for high-accuracy image recognition. Therefore, constantly being filmed by a camera may cause psychological burden on subjects.

[0031] The inventors focused on the above-mentioned problems and considered a configuration to solve them. As a result, they found that the above-mentioned problems can be solved by acquiring an image using a light source that projects a dot pattern of light onto a biological surface, detecting biological regions (e.g., human body regions) on the image based on the ratio (e.g., contrast described later) of the component due to directly reflected light (also called "surface reflected light") and the component due to scattered light inside the biological body (also called "internal scattered light") in the image, and separating the component due to directly reflected light and the component due to scattered light inside the biological body in the detected biological region by signal processing. In other words, the biological information detection device first detects a region on the image that is presumed to be a biological region, and measures biological information within that region. This method, as will be explained in detail later, significantly reduces the amount of computation required for image processing and enables high-speed and stable detection of biological information.

[0032] A biometric information detection device relating to one aspect of this disclosure is: A first light source projects multiple first dots, including living organisms, onto an object. An imaging device includes a plurality of first photodetector cells that detect second light from the object onto which the plurality of first dots are projected, and generates and outputs a first image signal showing a first image of the object onto which the plurality of first dots are projected, First arithmetic circuit and The second arithmetic circuit and It is equipped with.

[0033] The first image described above contains multiple pixels.

[0034] The first arithmetic circuit uses the first image signal to detect a first portion of the first image that corresponds to the living organism.

[0035] The second arithmetic circuit calculates biological information relating to the living organism using data from pixels within the first portion of the first image among the plurality of pixels.

[0036] The first arithmetic circuit can detect biological regions based on the ratio of the pixel signal in the region where the dot pattern is projected to the pixel signal in the surrounding region. For example, it can determine whether or not a living organism is present at the location corresponding to a specific pixel based on the ratio of the standard deviation to the mean value (referred to as "contrast") of a specific pixel in the image and a plurality of pixels arranged around that specific pixel. The second arithmetic circuit generates and outputs information about the living organism using mainly the signals from the region of the image where the dot pattern is not projected. With this configuration, biological information can be acquired with high accuracy.

[0037] In this specification, "biological information" refers to various types of information relating to the body, such as heart rate, blood flow rate, blood pressure, blood oxygen saturation, and respiration. In this specification, information indicating a person's state, such as their level of concentration or emotions, obtained from this information, is also included in "biological information."

[0038] (principle) The following explains the principle of a biometric information detection device that enables highly accurate acquisition of biological information.

[0039] In the biological information detection device of the embodiments of this disclosure, light with a wavelength range of approximately 650 nm to approximately 950 nm is used. This wavelength range is included in the red to near-infrared wavelength range. In this specification, the term "light" is used not only for visible light but also for infrared light. The above wavelength range is called the "biological window" and is known to have a low absorption rate in the body.

[0040] Figure 22 shows the light-emitting properties of oxyhemoglobin, deoxyhemoglobin, melanin, and water, respectively. This figure shows the wavelength dependence of the absorption coefficient and the scattering coefficient of light within the body. In the visible light region below 650 nm, absorption by blood (i.e., hemoglobin) is large, and in the wavelength range longer than 950 nm, absorption by water is large. Therefore, light in these wavelength ranges is not suitable for acquiring information from within the body. On the other hand, in the wavelength range from approximately 650 nm to approximately 950 nm, the absorption coefficients of hemoglobin and water are relatively low, and the scattering coefficient is large. The scattering coefficient is more than an order of magnitude larger than the absorption coefficient, and in the wavelength range of the "biological window," scattering becomes dominant in the interaction between the skin and near-infrared light. Therefore, light in this wavelength range, after entering the body, is strongly scattered and returns to the body surface. Since such optical properties are unique to skin, it becomes possible to distinguish the human body from other substances using this diffuse reflection property.

[0041] The biological information detection device in the embodiments of this disclosure primarily utilizes light in the wavelength range corresponding to the "window of life." For example, by using a dot array light source, it is possible to spatially separate and detect with high precision the light directly reflected from the biological surface and the light scattered within the body and returned, thereby efficiently acquiring biological information.

[0042] Figure 1A shows a schematic configuration of a biological information detection device in an exemplary embodiment of the present disclosure. The device comprises a light source 1, which is an array of point images (sometimes referred to herein as "array of point images" or "dot patterns"), which projects a plurality of discretely arranged point images onto an object including a living organism, and a camera 2. The light source 1 is positioned to project the plurality of point images onto the living organism 3. The imaging device 2 has an image sensor (also referred to as "image sensor"), which images the living organism surface 4, generates an image signal, and outputs it.

[0043] Figure 1B is a diagram illustrating the characteristics of the image of the biological surface acquired by the imaging device 2. Light L0 emitted from the light source 1 is reflected by the biological surface 4. The surface reflected light L1 reflected by the biological surface 4 retains the image of the arrayed point image from the light source 1. In contrast, the internally scattered light L2 that penetrates into the biological tissue 3, is scattered inside the tissue, and exits from the biological surface 4 loses the image of the arrayed point image from the light source 1 due to strong scattering within the tissue. By using the light source 1, the surface reflected light L1 and the internally scattered light L2 can be easily separated spatially.

[0044] The living organism 3 shown in Figure 1A is human skin, including the epidermis 33, dermis 34, and subcutaneous tissue 35. The epidermis 33 has no blood vessels, but the dermis 34 contains capillaries 31 and arterioles and veins 32. Since the epidermis 33 has no blood vessels, surface reflected light L1 does not contain information about blood. Because the epidermis 33 contains melanin pigment, which strongly absorbs light, surface reflected light L1 from the epidermis 33 becomes noise when acquiring blood information. Therefore, surface reflected light L1 is not only useless for acquiring blood information, but also hinders the acquisition of accurate blood information. In order to detect biological information with high accuracy, it is extremely important to suppress the influence of surface reflected light and efficiently acquire information from scattered light inside the body.

[0045] To solve the above problems, the embodiments of this disclosure have a novel configuration that spatially separates directly reflected light and scattered light from within the body using a light source and imaging device (or imaging system) that project an array of point images onto a living body. This makes it possible to measure information from within the living body with high precision without contact.

[0046] Conventionally, methods using polarized illumination, such as the one disclosed in Patent Document 6, have been used to separate directly reflected light from such biological surfaces. In methods using polarized illumination, a polarizer is used that has a polarization transmission axis perpendicular to the polarization direction of the illumination light reflected from the object being photographed. By imaging with a camera through such a polarizer, the influence of surface reflected light can be suppressed. However, with respect to reflection from surfaces with irregularities such as skin, there was a problem that the degree of polarization of the surface reflected light differed depending on the position, making it difficult to sufficiently separate the directly reflected light. According to the method disclosed herein, since directly reflected light and scattered light can be spatially separated, surface reflected light can be separated. This allows for more effective suppression of the effects of sunlight.

[0047] This disclosure includes, for example, the aspects described in the following sections.

[0048] [Item 1] The biometric information detection device relating to item 1 of this disclosure is A first light source projects multiple first dots, including living organisms, onto an object. An imaging device includes a plurality of first photodetector cells that detect second light from the object onto which the plurality of first dots are projected, and generates and outputs a first image signal showing a first image of the object onto which the plurality of first dots are projected, First arithmetic circuit and The second arithmetic circuit and Equipped with, The first image described above includes multiple pixels, The first arithmetic circuit uses the first image signal to detect a first portion of the first image that corresponds to the living organism, The second arithmetic circuit calculates biological information relating to the living organism using data from pixels within the first portion of the first image among the plurality of pixels.

[0049] [Item 2] In the biological information detection device described in item 1, The second light includes a third light from a position on the surface of the object onto which at least one of the plurality of first dots is projected, and a fourth light from a position on the surface of the object onto which the plurality of first dots are projected, but which surrounds the at least one first dot. The first arithmetic circuit detects the first portion of the first image using the first image signal corresponding to the third light and the first image signal corresponding to the fourth light from the first image signal, The second arithmetic circuit may calculate biological information relating to the living organism using data from the pixels corresponding to the fourth light among the pixels in the first portion of the first image.

[0050] [Item 3] In the biological information detection device described in item 2, The first arithmetic circuit may determine the ratio between the intensity of the first image signal corresponding to the third light and the intensity of the first image signal corresponding to the fourth light, and use this ratio to detect the first portion of the first image.

[0051] [Item 4] In the biological information detection device described in item 2, The first arithmetic circuit may detect the first portion of the first image using the ratio of the standard deviation to the mean value of the intensity of the first image signal corresponding to the third light and the intensity of the first image signal corresponding to the fourth light.

[0052] [Item 5] In a biological information detection device described in any of items 1 to 4, The first light may include light with a wavelength of 650 nm to 950 nm.

[0053] [Item 6] In a biological information detection device described in any of items 1 to 5, The aforementioned biological information includes the heart rate of the organism, the blood pressure of the organism, the blood flow rate of the organism, and the It may include at least one selected from the group consisting of the blood oxygen saturation of the living organism, the concentration of melanin pigment in the skin of the living organism, the presence or absence of blemishes in the skin of the living organism, and the presence or absence of birthmarks in the skin of the living organism.

[0054] [Item 7] In a biological information detection device described in any of items 1 to 6, The imaging device is A first bandpass filter that transmits the second light, An image sensor having an imaging surface on which the plurality of first photodetectors are arranged, wherein light transmitted through the first bandpass filter is incident on the imaging surface, It may also include

[0055] [Item 8] In a biological information detection device described in any of items 1 to 7, The second arithmetic circuit may use the time change of a value obtained by applying low-pass filtering to at least a portion of the pixel data to calculate at least one selected from the group consisting of the living body's heart rate, the living body's blood pressure, and the living body's blood flow rate as the biological information.

[0056] [Item 9] A biological information detection device described in any of items 1 to 8 is: The system further comprises a second light source that projects a plurality of second dots onto the object using a second light source, The first light includes light with a wavelength of 650 nm to 800 nm, The second light includes light with wavelengths between 800 nm and 950 nm. The imaging device further includes a plurality of second photodetectors that detect a fifth light from the object onto which the plurality of second dots are projected. The imaging device may generate and output a second image signal showing a second image of the object onto which the plurality of second dots are projected.

[0057] [Item 10] In the biological information detection device described in item 9, The imaging device is An image sensor having an imaging surface divided into a first region where the plurality of first photodetectors are arranged and a second region where the plurality of second photodetectors are arranged, A first optical system that forms the first image in the first region, A second optical system for forming the second image in the second region, It may also include

[0058] [Item 11] In the biological information detection device described in item 10, The imaging device is A first bandpass filter is placed on the path of the second light and transmits the second light, A second bandpass filter is positioned on the fifth light path and transmits the fifth light, It may also include

[0059] [Item 12] In the biological information detection device described in item 9, The imaging device is An imaging surface on which the plurality of first photodetectors and the plurality of second photodetectors are arranged, A plurality of first bandpass filters facing the plurality of first photodetectors and transmitting the second light, An image sensor including a plurality of second bandpass filters that face the plurality of second photodetectors and transmit the fifth light, An optical system that forms the first image and the second image on the imaging surface, It may also include

[0060] [Item 13] In the biological information detection device described in item 9, The imaging device is An imaging surface on which the plurality of first photodetectors, the plurality of second photodetectors, and the plurality of third photodetectors are arranged, A plurality of first bandpass filters facing the plurality of first photodetectors and transmitting the second light, A plurality of second bandpass filters that face the plurality of second photodetectors and transmit the fifth light, An image sensor including a plurality of third bandpass filters that face the plurality of third photodetectors and transmit visible light, An optical system that forms the first image and the second image on the imaging surface, It further includes, The plurality of third bandpass filters include a plurality of color filters with different transmission wavelength ranges from one another. The image sensor may generate and output a color image signal using the plurality of third photodetector cells.

[0061] [Item 14] In a biological information detection device described in any of items 9 to 13, The second arithmetic circuit may use the first image signal and the second image signal to calculate information indicating the blood oxygen saturation level of the living organism.

[0062] [Item 15] In a biological information detection device described in any of items 9 to 13, The second arithmetic circuit is, Using the first image signal and the second image signal, the blood flow rate and blood oxygen saturation of the living organism are calculated. Information may be generated using the blood flow rate of the organism and the blood oxygen saturation of the organism to indicate at least one selected from the group consisting of the organism's physical condition, emotions, and level of concentration.

[0063] [Item 16] In a biological information detection device described in any of items 9 to 13, When the first image and the second image include at least one selected from the group consisting of the forehead and nose of the living organism, The second calculation circuit uses the first image signal and the second image signal to calculate the time change in blood flow and the time change in blood oxygen saturation in at least one of the group consisting of the forehead and the nose. Even if the time-dependent changes in blood flow and blood oxygen saturation are used to generate information indicating at least one selected from the group consisting of the physical condition, emotions, and concentration level of the organism, good.

[0064] [Item 17] In a biological information detection device described in any of items 9 to 13, When the first image and the second image include the forehead and nose of the living organism, The second calculation circuit uses the first image signal and the second image signal to calculate the time change in blood flow rate and blood oxygen saturation in the forehead, and the time change in blood flow rate and blood oxygen saturation in the nose. Based on a comparison of the time-dependent changes in blood flow rate and blood oxygen saturation in the forehead and the time-dependent changes in blood flow rate and blood oxygen saturation in the nose, information may be generated indicating at least one selected from the group consisting of the body's physical condition, emotions, and level of concentration.

[0065] [Item 18] In a biological information detection device described in any of items 1 to 17, The first light source may be a laser light source.

[0066] [Item 19] In a biological information detection device described in any of items 1 to 18, The imaging device is An image sensor having an imaging surface on which the plurality of first photodetectors are arranged, An optical system that forms the first image on the imaging surface, An adjustment mechanism for adjusting the focus of the optical system, It further includes, The adjustment mechanism may maximize the contrast of the first image by adjusting the focus.

[0067] [Item 20] In a biological information detection device described in any of items 1 to 19, The first arithmetic circuit uses the first image signal to determine whether the first image contains at least one selected from the group consisting of the forehead, nose, mouth, eyebrows, and hair of the living organism. When it is determined that the first image includes at least one of the group consisting of the forehead, nose, mouth, eyebrows, and hair of the living organism, The second arithmetic circuit may also calculate biological information relating to the living organism.

[0068] [Item 21] In a biological information detection device described in any of items 1 to 20, The first arithmetic circuit may further calculate other biological information relating to the living organism using data from pixels in a second portion of the first image that is different from the first portion of the plurality of pixels.

[0069] [Item 22] In a biological information detection device described in any of items 1 to 20, The second calculation circuit may further determine whether the living organism has moved by comparing the position of the first portion in the first image at a first time point with the position of the first portion in the first image at a second time point.

[0070] In this disclosure, all or part of a circuit, unit, device, component or part, or all or part of a functional block in a block diagram, is a semiconductor device, semiconductor integrated circuit (IC), or LSI. It may be implemented by one or more electronic circuits, including large-scale integration. The LSI or IC may be integrated on a single chip or composed of multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. Here, we refer to them as LSI or IC, but the name may change depending on the degree of integration, and they may also be called system LSI, VLSI (very large-scale integration), or ULSI (ultra large-scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the manufacture of the LSI, or a reconfigurable logic device that allows for the reconfiguration of junction relationships within the LSI or the setup of circuit compartments within the LSI, can also be used for the same purpose.

[0071] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROM, optical disk, or hard disk drive, and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0072] The embodiments of this disclosure will be described in more detail below. The following embodiments relate to a biometric information detection device that measures biometric information non-contactually, mainly using the human face as the biological surface. However, the technology of this disclosure is not limited to the human face, but can also be applied to parts of the body other than the face or to the skin of animals other than humans.

[0073] (Embodiment 1) As a first embodiment, a system applying the technology of this disclosure to non-contact heart rate measurement will be described. With increasing healthcare awareness, the importance of continuous biometric information sensing is growing. A system that can continuously measure biometric information non-contact is important not only in hospitals but also for health management in daily life. The system of this embodiment can monitor heart rate and heart rate variability non-contact.

[0074] Figure 2 shows a schematic configuration of the biodetection system of this embodiment. As shown in Figure 2, the biodetection system of this embodiment comprises a light source 1 that emits light in the near-infrared wavelength range located away from the living body 3, an imaging device 2 which is a camera capable of recording an image of the irradiated living body surface 4, and a computer 20 connected to the light source 1 and the imaging device 2. The computer 20 can separate and measure the component of surface reflected light L1 and the component of scattered light L2 from the captured image. The computer 20 can detect whether a living body is present in the image based on the intensity of the surface reflected light L1 and the intensity of the scattered light L2. It can also calculate and output biometric information such as heart rate from the signal in the living body region of the image.

[0075] Light source 1 is designed to project a dot pattern onto the biological surface 4. The dot pattern is typically a collection of tiny bright spots arranged in two dimensions. Depending on the application, a one-dimensionally arranged dot pattern may also be used. In this embodiment, for example, the RPP017ES random dot pattern laser projector from Osela, Inc. can be used as light source 1. This laser light source emits near-infrared laser light at 830 nm and projects a laser dot pattern of 57,446 points within a 45° × 45° field of view.

[0076] Figure 3A shows an example of the configuration of an imaging device and the generated images and biological information. The imaging device 2, which is a camera, has an optical system 5 and a camera housing 6. The optical system 5 may be a collection of multiple lenses. Inside the camera housing 6 are an image sensor 7 and a bandpass filter 8 that transmits only light with a wavelength of 830 nm ± 10 nm, which is the wavelength of light from the light source 1.

[0077] When a person is used as the subject, the image sensor 7 acquires an image signal containing multiple point images with brightness corresponding to the infrared reflectance at each location. Part (a) of Figure 3A shows an example of an image represented by such an image signal. From this image signal, the arithmetic circuit in the computer 20 detects only the area of ​​the human body through signal processing, as shown in part (b) of Figure 3A. This detection is performed based on the ratio of the signal component due to surface reflected light L1 and the signal component due to internally scattered light L2.

[0078] As already mentioned, living organisms have a unique optical property called the "biological window" for wavelengths from red to near-infrared. Human skin has a small absorption coefficient and a large scattering coefficient in this wavelength range. Therefore, light that passes through the surface of the skin, which is the biological surface 4, undergoes repeated multiple scattering within the body and diffuses, emitting from the biological surface 4 over a wide area. For this reason, as shown in the enlarged view of part (c) of Figure 3A, in the region of the human body in the image, a region caused by internally scattered light L2 is generated around each bright spot caused by surface reflected light L1. In the above wavelength range, a characteristic of living organisms is that the ratio of scattered light to directly reflected light is high. In contrast, in non-living objects, the ratio of surface reflected light is overwhelmingly greater than that of scattered light. Therefore, it is possible to detect living organism regions based on the ratio of direct light to scattered light. Furthermore, biological information can be acquired at high speed using the signals of multiple pixels included in the living organism region in the obtained image. The detection of the human organism region in this embodiment, which utilizes the optical properties of skin, is faster and more accurate than conventional methods using image recognition. High-speed and high-precision human body detection using the detection of the human organism region and its information has made high-speed and high-precision biological information sensing possible.

[0079] Figure 3B is a block diagram showing the configuration of the computer 20. The computer 20 includes an input interface (IF) 21 electrically connected to the imaging device 2, a first arithmetic circuit 22 that performs signal processing to detect areas of the human body in the image, a second arithmetic circuit 23 that calculates biological information (pulsation in this embodiment) using image data within the detected human body area, a memory 25 for recording various data, a control circuit 26 for controlling the operation of the entire device, an output interface (IF) 24 for outputting data generated by the second arithmetic circuit 23, and a display 27 for displaying processing results. Each of the arithmetic circuits 22 and 23 may be, for example, an image processing circuit such as a digital signal processor (DSP). In Figure 3B, the arithmetic circuits 22 and 23 are represented by different blocks, but they may both be realized by a single circuit. The control circuit 26 may be, for example, an integrated circuit such as a central processing unit (CPU) or a microcomputer. The control circuit 26 performs control such as issuing a lighting instruction to the light source 1, an imaging instruction to the imaging device 2, and an arithmetic instruction to the arithmetic circuits 22 and 23 by executing a control program recorded in the memory 25. The control circuit 26 and the arithmetic circuits 22 and 23 may be implemented by a single integrated circuit. In the example in Figure 3B, the computer 20 is equipped with a display 27, but the display may be an external device electrically connected by wire or wireless. The computer 20 may acquire image information from the remote imaging device 2 by a communication circuit (not shown).

[0080] In the example shown in Figure 3A, the second arithmetic circuit 23 averages the signal components of the internally scattered light L2 within the human body region detected by the first arithmetic circuit 22. This averaging is performed, for example, for each frame of the video. As a result, data on the time variation of the average value of the signal components of the internally scattered light L2 is obtained, as illustrated in part (d) of Figure 3A. From this data, the heart rate (number of beats per unit time) can be determined by finding the period or frequency.

[0081] Furthermore, a similar system configuration allows for simultaneous measurement of respiration as well as heart rate. Figure 3C schematically illustrates such a respiration sensing system. The hardware in this example is exactly the same as that shown in Figure 3A. This configuration enables non-contact respiration monitoring through image signal processing.

[0082] The interval between breaths in a person is approximately 3-4 seconds (15-20 breaths / minute), and the expansion and contraction of the chest and abdominal wall due to breathing is approximately 5 mm in adults. If this chest movement can be measured with an imaging device, breathing can be monitored.

[0083] A method for monitoring respiration from chest movement using a near-infrared dot array light source is disclosed, for example, in Non-Patent Document 1. The system in Non-Patent Document 1 achieves highly accurate non-contact sensing of respiration by pre-determining the examination area for a stationary subject. The system in Non-Patent Document 1 is intended for non-contact respiration monitoring during sleep and is a large system that monitors only respiration.

[0084] In contrast, the system of this embodiment shown in Figure 3C enables simultaneous heart rate sensing and respiration sensing in a compact and inexpensive manner. Furthermore, it allows for stable measurements that track the subject's body movements.

[0085] The method for measuring respiration in this embodiment will be described below with reference to Figure 3C. The first arithmetic circuit 22 detects a human body from the near-infrared image acquired by the image sensor 7 (part (a) in Figure 3C) using the method described above, and estimates the face region from the data of the human body region (part (b) in Figure 3C). Furthermore, it estimates the position of the chest from the center of the face region. For example, the chest region is estimated to be a position shifted downward by approximately 1 to 1.2 times the vertical length of the face region from the center of the face. Then, the temporal positional variation of the dot array of this chest region (part (d) in Figure 3C) is measured. At that time, by averaging the positional variations of multiple dots, it is possible to measure positional variations below the pixel pitch. It should be noted here that even if the subject (chest) moves in the horizontal direction, the position of the captured dots hardly changes, and the position of the dots moves and is measured only when the depth direction changes. In methods using conventional images, both the horizontal and vertical movement of the subject are detected as movement on the pixels, and horizontal movement is detected with higher sensitivity, resulting in low measurement accuracy. In contrast, in this embodiment, by using a dot array light source, only vertical movement of the object is detected, and lateral movement is not detected. This enables highly accurate respiration monitoring. The amount of change in the dot array pattern of the chest region between frames can be determined by calculating the autocorrelation of the dot array pattern. This average amount of movement represents the vertical movement of the chest due to respiration. As shown in part (e) of Figure 3C, respiration monitoring becomes possible by plotting the average amount of movement of the dot pattern on the time axis. According to the configuration of this embodiment, highly accurate respiration sensing is possible while tracking the chest region, even when there is body movement.

[0086] In the example shown in Figure 3C, the method for measuring heart rate using pixel data in the forehead region of the image is the same as the method shown in Figure 3A. In the example in Figure 3C, both heart rate and respiration are measured, but only respiration may be measured.

[0087] Next, we will explain an example of a biometric detection method using actual data.

[0088] Figure 4A shows an example of an image acquired by a conventional imaging device that detects visible light. A human face F is visible in the center. The left side of Figure 4B shows the same scene as in Figure 4A, illuminated by a light source 1 with a wavelength of 830 nm, acquired by the imaging device 2 of this embodiment. In this image, the face F is recognized due to the strong reflection from the box B placed in the foreground. This is difficult. Therefore, in order to detect the human body, the first arithmetic circuit 22 directly calculates the contrast between reflected light and scattered light from the near-infrared image.

[0089] Figure 5 shows an example of a pixel region used for calculating contrast. Image data is recorded in memory 25 as two-dimensional intensity data. Let Pij be the data for the i-th pixel in the horizontal (x) direction and the j-th pixel in the vertical (y) direction. The contrast Cij of this (i,j) pixel is defined as follows. Cij = Sij / Aij

[0090] Here, Sij and Aij are the standard deviation and mean values ​​of the pixel data within a 7x7 pixel region centered on the (i,j) pixel, respectively. As the ratio of scattered light to directly reflected light increases, the standard deviation Sij decreases, and therefore the value of Cij decreases. After repeating this process for all pixels, the first arithmetic circuit 22 extracts only the pixels whose Cij value is within a predetermined range. As an example, the right-hand figure of Figure 4B shows a portion of the region where 0.2 < Cij < 0.47. In this figure, pixels whose Cij value is within the above range are displayed in white, and all other pixels are displayed in black. It can be seen that the living organism (i.e., face F) has been correctly extracted.

[0091] Thus, the first arithmetic circuit 22 in this embodiment calculates the contrast Cij, which is the ratio of the standard deviation to the mean value of a specific pixel in the image and a plurality of pixels arranged around that pixel. Based on this value, it can determine whether or not a living organism is present at the location corresponding to the specific pixel and output information indicating its presence or absence.

[0092] According to this embodiment, by utilizing the specific optical characteristics of a living body, it is possible to efficiently detect a living body hidden among many objects. Here, in order to obtain the contrast of an image (that is, the contrast between direct reflected light and scattered light), the average value and standard deviation value within a region of 7×7 pixels were obtained, but this is just an example. The size (i.e., the number of pixels) of the pixel region used for the contrast calculation is appropriately set according to the density of a plurality of point images formed by the light source 1 and the resolution of the imaging device 2. In order to suppress the variation in the calculation results, a plurality (for example, 3 or more) of point images may be included within the pixel region of the calculation target. By increasing the number of pixels in the region of the calculation target, the accuracy of the calculated value of the contrast is improved, but the resolution of the obtained image of the living body decreases. Therefore, the number of pixels in the region of the calculation target is appropriately set according to the configuration of the system and the purpose of use. Furthermore, not only the number of pixels of the calculation target but also the interval between pixels for which this process is repeated affects the processing speed. In the above process, the calculation was sequentially repeated for all pixels, but by increasing the pixel interval for which the calculation is performed, the resolution decreases but the processing speed can be increased. This pixel interval may also be appropriately set according to the configuration of the system and the purpose of use. Similarly, the predetermined range of the contrast is not limited to 0.2 < Cij < 0.47 and may be appropriately set according to the configuration of the system and the purpose of use.

[0093] By the above method, the first arithmetic circuit 22 detects the region of the human body in the two-dimensional image acquired by the imaging device 2. Next, the second arithmetic circuit 23 acquires biological information. Since the region of the human body in the image has already been specified by the first arithmetic circuit 22, biological information is generated using the pixel data within this region. The second arithmetic circuit 23 generates data on the time variation of the heart rate and respiration as biological information, for example, as shown in FIG. 3C. Thereby, it is possible to monitor the heart rate and heart rate variation non-contact.

[0094] Conventional bio-information sensing systems using cameras typically detect bio-information by averaging pixel data of biological regions in an image. In contrast, the bio-information sensing system of this embodiment uses a dot array light source, so it removes unnecessary surface reflected light components from the skin surface from a 2D image and selectively detects scattered light from inside the body that contains bio-information. This makes extraction possible. The projected dot array image (surface reflected light) is detected as points with high pixel values, while the components scattered within the body (internal scattered light) are detected around the dots as pixel values ​​lower than those of the dots themselves. By setting a threshold for light intensity and averaging the data of pixels excluding those with a light intensity above a certain level, internal scattered light can be efficiently extracted. This processing makes it possible to acquire biological information with high accuracy.

[0095] Figure 6 is a flowchart illustrating an example of the operations performed by the first arithmetic circuit 22 and the second arithmetic circuit 23 in this embodiment. Here, the operations of the arithmetic circuits 22 and 23 will be explained using the case where a moving image is acquired by the image sensor 7 as an example. Note that the following operations can be achieved by one or more processors executing a computer program stored in memory.

[0096] First, the first arithmetic circuit 22 extracts the human body region from the acquired video image (step S101). The method for extracting the human body region is as described above. Next, the second arithmetic circuit 23 removes the data from the central part of the dot array, which corresponds to the directly reflected light component, from the pixel value data within the extracted human body region using a preset threshold (step S102). Then, the second arithmetic circuit 23 calculates the average value of the pixel values ​​within the human body region (corresponding to the internal scattering component) (step S103). Steps S101 to S103 above are performed for each frame of the video image. The second arithmetic circuit 23 uses the data from frames over a predetermined period (for example, several seconds to tens of seconds) to calculate the period and amplitude of the time variation of the above average value (step S104). This makes it possible to obtain information about blood flow within the body. Because arterial blood pumped from the heart moves through the blood vessels with a wave called a pulse wave, the absorption rate and reflectance of near-infrared light change in conjunction with the pulsation. The heart rate can be determined from the period of this fluctuation in reflectance. Furthermore, blood pressure or blood flow can be estimated from the amplitude of the pulsation.

[0097] In the step of removing the directly reflected light component using a preset threshold (step S102), a fixed threshold can be used when the distance between the imaging device and the subject is constant. However, since the distance between the imaging device and the subject usually fluctuates, it is desirable to be able to handle such cases. For example, the average value of the pixel values ​​of the entire biological region to be processed may be calculated, and the threshold may be changed according to that average value. In such a configuration, the higher the average value of the pixel values, the higher the threshold can be set.

[0098] According to this embodiment, heart rate data such as that shown in part (d) of Figure 3A can be obtained. Numerous methods have been proposed for non-contact monitoring of heart rate using conventional visible and near-infrared cameras. In these conventional methods, the separation of surface reflected light components and scattered light components is insufficient, making them susceptible to ambient light interference in non-contact measurements, and stable and highly accurate measurements are difficult. In contrast, this embodiment enables stable and highly accurate heart rate measurement by spatially separating the surface reflected light component and scattered light component in the acquired image signal. For example, in conventional remote heart rate measurement using cameras, detection becomes unstable during conversation due to body movements associated with speaking, making highly accurate heart rate measurement impossible. By using the method of this embodiment, it is possible to stably measure the heart rate even with body movements similar to those during conversation.

[0099] According to this embodiment, the psychological stress of the subject can also be estimated. It is known that psychological stress can be estimated from the temporal fluctuations of heart rate. When the autonomic nervous system is functioning normally, the interval between heartbeats fluctuates, but it is known that stress reduces the fluctuations in the interval between heartbeats. The second calculation circuit 23 in this embodiment can also detect the presence or degree of psychological stress based on these changes in the fluctuations in the interval between heartbeats. Non-restrictive and non-contact heart rate sensing technology like that of this embodiment is important for performing stress sensing on a regular basis in daily life.

[0100] As described above, the system of this embodiment makes it possible to continuously monitor heart rate or blood pressure, including during sleep, without being restrained. This allows for the construction of systems that, for example, continuously monitor a patient's condition in a hospital and alert medical staff if an abnormality occurs. In ordinary households, it is also possible to monitor the nighttime heart rate of patients suffering from sleep apnea syndrome. Furthermore, since stress sensing can be easily performed in daily life as described above, it becomes possible to lead a more fulfilling daily life.

[0101] (Embodiment 2) Embodiment 1 described a system that detects human body regions from images and acquires biometric information from the human body regions within the images. Below, as a second embodiment, a typical application example using human body detection will be described. Human body detection is being developed, for example, to detect victims buried under rubble at disaster sites. Finding victims within 72 hours after a disaster is crucial in determining their survival rate. For this reason, a simple and stable biometric detection system is needed. Biometric detection technology is also used in the security and transportation fields. In the security field, biometric detection technology plays an important role in detecting intruders, and in the transportation field, it plays an important role in detecting pedestrians. There is a growing need for a system that can selectively detect living organisms (especially humans) from images containing various structures or objects. In Embodiment 1, living organisms can be detected by the operation of the first arithmetic circuit 22, but by further sensing biometric information (e.g., presence or absence of pulsation) in the area where living organisms are detected, more reliable and advanced biometric detection becomes possible. Below, the operation of this embodiment will be described for each application. The physical configuration in this embodiment is the same as the physical configuration in Embodiment 1.

[0102] (1) Finding victims during a disaster In the event of natural disasters such as earthquakes, tsunamis, and mudslides, the early detection of victims buried under rubble is particularly important from a life-saving perspective. There is a "72-hour barrier" where the survival rate is said to drop significantly after three days, making it necessary to quickly locate victims in chaotic situations. By using the system of this embodiment, it becomes possible to capture images and detect victims hidden under rubble in real time, even in situations where rubble is scattered. Because the system is small, it can be mounted on a UAV (Unmanned aerial vehicle, or drone). This makes it possible to acquire images remotely and search for survivors from a distance, even in disaster sites that are difficult to access due to the risk of secondary disasters.

[0103] In Embodiment 1, human body detection is possible by the biodetection processing performed by the first arithmetic circuit 22. However, in order to improve accuracy, this embodiment further utilizes the processing results of the second arithmetic circuit 23. For areas estimated to be living organisms by the first arithmetic circuit 22 (referred to as the biological area), the second arithmetic circuit 23 senses biological information (e.g., presence or absence of pulse). This makes it possible to detect a human body more reliably. By determining the presence or absence of body movement in the biological area, false detections can be reduced and reliability can be increased. The presence or absence of body movement can be determined, for example, by comparing multiple consecutive frames and determining whether or not there is a temporal change in the biological area. Furthermore, by using the pixel data in the biological area to determine the presence or absence of a heartbeat, the accuracy of human body detection can be dramatically improved. According to this embodiment, by combining biodetection using the optical properties of the skin, body movement detection by movement of the biological area, and heart rate measurement calculated from the signal intensity in the biological area, high-speed and reliable biodetection becomes possible. Furthermore, since the physical condition of the victim can be determined from the biological information, it becomes possible to determine the priority of rescue based on that data.

[0104] In the human body detection applications described below, (1) biodetection and (2) By using information from (3) body movement detection and heart rate detection in combination, more reliable biometric detection can be achieved.

[0105] (2) Monitoring use Surveillance cameras have become widespread and contribute to the safety and security of citizens' lives. As the number of surveillance cameras increases, it becomes important to consider who reviews the footage and how. Currently, it is difficult for people to constantly monitor the images, so images are often stored and reviewed after a problem (incident) occurs to understand the situation. For example, it is conceivable that the moment a problem occurs can be captured from real-time images, allowing for immediate response. By using the technology disclosed in this disclosure, it is possible to build a system that recognizes when a person enters the surveillance camera screen, issues a warning to the person in charge, and allows them to review the image in real time. The person in charge does not necessarily need to stand in front of the surveillance camera monitor; it is possible to build a system where a warning appears on the person's mobile device when a person is detected, and the image is displayed. Such a system is suitable for monitoring warehouses, back entrances of buildings, or places where access is restricted, where people do not normally enter. In addition, in places such as buildings where many cameras are centrally monitoring, highlighting camera footage that detects a person can help prevent anomalies from being overlooked, and facilitate early detection and countermeasures for anomalies.

[0106] For surveillance purposes, in addition to detecting suspicious individuals, the system can acquire more important information by sensing biometric information (presence or absence of pulse) using the second arithmetic circuit 23. The psychological state of a suspicious person can be estimated from their heart rate or heart rate variability in the surveillance image. From the estimated state of tension, the person's level of danger (attention level) can be estimated. Security systems are being developed to detect individuals with a high potential to commit crimes in crowds at airports or commercial facilities using camera footage. The biometric detection and sensing system of this embodiment can also be applied to such purposes.

[0107] In surveillance applications, the traditional method of human judgment of surveillance images is being replaced by a system that uses computer-based object recognition, thanks to advancements in image recognition technology. In such applications, it is common to send images to a host computer for recognition. However, this requires sending image data to a computer, leading to problems such as increased communication volume, decreased communication speed, and increased load on the host computer. If primary image recognition and judgment were possible with the surveillance camera, the burden of communication, recording, and computation could be significantly reduced. However, a challenge remained: insufficient reliability in this recognition could lead to missed events. The human detection method of this embodiment can detect humans with high reliability, making it possible to selectively send only the partial image containing the human to the host computer when a person is detected. As a result, more efficient operation of the surveillance system becomes possible.

[0108] Furthermore, advancements in image recognition technology have made it possible to identify individuals with high accuracy using images. Currently, the common method for identifying individuals from images involves sending the image to a host computer for recognition, but as mentioned above, the burden of communication, recording, and computation remains a challenge. During computation, the process of extracting the face for face recognition places a significant burden on the system. Using the detection method of this embodiment, the face portion can be easily extracted from the image. Therefore, it becomes possible to send only the face portion to the host computer for individual identification, significantly reducing the burden of individual identification. Moreover, for a limited number of people, by pre-registering their characteristics on the surveillance camera side, it becomes possible to instantly identify individuals on the camera side without going through the host computer.

[0109] (3) Automotive applications By installing the system of this embodiment in a car, pedestrians on the road can be constantly recognized, making it safer. This system enables safe driving. It can detect people and warn the driver even when they are hidden behind objects and visibility is poor. In autonomous driving, it is anticipated that a problem will arise in situations where stopping by braking is not possible and an accident cannot be avoided regardless of which way the vehicle changes direction. In such cases, it is effective to detect a person with this system and change the vehicle's course to avoid the person. For such applications, high-precision and high-speed detection of people is required, making the system of this embodiment particularly suitable.

[0110] (4) Human body detection switch There are a wide range of applications for detecting a human body and switching power on and off. For example, applications include detecting a person in a room and controlling the switch of equipment such as an air conditioner or light, controlling automatic doors with high precision, detecting pedestrians at a crosswalk and controlling pedestrian signals, and changing the brightness of lighting on a vending machine. This embodiment is applicable to these applications. Using the system of this embodiment, a highly functional switch that does not react to objects or pets but is sensitive only to people can be realized. For such applications, a compact human body detection sensor unit can be constructed by integrating the light source, imaging device, and processing circuit of this system.

[0111] (5) Biometric authentication Biometric authentication methods such as fingerprint, iris, and vein recognition are widely used as methods of personal identification. As their use expands, so do cases and risks of biometric authentication fraud. In image-based authentication, image duplication techniques such as copying have been used until now. In recent years, with the expansion of iris recognition and 3D printing, the risk of fraud using even more accurate duplication has increased. A two-stage authentication system is effective as a countermeasure against such risks. For example, a method in which the biometric detection system of this embodiment confirms that the subject is a living being before performing normal biometric authentication is effective. By confirming that the subject is a living being using the biometric detection system of this system, it is possible to increase the reliability of biometric authentication.

[0112] Of the applications described above, in applications (1) to (4), the second arithmetic circuit 23 may generate image data in which the region determined to be a human body is superimposed on a visible image and display it on the display. A single image showing the human body region, or an image in which an infrared image and a near-infrared image showing the human body region are superimposed, differ from the human visual image, so even when a human body is detected, there are problems in the human body's position recognition. To solve this problem, a visible camera may be added to the system shown in Figure 3A. The second arithmetic circuit 23 may superimpose the visible image obtained from the visible camera and the infrared image obtained from the image sensor 7 to generate image data in which the human body region is superimposed on the visible image. By highlighting the human body region on the visible image, visibility can be improved. In applications where a human makes a judgment after detecting a human body, a system that can superimpose a visible image and an image of the human body region is more effective.

[0113] Furthermore, when using a system with an additional visible-view camera, the second arithmetic circuit 23 may extract the contour lines of the image from the visible-view image and remove the parts corresponding to the contour lines from the area estimated to be the human body. This is because the infrared reflectivity may change significantly at the contour areas of objects, which may lead to the misdetection of the contour area of ​​an object as the human body. By removing the contour area, a human body image with less noise can be obtained.

[0114] To acquire both visible light and near-infrared images for human detection with a single camera, for example, one can remove the visible light cut filter from the camera and switch between visible light and near-infrared light every frame by linking the illumination light to the camera's frame rate. With such a configuration, both visible and near-infrared images can be acquired with a single camera. The advantage of this method is that one camera Since the camera can acquire both visible and near-infrared images, there is no parallax between cameras, making it easy to overlay images.

[0115] (Embodiment 3) As a third embodiment, a more specific application example combining human body detection and human body information sensing will be described. As mentioned above, the system of this disclosure can quickly detect a human body and acquire biometric information such as heart rate from the detected human body area with high speed and accuracy. This can be used to realize a monitoring system in personal spaces such as bathrooms, toilets, and bedrooms. Consideration for privacy is especially important in personal spaces. In systems that constantly photograph subjects with high-resolution cameras and use those images, there are concerns about privacy violations due to image leakage, and psychological burdens during filming due to the presence of the camera. These problems are solved by the monitoring system of this embodiment.

[0116] With the aging population, it is estimated that between 10,000 and 20,000 people die in the bath each year in Japan. This number is far greater than the 4,000 to 5,000 traffic fatalities. Causes of death in the bathroom include accidents (drowning) and illnesses (heart attacks and strokes due to brain disease). Deaths are more common among the elderly, occur more frequently in winter, and the annual number is increasing with the aging population. In both cases of accidents and illnesses, many of the deaths in the bathroom could have been prevented if the abnormality had been detected early. Because the bathroom is a closed, private space, detection is often delayed, leading to death in many cases. There is a strong need for a system that can monitor individuals in the bathroom while respecting their privacy.

[0117] Figure 7A is a schematic diagram showing the biological information detection device and its processing in this embodiment. The configuration of the biological information detection device in this embodiment is basically the same as the configuration shown in Figure 3A. However, in this embodiment, considering use in a bathroom, the light source 1 and the imaging device 2, which is a camera, are housed in a waterproof housing 15. The housing 15 has an opening on the front so as not to block the light from the light source 1 and the light returning from the subject 3. The opening is provided with a filter 16 that cuts out visible light and transmits near-infrared light. Near-infrared light emitted from the light source 1 passes through the filter 16 and is incident on the subject 3. The near-infrared light reflected by the subject 3 passes through the filter 16 again, passes through the optical system 5, which is the lens of the imaging device 2, and the bandpass filter 8, and is incident on the image sensor 7.

[0118] The biological information detection device shown in Figure 7A further includes a speaker 18 that emits an alarm (warning sound) and a control device 17. The control device 17 is connected to and controls the imaging device 2, the light source 1, and the speaker 18. The control device 17 is an element corresponding to the computer 20 shown in Figure 2, and includes a first arithmetic circuit 22, a second arithmetic circuit 23, a memory 25, and a control circuit 26, as shown in Figure 3B. When the control circuit 26 in the control device 17 detects an abnormality, it instructs the speaker 18 to emit an alarm.

[0119] In this embodiment, the system is waterproofed, and care is taken to ensure that the imaging device 2 is not visible to the human eye. This makes it possible to reduce the psychological burden of being filmed in a bathroom. The basic system configuration and signal processing are almost the same as in Embodiment 1.

[0120] The actual monitoring algorithm in this embodiment will be described below with reference to Figures 7B and 7C. In this embodiment, the biometric information detection device (also referred to as the "monitoring system") shown in Figure 7A is installed in the corner of the bathroom, and as shown in part (a) of Figure 7B, it is possible to monitor the entire bathroom. Human body detection (part (b) of Figure 7B), body movement detection (part (c) of Figure 7B), and detection of heart rate abnormalities (part (d) of Figure 7B) are performed from the near-infrared image data acquired by imaging. If there is no body movement after a human body has been detected, the first alarm is issued. Alarm 1 is issued to the person, for example, while they are bathing, to alert them. Furthermore, if a heart rate abnormality is detected, a second alarm (Alarm 2) is issued to, for example, someone outside the bathroom. The operation of the monitoring system of this embodiment will be explained in more detail below with reference to the flowchart in Figure 7C.

[0121] Figure 7C is a flowchart showing the operation of the monitoring system of this embodiment. First, the first arithmetic circuit 22 detects a human body based on the acquired near-infrared image data in the same manner as in Embodiment 1 (step S201). If a human body is detected, the system proceeds to the next step, body motion detection, S202. At this time, the image data used for human body detection is not recorded in the storage device (for example, the memory 25 shown in Figure 3B), and is overwritten with the image data of the next frame of the video, leaving only the data of the human body region. In this way, since no image data that can identify an individual is left, privacy is protected.

[0122] Next, the second arithmetic circuit 23 detects body movement by comparing data between multiple consecutive frames for the detected human body area (step S202). For example, if there is no body movement for a certain period of time (e.g., 30 seconds) or longer, alarm 1 is issued to the person (step S203). This could be an alarm such as, "Are you awake? It's dangerous in the bath. If you are awake, press the OK button." Alarm 1 is intended to alert the person and confirm their status. If there is no body movement, the second arithmetic circuit 23 further measures pulsation (step S204). If the pulsation is small or no pulsation can be detected, alarm 2 is issued (step S205). This is an alarm for people outside the bathroom (family, caregivers, ambulance, etc.). This alarm may be intended to confirm and request assistance via voice alert, telephone, or the internet to a target person pre-configured by the system.

[0123] According to this embodiment, a simple system configuration enables three stages of detection: (1) human body detection, (2) body movement detection, and (3) heart rate measurement. Therefore, highly reliable monitoring can be achieved.

[0124] In the example above, three stages of detection—(1) human body detection, (2) body movement detection, and (3) heart rate measurement—are performed in a step-by-step manner, but they do not have to be performed in this order. For example, after human body detection, body movement detection and heart rate measurement may be performed in parallel. This allows for continuous monitoring of the bather's heart rate and enables appropriate advice to be given to the bather. Many drowning deaths occur due to changes in heart rate caused by vasoconstriction due to the temperature difference between the changing room and the bathroom, and due to dizziness (overheating) caused by a decrease in blood flow to the brain and heart due to increased blood flow on the body surface, resulting in orthostatic hypotension. By continuously monitoring the heart rate as in this embodiment, changes in the bather's physical condition can be detected in real time. By providing feedback to the bather when a change in physical condition is detected, the aforementioned accidents can be prevented. For example, if the heart rate increases significantly, a message such as "Be careful of dizziness. When standing up, hold onto the handrail and stand up slowly" can be sent.

[0125] In monitoring systems for private spaces such as bathrooms, toilets, and bedrooms, protecting privacy is especially important. In this embodiment, near-infrared images acquired by the camera are used solely for human detection, and the image data itself is not recorded on a storage medium; instead, it is constantly overwritten with data from the next frame after human detection processing. Furthermore, the system of this embodiment is designed without an image data output mechanism. Therefore, it is impossible to obtain image data from the outside. Care has been taken to ensure that privacy is not violated even in the event of an attack by a malicious hacker. In addition, because the camera can be made invisible from the outside by using near-infrared light, monitoring is possible without the person feeling that they are being filmed. In spatial monitoring systems, ensuring privacy, both in terms of hardware and psychological aspects, is particularly important. This embodiment makes it possible to monitor the home while respecting privacy.

[0126] (Embodiment 4) As a fourth embodiment, a non-contact system for measuring blood oxygen saturation will be described. The major role of blood is to receive oxygen from the lungs and transport it to the tissues, and to receive carbon dioxide from the tissues and circulate it back to the lungs. Approximately 15g of hemoglobin is present in 100ml of blood. Hemoglobin bound to oxygen is called oxyhemoglobin (HbO2), Hemoglobin that is not bound to oxygen is called reduced hemoglobin (Hb). As shown in Figure 2, the light absorption properties of oxyhemoglobin and reduced hemoglobin are different. Oxyhemoglobin absorbs infrared light with wavelengths above approximately 830 nm relatively well, while reduced hemoglobin absorbs red light. It absorbs light (for example, at a wavelength of 660 nm) relatively strongly. For near-infrared light at a wavelength of 830 nm, there is no difference in the absorption rates of the two. Therefore, in this embodiment, transmitted light at these two wavelengths, 660 nm and 830 nm, is measured. From the ratio of transmitted infrared light to red light, the ratio of the two types of hemoglobin (oxygen saturation) can be determined. Oxygen saturation is a value that indicates how much of the hemoglobin in the blood is bound to oxygen. Oxygen saturation is defined by the following formula. Oxygen saturation = C(HbO2) / [C(HbO2)+C(Hb)] × 100(%) Here, C(Hb) represents the concentration of reduced hemoglobin, and C(HbO2) represents the concentration of oxyhemoglobin.

[0127] While the body contains components other than blood that absorb light in the red to near-infrared wavelength range, the time-varying light absorption rate is mainly due to hemoglobin in arterial blood. Therefore, blood oxygen saturation can be measured with high accuracy based on the fluctuations in absorption rate. Arterial blood pumped from the heart moves through the blood vessels as a pulse wave. Venous blood, on the other hand, does not have a pulse wave. Light irradiated onto the body is absorbed by various layers of the body, including arteries, veins, and tissues other than blood, and then penetrates the body. In this process, the thickness of tissues other than arteries does not change over time. Therefore, scattered light from within the body shows a time-varying intensity change in accordance with the change in the thickness of the arterial blood layer due to pulsation. This change reflects the change in the thickness of the arterial blood layer and does not include the influence of venous blood and tissues. Therefore, information about arterial blood can be obtained by focusing only on the fluctuating component of scattered light. By measuring the period of the component that changes over time, the pulse rate can also be determined.

[0128] Figure 8 shows the configuration of the system in this embodiment. This system comprises two array point image light sources 101 and 102, which are positioned away from the living organism 3 and emit near-infrared wavelength light (e.g., wavelength 830 nm) and red wavelength light (e.g., wavelength 660 nm), respectively; two cameras, imaging devices 201 and 202, which are capable of recording the irradiated living organism surface; and a computer 20 that separates and measures the direct reflected light intensity at the living organism surface and the scattered light intensity within the body from the acquired images, and calculates biological information from the direct reflected light intensity and the scattered light intensity. In this case, two light sources 101 and 102 with different wavelengths and imaging devices 201 and 202 corresponding to each light source are equipped to measure blood oxygen saturation.

[0129] Figure 9 shows the configuration of the imaging device. Each of the imaging devices 201 and 202 has an optical system 5 which is a lens and a camera housing 6. The camera housing 6 is equipped with an image sensor 7 and a bandpass filter 802 which selectively transmits near-infrared light (wavelength 830nm). The camera housing 6 is equipped with an image sensor 7 and a bandpass filter 801 which selectively transmits red light (wavelength 660nm).

[0130] For the light source 101, for example, the RPP017ES random dot pattern laser projector from Osela, Inc. of the United States can be used. This laser light source is an 830nm near-infrared laser. - This is a light source that projects a laser dot pattern of 57,446 points within a 45 x 45° field of view. For light source 102, for example, the RPP016ES random dot pattern laser projector from Osela, Inc. in the United States can be used. This laser light source is a 660nm red laser light source that projects a laser dot pattern of 23,880 points within a 35 x 35° field of view.

[0131] The computer 20 controls the two imaging devices 201 and 202 and the light sources 101 and 102 so that they work in conjunction to capture images simultaneously. As a result, images of two different wavelengths of light are generated from the imaging devices 201 and 202, for example, as shown on the right side of Figure 9.

[0132] Figure 10 shows the transmittance characteristics of bandpass filters 801 and 802. Bandpass filter 801 has a transmission center wavelength of 830 nm and a bandwidth of 10 nm. Bandpass filter 802 has a transmission center wavelength of 660 nm and a bandwidth of 10 nm. The transmission center wavelengths of bandpass filters 801 and 802 coincide with the center wavelengths of light sources 101 and 102, respectively. Therefore, imaging device 201 acquires an image using light with a wavelength of 830 nm, and imaging device 202 acquires an image using light with a wavelength of 660 nm.

[0133] The first arithmetic circuit 22 in the computer 20, similar to Embodiment 1, first extracts the human body region from the moving image. It then performs threshold-based data selection on the pixel data within that region to remove the directly reflected light component. After that, it calculates the average value of the pixel values ​​within the measurement region. The above processing is performed for the 830nm and 660nm imaging devices, respectively. The average value calculated in this way indicates the intensity of diffusely reflected light from the living body.

[0134] Figure 11 shows an example of the time evolution of the obtained diffuse reflected light intensity. For both near-infrared light (wavelength 830 nm) and red light (wavelength 660 nm), the reflected light intensity fluctuates over time. Here, the intensity of light from light sources 101 and 102 at the biological surface is denoted as Ii(830) and Ii(660), respectively, and the time-averaged values ​​of the fluctuating components of diffuse reflected light from the biological body are denoted as ΔI(830) and ΔI(660), respectively. Blood oxygen saturation SpO2 is calculated using the following formula. SpO2= a+b*(log(ΔI(660) / Ii(660))) / (log(ΔI(830) / Ii(830))) A and B in the above equation can be determined from their relationship to the measurements of an existing pulse oximeter.

[0135] To verify the accuracy of the measuring device, the system was used to measure oxygen saturation at the fingertips, not the forehead. A blood pressure cuff was used to pressurize the upper arm at a constant pressure (200 mmHg) to stop blood flow, and oxygen saturation was measured at the fingertips.

[0136] For comparison, a commercially available finger-clamping pulse oximeter was attached to the index finger, and the oxygen saturation of the middle finger was measured non-contact using this system. The first measurement determined the above-mentioned a and b, and subsequent measurements measured blood oxygen saturation (SpO2).

[0137] Figure 12 shows a comparison of measurements obtained using a contact-type pulse oximeter and measurements obtained using this embodiment. The results are generally consistent, indicating that the measurements are accurate. This embodiment allows for simultaneous measurement of not only blood oxygen saturation but also pulse rate from the pulse wave shown in Figure 11.

[0138] It is known that stress and fatigue can be measured from the fluctuations or frequency characteristics of pulse waves. By using the system of this embodiment, it is possible to estimate the psychological state, such as stress, and physical condition of a subject from pulse waves in a non-contact manner.

[0139] (Embodiment 5) As a fifth embodiment, a method for measuring blood oxygen saturation using a single camera will be described. In the fourth embodiment, two cameras were used to acquire signals from different light source wavelengths with each camera. This method has the advantage of being able to reuse existing cameras, but the system configuration becomes complex because two cameras are used in conjunction to capture images. There is also the challenge that the data acquired will be two separate video data sets, making it complex to process the data in a synchronized manner. To avoid this, in this embodiment, a camera capable of simultaneously acquiring image data for two wavelengths with a single camera has been realized.

[0140] Figure 13 shows the configuration of the biological information detection device of this embodiment. This device has the structure of a two-lens stereo camera having two imaging devices 201 and 202. Therefore, in this specification, such a system will be referred to as the "stereo camera system". The biological information detection device, which is a camera, has a first laser point image light source, light source 101 (wavelength 830 nm), and a second laser point image light source, light source 102 (wavelength 760 nm). The reflected light from the living body illuminated by light sources 101 and 102 passes through bandpass filters 801 and 802, respectively, and its direction of propagation is bent by 90 degrees by mirrors 901 and 902, and is imaged onto the imaging surface of image sensors 701 and 702 by optical systems 501 and 502, which are lenses. The bandpass filters 801 and 802 are narrow-band bandpass filters that transmit only light with wavelengths of 830 ± 15 nm and 760 ± 15 nm, respectively, which correspond to the wavelengths of the two light sources.

[0141] When the shutter button 11 is pressed, the two light sources 101 and 102 light up, and simultaneously the image sensors 701 and 702 acquire an image of the living organism. The acquired image is converted into a stereo image format by an image processing processor (corresponding to the arithmetic circuit 22 or 23 in Figure 3B), and after image signal processing, it is stored in a storage device (corresponding to memory 24 in Figure 3B). Subsequent processing is the same as in Embodiment 3 or 4.

[0142] According to this embodiment, by configuring the imaging system as a single stereo camera, the entire system becomes compact, and the configuration of the signal processing system, from subsequent image signal processing to oxygen saturation calculation, can be simplified. This makes it possible to achieve both ease of operation and high speed.

[0143] For the two light sources, wavelengths such as 760 nm and 830 nm in the near-infrared region can be used. The absorption difference between oxyhemoglobin and deoxyhemoglobin is greater at 660 nm than at 760 nm, as used in embodiments 2 and 3, allowing for more accurate measurement of oxygen saturation. However, since 660 nm is in the visible region, using this wavelength may cause discomfort to the subject. Furthermore, fluorescent and LED lighting contain components at 660 nm, making them susceptible to ambient light. In this embodiment, 760 nm was selected considering this. Since the absorption peak of deoxyhemoglobin is at 760 nm, using wavelengths between 760-780 nm is effective when the short-wavelength light source is set in the near-infrared region. The wavelengths used are not limited to those mentioned above and should be appropriately selected according to the application and operating environment.

[0144] (Embodiment 6) As a sixth embodiment, another method for measuring blood oxygen saturation using a single camera will be described. In the fifth embodiment, the single camera was configured as a stereo camera system including two optical systems and two image sensors. In this embodiment, a system is employed in which two different images corresponding to two wavelengths are acquired with a single image sensor by splitting the image using multiple lenses. This embodiment's method is referred to as the "stereo lens method." The stereo lens method system will be described with reference to Figure 14.

[0145] Figure 14 is a schematic cross-sectional view showing a part of the biological information detection device of this embodiment. Although not shown in Figure 14, the biological information detection device includes, for example, two light sources within the camera housing 6 that project dot patterns using light of two wavelengths, 830 nm and 760 nm. As shown in Figure 14, the optical system 5 has two sets of lenses, optical systems 501 and 502, inside. Optical systems 501 and 502 are designed to form images on two different regions of the imaging surface of the image sensor 7, respectively. In front of optical systems 501 and 502 are two narrow-band bandpass filters 801 and 802, respectively, which transmit light of 830 nm and 760 nm, corresponding to the wavelengths of the two light sources.

[0146] With this configuration, two images can be acquired simultaneously using two different wavelengths of light with a single image sensor 7. The second control circuit 23 calculates blood oxygen saturation from these two images in the same manner as in embodiments 3 to 5. According to this embodiment, since a single image signal contains information from two images taken simultaneously at two different wavelengths, the calculation process becomes easier.

[0147] The results of stress sensing using this stereo lens system are described below. Patent documents 4 and 5 disclose a method for detecting a decrease in temperature around the nose caused by stress (tension) or concentration using thermography. Psychological changes reduce blood flow in the nasal area, which in turn causes a decrease in temperature in the nasal area. Detecting this using thermography is a common practice. Changes in facial temperature are caused by changes in blood flow. If changes in blood flow can be measured with high accuracy, stress sensing can be performed with higher accuracy and responsiveness than measuring the surface temperature changes that result from changes in blood flow.

[0148] Figure 15A shows the results of stress sensing performed using the biological information detection device of this embodiment. As a stressor, a cold water load was applied by placing the right hand in cold water (ice water). For comparison, the temperature changes of the nose and cheek, enclosed by dotted lines in Figure 15B, were measured using thermography. Figure 15A shows these measurement results. The temperature of the nose gradually decreased over about 3 minutes after the start of the cold water load, and stabilized at a decrease of about 1.2°C. After the load ended, the temperature returned to normal over about 3 minutes. On the other hand, the temperature of the cheek was hardly affected by the cold water load and remained stable.

[0149] Figure 15C shows the changes in blood flow rate and blood oxygen saturation obtained using the bio-information detection device of this embodiment, which employs a stereo lens system. From the blood flow rate and oxygen saturation (SpO2) data in the face, data for the nasal and cheek regions, shown by dotted lines in Figure 15B, were extracted. The solid line shows the change in blood flow rate over time, and the dotted line shows the change in oxygen saturation (ΔSpO2) over time. As shown in Figure 15C, the blood flow rate in the nasal region showed a decreasing trend immediately after cold / thermal stimulation, indicating a high time responsiveness. On the other hand, the blood flow rate in the cheek region hardly changed. Regarding oxygen saturation, a decrease in oxygen saturation was observed in the nasal region along with a decrease in blood flow, but there was almost no change in the cheek region.

[0150] As these results show, a wealth of data can be obtained by measuring blood flow and oxygen saturation in different parts of the face. Based on this data, it is possible to detect emotions (i.e., feelings), physical condition, or concentration levels with high accuracy. Changes in blood flow due to the influence of the autonomic nervous system. Since blood flow varies depending on the area of ​​the face, it is particularly important to measure changes in blood flow in specific areas using a camera. In this process, measuring areas with less change in blood flow simultaneously and using them as a reference can improve the accuracy of the measurement.

[0151] (Embodiment 7) As a seventh embodiment, another method for measuring blood oxygen saturation using a single camera is described. I will reveal it.

[0152] Figure 16 is a schematic cross-sectional view showing the configuration of the biological information detection device in this embodiment. This device includes a stereo adapter 10 that can be attached to an optical system 5, which is a normal lens. The stereo adapter 10 is an attachment comprising four mirrors 151, 152, 153, and 154, and two bandpass filters 801 and 802. By using the stereo adapter 10, two images corresponding to two wavelengths can be formed in two different areas of the imaging surface of the image sensor 7. This method is called the "stereo adapter method".

[0153] In the stereo adapter system, two sets of opposing mirrors are used to acquire two different images corresponding to two different wavelengths using a single image sensor 7. Although not shown in Figure 16, two light sources emitting light at two wavelengths, 830 nm and 760 nm, are actually built into the camera housing 6. The stereo adapter 10 is attached to the front of the optical system 5. Two sets of mirrors (a pair of mirrors 151 and 152, and a pair of mirrors 153 and 154) bend the optical path twice before introducing it into the optical system 5. Narrowband bandpass filters 801 and 802, which transmit light at wavelengths of 830 nm and 760 nm corresponding to the wavelengths of the light sources, are installed between the optical system 5 and the mirrors 151, 152, 153, and 154.

[0154] This biometric information detection device can acquire images of two wavelengths at the same time using a single image sensor 7. The basic concept is the same as in Embodiment 5. The stereo lens method has the advantage of allowing for a smaller lens, thus enabling a smaller overall system. On the other hand, the stereo adapter method results in a larger overall system, but it has the advantage of allowing the use of high-performance camera lenses to improve resolution, as well as the use of lenses with different magnifications and zoom lenses. The advantage of the stereo adapter method is that it increases the flexibility of the system.

[0155] In this embodiment, we investigated the detection of human emotions using a biometric information detection device, which is a camera. As described in Embodiment 5, it is possible to reliably detect human emotions or feelings such as stress based on blood flow. Changes in human emotions or feelings activate the autonomic nervous system, which changes the blood flow on the skin surface. This change in blood flow causes a change in complexion. Humans normally detect a person's emotions or physical condition from these subtle changes in complexion. It is believed that doctors known as masters can diagnose a patient's physical condition or cause of illness simply by looking at their face because they can distinguish physical changes from subtle changes in the patient's complexion. It is also said that subtle changes in complexion, along with subtle changes in facial expression, play an important role when a person with keen senses reads another person's emotions. Furthermore, in the fields of games, animation, and computer graphics, which have made remarkable progress in recent years, research is widely being conducted on subtly changing the complexion of characters in order to give a natural impression or realism to a scene. As these examples show, facial color reflects a person's emotions and physical condition, and research is underway to interpret emotions by measuring facial color (e.g., Non-Patent Document 2). However, attempts to directly measure emotions from facial color are difficult to perform consistently and are not practical. This is because there are individual differences in changes in facial color, and these changes are subtle and strongly affected by ambient light and cameras, making consistent measurement difficult. There is a need for a method to measure changes in facial color more stably and accurately using methods other than colorimetry.

[0156] It is known that a person's complexion is primarily determined by the amount of melanin pigment contained in the skin surface (dermis) and the concentration of hemoglobin (oxyhemoglobin and deoxygenated (reduced) hemoglobin) in the blood. Since melanin pigment does not change in a short period of time (it changes due to aging, sunburn, etc.), emotional changes can be reliably measured by measuring blood flow. The method, in order to detect emotional changes, does not measure facial color but directly measures the blood flow of oxygenated and deoxygenated hemoglobin that is changing facial color. As explained in Embodiment 5, changes in facial blood flow differ depending on the area. This is because different areas of the face are more susceptible to the influence of the autonomic nervous system. For example, the nose has many arteriovenous anastomoses and is easily influenced by the autonomic nervous system, while the forehead is less susceptible to the influence of cutaneous vasoconstrictor nerves. The calculation circuit 22 in this embodiment calculates the blood flow of multiple different areas and compares them, thereby enabling the detection of emotional changes with high accuracy.

[0157] The following describes the measurement of changes in blood flow associated with emotional changes. A stereo adapter camera, as shown in Figure 16, was used to measure blood flow. The subject sat in front of the camera, and their face was captured by the camera. From a stable state, the subject watched videos that evoked emotions such as fear, laughter, surprise, and disgust, and color images of the subject's face were acquired. Changes in emotion were read from the changes in facial expressions in the video scenes and color images, and the changes in blood flow at that time were measured. Blood flow was measured in the nasal and forehead areas, as shown by the dotted lines in Figure 17A.

[0158] Figure 17B shows the time-dependent changes in total blood flow (oxyhemoglobin and deoxygenated hemoglobin) and the proportion of oxyhemoglobin in the blood flow (oxygen saturation) when laughter is induced. It can be seen that both total blood flow and blood oxygen saturation change significantly with the emotional change of laughter. Similar studies were conducted for other emotions. The results are shown in Figure 18. Figure 18 is a plot of the relationship between blood flow and oxygen saturation that occurred during each emotional change, with oxygen saturation on the x-axis and total blood flow on the x-axis. The changes in total blood flow and blood oxygen saturation when other emotions other than laughter were induced, such as sadness, surprise, depression, fear, disgust, anger, concentration, and happiness, were calculated in the same way as above. The same measurements were performed on 12 subjects. Figure 18 shows the average values ​​of the experimental results for the 12 subjects. Although there were individual differences, the changes in total blood flow and blood oxygen saturation showed a similar trend in almost all subjects. These results demonstrate that emotional changes can be detected by at least one of blood flow and oxygen saturation.

[0159] As shown in Figure 17B, the relationship between oxygen saturation and blood flow differs depending on the area of ​​the face. Therefore, by determining the blood flow and oxygen saturation at multiple points on the face, it is possible to sense emotions with higher accuracy. In the emotion sensing test conducted with this embodiment, measurements were taken at three points: the forehead, cheeks, and nose. The changes in oxygen saturation and blood flow due to emotional changes differed between the forehead, cheeks, and nose. Therefore, by creating a table in advance that shows the relationship between the changes in oxygen saturation and blood flow at each point, and calculating the correlation with the measured values ​​of oxygen saturation and blood flow, it is possible to detect emotional changes with high accuracy.

[0160] (Embodiment 8) As an eighth embodiment, a method for measuring blood oxygen saturation using a single camera without image splitting by an optical system will be described. Embodiments 3 to 7 described a method in which light from two light sources corresponding to two wavelengths is split and sensed, and biological information such as oxygen saturation is calculated. The biological information detection device of this embodiment does not split the image, but acquires two image signals of different wavelengths by an image sensor.

[0161] Figure 19A is a schematic diagram showing the configuration of the biological information detection device in this embodiment. This device separates two images corresponding to two wavelengths using an image sensor 703 rather than an optical system. Although the point light source is omitted in Figure 19A, in reality, two light sources emitting light of two wavelengths, 830 nm and 760 nm, are built into the camera housing 6. A bandpass filter 8 that transmits light with wavelengths between 730 nm and 850 nm is placed in front of the optical system 5, which is the lens of the camera. The bandpass filter 8 transmits visible light and long-wavelength red light. External light is cut off. The light that passes through the bandpass filter 8 is imaged onto the imaging surface of the image sensor 703 by the optical system 5. The image sensor 703 used here differs from ordinary image sensors in that it has two types of bandpass filters that transmit near-infrared light.

[0162] Figure 19B shows multiple filters facing multiple photodetector cells arranged on the imaging surface of the image sensor 703. The image sensor 703 has filter IR1, which selectively transmits light in the 680-800nm ​​range, and filter IR2, which selectively transmits light with wavelengths of 800nm ​​or greater. Filters IR1 and IR2 are arranged in a checkerboard pattern. The diagram below Figure 19B shows an example of the wavelength dependence of the transmittance of filters IR1 and IR2. The image sensor 703 detects two images from two light sources with wavelengths of 760nm and 830nm using multiple photodetector cells (also called pixels).

[0163] The first arithmetic circuit 22 (Figure 3B) individually reads data from multiple photodetector cells corresponding to a wavelength of 760 nm and data from multiple photodetector cells corresponding to a wavelength of 830 nm from the image sensor 703. For each image, it detects the human body region. Then, as shown in Figure 19C, the second arithmetic circuit 23 (Figure 3B) interpolates the missing pixel data in each data set to generate an image with a wavelength of 760 nm and an image with a wavelength of 830 nm. The second arithmetic circuit 23 calculates blood flow rate and oxygen saturation from these two images. Since these two images completely overlap, the calculation is simpler compared to calculating blood flow rate and oxygen saturation from two separated images. A challenge with this method is that, compared to using bandpass filters corresponding to each light source, the shielding ability of the filters is lower, raising concerns about color mixing between light sources.

[0164] (Embodiment 9) As a ninth embodiment, a biological information detection device capable of acquiring not only two images corresponding to two wavelength light sources without dividing the image, but also a color image, will be described.

[0165] Figure 20A shows the configuration of the biological information detection device in this embodiment. Although the point image light source is omitted in Figure 20A, two light sources emitting light of two wavelengths, 830 nm and 760 nm, are built into the camera housing 6. In this embodiment, a bandpass filter is not provided in front of the optical system 5, which is a lens, in order to acquire a color image. Visible light and illumination light from the laser light source are imaged onto the imaging surface of the image sensor 704 by the optical system 5. The image sensor 704 used here differs from a normal image sensor in that it includes a photodetector cell for acquiring a color image and two types of photodetector cells for acquiring a near-infrared image.

[0166] Figure 20B shows multiple bandpass filters (or color filters) arranged on the imaging surface of the image sensor 704. The diagram below Figure 20B shows the wavelength dependence of the relative sensitivity of pixels facing each filter. As shown in Figure 20B, three types of color filters (R, G, and B filters) that transmit blue, green, and red light respectively, as well as filter IR-1 that transmits light above 650 nm and filter IR-2 that transmits light above 800 nm, are arranged on the imaging surface. The arrangement of two green filters placed diagonally adjacent to each other, with the red and blue filters placed diagonally opposite, is the same as the Bayer array of a normal image sensor. The difference from conventional image sensors is that the two filters IR-1 and IR-2 are placed next to the four basic filter units of the Bayer array.

[0167] The filter IR1 in Embodiment 8 and the filter IR-1 in this embodiment have different transmission wavelength ranges. While the filter IR1 in Embodiment 8 was a relatively narrow-band filter that transmitted light in the 650-800nm ​​wavelength range, this embodiment uses a filter that transmits light in the 650nm wavelength range and above. This is for the image sensor 70 This is to simplify the manufacturing process of step 3. However, it is not limited to this, and it is also possible to use the filter shown in Embodiment 8. Filter IR-1 in this embodiment is sensitive to both 760 nm and 830 nm. Therefore, the second calculation circuit 23 calculates the signal corresponding to 760 nm by subtracting the signal of the photodetector cell facing filter IR-2 from the signal of the photodetector cell facing filter IR-1, and then determines the blood oxygen saturation. As a result, as shown in Figure 20C, a red, blue, and green image (color image) and images of each wavelength, 760 nm and 830 nm, are obtained from the image sensor 704.

[0168] This method is even more prone to color mixing than Embodiment 8. However, it allows for the simultaneous acquisition of color images and information on blood flow and blood oxygen saturation using a simple single-camera system. The advantage of this system is that, because it acquires both visible and near-infrared images with a single camera, it is possible to obtain visible and near-infrared images without parallax. This is particularly effective for applications where visible and near-infrared images are to be overlaid and displayed.

[0169] This section describes an example configuration of a bio-information sensing camera using a multispectral sensor that supports five wavelengths: two wavelengths in the infrared region and three wavelengths in the visible region (red, blue, and green). With the human body detection method described in Embodiment 1, color imaging and human body detection are possible by measuring four wavelengths: one wavelength in the infrared region and three wavelengths in the visible region (red, blue, and green). For such applications, for example, a multispectral sensor with four types of color filters corresponding to four wavelengths, as shown in Figure 20D, can be used. The color filter assigns a near-infrared (IR) pixel to one of the two green pixels in the Bayer array typically used in image sensors. Here, assuming a camera compatible with a system that uses 850nm near-infrared illumination, a filter that selectively transmits 850nm was selected as the near-infrared filter. By using such a camera, it becomes possible to use a single camera system for both a normal color camera and a bio-detection camera. Only one surveillance camera is needed, and extracting color images of the detected human body becomes easier than using two cameras. Here, a color filter corresponding to 850nm was used, but it is possible to change the near-infrared filter depending on the near-infrared light source used.

[0170] (Other embodiments) The embodiments described above illustrate the present disclosure, but the disclosure is not limited to the embodiments described above, and various modifications are possible. The processes described for each of the embodiments described above may also be applicable to other embodiments. Examples of other embodiments are described below.

[0171] In the embodiments described above, a laser light source is used as the array point image light source, but other types of light sources may be used. For example, it is possible to use a less expensive LED light source. However, LED light sources have lower directivity and tend to spread out compared to laser light sources. For this reason, when using an LED light source, a dedicated focusing optical system may be used, or the distance between the object to be imaged and the camera may be restricted.

[0172] The biological information detection device may include an adjustment mechanism for adjusting the focus of the optical system. Such an adjustment mechanism can be implemented, for example, by a motor (not shown) and a control circuit 26 shown in Figure 3B. Such an adjustment mechanism adjusts the focus of the optical system to maximize the contrast of the dot pattern image projected onto the object by the light source. This improves the accuracy of the contrast calculation described in Embodiment 1.

[0173] The first arithmetic circuit 22 detects biological regions using the image signal output from the image sensor. If multiple biological regions (regions of different people or the faces and hands of the same person) are detected in the image, the circuit may determine which biological region to detect based on the size or shape of the detected region.

[0174] The second arithmetic circuit 23 may generate information about the epidermis, including at least one of the melanin pigment concentration, the presence or absence of freckles, and the presence or absence of birthmarks, based on the image signal. As mentioned above, the epidermis contains melanin pigment, which strongly absorbs light. Freckles and birthmarks are caused by an increase in melanin pigment. Therefore, the melanin pigment concentration, freckles, and birthmarks can be detected based on the light intensity distribution from the surface of the living body. The second arithmetic circuit 23 may, for example, extract a direct reflected light component from the surface of the living body from the image signal and generate information about the epidermis, including at least one of the melanin pigment concentration, the presence or absence of freckles, and the presence or absence of birthmarks, based on this direct reflected light component. The direct reflected light component can be obtained, for example, by determining whether the contrast in Embodiment 1 exceeds a predetermined threshold, or by removing low-frequency components in the image signal.

[0175] This disclosure describes a double-camera system using two cameras (Figure 8), a stereo-camera system in which two sets of optical systems and two sets of image sensors are mounted on one camera (Figure 13), a stereo-lens system using two sets of lenses and one image sensor (Figure 14), a stereo-adapter system using a lens adapter to use one lens and one image sensor (Figure 16), and a system that splits an image using an image sensor (Figures 19A and 20A). As already mentioned, each system has its advantages and disadvantages, so the optimal system can be selected depending on the application.

[0176] As described above, according to the embodiments of this disclosure, it is possible to measure not only heart rate and blood flow but also blood oxygen saturation without restraining the subject or without bringing detection devices such as sensors into contact with the subject. It is also possible to estimate the subject's emotions or physical condition from the blood flow and oxygen saturation measurements taken from different parts of the subject. [Industrial applicability]

[0177] The biometric information detection device described herein is useful for detecting biometric information such as a subject's heart rate, blood flow rate, blood pressure, blood oxygen saturation, emotions, and physical condition. [Explanation of Symbols]

[0178] 1, 101, 102 light source 2, 201, 202 Imaging devices 3 Living organisms 4. Biological surface 5, 501, 502 optical system 6 Camera housing 7, 701, 702, 703, 704 Image Sensors 8, 801, 802 bandpass filters 901, 902, 151, 152, 153, 154 Miller 11 Shutter button 15 cabinets 16 filters 20 computers 21 Input Interfaces 22 1st calculation circuit 23 Second arithmetic circuit 24 output interfaces 25 memory 26 Control circuits 27 displays 31 Capillaries 32 Arteries and veins 33 Epidermis 34 Dermis 35 Subcutaneous tissue Light from the L0 light source L1 surface reflected light L2 Internally scattered light

Claims

1. A light source that projects multiple dots onto a target using the first light, An image sensor that detects a second light from the object onto which the plurality of dots are projected and generates one or more image signals, A processing circuit is provided, The aforementioned processing circuit is Based on the one or more image signals, biological information corresponding to the target heart rate is generated. Based on the one or more image signals, information in the depth direction of the target is generated. Processing device.

2. The processing circuit generates the biological information based on a plurality of first pixels corresponding to the target face, which are included in the one or more image signals. The apparatus according to claim 1.

3. The processing circuit generates the depth information based on a plurality of second pixels included in the one or more image signals that correspond to positions different from the target face. The apparatus according to claim 2.

4. The aforementioned processing circuit is Based on the one or more image signals mentioned above, a first part corresponding to a living organism is detected. Based on the results of detecting the first portion, the plurality of first pixels are determined. The apparatus according to claim 2.

5. The aforementioned processing circuit is Based on the aforementioned biological information, an abnormality in the heart rate is detected. When an abnormality in the heart rate is detected, a signal is output to alert the area outside the space in which the object is located. The apparatus according to claim 1.

6. A method of processing performed by a computer, A light source is used to project multiple dots onto a target using the first light source. The image sensor is made to detect a second light from the object onto which the plurality of dots are projected, thereby generating one or more image signals. Based on the one or more image signals, biological information corresponding to the target heart rate is generated. Based on the one or more image signals, information in the depth direction of the target is generated. Processing method.

7. A light source is used to project multiple dots onto a target using the first light source. The image sensor is made to detect a second light from the object onto which the plurality of dots are projected, thereby generating one or more image signals. Based on the one or more image signals, biological information corresponding to the target heart rate is generated. Based on the one or more image signals, information in the depth direction of the target is generated. A program that causes a computer to perform a task.

Images