Biological measurement device and biological measurement method

Abstract

A biological measuring device according to one aspect of the present invention comprises a light source for irradiating light to a user's head, an image sensor, a control circuit for controlling the light source and the image sensor, and a signal processing circuit. The control circuit causes the light source to emit the light, and causes the image sensor to detect at least a portion of the reflected light returned from the head due to the irradiation of the light and output an image signal. The signal processing circuit generates brain activity data representing the state of the user's brain based on the image signal, and stops outputting the brain activity data based on at least one selected from the group consisting of the image signal and the sensor signal, which is a signal output from a sensor that detects changes in the user's surrounding environment that affect the brain activity data.

Description

Technical Field

[0001] The present invention relates to a biological measuring device and a biological measuring method. Background Art

[0002] In the field of biological measurement, a method is used in which light is emitted toward an object and internal information of the object is obtained from the light that has penetrated the interior of the object. In this method, a surface reflection component reflected from the surface of the object may become noise. If the noise caused by the surface reflection component is removed, the desired internal information can be obtained with high accuracy.

[0003] Patent Document 1 discloses an imaging device that measures internal information of an object in a non-contact manner while suppressing noise caused by surface reflection components.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-202328 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, in the above-mentioned conventional imaging device, when the object moves during measurement, the correction requires computational cost.

[0009] An object of the present invention is to provide a biological measuring device that measures the state of an object by a method that is cheaper than conventional methods.

[0010] Means for solving problems

[0011] A biological measuring device according to one technical solution of the present invention comprises a light source for irradiating light to the head of a user, an image sensor, a control circuit for controlling the light source and the image sensor, and a signal processing circuit. The control circuit causes the light source to emit the light, and causes the image sensor to detect at least a portion of the reflected light returned from the head due to the irradiation of the light and output an image signal. The signal processing circuit generates brain activity data representing the state of the brain of the user based on the image signal, and stops outputting the brain activity data based on at least one selected from the group consisting of the image signal and the sensor signal, which is a signal output from a sensor that detects changes in the surrounding environment of the user that affect the brain activity data.

[0012] Effects of the Invention

[0013] According to the living body measuring device according to one aspect of the present invention, even when the object moves during measurement or the surrounding environment of the object changes, it is possible to measure the internal information of the object by an inexpensive method. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A It is a diagram schematically showing an example of the living body measuring device according to the present embodiment.

[0015] Figure 1B This is a diagram showing a schematic configuration example of one pixel of an image sensor.

[0016] Figure 1C This is a diagram showing an example of the structure of an image sensor.

[0017] Figure 1D This is a diagram schematically showing an example of the operation within one frame according to the present embodiment.

[0018] Figure 1E This is a flowchart schematically showing the operation related to the light source and the image sensor performed by the control circuit.

[0019] Figure 1F Schematically shows a light signal reaching an image sensor when rectangular pulse light emitted from a light source returns from a user.

[0020] Figure 1G FIG. 2 is a diagram schematically showing an example of a time chart when detecting a surface reflection component.

[0021] Figure 1H FIG. 2 is a diagram schematically showing an example of a time chart when detecting an internal scattered component.

[0022] Figure 2 This is a flowchart showing an example of a process of measuring biological measurement data of a user according to the present embodiment.

[0023] Figure 3A This is a diagram for explaining an example of the operation of processing the biological measurement data of the user according to the present embodiment.

[0024] Figure 3B This is a diagram schematically showing the relationship between changes in body exercise amount and the ineffective period.

[0025] Figure 3C This is a diagram schematically showing the relationship between the moving speed of the head and the invalid period.

[0026] Figure 3D This is a diagram for explaining an example of the operation of processing for measuring biological measurement data of a user according to the present embodiment.

[0027] Figure 3EIt is a diagram schematically showing an example of the living body measuring device according to the present embodiment.

[0028] Figure 4 It is a diagram schematically showing an example of the living body measuring device according to the present embodiment.

[0029] Figure 5A This is a flowchart showing an example of a process of measuring biological measurement data of a user according to the present embodiment.

[0030] Figure 5B This is a diagram showing an example of the relationship between the difference value and the reliability.

[0031] Figure 6 This is a diagram for explaining an example of the operation of processing the biological measurement data of the user according to the present embodiment.

[0032] Fig. 7A It is a diagram schematically showing an example of the living body measuring device according to the present embodiment.

[0033] Figure 7B This is a diagram schematically showing an example of the arrangement of various parts when the biological measurement device is installed inside a car.

[0034] Figure 7C This is a diagram schematically showing an example of the arrangement of various parts when the living body measurement device is additionally provided in a game machine or a suction device.

[0035] Figure 8 This is a flowchart showing an example of a process of measuring biological measurement data of a user according to the present embodiment.

[0036] Fig. 9 This is a diagram for explaining an example of the operation of processing the biological measurement data of the user according to the present embodiment.

[0037] Fig. 10A This is a graph showing the relationship between time and the moving distance of the head.

[0038] Fig. 10B It is a graph showing the relationship between time and oxygenated hemoglobin concentration. DETAILED DESCRIPTION

[0039] (Process for achieving one aspect of the present invention)

[0040] In the field of biological measurement, it is known that changes in the user's state affect the measurement results. For example, in the measurement of brain activity information using near infrared spectroscopy (NIRS), the movement of the user's head is called body movement. Body movement is believed to cause irregular changes in the signal indicating changes in cerebral blood flow.

[0041] When the measurement of brain activity information is for research purposes, the operator of the NIRS device can adopt a process in which the operator determines the irregularity of the signal, notifies the user that "the measurement is wrong", and prompts the user to measure again. However, it is not easy to adopt such a process for daily use such as measuring the user's brain activity information.

[0042] Furthermore, when the measurement device measures brain activity information from the user's head in a non-contact manner, the movement of the head changes the relative positional relationship between the measurement device and the user, thus further increasing the influence on the detection signal of cerebral blood flow changes.

[0043] Patent document 1 discloses a method for correcting the position deviation accompanying movement by pattern matching of cerebral blood flow distribution between frames. The pattern of cerebral blood flow distribution has a low spatial frequency. That is, the cerebral blood flow distribution represents a flat distribution in space. Therefore, it is not necessarily easy to detect the feature points required for matching. In addition, it takes computational cost to restore the cerebral blood flow distribution so that the head does not move in all frames.

[0044] Based on the above studies, the present inventors have conceived of a living body measuring device described in the following items.

[0045] [Project 1]

[0046] The biological measuring device related to the first item includes a light source for irradiating light to the head of the user, an image sensor, a control circuit for controlling the light source and the image sensor, and a signal processing circuit. The control circuit causes the light source to emit the light, and causes the image sensor to detect at least a portion of the reflected light returned from the head due to the irradiation of the light and output an image signal. The signal processing circuit generates brain activity data representing the state of the brain of the user based on the image signal, and stops the output of the brain activity data based on at least one selected from the group consisting of the image signal and the sensor signal, which is a signal output from a sensor that detects changes in the surrounding environment of the user that affect the brain activity data.

[0047] [Project 2]

[0048] In the biological measuring device related to the first item, the light may be pulse light; the control circuit may cause the image sensor to output the first signal as the image signal, and the first signal may be a signal obtained by detecting a component included in the reflected pulse light during a period from when the intensity of the reflected pulse light returned from the head due to the irradiation of the pulse light starts to decrease to when the intensity of the reflected pulse light ends.

[0049] [Item 3]

[0050] In the biological measuring device related to the first or second item, it may be that the control circuit causes the light source to repeatedly emit the light and the image sensor to repeatedly output the image signal during a specified period; the signal processing circuit also calculates a first value based on at least one selected from the group consisting of the image signal and the sensor signal, and stops outputting the brain activity data during a first period in the specified period in which the first value meets a predetermined condition.

[0051] [Item 4]

[0052] In the living body measurement device according to item 3, the signal processing circuit may output a signal indicating that the brain activity data is invalid during the first period.

[0053] [Item 5]

[0054] In the living body measurement apparatus according to item 3, the signal processing circuit may output, during the first period, data identical to the brain activity data generated before the first period.

[0055] [Item 6]

[0056] In the living body measuring device related to the third item, the signal processing circuit may output data obtained by interpolating the brain activity data generated before the first period and the brain activity data generated after the first period as the brain activity data in the first period.

[0057] [Item 7]

[0058] In the biological measuring device related to the third item, the signal processing circuit may stop outputting the brain activity data in at least one period selected from the group consisting of a second period before the start of the first period and a third period after the end of the first period, in addition to the first period.

[0059] [Item 8]

[0060] In the living body measuring device according to item 7, the third period may be longer than the second period.

[0061] [Item 9]

[0062] In the biological measuring device according to any one of items 3 to 8, the frequency of calculating the first value may be greater than the frequency of generating the brain activity data.

[0063] [Item 10]

[0064] In the biological measuring device related to any one of items 1 to 9, it may be that the light is pulse light; the control circuit causes the image sensor to output a second signal as the image signal, the second signal being a signal obtained by detecting a component included in the reflected pulse light before the intensity of the reflected pulse light returned from the head due to the irradiation of the pulse light begins to decrease; and the signal processing circuit stops outputting the brain activity data based on the second signal.

[0065] [Item 11]

[0066] In the biological measuring device related to Item 10, the signal processing circuit may also calculate the displacement of the head relative to the reference position or the moving speed of the head based on the second signal, and stop outputting the brain activity data when the absolute value of the displacement or the absolute value of the moving speed exceeds a threshold value.

[0067] [Item 12]

[0068] In the biological measuring device related to Item 10, the signal processing circuit may also calculate the brightness value of the head or the changing speed of the brightness value of the head based on the second signal, and stop outputting the brain activity data when the absolute value of the brightness value or the absolute value of the changing speed of the brightness value exceeds a threshold value.

[0069] [Item 13]

[0070] In the living body measuring device related to item 10, the signal processing circuit may further calculate the area of ​​a predetermined region in the head based on the second signal, and stop outputting the brain activity data when the area is smaller than a threshold value.

[0071] [Item 14]

[0072] In the biological measuring device according to any one of items 1 to 9, the signal processing circuit may further calculate a second value using the brain activity data, and stop outputting the brain activity data when an absolute value of a rate of change of the second value exceeds a threshold value.

[0073] [Item 15]

[0074] In the living body measuring device according to the first item, the sensor may be an acceleration sensor provided in the surrounding environment.

[0075] [Item 16]

[0076] In the living body measuring device according to the first item, the sensor may be an illuminance sensor provided in the surrounding environment.

[0077] [Item 17]

[0078] In the biological measuring device according to the first item, the sensor may be at least one selected from the group consisting of a steering angle sensor, a gear position sensor, and a speed sensor disposed in a vehicle driven by the user.

[0079] [Item 18]

[0080] The biological measuring device related to item 18 comprises: a light source for irradiating light to the head of the user; an image sensor; a control circuit for controlling the light source and the image sensor; and a signal processing circuit. The control circuit causes the light source to emit the light, and causes the image sensor to detect at least a portion of the reflected light returned from the head due to the irradiation of the light and output an image signal. The signal processing circuit generates brain activity data representing the state of the brain of the user based on the image signal, calculates the reliability of the brain activity data based on at least one selected from the group consisting of the image signal and the sensor signal, and outputs reliability data representing the reliability, wherein the sensor signal is a signal output from a sensor that detects changes in the surrounding environment of the user that affect the brain activity data.

[0081] [Item 19]

[0082] In the living body measuring device according to item 18, the signal processing circuit may output the reliability data together with the brain activity data.

[0083] [Item 20]

[0084] In the biological measuring device related to item 18 or 19, the signal processing circuit may further calculate a first value based on at least one selected from the group consisting of the image signal and the sensor signal, and the reliability of the calculation is lower as the first value is farther from a predetermined value.

[0085] [Item 21]

[0086] In the biological measuring device related to item 18 or 19, the signal processing circuit may further calculate a first value based on at least one selected from the group consisting of the image signal and the sensor signal, and the reliability of the calculation is lower as the period during which the first value exceeds a preset value is longer.

[0087] [Item 22]

[0088] In the biological measuring device related to item 18 or 19, the signal processing circuit may further calculate the first value based on at least one selected from the group consisting of the image signal and the sensor signal, and the longer the period during which the first value exceeds a predetermined value during a certain period, the lower the reliability of the calculation.

[0089] [Item 23]

[0090] In the biological measuring device according to any one of items 18 to 23, the sensor may be at least one selected from the group consisting of a steering angle sensor, a gear position sensor, and a speed sensor disposed in a vehicle driven by the user.

[0091] [Item 24]

[0092] In the biological measuring device related to any one of items 18 to 23, it may be that the light is pulse light; the control circuit causes the image sensor to output a second signal as the image signal, the second signal being a signal obtained by detecting a component included in the reflected pulse light before the intensity of the reflected pulse light returned from the head due to the irradiation of the pulse light begins to decrease; and the signal processing circuit calculates the reliability based on the second signal.

[0093] [Item 25]

[0094] In the biological measuring device related to the first or second item, it may be that the control circuit causes the light source to repeatedly emit the light and the image sensor to repeatedly output the image signal during a specified period; the signal processing circuit also calculates the first value based on at least one selected from the group consisting of the image signal and the sensor signal, and stops outputting the brain activity data during the specified period, starting from the first period in which the first value meets a preset condition and ending after a delay time until the first period ends.

[0095] [Item 26]

[0096] The living body measurement method related to Item 26 includes the following processing: causing a light source to emit light to illuminate the user's head; causing an image sensor to detect at least a portion of reflected light returned from the head due to the illumination of the light and output an image signal; generating brain activity data representing the state of the user's brain based on the image signal; and stopping the output of the brain activity data based on at least one selected from the group consisting of the image signal and the sensor signal, the sensor signal being a signal output from a sensor that detects changes in the user's surrounding environment that affect the brain activity data.

[0097] [Item 27]

[0098] The living body measurement method related to Item 27 includes the following processing: causing a light source to emit light to illuminate the user's head; causing an image sensor to detect at least a portion of reflected light returned from the head due to the illumination of the light and output an image signal; generating brain activity data representing the state of the user's brain based on the image signal; and calculating the reliability of the brain activity data based on at least one selected from the group consisting of the image signal and the sensor signal, and outputting reliability data representing the reliability, wherein the sensor signal is a signal output from a sensor that detects changes in the user's surrounding environment that affect the brain activity data.

[0099] [Item 28]

[0100] The program according to Item 28 is a program for use in a living body measurement device, the living body measurement device comprising a light source for irradiating light to a head of a user, an image sensor, a control circuit for controlling the light source and the image sensor, and a signal processing circuit; the control circuit causes the light source to emit the light, and causes the image sensor to detect at least a portion of reflected light returned from the head due to the irradiation of the light and output an image signal; the program causes the signal processing circuit to generate living body measurement data representing a state of the user based on the image signal output from the image sensor, determine whether to output the living body measurement data based on a sensor signal output from the image sensor and / or other sensors that detect changes in the surrounding environment of the user that affect the living body measurement data, or calculate the reliability of the living body measurement data based on the sensor signal, and output reliability data representing the reliability.

[0101] The embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, configuration positions and connection forms of components, steps, and the order of steps shown in the following embodiments are examples and do not limit the present invention. In addition, among the components of the following embodiments, the components that are not described in the independent claims representing the highest concept are described as arbitrary components.

[0102] In the present invention, all or part of a circuit, unit, device, component or part, or all or part of a functional block in a block diagram may be performed by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC) or an LSI (large scale integration). An LSI or an IC may be integrated into one chip or may be composed of a combination of multiple chips. For example, functional blocks other than storage elements may be integrated into one chip. Here, it is referred to as LSI or IC, but the name may change depending on the degree of integration, and may also be referred to as system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA) that can be programmed after the manufacture of the LSI, or a reconfigurable logic device that can reconfigure the bonding relationship within the LSI or set the circuit division within the LSI may also be used for the same purpose.

[0103] Furthermore, all or part of the functions or operations of a circuit, unit, device, component or part can be executed by software processing. In this case, the software is recorded in one or more non-temporary recording media such as ROMs, optical disks, hard disk drives, etc. When the software is executed by a processor, the functions determined by the software are executed by the processor and peripheral devices. The system or device may also have one or more non-temporary recording media with software recorded, a processor, and necessary hardware devices, such as interfaces.

[0104] Hereinafter, the living body measuring device according to the present embodiment will be described in detail with reference to the drawings.

[0105] (First embodiment)

[0106] [1. Biological Measurement Device]

[0107] First, refer to Figure 1A 3 , the configuration and operation of the living body measurement device 10 according to the first embodiment will be described.

[0108] Figure 1A1 is a diagram schematically showing an example of a biological measuring device 10 according to the present embodiment. The biological measuring device 10 includes an imaging unit 121, a measuring unit 110, and a signal processing circuit 122. The imaging unit 121 includes a light source 101, an image sensor 102 including a photoelectric conversion unit 103 and a charge storage unit 104, a control circuit 105 including a light source control unit 106 and a sensor control unit 107, and an image signal acquisition unit 108. The signal processing circuit 122 includes a biological measurement data generation unit 109 and an output determination unit 111.

[0109] [1-1. Light source 101]

[0110] The light source 101 emits light toward the target part of the user 100. The target part of the user 100 is, for example, the head, more specifically, the forehead. The light emitted from the light source 101 and reaching the user 100 is divided into a surface reflection component I1 reflected by the surface of the user 100 and an internal scattering component I2 scattered by the inside of the user 100. The internal scattering component I2 is a component that is reflected or scattered once or multiple times inside the biological body. When light is irradiated to the forehead of the user 100, the internal scattering component I2 refers to a component that reaches a part about 8 mm to 16 mm inside the surface of the forehead, such as the brain, and returns to the biological measuring device 10 again. The surface reflection component I1 includes three components: a direct reflection component, a diffuse reflection component, and a diffuse reflection component. The direct reflection component is a reflection component where the incident angle is equal to the reflection angle. The diffuse reflection component is a component that is diffused and reflected by the concave-convex shape of the surface. The diffuse reflection component is a component that is scattered and reflected by the internal tissue near the surface. When light is emitted toward the forehead of the user 100, the scattered reflection component is a component scattered and reflected inside the epidermis. Hereinafter, in the present invention, it is assumed that the surface reflection component I1 reflected by the surface of the user 100 includes these three components. The surface reflection component I1 and the internal scattered component I2 change their traveling directions by reflection or scattering, and part of them reaches the image sensor 102.

[0111] First, the method of obtaining the internal scattered component I2 is described. The light source 101 repeatedly emits pulsed light at a predetermined time interval or a predetermined timing in accordance with the instruction from the light source control unit 106. The pulsed light emitted from the light source 101 can be, for example, a rectangular wave with a falling period close to zero. In this specification, the "falling period" refers to the period from when the intensity of the pulsed light starts to decrease to when the decrease ends. Usually, the light incident on the user 100 propagates along various paths in the user 100 and is emitted from the surface of the user 100 with a time difference. Therefore, the rear end of the internal scattered component I2 of the pulsed light has a widening. When the target part is the forehead, the widening of the rear end of the internal scattered component I2 is about 4ns. If this situation is taken into account, the falling period of the pulsed light can be set to, for example, less than 2ns, which is less than half of it. The falling period can also be less than 1ns, which is another half of it. The rising period of the pulsed light emitted from the light source 101 is arbitrary. In this specification, the "rising period" refers to the period from when the intensity of the pulsed light starts to increase to when the increase ends. In the detection of the internal scattered component I2 in this embodiment, the falling part of the pulse light is used, and the rising part is not used. The rising part of the pulse light can be used for the detection of the surface reflected component I1. The light source 101 can be a laser such as LD. The light emitted from the laser has a steep time response characteristic in which the falling part of the pulse light is approximately at right angles to the time axis.

[0112] The wavelength of the light emitted from the light source 101 can be, for example, any wavelength included in the wavelength range of 650 nm to 950 nm. This wavelength range is included in the wavelength range from red to near infrared. In this specification, the term "light" is used not only for visible light but also for infrared. The above wavelength range is called the "biological window" and has the property of being less easily absorbed by water and skin in a living body. In the case of using a living body as a detection object, the detection sensitivity can be increased by using light in the above wavelength range. In the case of detecting changes in blood flow in the skin and brain of the user 100 as in the present embodiment, it can be considered that the light used is mainly absorbed by oxidized hemoglobin (HbO2) and deoxygenated hemoglobin (Hb). The wavelength dependence of light absorption is different in oxidized hemoglobin and deoxygenated hemoglobin. Generally, if the blood flow changes, the concentration of oxidized hemoglobin and deoxygenated hemoglobin changes. Therefore, the degree of light absorption also changes. Therefore, if the blood flow changes, the amount of light detected also changes over time.

[0113] The light source 101 may emit light of two or more wavelengths included in the above-mentioned wavelength range. Such light of multiple wavelengths may be emitted from multiple light sources, respectively.

[0114] In the biological measuring device 10 of this embodiment, since the user 100 is measured in a non-contact manner, a light source 101 designed in consideration of the effect on the retina can be used. For example, a light source 101 that satisfies the level 1 of the laser safety standard established by various countries can be used. When the level 1 is satisfied, low-illuminance light with a radiation emission limit (AEL) of less than 1 mW is irradiated to the subject 500. In addition, the light source 10 itself does not need to satisfy the level 1. For example, the level 1 of the laser safety standard can be satisfied by providing a diffuser plate or an ND filter in front of the light source 10 to diffuse or attenuate the light.

[0115] The pulse light emitted from the light source 101 does not need to be an extremely short pulse light. The pulse width is arbitrary. When irradiating light to the forehead in order to measure cerebral blood flow, the amount of light of the internal scattered component I2 is several thousand to tens of thousands of times smaller than the amount of light of the surface reflected component I1, which is very small. Furthermore, if the laser safety standard is taken into consideration, the amount of light emitted is small, so the detection of the internal scattered component I2 becomes difficult. Therefore, if the light source 101 emits pulse light with a relatively large pulse width, the cumulative amount of the internal scattered component I2 accompanied by the time delay is increased, and the detection light amount can be increased to improve the SN ratio.

[0116] The light source 101 emits pulse light with a pulse width of, for example, 3 ns or more. Generally, the temporal spread of light scattered in living tissues such as the brain is about 4 ns.

[0117] The light source 101 may also emit pulsed light with a pulse width of more than 5 ns, and further more than 10 ns. On the other hand, if the pulse width is too large, unused light increases and becomes a waste. Therefore, the light source 101, for example, emits pulsed light with a pulse width of less than 50 ns. Alternatively, the light source 101 may also emit pulsed light with a pulse width of less than 30 ns, and further less than 20 ns.

[0118] In addition, the irradiation pattern of the light source 101 may be, for example, a uniform intensity distribution in the irradiation area. In the biological measuring device 10 of this embodiment, the surface reflection component I1 can be separated from the internal scattering component I2 in time and reduced. Therefore, a light source 101 having an emission pattern with a uniform intensity distribution can be used. An irradiation pattern with a uniform intensity distribution can also be formed by diffusing the light emitted from the light source 101 with a diffusion plate.

[0119] In this embodiment, unlike the conventional technique, it is also possible to detect the internal scattered component I2 emitted from the position of the user 100 irradiated with the light from the light source 101. By irradiating the user 100 with light over a wide range in space, it is also possible to improve the measurement resolution.

[0120] [1-2. Image sensor 102]

[0121] The image sensor 102 detects at least a portion of the reflected pulse light emitted from the light source 101 and returned from the object part of the user 100. The image sensor 102 outputs one or more image signals corresponding to the intensity of the detected light. The image signal acquisition unit 108 acquires the image signal output from the image sensor 102. The image signal acquisition unit 108 sends the acquired image signal to the biological measurement data generation unit 109 and the measurement unit 110 described later. When brain activity data representing the state of brain activity is generated from the image signal as biological measurement data, the image signal is a signal corresponding to the intensity contained in at least a portion of the falling period of the reflected pulse light.

[0122] The image sensor 102 includes a photoelectric conversion unit 103 including a plurality of photoelectric conversion elements and a charge storage unit 104. Specifically, the image sensor 102 includes a plurality of light detection units arranged in two dimensions, and acquires two-dimensional information of the user 100 at a time. In this specification, a light detection unit is also referred to as a "pixel". The image sensor 102 is any image sensor such as a CCD image sensor or a CMOS image sensor.

[0123] The image sensor 102 is provided with an electronic shutter. The electronic shutter is a circuit that controls the timing of imaging. In the present embodiment, the sensor control unit 107 of the control circuit 105 has the function of an electronic shutter. The electronic shutter controls the period of signal accumulation for converting the received light into an effective electrical signal and accumulating it, and the period of stopping signal accumulation. The signal accumulation period may also be referred to as an "exposure period" or a "photography period". In the following description, the width of the exposure period may be referred to as a "shutter width". The time from the end of one exposure period to the beginning of the next exposure period may be referred to as a "non-exposure period". In the following, the state of exposure may be referred to as "OPEN" and the state of stopping exposure may be referred to as "CLOSE".

[0124] The image sensor 102 can adjust the exposure period and the non-exposure period in the sub-nanosecond range, for example, 30ps to 1ns, by an electronic shutter. In the biological measuring device 10 of the present embodiment, when at least brain activity data is generated, it is not necessarily necessary to correct the light amount of the subject. Therefore, the shutter width does not need to be larger than the pulse width. Therefore, the shutter width can be set to a value of, for example, 1ns or more and 30ns or less. According to the biological measuring device 10 of the present embodiment, the shutter width can be reduced. Therefore, the influence of the dark current included in the detection signal can be reduced.

[0125] When emitting light toward the forehead of the user 100 to detect information such as cerebral blood flow, the attenuation rate of the light inside is very large. For example, compared with the incident light, the emitted light will be attenuated to about 1 in 1,000,000. Therefore, if the internal scattered component I2 is to be detected, there will be insufficient light through the irradiation of only one pulse. In the irradiation at level 1 of the laser safety standard, the light amount is weak. In this case, the light source 101 emits pulsed light multiple times, and correspondingly, the image sensor 102 is also exposed multiple times through the electronic shutter. In this way, the detection signal can be accumulated to improve the sensitivity.

[0126] Hereinafter, a configuration example of the image sensor 102 will be described.

[0127] The image sensor 102 may include a plurality of pixels arranged two-dimensionally on an imaging surface. Each pixel may include a photoelectric conversion element such as a photodiode and one or more charge accumulation units. The following describes an example in which each pixel includes a photoelectric conversion element and two charge accumulation units. The photoelectric conversion element generates a signal charge corresponding to the amount of received light by photoelectric conversion. One of the two charge accumulation units accumulates the signal charge generated by the surface reflection component I1 of the pulse light, and the other accumulates the signal charge generated by the internal scattering component I2 of the pulse light.

[0128] In order to obtain the internal scattered component I2, the control circuit 105 causes the light source 101 to emit one or more pulsed lights. The control circuit 105 causes the image sensor 102 to detect the components included in the falling period in each pulsed light returned from the object part of the user 100 for each pixel of the image sensor 102. The components include the internal scattered component I2. The control circuit 105 causes the image sensor 102 to output a signal obtained by the detection. In this embodiment, the image signal based on the internal scattered component I2 is used to generate biological measurement data. Here, the light source 101 may also emit light of two wavelengths.

[0129] In order to obtain the surface reflection component I1, the control circuit 105 causes the light source 101 to emit one or more pulsed lights. The control circuit 105 causes the image sensor 102 to detect the component contained in the falling period before each pulsed light returned from the object part of the user 100 for each pixel of the image sensor 102. This component includes the surface reflection component I1. In addition, before the falling period refers to before the intensity of each pulsed light begins to decrease. The control circuit 105 causes the image sensor 102 to output a signal obtained by the detection. If the component contained in the falling period before each pulsed light is detected, the SN ratio of the image signal can be improved. In this specification, "detecting the component contained in the falling period before each pulsed light" also includes, for example, detecting the component contained in the rising period of each pulsed light, or detecting the entirety of each pulsed light. In this embodiment, the image signal based on the surface reflection component I1 is used to measure a state associated with the validity of biological measurement data.

[0130] Figure 1B 2 is a diagram showing a schematic configuration example of one pixel 201 of the image sensor 102. Figure 1B The structure of one pixel 201 is schematically shown, but does not necessarily reflect the actual structure. Figure 1B The pixel 201 shown includes a photodiode 203 that performs photoelectric conversion, a first floating diffusion layer (Floating Diffusion) 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207 as charge storage portions, and a drain 202 that discharges signal charges.

[0131] The photons incident on each pixel due to the emission of one pulse light are converted into signal electrons as signal charges by the photodiode 203. The converted signal electrons are discharged to the drain 202 or distributed to any one of the first floating diffusion layer 204 to the fourth floating diffusion layer 207 according to the control signal input from the control circuit 105.

[0132] The emission of pulse light from the light source 101, the accumulation of signal charges in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, and the discharge of signal charges to the drain 202 are repeated in this order. This repetitive action is high-speed, for example, it can be repeated tens of thousands to hundreds of millions of times within the time of one frame of a moving image. The time of one frame is, for example, about 1 / 30 second. The pixel 201 finally generates four image signals based on the signal charges accumulated in the first floating diffusion layer 204 to the fourth floating diffusion layer 207 and outputs them.

[0133] The control circuit 105 in this example causes the light source 101 to repeatedly emit pulse light having a wavelength λ1 and a pulse light having a wavelength λ2 in sequence. By selecting two wavelengths having different absorption rates in the internal tissue of the user 100 as the wavelength λ1 and the wavelength λ2, the state of the user 100 can be analyzed. For example, a wavelength longer than 805 nm can be selected as the wavelength λ1, and a wavelength shorter than 805 nm can be selected as the wavelength λ2. Thus, changes in the concentration of oxygenated hemoglobin and the concentration of deoxygenated hemoglobin in the blood of the user 100 can be detected.

[0134] The control circuit 105 first causes the light source 101 to emit a pulse light of wavelength λ1. The control circuit 105 causes the first floating diffusion layer 204 to accumulate signal charges during the first period when the internal scattered component I2 of the pulse light of wavelength λ1 is incident on the photodiode 203. In order to obtain the internal scattered component I2, the pulse light is emitted at a predetermined timing. This pulse light is referred to as the first pulse light.

[0135] Next, the control circuit 105 causes the second floating diffusion layer 205 to accumulate signal charges during the second period when the surface reflection component I1 of the pulse light of wavelength λ1 is incident on the photodiode 203. In order to obtain the surface reflection component I1, the pulse light is emitted at a predetermined timing different from that when the internal scattering component is obtained. This pulse light is referred to as the second pulse light.

[0136] Next, the control circuit 105 causes the light source 101 to emit pulse light of wavelength λ2. The control circuit 105 causes the third floating diffusion layer 206 to accumulate signal charges during the third period when the internal scattered component I2 of the pulse light of wavelength λ2 is incident on the photodiode 203.

[0137] Next, the control circuit 105 causes the fourth floating diffusion layer 207 to accumulate signal charges during the fourth period in which the surface reflection component I1 of the pulse light having the wavelength λ2 is incident on the photodiode 203 .

[0138] In this way, after the emission of the pulse light of wavelength λ1 is started, the control circuit 105 causes the first floating diffusion layer 204 and the second floating diffusion layer 205 to accumulate the signal charge from the photodiode 203 in sequence with a predetermined time difference. Then, after the emission of the pulse light of wavelength λ2 is started, the control circuit 105 causes the third floating diffusion layer 206 and the fourth floating diffusion layer 207 to accumulate the signal charge from the photodiode 203 in sequence with the predetermined time difference. The above operation is repeated multiple times. In order to estimate the amount of disturbance light and ambient light, a period during which signal charge is accumulated in other floating diffusion layers (not shown) in a state where the light source 101 is turned off can be set. By subtracting the signal charge amount of the other floating diffusion layers from the signal charge amount of the first floating diffusion layer 204 to the fourth floating diffusion layer 207, a signal after the disturbance light and ambient light components are removed can be obtained.

[0139] In addition, in the present embodiment, the number of charge accumulation units is set to 4, but a plurality of numbers more than two may be set depending on the purpose. For example, when only one wavelength is used, the number of charge accumulation units may be 2. In addition, in the application where one wavelength is used and the surface reflection component I1 is not detected, the number of charge accumulation units per pixel may be 1. In addition, even when two or more wavelengths are used, if the video recording using each wavelength is performed in different frames, the number of charge accumulation units may be 1. In addition, as described later, if the detection of the surface reflection component I1 and the detection of the internal scattering component I2 are performed in different frames, respectively, the number of charge accumulation units may be 1.

[0140] Figure 1C 1 is a diagram showing an example of the structure of the image sensor 102. Figure 1C In the example shown, the area surrounded by the double-dashed line frame corresponds to one pixel 201. The pixel 201 includes one photodiode. Figure 1C Only four pixels arranged in two rows and two columns are shown, but a greater number of pixels may be configured in practice. The pixel 201 includes four floating diffusion layers, namely, the first floating diffusion layer 204 to the fourth floating diffusion layer 207. The signals accumulated in the four floating diffusion layers, namely, the first floating diffusion layer 204 to the fourth floating diffusion layer 207, are processed like the signals of four pixels of a normal CMOS image sensor and are output from the image sensor 102.

[0141] Each pixel 201 has four signal detection circuits. Each signal detection circuit includes a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. In this example, the reset transistor 310 is connected to the Figure 1BThe pulse input into the gate of the reset transistor 310 corresponds to the drain 202 shown, and the pulse corresponding to the drain discharge pulse. Each transistor is, for example, a field effect transistor formed on a semiconductor substrate, but is not limited to this. As shown in the figure, one of the input terminal and the output terminal of the source follower transistor 309 is connected to one of the input terminal and the output terminal of the row selection transistor 308. The above-mentioned one of the input terminal and the output terminal of the source follower transistor 309 is typically the source. The above-mentioned one of the input terminal and the output terminal of the row selection transistor 308 is typically the drain. The gate of the source follower transistor 309, which serves as a control terminal, is connected to the photodiode 203. The signal charge of holes or electrons generated by the photodiode 203 is accumulated in the floating diffusion layer, which serves as a charge accumulation unit between the photodiode 203 and the source follower transistor 309.

[0142] Although in Figure 1C Although not shown, the first to fourth floating diffusion layers 204 to 207 are connected to the photodiode 203. A switch may be provided between the photodiode 203 and each of the first to fourth floating diffusion layers 204 to 207. The switch switches the conduction state between the photodiode 203 and each of the first to fourth floating diffusion layers 204 to 207 according to the signal accumulation pulse from the control circuit 105. Thus, the start and stop of the accumulation of signal charges to each of the first to fourth floating diffusion layers 204 to 207 are controlled. The electronic shutter of this embodiment has a mechanism for such exposure control.

[0143] The signal charges accumulated in the first floating diffusion layer 204 to the fourth floating diffusion layer 207 are read out by turning on the gate of the row selection transistor 308 by the row selection circuit 302. At this time, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified according to the signal potential of the first floating diffusion layer 204 to the fourth floating diffusion layer 207. The analog signal based on the current read out from the vertical signal line 304 is converted into digital signal data by the analog-digital conversion circuit 307 connected to each column. The digital signal data is read out by each column by the column selection circuit 303 and output from the image sensor 102. After reading out one row, the row selection circuit 302 and the column selection circuit 303 read out the next row, and the information of the signal charges of the floating diffusion layers of all rows is read out in the same manner. After reading out all the signal charges, the control circuit 105 resets all the floating diffusion layers by turning on the gate of the reset transistor 310. Thus, the imaging of one frame is completed. Similarly, by repeating the high-speed imaging of frames, the imaging of a series of frames by the image sensor 102 is completed.

[0144] In this embodiment, an example of a CMOS type image sensor 102 is described, but the image sensor 102 may be a CCD type, a single photon counting type element, or an amplification type image sensor such as an EMCCD or ICCD.

[0145] Figure 1D FIG. 2 is a diagram schematically showing an example of an operation within one frame of the present embodiment. Figure 1D As shown, the emission of the pulse light of wavelength λ1 and the emission of the pulse light of wavelength λ2 may be switched alternately multiple times within one frame. This can reduce the time difference in the acquisition timing of the detection images based on the two wavelengths, and enable the imaging based on the pulse light of the two wavelengths to be performed almost simultaneously.

[0146] In this embodiment, the image sensor 102 detects both the surface reflection component I1 and the internal scattering component I2 of the pulse light. Based on the temporal or spatial changes in the internal scattering component I2, biological measurement data of the user 100 can be generated. On the other hand, based on the temporal or spatial changes in the surface reflection component I1, data related to the validity of the biological measurement data can be measured.

[0147] In addition, in this specification, a signal related to the validity of biological measurement data may be referred to as a "validity signal".

[0148] [1-3. Control circuit 105]

[0149] The control circuit 105 controls the above-mentioned actions of the light source 101 and the image sensor 102. Specifically, the control circuit 105 adjusts the time difference between the emission timing of the pulse light of the light source 101 and the shutter timing of the image sensor 102. Hereinafter, this time difference may be referred to as "phase" or "phase delay". The "emission timing" of the pulse light of the light source 101 refers to the time when the pulse light emitted from the light source 101 starts to rise. The control circuit 105 can adjust the phase by changing the emission timing or by changing the shutter timing.

[0150] The control circuit 105 may also be configured to remove an offset component from a signal detected by a light receiving element of the image sensor 102. The offset component is a signal component caused by ambient light such as sunlight or fluorescent light, or interference light. For example, by detecting a signal with the image sensor 102 in a state where the drive of the light source 101 is turned off and no light is emitted, the offset component caused by the ambient light or interference light is estimated.

[0151] The control circuit 105 may be an integrated circuit including a processor such as a central processing unit (CPU) or a microcomputer and a memory. The control circuit 105 adjusts the emission timing and shutter timing, estimates the offset component, and removes the offset component by executing a program recorded in the memory.

[0152] Figure 1E 1 is a flowchart showing the operation of the light source 101 and the image sensor 102 performed by the control circuit 105. The control circuit 105 includes a light source control unit 106 and a sensor control unit 107, and generally executes the following operations. Figure 1G Here, the operation in the case of detecting only the internal scattered component I2 will be described.

[0153] In step S101, the light source control unit 106 first causes the light source 101 to emit pulse light for a predetermined time. At this time, the electronic shutter of the image sensor 102 is in a state of stopping exposure. The sensor control unit 107 stops the electronic shutter from exposing until a portion of the pulse light is reflected by the surface of the user 100 and reaches the image sensor 102. Next, in step S102, the sensor control unit 107 starts exposing the electronic shutter at the timing when another portion of the pulse light is scattered inside the user 100 and reaches the image sensor 102. After the predetermined time has passed, in step S103, the sensor control unit 107 stops the electronic shutter from exposing. Next, in step S104, the control circuit 105 determines whether the number of times the above-mentioned signal accumulation has been performed has reached a predetermined number of times. If the determination is "no", steps S101 to S103 are repeated until the determination is "yes". If it is determined as “YES” in step S104 , then in step S105 , the sensor control unit 107 causes the image sensor 102 to generate and output a signal representing an image based on the signal charges accumulated in each floating diffusion layer.

[0154] Through the above operation, it is possible to detect the components of light scattered inside the measurement object with high sensitivity. In addition, multiple emission and exposure are not essential, but are performed as needed.

[0155] [1-4. Measurement unit 110]

[0156] The measuring unit 110 measures a validity signal indicating the validity of biological measurement data, and sends the measurement result as a signal to the output determining unit 111 described later.

[0157] In this embodiment, the effectiveness signal is measured based on an image including temporal or spatial changes in the surface reflection component I1 . The image is acquired from the image sensor 102 .

[0158] The measuring unit 110 may also include an operation circuit for performing operation processing such as image processing. Such an operation circuit may be implemented by a combination of a programmable logic device (PLD) such as a digital signal processor (DSP), a field programmable gate array (FPGA), or a central processing unit (CPU), or an image processing operation processor (GPU) and a computer program.

[0159] In other embodiments described below, the effectiveness signal may be measured based on an image including temporal or spatial changes in the internal scattering component I2.

[0160] In other embodiments described later, the validity signal may be measured based on something other than the image output from the image sensor 102. In this case, the measuring unit 110 may include a sensor disposed inside or outside the biological measurement device 10. The sensor measures changes in the surrounding environment of the user 100 and outputs a sensor signal indicating the change. The sensor signal may be, for example, a signal indicating the amount of change in the physical change of the surrounding environment that affects the biological measurement data. The sensor signal may be, for example, a signal indicating that the surrounding environment is in a specific state that affects the biological measurement data. "Influencing the biological measurement data" includes mixing noise into the biological measurement data or the biological measurement data becoming unmeasurable. The sensor may also be at least one selected from the group consisting of an illumination sensor, an acceleration sensor, a velocity sensor, a steering angle sensor, and a gear position sensor.

[0161] [1-4. Signal processing circuit 122]

[0162] The signal processing circuit 122 processes the signal output from the image sensor 102. The signal processing circuit 122 includes a biological measurement data generating unit 109 and an output determining unit 111. Figure 1A In the example shown, the signal processing circuit 122 and the measuring unit 110 are separated, but they may be integrated.

[0163] The biological measurement data generation unit 109 generates biological measurement data of the user 100 based on the image signal output from the image sensor 102. When the biological measurement data is brain activity data of the user 100, the biological measurement data generation unit 109 processes the image signal including the temporal or spatial change of the internal scattered component I2, generates moving image data indicating the temporal change of cerebral blood flow, and sends a signal based on the generation result to the output determination unit 111. The temporal change of cerebral blood flow is, for example, the temporal change of the concentration of oxygenated hemoglobin and / or deoxygenated hemoglobin.

[0164] When the biological measurement data is the brain activity data of the user 100, the biological measurement data generating unit 109 is not limited to the moving image data of the cerebral blood flow, and may generate other data related to the cerebral blood flow. The data related to the cerebral blood flow is, for example, the psychological state of the user 100 estimated from the moving image data of the cerebral blood flow.

[0165] It is known that there is a close relationship between changes in blood components such as cerebral blood flow or hemoglobin and human neural activity. For example, the activity of nerve cells changes according to changes in human emotions, thereby changing cerebral blood flow or blood components. Therefore, if biological information such as changes in cerebral blood flow or blood components can be measured, the psychological state of user 100 can be inferred. The psychological state of user 100, for example, involves mood, emotion, health status or temperature sensation. Mood, for example, is pleasant or unhappy. Emotions, for example, are peace of mind, uneasiness, sadness or anger. Health status, for example, is healthy or tired. Temperature sensation, for example, is hot, cold or stuffy. In addition, derived from this, indicators indicating the degree of brain activity are also included in the psychological state. Such indicators are, for example, proficiency, familiarity or concentration. In this specification, such data related to cerebral blood flow are collectively referred to as brain activity data.

[0166] The output determination unit 111 determines whether to output the biological measurement data sent from the biological measurement data generation unit 109 based on the measurement result sent from the measurement unit 110. The measurement result includes a value calculated by the measurement unit 110 based on the image signal output from the image sensor 102 and / or the sensor signal output from the above sensor.

[0167] At the timing when the output of the biological measurement data is stopped, the output determination unit 111 outputs a signal indicating that the biological measurement data at that timing is invalid. Alternatively, the output determination unit 111 continues to output the same data as the immediately previous valid biological measurement data. Alternatively, at the timing when the biological measurement data becomes valid again, the output determination unit 111 outputs data interpolated from the immediately previous valid biological measurement data.

[0168] The signal processing circuit 122, like the measuring unit 110, may also include an operation circuit for performing operation processing such as image processing. Such an operation circuit may be implemented, for example, by a programmable logic device (PLD) such as a digital signal processor (DSP), a field programmable gate array (FPGA), or a central operation processing unit (CPU), or a combination of an operation processor (GPU) for image processing and a computer program. In addition, the signal processing circuit 122 and the control circuit 105 may be either a combined circuit or a separate circuit. In addition, the signal processing circuit 122 may also be a component of an external device such as a server located at a distance. In this case, the external device such as the server transmits and receives data to and from the light source 101, the image sensor 102, and the control circuit 105 through wired communication or wireless communication.

[0169] [1-5. Others]

[0170] The biological measuring device 10 may also include an imaging optical system that forms a two-dimensional image of the user 100 on the light receiving surface of the image sensor 102. The optical axis of the imaging optical system is substantially orthogonal to the light receiving surface of the image sensor 102. The imaging optical system may also include a zoom lens. By changing the position of the zoom lens, the magnification of the two-dimensional image of the user 100 changes. As a result, the resolution of the two-dimensional image on the image sensor 102 changes. Therefore, even if the distance to the user 100 is long, it is possible to enlarge the desired measurement area and observe it in detail.

[0171] In addition, the biological measuring device 10 may be provided with a bandpass filter between the user 100 and the image sensor 102 that allows only light of a wavelength band emitted from the light source 101 or light near the wavelength band to pass. This can reduce the influence of interference components such as ambient light. The bandpass filter may be composed of a multilayer film filter or an absorption filter. The bandwidth of the bandpass filter may be set to have a width of about 20 to 100 nm in consideration of the temperature of the light source 101 and the wavelength variation accompanying oblique incidence to the filter.

[0172] In addition, the biological measurement device 10 may include polarizing plates between the light source 101 and the user 100, and between the image sensor 102 and the user 100. In this case, the polarization directions of the polarizing plate disposed on the light source 101 side and the polarizing plate disposed on the image sensor 102 side are in a cross-Nichol relationship. Thus, it is possible to prevent the regular reflection component, which is a component having the same incident angle and reflection angle among the surface reflection components I1 of the user 100, from reaching the image sensor 102. That is, it is possible to reduce the amount of light of the surface reflection component I1 reaching the image sensor 102.

[0173] [2. Operation of light source and image sensor]

[0174] The biological measuring device 10 of this embodiment can detect the surface reflection component I1 and the internal scattering component I2 separately. When the user 100 is a person and the object part is the forehead, the signal intensity based on the internal scattering component I2 to be detected is very small. This is not only because the amount of light emitted to meet the safety standard of the laser is very small as mentioned above, but also because the scattering and absorption of light caused by the scalp, cerebral myeloid fluid, skull, gray matter, white matter and blood flow are large. Furthermore, the change in signal intensity caused by the change in blood flow or components in the blood flow during brain activity is equivalent to a few tens of times the signal intensity before the change, which becomes very small. Therefore, in this embodiment, when taking pictures, the surface reflection component I1 that is several thousand to tens of thousands times the signal component to be detected is removed as much as possible.

[0175] Hereinafter, an example of the operation of the light source 101 and the image sensor 102 in the biological measurement device 10 of the present embodiment will be described.

[0176] like Figure 1A As shown, if the light source 101 irradiates the user 100 with pulsed light, a surface reflection component I1 and an internal scattering component I2 are generated. Part of the surface reflection component I1 and the internal scattering component I2 reaches the image sensor 102. The internal scattering component I2 is emitted from the light source 101 and passes through the inside of the user 100 before reaching the image sensor 102. Therefore, the optical path length of the internal scattering component I2 is longer than the optical path length of the surface reflection component I1. Therefore, the time when the internal scattering component I2 reaches the image sensor 102 is later than the time when the surface reflection component I1 reaches the image sensor 102 on average.

[0177] Figure 1F: This is a diagram schematically showing the optical signal arriving at the image sensor 102 when the rectangular pulse light emitted from the light source 101 returns from the user 100. In signals (a) to (d), the horizontal axis represents time (t). In signals (a) to (c), the vertical axis represents intensity, and in signal (d), the vertical axis represents the open (OPEN) or closed (CLOSE) state of the electronic shutter. Signal (a) represents the surface reflection component I1. Signal (b) represents the internal scattering component I2. Signal (c) represents the total component of the surface reflection component I1 shown in signal (a) and the internal scattering component I2 shown in signal (b). As shown in signal (a), the surface reflection component I1 maintains a rectangular shape. On the other hand, as shown in signal (b), the internal scattering component I2 is the total of light that has passed through various optical path lengths. Therefore, signal (b) shows a characteristic such as the rear end tailing of the pulse light. In other words, since the falling period of the internal scattered component I2 is longer than the falling period of the surface reflected component I1, the proportion of the internal scattered component I2 is increased and extracted from the light signal shown in signal (c). For this reason, as shown in signal (d), the exposure of the electronic shutter starts after the rear end of the surface reflected component I1. After the rear end refers to when the surface reflected component I1 falls or thereafter. The shutter timing of the electronic shutter is adjusted by the control circuit 105. As described above, the biological measuring device 10 of this embodiment distinguishes and detects the surface reflected component I1 from the internal scattered component I2 that reaches the deep part of the object. Therefore, the emission pulse width and the shutter width are arbitrary. Therefore, unlike the conventional method of using a fast scan camera, the cost can be greatly reduced by a simple structure.

[0178] exist Figure 1F In the example shown in the signal (a), the rear end of the surface reflection component I1 drops vertically. In other words, the time from the start of the drop to the end of the drop of the surface reflection component I1 is zero. However, in reality, there are cases where the waveform of the pulse light emitted from the light source 101 itself is not completely vertical, there are fine irregularities on the surface of the user 100, and / or scattering occurs in the epidermis. In these cases, the rear end of the surface reflection component I1 does not drop vertically. In addition, since the user 100 is an opaque object, the amount of light of the surface reflection component I1 is much larger than the amount of light of the internal scattering component I2. Therefore, even in the case where the rear end of the surface reflection component I1 slightly extends from the vertical drop position, the internal scattering component I2 may be masked. In addition, during the readout period of the electronic shutter, a time delay associated with the movement of electrons may occur. Therefore, there is a possibility that the above cannot be achieved. Figure 1FThe ideal binary readout situation is shown in the signal (d). Therefore, the control circuit 105 may also delay the shutter timing of the electronic shutter slightly after the surface reflection component I1 just drops. For example, the shutter timing of the electronic shutter may be delayed by about 0.5ns to 5ns. In addition, the control circuit 105 may adjust the emission timing of the light source 101 instead of adjusting the shutter timing of the electronic shutter. The control circuit 105 adjusts the time difference between the shutter timing of the electronic shutter and the emission timing of the light source 101. When measuring the blood flow or the changes in the components in the blood flow during brain activity in a non-contact manner, if the shutter timing is delayed too much, the internal scattered component I2, which is originally small, is further reduced. Therefore, the shutter timing may also be kept near the rear end of the surface reflection component I1. The time delay caused by the scattering of the user 100 is 4ns. Therefore, the maximum delay amount of the shutter timing is about 4ns.

[0179] The plurality of pulse lights emitted from the light source 101 may be exposed at shutter timings of the same phase. In this way, the detection light amount of the internal scattered component I2 is amplified.

[0180] Alternatively, instead of or in addition to placing a bandpass filter between the user 100 and the image sensor 102, the control circuit 105 may estimate the offset component by performing photography with the same exposure time without causing the light source 101 to emit light. The estimated offset component is subtracted from the signal detected by each pixel of the image sensor 102. In this way, the dark current component generated in the image sensor 102 can be removed.

[0181] The internal scattered component I2 includes internal information of the user 100, such as cerebral blood flow information. The amount of light absorbed by the blood changes according to the temporal change of the cerebral blood flow of the user 100. As a result, the amount of detection light obtained by the image sensor 102 also increases or decreases accordingly. Therefore, by monitoring the internal scattered component I2, the brain activity state can be estimated according to the change of the cerebral blood flow of the user 100. In this specification, the signal representing the internal scattered component I2 among the signals output from the image sensor 102 is sometimes referred to as a "brain activity signal". The brain activity signal can include the increase or decrease information of the cerebral blood flow of the user 100.

[0182] Next, an example of a method for detecting the surface reflection component I1 is described. The surface reflection component I1 includes surface information of the user 100. The surface information is, for example, blood flow information of the face and scalp. The image sensor 102 detects the surface reflection component I1 in the optical signal of the pulse light emitted from the light source 101, reaching the user 100 and returning to the image sensor 102 again.

[0183] Figure 1GFIG. 1 is a diagram schematically showing an example of a time chart for detecting the surface reflection component I1. Figure 1G As shown, the shutter may be set to OPEN before the pulse light reaches the image sensor 102, and the shutter may be set to CLOSE before the rear end of the pulse light arrives. By controlling the shutter in this way, the mixing of the internal scattered component I2 can be reduced. As a result, the proportion of light passing through the surface near the user 100 can be increased. The timing of shutter CLOSE may also be set to just after the light reaches the image sensor 102. Thus, signal detection of the proportion of the surface reflection component I1 with a relatively short optical path length is increased. By obtaining the signal of the surface reflection component I1, the pulse rate of the user 100 or the degree of oxidation of the facial blood flow can also be detected. As another method of obtaining the surface reflection component I1, the image sensor 102 may detect the entire pulse light emitted from the light source 101, or detect the continuous light emitted from the light source 101.

[0184] Figure 1H 1 is a diagram schematically showing an example of a time chart when detecting the internal scattered component I2. By opening the shutter while the rear end portion of the pulse reaches the image sensor 102, the signal of the internal scattered component I2 can be acquired.

[0185] The surface reflection component I1 may be detected by a device other than the biological measuring device 10 that obtains the internal scattering component I2. A device different from the device that obtains the internal scattering component I2, or other equipment such as a sphygmograph or a Doppler blood flowmeter may be used. In this case, the other equipment is used in consideration of timing synchronization between the equipment, interference of light, and alignment of the detection part. If time-sharing video is performed by the same camera or the same sensor as in the present embodiment, time and space deviations are less likely to occur. In the case where the signals of both the surface reflection component I1 and the internal scattering component I2 are obtained by the same sensor, as in Figure 1G and Figure 1H As shown in FIG. 1 , the components to be obtained may be switched every frame. Figures 1B to 1D As described above, the components to be acquired are alternately switched at high speed within one frame. In this case, the detection time difference between the surface reflection component I1 and the internal scattering component I2 can be reduced. In addition, the signals of both the surface reflection component I1 and the internal scattering component I2 can be acquired from the same pulse light.

[0186] Furthermore, it is also possible to use light of two wavelengths to obtain the signals of the surface reflection component I1 and the internal scattering component I2. For example, pulsed light of two wavelengths of 750 nm and 850 nm may be used. Thus, the concentration change of oxyhemoglobin and the concentration change of deoxyhemoglobin can be calculated based on the change of the detected light amount at each wavelength. When the surface reflection component I1 and the internal scattering component I2 are obtained at two wavelengths, for example, as shown in FIG. Figures 1B to 1D As described above, a method of rapidly switching the four types of charge accumulation within one frame can be used. This method can reduce the time variation of the detection signal.

[0187] The biological measuring device 10 can emit pulsed near-infrared light or visible light toward the forehead of the user 100, and detect changes in the amount of oxyhemoglobin or pulse in the scalp or face based on the temporal changes in the surface reflection component I1. The light source 101 emits near-infrared light or visible light to obtain the surface reflection component I1. If it is near-infrared light, it can be measured regardless of day or night. In the case of measuring the pulse, visible light with higher sensitivity can also be used. If it is daytime, sunlight or indoor light sources as interference light can be used instead of lighting. In the case of insufficient light, it can also be strengthened by a dedicated light source. The internal scattered component I2 includes light components that reach the brain. By measuring the temporal changes in the internal scattered component I2, the temporal increase or decrease in cerebral blood flow can be measured.

[0188] The light reaching the brain also passes through the scalp and the surface of the face. Therefore, the changes in the blood flow of the scalp and the face are also detected by superimposing. In order to remove or reduce the influence thereof, the biological measurement data generation unit 109 may perform a process of subtracting the surface reflection component I1 from the internal scattering component I2 detected by the image sensor 102 when using the brain activity data as the biological measurement data. In this way, pure brain blood flow information after removing the blood flow information of the scalp and the face can be obtained. In the subtraction method, for example, a method of subtracting a value obtained by multiplying the signal of the surface reflection component I1 by a certain coefficient greater than 1 determined in consideration of the optical path length difference from the signal of the internal scattering component I2 may be used. This coefficient may be calculated by simulation or experiment based on the average value of the optical constant of a normal human head, for example. Such a subtraction process can be easily performed when the same camera or sensor is used to measure light of the same wavelength. This is because it is easy to reduce the deviation in time and space, and it is easy to make the characteristics of the scalp blood flow component contained in the internal scattering component I2 consistent with the characteristics of the surface reflection component I1.

[0189] There is a skull between the brain and the scalp. Therefore, the two-dimensional distribution of cerebral blood flow and the two-dimensional distribution of scalp and face blood flow are independent. Therefore, based on the signal detected by the image sensor 102, the two-dimensional distribution of the internal scattering component I2 and the two-dimensional distribution of the surface reflection component I1 can be separated using a statistical method such as independent component analysis or principal component analysis.

[0190] [3.Biological measurement process]

[0191] Next, an example of a method for measuring the biological measurement data of the user 100 using the above-mentioned biological measurement device 10 is described. In this embodiment, the measuring unit 110 measures the amount of body movement of the user 100 based on the image signal including the surface reflection component I1. In this embodiment, the amount of body movement is the movement distance of the target part. Here, the target part is the head of the user 100. The movement distance can also be said to be the displacement from the reference position. In this embodiment, based on the measurement result, the output determination unit 111 determines whether to output the biological measurement data.

[0192] Figure 2 This is a flowchart showing an example of a process of measuring biological measurement data of the user 100 according to the present embodiment.

[0193] In step S201, the biological measurement device 10 performs initial settings before measuring biological measurement data. In step S201, the control circuit 105 optimally adjusts the timing of emitting pulse light from the light source 101 and the shutter timing of the image sensor 102 according to the distance between the biological measurement device 10 and the user 100. In addition, in step S201, the control circuit 105 causes the image signal acquisition unit 108 to send the image signal including the surface reflection component I1 to the measurement unit 110; and the measurement unit 110 calculates the position of the head in the initial state and stores it in a memory (not shown) in the measurement unit 110.

[0194] In step S202 , the control circuit 105 causes the image signal acquisition unit 108 to send an image signal indicating an internal image including the internal scattering component I2 to the biological measurement data generation unit 109 .

[0195] In step S203 , the control circuit 105 causes the image signal acquisition unit 108 to send an image signal representing a surface image including the surface reflection component I1 to the measurement unit 110 .

[0196] The order of step S202 and step S203 may also be reversed.

[0197] In step S204, the measuring unit 110 calculates the position of the object part based on the image signal representing the surface image, and calculates the difference value between the position of the object part in the initial state stored in the memory, thereby calculating the amount of body movement. The difference value is the displacement of the object part. In the calculation of the position of the object part based on the image signal, a well-known image processing method such as feature point extraction using edge detection can be used. Compared with the two-dimensional image of cerebral blood flow, the two-dimensional image of the face contains more components with high spatial frequency. Therefore, the two-dimensional image of the face is advantageous in that it is easier to extract feature points. The measuring unit 110 sends the amount of body movement to the output determination unit 111 as a difference value based on the initial state.

[0198] In step S205, the output determination unit 111 determines whether the amount of body movement is below a threshold value. The threshold value is a value within a range of, for example, 1 mm to 30 mm. When a two-dimensional distribution of cerebral blood flow is obtained as biological measurement data, the threshold value can be increased if the desired resolution of the two-dimensional distribution is relatively low.

[0199] When the amount of body movement is less than or equal to the threshold value, in step S206, the biological measurement data generation unit 109 generates biological measurement data based on the image signal representing the internal image, and sends the biological measurement data to the output determination unit 111. In step S207, the output determination unit 111 outputs the biological measurement data. The output may be displayed on a display unit (not shown) of the biological measurement device 10, for example, or may be used for controlling a higher-level system (not shown).

[0200] When the amount of body movement exceeds the threshold value, step S206 and step S207 are skipped, and the output determination unit 111 stops outputting the biological measurement data. Here, the output determination unit 111 may output a signal indicating that the biological measurement data at that timing is invalid instead of stopping the output of the biological measurement data. Alternatively, the output determination unit 111 may continue to output the same data as the immediately previous valid biological measurement data. Alternatively, the output determination unit 111 may output data interpolated from the immediately previous valid biological measurement data at the timing when the biological measurement data becomes valid again.

[0201] In step S208 , it is determined whether the measurement for a predetermined period has been performed. This determination may be made by the biological measuring device 10 or by a higher-level system connected to the output of the biological measuring device 10 .

[0202] The above-mentioned "prescribed period" may be, for example, a period until the psychological state of the user 100 can be estimated. Alternatively, the "prescribed period" may be a period until a series of tasks given to the user 100 are completed. Alternatively, the "prescribed period" may be a period until the user 100 completes a series of operations. The series of operations may be, for example, driving a car or operating a game console.

[0203] When the measurement cannot be performed for the predetermined period, the biological measuring device 10 repeats the flow from step S202 to step S208. When the measurement can be performed for the predetermined period, the biological measuring device 10 ends the measurement.

[0204] The above-mentioned operation of the biological measurement device 10 of the present embodiment can be summarized as follows. The control circuit 105 causes the light source 101 to emit light and causes the image sensor 102 to output an image signal. The biological measurement data generation unit 109 generates biological measurement data based on the image signal. The output determination unit 111 determines whether to output the biological measurement data. The control circuit 105, the biological measurement data generation unit 109, and the output determination unit 111 repeatedly perform the above-mentioned operation. The output determination unit 111 stops the output of the biological measurement data during a period in which the value calculated based on the image signal output from the image sensor 102 meets a preset condition. In the above-mentioned example, the value calculated based on the image signal output from the image sensor 102 is a differential value indicating the displacement amount of the object part calculated by the measurement unit 110. The preset condition is that the differential value exceeds a threshold value.

[0205] [4. Actions during biological measurement]

[0206] Figure 3A 1 is a diagram for explaining an example of the operation of processing the biological measurement data of the user 100 according to the present embodiment. Figure 3A In the example shown, the target portion is the head of the user 100 .

[0207] exist Figure 3A In the examples shown in parts (a) to (d), the horizontal axis direction represents the passage of frames or time. Part (a) schematically shows the change of the surface image, which is an image containing the surface reflection component I1. Part (b) shows the change of the amount of body movement. Part (c) shows the determination result. Part (d) shows the change of the oxygenated hemoglobin concentration in the forehead area as brain activity data as an example of biological measurement data.

[0208] Each frame is assigned a number, but one or more frames may be inserted between each frame. The frame rate or measurement frequency may also be, for example, in the range of 1fps (frame per second) to 30fps. The frequency of measuring body movement and the frequency of generating biological measurement data may also be different. Changes in cerebral blood flow change slowly from 1 second to several seconds. On the other hand, body movement obtained as an image in a non-contact manner may also change at a faster rate than the cerebral blood flow. Therefore, in order to detect body movement more accurately, the frequency of measuring body movement may be greater than the frequency of generating cerebral blood flow data.

[0209] exist Figure 3A In the example shown in part (a) of FIG. 1 , the amount of body movement measured by the measuring unit 110 exceeds the threshold value due to the movement of the object part in frame 3 and frame 4. There is a correlation between the amount of body movement and the irregularity of the generated biological measurement data. Therefore, during the period when the amount of body movement exceeds the threshold value, the output determination unit 111 determines that the biological measurement data is invalid and stops the output of the biological measurement data. As a result, Figure 3A As shown in part (d) of FIG. 1 , brain activity data as biological measurement data is output in frames 1, 2, and 5. Figure 3A In the example shown in part (b), the period during which the amount of body movement exceeds the threshold is recorded as an "invalid period".

[0210] According to the above-described embodiment, it is possible to measure biological measurement data in a non-contact manner without urging the user 100 to measure again. Thus, a biological measurement device capable of daily measurement can be realized. In addition, calculations accompanying the restoration of cerebral blood flow distribution are not required. Therefore, the calculation cost can be reduced, and biological measurement based on an inexpensive method can be performed.

[0211] Figure 3B is a diagram schematically showing the relationship between changes in body movement and ineffective periods. Figure 3A In the action shown in part (b) of the embodiment, the period during which the amount of body movement exceeds the threshold is set as an invalid period. Figure 3B As shown in part (a) of FIG. 1 , the period before the amount of physical exercise exceeds the threshold value may also be included in addition to the period during which the amount of physical exercise exceeds the threshold value. f1 and the period after the amount of physical activity falls below the threshold r1 The period of time during which the amount of physical exercise exceeds the threshold value may also include the period d f1 and period d r1 The period of one of the two is set as an invalid period. For example, the period d f1 and period d r1Thus, the output determination unit 111 may determine the time period d before the start of the period other than the period when the value calculated based on the image signal output from the image sensor 102 meets the preset condition. f1 and / or the period after the end of r1 If the value does not meet the condition, the output of biological measurement data is also stopped.

[0212] In addition, the timing of starting the invalid period may also be after the amount of body movement exceeds the threshold. That is, when the timing of the amount of body movement exceeding the threshold is used as a reference, d f1 The reason for this is that there may be a delay time for system processing from when the amount of body movement exceeds the threshold to when the invalid period is set.

[0213] Furthermore, if Figure 3B As shown in part (b), the period after the end of r2 than the period before the start f2 For example, during the period d f2 is equivalent to a period of 0.5 frames, and the period d r2 This is a period equivalent to 4.5 frames. The reason for this will be described later in the embodiments.

[0214] In this embodiment, the amount of body movement is calculated based on the image signal containing the surface reflection component I1. On the other hand, the amount of body movement can also be calculated based on the image signal containing the internal scattering component I2. The image signal containing the internal scattering component I2 also contains information on the outer shape of the object. However, the image signal containing the surface reflection component I1 can further increase the amount of detection light. Therefore, the image signal containing the surface reflection component I1 is more advantageous than the image signal containing the internal scattering component I2 in terms of SN ratio.

[0215] In addition, in the present embodiment, the displacement of the object part is set as the measurement object, but the measurement object is not limited to this. The measurement object may also be the moving speed of the object part. In the frame where the absolute value of the moving speed of the object part exceeds the threshold, the output of the brain activity data may also be stopped. This is because it can be considered that body movement has occurred in the frame. In other words, the moving speed of the object part is the moving distance of the object part between frames.

[0216] Figure 3C Schematically shows the relationship between the moving speed of the object part and the invalid period. As an example of a method for setting the invalid period, Figure 3CAs shown, the invalid period can also be set from the time when the moving speed exceeds the positive threshold to the time when it falls below the positive threshold, and then exceeds the negative threshold after falling below the negative threshold, or from the time when the moving speed falls below the negative threshold to the time when it exceeds the negative threshold, and then falls below the positive threshold after exceeding the positive threshold.

[0217] Furthermore, the measurement target does not need to be related to the movement of the target portion, and may be the brightness value of the target portion.

[0218] Figure 3D 1 is a diagram for explaining an example of the operation of processing the biological measurement data of the user 100 according to the present embodiment. Figure 3D In the example shown, the object portion is the head of the user 100. Figure 3D In the example shown in part (a) of FIG. 1 , the measuring unit 110 measures the brightness value of the target part based on the image including the internal scattered component I2, that is, the internal image. The period in which the brightness value of the target part exceeds the threshold is set as an invalid period. The output determination unit 111 stops the output of brain activity data in frame 2 and frame 3 included in the invalid period. If the brightness value of the target part in the image including the internal scattered component I2 is close to the saturation value, the biological measurement data cannot be correctly generated. Therefore, by determining the validity based on the brightness value of the target part, the biological measurement device 10 can output only the correct biological measurement data.

[0219] exist Figure 3D In the example shown, the period during which the brightness value of the target portion exceeds the threshold is set as the invalid period. On the other hand, the period during which the brightness value of the target portion is lower than the threshold may be set as the invalid period. This is because when the brightness value of the target portion of the image including the internal scattered component I2 is extremely low, the SN ratio of the image is insufficient. In this case, correct biological measurement data cannot be generated.

[0220] In addition, the measurement object may also be the speed of change of the brightness value of the object portion. In a frame where the absolute value of the speed of change of the brightness value exceeds a threshold, the output of brain activity may be stopped. This is because it can be considered that the brightness in the frame is extremely high or low.

[0221] In addition, the measuring unit 110 may measure the area of ​​a predetermined region in the target part for measuring the biological measurement data based on the image signal. The area of ​​the predetermined region in the target part is, for example, the area of ​​the forehead. In the calculation of the area of ​​the forehead based on the image signal, for example, a well-known image processing method such as feature point extraction using edge detection described above may be used. If the forehead contains a lot of hair, the output of the biological measurement data may be stopped when the area of ​​the forehead is lower than a threshold. This is because if the area of ​​the target part is small, the accuracy of the generated biological measurement data decreases.

[0222] Furthermore, the measurement object may be the biological measurement data itself.

[0223] Figure 3E FIG. 1 is a diagram schematically showing an example of a biological measuring device 10 according to the present embodiment. Figure 3E In the example shown, Figure 1A Unlike the example shown, the result of generating the biological measurement data is also sent to the measuring unit 110. The measuring unit 110 measures the rate of change of the biological measurement data. The biological measurement data is, for example, the oxygenated hemoglobin concentration or the deoxygenated hemoglobin concentration as brain activity data. The change of cerebral blood flow changes slowly from 1 second to several seconds. On the other hand, body movement and brightness changes of the object part affect the measured value of cerebral blood flow at a faster rate than the cerebral blood flow. Therefore, by determining the validity based on the rate of change of the biological measurement data, the biological measurement device 10 can output only the correct biological measurement data. The output determination unit 111 can also stop the output of the biological measurement data when the absolute value of the rate of change of the value calculated based on the biological measurement data exceeds the threshold value.

[0224] (Second embodiment)

[0225] Reference Figures 4 to 6 The structure and operation of the living body measurement device 20 according to the second embodiment will be described.

[0226] Figure 4 1 is a diagram schematically showing an example of a biological measurement device 20 according to the present embodiment. In the present embodiment, the measurement unit 110 measures the amount of body movement of the user 100 based on the image signal including the surface reflection component I1. In the present embodiment, the amount of body movement of the user 100 is the movement distance of the object part. The output determination unit 401 calculates the reliability of the biological measurement data based on the measurement result, and outputs the reliability data indicating the reliability together with the biological measurement data. Thus, in the second embodiment, unlike the first embodiment, the output determination unit 401 outputs not only the biological measurement data but also the reliability data.

[0227] Figure 5A This is a flowchart showing an example of a process of measuring biological measurement data of the user 100 according to the present embodiment.

[0228] The implementation of the initial setting in step S501, the acquisition of the surface image in step S502, the acquisition of the internal image in step S503, and the calculation of the body movement amount in step S504 are respectively the same as the operations in steps S201 to S204 of the first embodiment.

[0229] In step S505 , the output determination unit 401 calculates reliability based on the amount of body movement calculated from the difference value with respect to the initial state.

[0230] Figure 5B is a diagram showing an example of the relationship between the difference value and the reliability. When the difference value is 0 mm, the measurement result can be made most reliable. Figure 5B As shown in , when the difference value is 0 mm, the reliability can also be set to 100%. Figure 5B As shown, the reliability may be set to a lower value as the difference value is larger. In this way, the output determination unit 401 may calculate a lower reliability as the value calculated based on the image signal output from the image sensor 102 deviates from a preset value.

[0231] The biological measurement device 20 of this embodiment may output biological measurement data regardless of the amount of body movement. In step S506 , the biological measurement data generation unit 109 generates biological measurement data based on the image signal and sends the biological measurement data to the output determination unit 401 .

[0232] In step S507, the output determination unit 401 outputs the reliability data together with the biological measurement data. This output may be displayed on a display unit (not shown) of the biological measurement device 20, or may be used for controlling a higher-level system (not shown).

[0233] The operation in step S508 is the same as the operation in step S208 in the first embodiment.

[0234] Figure 6 1 is a diagram for explaining an example of the operation of processing the biological measurement data of the user 100 according to the present embodiment. Figure 6 In the example shown, the target portion is the head of the user 100 .

[0235] exist Figure 6 In the examples shown in parts (a) to (d) of Figure 3A Similarly, the horizontal axis direction represents the passage of frames or time as shown in the examples shown in parts (a) to (d). Part (a) schematically shows a surface image as an image containing the surface reflection component I1. Part (b) shows the change in the amount of body movement. Part (c) shows the change in the reliability value. Part (d) shows the change in the oxygenated hemoglobin concentration in the forehead area as brain activity data as an example of biological measurement data.

[0236] In the second embodiment, unlike the first embodiment, the reliability is calculated based on the amount of body movement in each frame. Figure 6 In the example shown in part (c), Figure 6 The output determination unit 401 calculates the reliability as a lower value as the amount of body movement shown in part (b) is larger. The amount of body movement is a difference value of the position of the object part based on the initial state.

[0237] In frame 3 and frame 4, the reliability becomes a low value due to the movement of the object part. On the other hand, the output determination unit 401 may output the biological measurement data in all frames. The output determination unit 401 also outputs the reliability data together with the biological measurement data.

[0238] Through the above-described embodiment, it is possible to measure the biological measurement data in a non-contact manner without urging the user 100 to measure again. As a result, a biological measurement device capable of daily measurement can be realized. In addition, calculations accompanying the restoration of cerebral blood flow distribution are not required. Therefore, the calculation cost can be reduced, and biological measurement based on an inexpensive method can be performed. Furthermore, a special effect can be obtained in that the upper system can use the reliability data to comprehensively determine the validity of the biological measurement data.

[0239] In the present embodiment, the reliability is calculated to be a lower value as the measurement result in the measuring unit 110 is further away from the threshold value. However, the calculation method of the reliability is not limited to this.

[0240] The reliability may not be the value of the measurement result itself, but may be calculated based on the temporal trend of the measurement result. For example, the longer the invalid period during which the measurement result exceeds the threshold continues, the lower the reliability is calculated. In other words, the output decision unit 111 may calculate the reliability lower the longer the period during which the value calculated based on the image signal output from the image sensor 102 exceeds the preset value.

[0241] Alternatively, the higher the proportion of the invalid period in a certain period, the lower the reliability may be calculated. In other words, the longer the period during which the value calculated based on the image signal output from the image sensor 102 exceeds a preset value in a certain period, the lower the reliability may be calculated. When the validity and invalidity of the measured value are switched in a short time, it is effective to use such a reliability calculation method.

[0242] (Third embodiment)

[0243] Reference FIG. 7A to FIG. 9 The structure and operation of the living body measurement device 30 according to the third embodiment will be described.

[0244] Fig. 7A701 is a diagram schematically showing an example of a biological measurement device 30 according to the present embodiment. The biological measurement device 30 according to the present embodiment includes an imaging unit 701, a signal processing circuit 702, and a measuring unit 703. In the present embodiment, the measuring unit 703 includes an acceleration sensor provided near the user 100 as a surrounding environment, and measures acceleration. The output determination unit 111 determines whether to output biological measurement data based on the measurement result.

[0245] Figure 7B 1 is a diagram schematically showing an example of the arrangement of the various parts when the biological measuring device 30 is installed in the interior of a car. The imaging unit 701, the signal processing circuit 702, and the measuring unit 703 may be separately arranged. The measuring unit 703 is arranged near the user 100. The "arranged near" mentioned here means arranged at a distance at which the acceleration measured by the measuring unit 703 is at least correlated with the body movement of the user 100 himself.

[0246] Figure 7C This is a diagram schematically showing an example of the arrangement of various parts when a biological measuring device is additionally provided in a game machine or an attraction device. Figure 7B Similarly, the camera unit 701, the signal processing circuit 702, and the measuring unit 703 may be separately configured. The user 100 operates the controller 705 based on, for example, visual information from the display 704. The measuring unit 703 is attached to, for example, headphones or a head-mounted display worn on the head of the user 100. The measuring unit 703 detects the acceleration of the object part accompanying the movement of the user. In addition, the measuring unit 703 may detect, for example, a change in the illumination of the object part accompanying a change in the display information of the display 704.

[0247] Figure 8 This is a flowchart showing an example of a process of measuring biological measurement data of the user 100 according to the present embodiment.

[0248] In step S801, the biological measurement device 30 performs initial settings before measuring biological measurement data. In step S801, similarly to the first embodiment, the control circuit 105 optimally adjusts the timing of emitting pulse light from the light source 101 and the shutter timing of the image sensor 102 according to the distance between the biological measurement device 30 and the user 100.

[0249] In step S802, the measuring unit 703 measures acceleration. When the biological measuring device 30 is installed inside a car, the measurement of acceleration corresponds to the measurement of vibration of the car.

[0250] The frequency of measuring acceleration may be higher than the frequency of generating biological measurement data. Thus, the tracking performance of the measurement value in the case of a sudden vibration is improved. As a result, the accuracy of the determination in the output determination unit 111 can be improved. In this embodiment, as an example, it is assumed that the frequency of measuring acceleration is 10 times the frequency of calculating brain activity data.

[0251] In step S803, the acceleration values ​​are sequentially stored in a memory (not shown) in the measuring unit 703. In step S804, the loop of steps S802 and S803 is repeated, for example, 10 times through a process of determining the number of acceleration measurements. There is no limit to the number of measurements.

[0252] In step S805, the measuring unit 703 refers to the contents of the memory and sends the maximum value from the stored 10 acceleration values ​​to the output determination unit 111. Alternatively, the average value of 10 times may be used instead of the maximum value. In this embodiment, a value at least related to the body movement of the user 100 is calculated based on the 10 acceleration values.

[0253] In step S806, the output determination unit 111 determines whether the maximum value of the acceleration value received from the measurement unit 703 is less than a threshold value. The threshold value is, for example, acceleration in one of the directions of front-back, left-right, and vertical, and is within a range of 0.1G to 1G (G = 9.8m / s 2 ) in the range of .

[0254] In step S807, the control circuit 105 causes the image signal acquisition unit 108 to send the image signal including the internal scattered component I2 to the biological measurement data generation unit 109. The operations in steps S808, S809 and S810 are respectively the same as the operations in steps S206, S207 and S208 of the first embodiment.

[0255] Fig. 9 1 is a diagram for explaining an example of the operation of processing the biological measurement data of the user 100 according to the present embodiment. Fig. 9 In the example shown, the target portion is the head of the user 100 .

[0256] exist Fig. 9 In the examples shown in parts (a) to (d), the horizontal axis direction represents the passage of frames or time. Part (a) schematically shows the state of the housing of a car or an attraction device. Part (b) shows the change in acceleration. Part (c) shows the determination result. Part (d) shows the change in the concentration of oxygenated hemoglobin in the forehead area as brain activity data as an example of biological measurement data.

[0257] In the third embodiment, unlike the first embodiment, the measuring unit 703 measures acceleration. The acceleration is measured because there is a correlation between the acceleration near the user 100 and the amount of body movement of the user 100 himself.

[0258] exist Fig. 9 In the example shown, in frames 3 and 4, the housing of the car or the attraction device vibrates. Therefore, the acceleration measured by the measuring unit 703 exceeds the threshold value. There is a correlation between the amount of body movement and the irregularity of the generated biological measurement data. Therefore, the output determination unit 111 determines that the biological measurement data is invalid during the invalid period when the acceleration exceeds the threshold value, and stops the output of the biological measurement data. As a result, Fig. 9 As shown in part (d) of FIG. 1 , brain activity data as biological measurement data is output in frame 1 , frame 2 , and frame 5 .

[0259] Through the above-described embodiment, it is possible to measure biological measurement data in a non-contact manner without urging the user 100 to measure again. As a result, a biological measurement device capable of daily measurement can be realized. In addition, calculations accompanying the restoration of cerebral blood flow distribution are not required. Therefore, the calculation cost can be reduced, and biological measurement can be performed using an inexpensive method. In addition, since it is not necessary to measure the amount of body movement based on image signals, a special effect of further reducing the calculation cost can be obtained.

[0260] In addition, the measurement object of the measuring unit 703 is not limited to acceleration as long as it is at least related to body movement. For example, when the biological measuring device 30 is installed in a car, the measuring unit 703 may include a speed sensor to measure the speed. In addition, the measuring unit 703 may include a steering angle sensor to measure the steering angle of the car.

[0261] Furthermore, the measuring unit 703 may include a gear position sensor to measure the gear position of the vehicle. In this case, the output determination unit 111 stops outputting the biological measurement data during the period in which the gear position measurement result indicates a preset state. The output determination unit 111 may also stop outputting the biological measurement data. Figure 3B As shown, in addition to this period, the output of the biological measurement data is also stopped in the period before the start and / or the period after the end of this period. The preset state is, for example, a state in which the gear is in the reverse position. In this state, in order to move the car backward, the user 100 drives while looking behind. As a result, the object part of the user 100 is displaced from the reference position, and the body movement of the user 100 occurs.

[0262] Furthermore, the measurement object of the measuring unit 703 may not be related to the body movement of the user 100. The measuring unit 703 may include an illuminance sensor provided near the user 100 as the surrounding environment to measure the illuminance. In this case, the output determination unit 111 sets the period during which the illuminance value exceeds the threshold value as an invalid period, and stops the output of the biological measurement data. If the brightness value of the object part in the image including the internal scattering component I2 is close to the saturation value, the biological measurement data cannot be correctly generated. Therefore, when there is at least a correlation between the illuminance value and the brightness value of the object part, the validity is determined based on the brightness value of the object part, so that the biological measurement device 30 can output only the correct biological measurement data.

[0263] (Example)

[0264] Hereinafter, examples implemented to confirm the principle of the present invention will be described.

[0265] The first embodiment Figure 1A The biological measurement device 10 shown is installed so as to face the head of the user 100 as the subject. In order to prevent accidental body movement, the face of the user 100 is placed on a jaw support. After the user 100 is relaxed, the user's head is photographed for 30 seconds, and brain activity data is measured as biological measurement data.

[0266] The wavelength of the light source is set to 750nm and 850nm. The image signal containing the surface reflection component I1 is obtained at 30fps. The image signal containing the internal scattering component I2 is obtained at 5fps. The resolution of the image signal is set to 320 pixels × 240 pixels. A range of 50 pixels × 50 pixels including the central part of the forehead is used as the target area, and spatial averaging is performed, and temporal averaging is performed in real time. The oxygenated hemoglobin concentration is calculated based on the image signal containing the internal scattering component I2 of two wavelengths, and the change of the calculated oxygenated hemoglobin concentration is monitored.

[0267] At the same time, based on the image signal including the surface reflection component I1, the change in the moving distance of the face from the measuring unit 110 is monitored. The Kanade-Lucas-Tomasi (KLT) algorithm is used for extraction and tracking of facial feature points.

[0268] During the measurement, the user 100 intentionally moved his head to the right only once, and immediately returned to the original position.

[0269] Fig. 10A This is a graph showing the relationship between time and the moving distance of the head. Fig. 10B is a graph showing the relationship between time and oxygenated hemoglobin concentration. Fig. 10AIn the example shown, X represents the change in the left-right direction, Y represents the change in the up-down direction, and Z represents the change in the depth direction. The moving distance is the displacement amount relative to the initial value in each of the X, Y, and Z directions. Fig. 10B In the example shown, the oxygenated hemoglobin concentration is set as a change amount relative to an initial value. The change amount is expressed in an arbitrary unit.

[0270] like Fig. 10A and Fig. 10B As shown, the oxygenated hemoglobin concentration fluctuates greatly in synchronization with the time of head movement from about 11 seconds to 12 seconds. Therefore, it can be seen that there is a correlation between body movement represented by head movement and irregular fluctuations in brain activity data.

[0271] In addition, according to Fig. 10A and Fig. 10B The results shown show that the change in the oxyhemoglobin concentration occurs with a slight delay from the start of the movement of the head. It can also be seen that the change in the oxyhemoglobin concentration is stable with a delay from the end of the movement of the head. The following two reasons can be considered. (1) The brain is in a state of floating in the myelin fluid inside the skull. Therefore, the movement of the brain is delayed in time compared to the movement of the skull. (2) In the measurement of the oxyhemoglobin concentration, time averaging is performed in real time. Therefore, the measured value is easily affected by the change immediately before.

[0272] according to Fig. 10A and Fig. 10B The results shown can also be judged as valid and the output of brain activity data can be stopped when the following two conditions are met. (1) The threshold for the moving distance in any of the X, Y, and Z directions is set to 5 mm. (2) The period from 0.5 seconds before exceeding the threshold to 4.5 seconds after falling below the threshold is set as the invalid period. These two conditions can eliminate irregular changes in the measured value of the oxygenated hemoglobin concentration.

[0273] In the above example, cerebral blood flow data is used as an example of biological measurement data, but the present invention is not limited thereto. The biological measurement data may be, for example, data representing at least one selected from the group consisting of scalp blood flow, pulse, sweating, respiration, and body temperature. This data can be obtained from the surface reflection component I1.

[0274] The present invention also includes procedures and methods for the actions performed by the signal processing circuit 122 and the signal processing circuit 702 .

[0275] Industrial Applicability

[0276] The biological measuring device of the present invention can be used as a camera or measuring equipment that obtains internal information of a user in a non-contact manner. The biological measuring device can also be applied to biological or medical sensing, sensing of a driver of a car, sensing of a user in a game machine or attraction, sensing of a learner in an educational institution, or sensing of a worker in a workplace.

[0277] Description of symbols

[0278] 10, 20, 30 Biological measuring device

[0279] 100 users

[0280] 101 Light Source

[0281] 102 Image Sensor

[0282] 103 Photoelectric conversion unit

[0283] 104 Charge storage unit

[0284] 105 Control Circuit

[0285] 106 Light source control unit

[0286] 107 Sensor control unit

[0287] 108 Image signal acquisition unit

[0288] 109 Biological measurement data generation unit

[0289] 110, 703 Measurement Department

[0290] 111, 401 Output decision unit

[0291] 121, 701 Camera Department

[0292] 122, 702 signal processing circuit

[0293] 201 pixels

[0294] 202 Drain

[0295] 203 Photodiode

[0296] 204 1st floating diffusion layer

[0297] 205 Second floating diffusion layer

[0298] 206 3rd floating diffusion layer

[0299] 207 4th floating diffusion layer

[0300] 302 row selection circuit

[0301] 303 Column selection circuit

[0302] 304 vertical signal line

[0303] 305 Source Follower Power Supply

[0304] 306 Source Follower Load

[0305] 307 Analog-digital conversion circuit

[0306] 308 row selection transistor

[0307] 309 Source Follower Transistor

[0308] 310 Reset transistor

[0309] 704 Display

[0310] 705 Controller

Claims

1. A biological measuring device, wherein: have: a light source for irradiating light toward the user's head; Image sensor; A control circuit, controlling the light source and the image sensor; and Signal processing circuit; The above control circuit performs the following control: causing the light source to emit the light; causing the image sensor to detect at least a portion of the reflected light returned from the head due to the irradiation of the light and output an image signal; The above signal processing circuit performs the following processing: generating brain activity data representing a state of the user's brain based on the image signal; calculating a first value based on at least one selected from the group consisting of the image signal and a sensor signal, the sensor signal being a signal output from a sensor that detects a change in the user's surrounding environment that affects the brain activity data, The output of the brain activity data is stopped during a first period in which the first value meets a preset condition and during at least one period selected from the group consisting of a second period before the start of the first period and a third period after the end of the first period. The at least one period is a period during which the first value does not satisfy a predetermined condition. The image sensor continues to output the image signal during a period in which the signal processing circuit stops outputting the brain activity data.

2. The biological measuring device according to claim 1, wherein: The above light is pulsed light; The control circuit causes the image sensor to output the first signal as the image signal, wherein the first signal is a signal obtained by detecting components contained in the reflected pulse light during the period from when the intensity of the reflected pulse light returned from the head due to the irradiation of the pulse light begins to decrease to when the decrease ends.

3. The biological measuring device according to claim 1 or 2, wherein: The control circuit causes the light source to repeatedly emit the light during a predetermined period, and causes the image sensor to repeatedly output the image signal.

4. The biological measuring device according to claim 3, wherein: The signal processing circuit outputs a signal indicating that the brain activity data is invalid during the first period.

5. The biological measuring device according to claim 3, wherein: The signal processing circuit outputs, during the first period, data identical to the brain activity data generated before the first period.

6. The biological measuring device according to claim 3, wherein: The signal processing circuit outputs data obtained by interpolating the brain activity data generated before the first period and the brain activity data generated after the first period as the brain activity data in the first period.

7. The biological measuring device according to claim 1, wherein: The third period is longer than the second period.

8. The biological measuring device according to claim 1, wherein: The frequency of calculating the first value is greater than or equal to the frequency of generating the brain activity data.

9. The biological measuring device according to claim 1 or 2, wherein: The above light is pulsed light; The control circuit causes the image sensor to output a second signal as the image signal, the second signal being a signal obtained by detecting a component included in the reflected pulse light before the intensity of the reflected pulse light returned from the head due to the irradiation of the pulse light begins to decrease; The signal processing circuit stops outputting the brain activity data based on the second signal.

10. The biological measuring device according to claim 9, wherein: The above signal processing circuit performs the following processing: further calculating, based on the second signal, a displacement of the head relative to a reference position or a moving speed of the head; When the absolute value of the displacement or the absolute value of the moving speed exceeds a threshold value, the output of the brain activity data is stopped.

11. The biological measuring device according to claim 9, wherein: The above signal processing circuit performs the following processing: further calculating the brightness value of the head or the speed of change of the brightness value of the head based on the second signal; When the absolute value of the brightness value or the absolute value of the change speed of the brightness value exceeds a threshold value, the output of the brain activity data is stopped.

12. The biological measuring device according to claim 9, wherein: The above signal processing circuit performs the following processing: further calculating the area of ​​the predetermined region in the head based on the second signal; When the area is smaller than the threshold, the output of the brain activity data is stopped.

13. The biological measuring device according to claim 1 or 2, wherein: The above signal processing circuit performs the following processing: The second value is also calculated using the above brain activity data; When the absolute value of the change speed of the second value exceeds a threshold value, the output of the brain activity data is stopped.

14. The biological measuring device according to claim 1, wherein: The sensor is an acceleration sensor arranged in the surrounding environment.

15. The biological measuring device according to claim 1, wherein: The sensor is an illumination sensor arranged in the surrounding environment.

16. The biological measuring device according to claim 1, wherein: The sensor is at least one selected from the group consisting of a steering angle sensor, a gear position sensor, and a speed sensor disposed in the vehicle driven by the user.

17. The biological measuring device according to claim 1 or 2, wherein: The control circuit causes the light source to repeatedly emit the light during a predetermined period, and causes the image sensor to repeatedly output the image signal. The above signal processing circuit performs the following processing: During the period from the first period in which the first value meets a preset condition to the end of the first period after a delay time has elapsed, the output of the brain activity data is stopped in the prescribed period.

18. A biological measurement method, wherein: This includes the following processing: causing the light source to emit light that illuminates the user's head; causing an image sensor to detect at least a portion of the reflected light returned from the head due to the irradiation of the light and output an image signal; generating brain activity data representing a state of the user's brain based on the image signal; calculating a first value based on at least one selected from the group consisting of the image signal and a sensor signal, the sensor signal being a signal output from a sensor that detects a change in the user's surrounding environment that affects the brain activity data, The output of the brain activity data is stopped during a first period in which the first value meets a preset condition and during at least one period selected from the group consisting of a second period before the start of the first period and a third period after the end of the first period. The at least one period is a period during which the first value does not satisfy a predetermined condition. While the output of the brain activity data is stopped, the image sensor continues to output the image signal.

Images