Information processing method and information processing system in a computer

Abstract

To provide a novel method to estimate a state of a subject.SOLUTION: A biological measurement apparatus according to one aspect of the present disclosure is provided with: a first detector that detects and outputs a brain activity signal indicating the state of brain activity of a subject; and a signal processing circuit. The signal processing circuit acquires the brain activity signal, acquires a biological signal of the subject that is different from the brain activity signal, determines which of a wakefulness state, a sleep state, and an impaired consciousness state the subject is in on the basis of the biological signal and the brain activity signal, and generates and outputs a signal indicating the state of the subject.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method and an information processing system in a computer. [Background technology]

[0002] Conventionally, there are known methods for determining the state of a subject from the subject's movements or an image obtained by photographing the subject. For example, Patent Document 1 discloses a method for determining whether a driver is dozing off based on the driver's steering operation. Furthermore, Patent Document 2 discloses a method for determining whether a driver is dozing off by detecting the degree of eye opening from an image obtained by photographing the driver with a camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-310738 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-143889 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques have required the subject's movements to detect the subject's condition, and the conditions that can be determined are limited.

[0005] The present disclosure provides a novel technique that makes it possible to distinguish between various states of a subject without requiring the subject to move. [Means for solving the problem]

[0006] An information processing method in a computer according to one aspect of the present disclosure includes acquiring a brain activity signal indicating a state of brain activity of a subject, acquiring a biosignal of the subject that is different from the brain activity signal, and determining whether the subject is in a state of impaired consciousness in which the subject is unconscious based on the brain activity signal and the biosignal.

[0007] A general or specific aspect of the present disclosure may be realized as an apparatus, a system, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of an apparatus, a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0008] According to the technology of the present disclosure, it is possible to determine whether or not a subject is in a state of impaired consciousness by using brain activity information and other biological information. [Brief explanation of the drawings]

[0009] [Figure 1A] FIG. 1A is a schematic diagram illustrating a bioinstrumentation device according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1B is a diagram showing an example of the change over time in the intensity of light reaching the image sensor. [Figure 1C] FIG. 1C is a diagram in which the horizontal axis represents the width of the input pulse light and the vertical axis represents the amount of light detected by the sensor. [Figure 1D] FIG. 1D is a diagram showing an example of a schematic configuration of one pixel of an image sensor. [Figure 1E] FIG. 1E is a diagram illustrating an example of the configuration of an image sensor. [Figure 1F] FIG. 1F is a diagram showing an example of the movement within one frame. [Figure 1G] FIG. 1G is a flowchart showing an outline of the operation of the control circuit. [Figure 2] FIG. 2 is a diagram for explaining a method for detecting the internal scattering component of pulsed light. [Figure 3A]FIG. 3A is a diagram showing an example of a timing chart when detecting a surface reflection component. [Figure 3B] FIG. 3B is a diagram showing an example of a timing chart when detecting an internal scattering component. [Figure 4A] FIG. 4A is a diagram schematically showing the relationship between a person's pulse rate, brain activity, and physical condition. [Figure 4B] FIG. 4B is a diagram schematically showing the relationship between the amount of sweating, the amount of brain activity, and the physical condition of a person. [Figure 4C] FIG. 4C is a diagram showing a schematic diagram of the relationship between a person's body temperature, brain activity, and physical condition. [Figure 4D] FIG. 4D is a diagram schematically showing the relationship between a person's respiratory rate, brain activity, and physical condition. [Figure 5] FIG. 5 is a diagram showing how a biometric device detects the vital signs of a passenger in an autonomously driven taxi. [Figure 6] FIG. 6 is a diagram showing another example of the configuration of the bioinstrumentation device. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the bioinstrumentation device. [Figure 8] FIG. 8 is a diagram showing another application example of the bioinstrumentation device. [Figure 9] FIG. 9 is a diagram showing yet another application example of the bioinstrumentation device. [Figure 10] FIG. 10 is a diagram showing yet another application example of the bioinstrumentation device. [Figure 11A] FIG. 11A is a diagram showing a bioinstrumentation device integrated with a head-mounted display, which is yet another application example of the bioinstrumentation device. [Figure 11B] FIG. 11B is a diagram showing a state in which a subject is using the bioinstrumentation device shown in FIG. 11A. [Figure 12] FIG. 12 is a diagram showing yet another application example of the bioinstrumentation device. [Figure 13] FIG. 13 is a diagram showing still another application example of the bioinstrumentation device. [Figure 14]FIG. 14 is a diagram showing yet another application example of the bioinstrumentation device. [Figure 15] FIG. 15 is a diagram showing yet another example of the configuration of the bioinstrumentation device. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure includes a biometric measurement device, a biometric measurement method, and a determination device described in the following items.

[0011] [Item 1] The bioinstrumentation device according to item 1 of the present disclosure includes a first detector that detects and outputs a brain activity signal indicating a state of brain activity of a subject; a signal processing circuit; Equipped with.

[0012] The signal processing circuit acquiring the brain activity signal; acquiring a biological signal of the subject that is different from the brain activity signal; determining whether the subject is in an awake state, a sleeping state, or a state of impaired consciousness based on the biological signal and the brain activity signal; A signal indicative of the subject's condition is generated and output.

[0013] According to the above aspect, by utilizing both the biological signal and the brain activity signal, the state of the subject can be determined more accurately than ever before.

[0014] [Item 2] In the bioinstrumentation device according to item 1, the first detector may further detect and output the biosignal.

[0015] [Item 3] The bioinstrumentation device according to item 1 may further include a second detector that detects and outputs the biosignal.

[0016] [Item 4] The bioinstrumentation device according to any one of items 1 to 3 includes: a light source that emits pulsed light to be irradiated onto the head of the subject; a control circuit; Furthermore, the first detector detects at least a portion of the reflected pulsed light returning from the head; The control circuit causing the light source to emit the pulsed light; The first detector may detect and output, as the brain activity signal, an internally scattered component of the reflected pulsed light that is scattered within the brain of the subject.

[0017] [Item 5] In the biomeasurement device according to any one of items 1 to 4, the biosignal may include at least one piece of information selected from the group consisting of pulse rate, sweat rate, respiratory rate, and body temperature.

[0018] According to the above aspect, it is possible to determine the condition of the subject based on at least one piece of biological information selected from the group consisting of pulse rate, sweat rate, respiratory rate, and body temperature, and to determine the condition of the subject based on brain activity signals.

[0019] [Item 6] In the bioinstrumentation device according to any one of items 1 to 5, the brain activity signal may include information on changes in cerebral blood flow of the subject.

[0020] According to the above aspect, the brain activity state of the subject can be estimated based on information relating to changes such as increases and decreases in the cerebral blood flow of the subject.

[0021] [Item 7] In the bioinstrumentation device according to any one of items 1 to 6, the signal processing circuit After acquiring the biological signal, the brain activity signal may be acquired.

[0022] According to the above aspect, the state of the subject is first determined based on a biological signal other than a brain activity signal, and then the brain activity state can be determined based on the brain activity signal.

[0023] [Item 8] In the biomeasurement device described in any one of items 1 to 7, the signal processing circuit may determine whether the subject's pulse is bradycardia or tachycardia based on the biosignal and the brain activity signal and output the result.

[0024] According to the above aspect, it is possible to determine whether the pulse of the subject is bradycardia or tachycardia.

[0025] [Item 9] In the bioinstrumentation device according to any one of items 1 to 8, the biosignal may include information about blood flow at the surface of the subject's skin.

[0026] According to the above aspect, the condition of the subject can be determined based on information about the blood flow on the surface of the subject's skin.

[0027] [Item 10] In the bioinstrumentation device according to item 4, the control circuit controls the first detector to: a surface reflection component reflected by the surface of the skin of the subject is further detected from the reflected pulsed light; A signal indicating the variation in the surface reflection component may be output as the biological signal.

[0028] According to the above aspect, it is possible to obtain information on both blood flow on the surface of the subject's skin and information on cerebral blood flow using a single detector.

[0029] [Item 11] In the bioinstrumentation device according to item 10, the control circuit controls the first detector to: detecting the internal scattering component by detecting a portion of the reflected pulsed light after the intensity of the reflected pulsed light starts to decrease; The surface reflection component may be detected by detecting at least a portion of the reflected pulsed light before the intensity of the reflected pulsed light starts to decrease.

[0030] According to the above aspect, the internal scattering component and the surface reflection component of the pulsed light returning from the subject's head can be detected with a high S / N ratio.

[0031] [Item 12] In the bioinstrumentation device according to item 10 or 11, the first detector is an image sensor including a plurality of pixels arranged two-dimensionally, Each of the plurality of pixels is a photoelectric conversion element that converts received light into a signal charge; a first charge storage unit that stores the signal charge corresponding to the surface reflection component; a second charge accumulation unit that accumulates the signal charge corresponding to the internal scattering component; may also include:

[0032] According to the above aspect, a two-dimensional image of the subject can be acquired.

[0033] [Item 13] In the biomeasurement device described in item 4, the time from the start of irradiation of the pulsed light to the acquisition of the brain activity signal may be longer than the time from the start of irradiation of the pulsed light to the acquisition of the biosignal.

[0034] According to the above aspect, first, a diagnosis based on a biological signal other than a brain activity signal can be performed, and then a diagnosis based on the brain activity signal can be performed.

[0035] [Item 14] In the bioinstrumentation device according to any one of items 1 to 13, the signal processing circuit making a first determination regarding a state of the subject based on the biological signal, and outputting a first signal indicating a result of the first determination; After outputting the first signal, a second determination regarding the state of the subject may be made based on the brain activity signal, and a second signal indicating the result of the second determination may be output.

[0036] According to the above aspect, first, a diagnosis based on a biological signal other than a brain activity signal can be performed, and then a diagnosis based on the brain activity signal can be performed.

[0037] [Item 15] The biometric device according to item 14 further includes a communication circuit for communicating with an external device outside the biometric device, When the signal processing circuit determines that the subject is not in a healthy state based on at least one selected from the group consisting of the biological signal and the brain activity signal, the communication circuit may notify the external device that the subject is not in a healthy state.

[0038] According to the above aspect, if it is determined that the subject is not in a healthy state based on the biological signals and / or the brain activity signals, a notification can be sent to an external device at, for example, a medical institution, so that even if the subject's condition is critical, action can be taken quickly, such as arranging for an ambulance.

[0039] [Item 16] A biometric measurement method according to item 17 of the present disclosure includes the steps of acquiring a brain activity signal from a subject; acquiring a biological signal of the subject that is different from the brain activity signal; and The method includes a step of determining whether the state of the subject is awake, asleep, or in a state of impaired consciousness based on the biological signal and the brain activity signal.

[0040] According to the above aspect, by utilizing both the biological signal and the brain activity signal, the state of the subject can be determined more accurately than ever before.

[0041] [Item 17] In the bioinstrumentation method according to item 16, the step of acquiring the brain activity signal comprises: a step of irradiating pulsed light onto the head of the subject using a light source in a bioinstrumentation device including a light source, a photodetector, a signal processing circuit, and a control circuit; and detecting and outputting, as the brain activity signal, an internally scattered component of the reflected pulsed light returned from the head, the internally scattered component being scattered within the brain of the subject, using the photodetector; The determining step may be performed using the signal processing circuit.

[0042] [Item 18] In the biomeasurement method described in Item 17, based on the results of the judgment of the subject's condition, the control circuit may output a notification to the subject, control equipment surrounding the subject that is different from the biomeasurement device, or communicate with an external device outside the biomeasurement device.

[0043] [Item 19] In the biometric measurement method according to any one of items 16 to 18, the biosignal may include at least one piece of information selected from the group consisting of pulse rate, sweat rate, respiratory rate, and body temperature.

[0044] According to the above aspect, it is possible to determine the condition of the subject based on at least one piece of biological information selected from the group consisting of pulse rate, sweat rate, respiratory rate, and body temperature, and to determine the condition of the subject based on brain activity signals.

[0045] [Item 20] In the bioinstrumentation method according to any one of items 16 to 19, the brain activity signal may include information on changes in cerebral blood flow of the subject.

[0046] According to the above aspect, the brain activity state of the subject can be estimated based on information relating to changes such as increases and decreases in the cerebral blood flow of the subject.

[0047] [Item 21] In the biomeasurement method according to any one of items 16 to 20, the step of acquiring the brain activity signal may be performed after the step of acquiring the biosignal.

[0048] According to the above aspect, the state of the subject is first determined based on a biological signal other than a brain activity signal, and then the brain activity state can be determined based on the brain activity signal.

[0049] [Item 22] In the bioinstrumentation method according to any one of Items 16 to 21, the determining step may include determining whether the pulse of the subject is bradycardia or tachycardia.

[0050] According to the above aspect, it is possible to determine whether the pulse of the subject is bradycardia or tachycardia.

[0051] [Item 23] In the bioinstrumentation method according to any one of items 16 to 22, the biosignal may include information about blood flow at the surface of the subject's skin.

[0052] According to the above aspect, the condition of the subject can be determined based on information about the blood flow on the surface of the subject's skin.

[0053] [Item 24] In the biomeasurement method according to item 17 or 18, the biomeasurement device further includes a memory that stores a first reference value that is a reference value of the biosignal and a second reference value that is a reference value of the brain activity signal, The determining step may be performed by comparing the biological signal with the first reference value and comparing the brain activity signal with the second reference value.

[0054] According to the above aspect, the state of the subject can be determined based on the biological signal and the brain activity signal by comparing them with two different reference values.

[0055] [Item 25] In the biomeasurement method according to item 17 or 18, the biomeasurement device further includes a memory that stores a data table that represents a relationship between the brain activity signal, the biosignal, and a state of the subject, The determining step may be performed by referring to the data table.

[0056] According to the above aspect, the physical state of the subject can be determined more accurately by making an evaluation with reference to a data table showing the relationship between brain activity signals, biological signals, and physical states.

[0057] [Item 26] The biomeasurement method according to item 17 or 18 further includes a step of acquiring time series data of the biosignal and the brain activity signal, the biometric measurement device further includes a memory that stores reference time-series data corresponding to the time-series data; The determining step may include at least one selected from the group consisting of comparing the time series data with reference time series data, calculating statistics of the time series data, and performing machine learning of a correlation between the time series data and the condition of the subject.

[0058] According to the above aspect, the condition of the subject can be determined more accurately.

[0059] [Item 27] The biometric measurement method according to item 17 or 18 includes the steps of: acquiring two-dimensional image data corresponding to the biosignal and the brain activity signal; further comprising a step of extracting a feature amount included in the two-dimensional image data, the biometric device further includes a memory that stores a reference feature corresponding to the feature; The determining step may include at least one selected from the group consisting of comparing the feature with a reference feature, calculating a statistical value of the feature, and performing machine learning of a correlation between the feature and the condition of the subject.

[0060] [Item 28] The determination device according to item 28 of the present disclosure includes one or more memories, When in operation, Acquiring brain activity signals from the subject; acquiring a biological signal of the subject that is different from the brain activity signal; and Based on the biological signal and the brain activity signal, it is determined whether the subject is in an awake state, a sleeping state, or a state of impaired consciousness. A circuit is provided.

[0061] [Item 29] A bioinstrumentation device according to item 29 of the present disclosure includes: a light source that emits pulsed light to be irradiated onto the head of a subject; a photodetector that detects at least a portion of the reflected pulsed light returning from the head; a control circuit for controlling the light source and the photodetector; a signal processing circuit; Equipped with.

[0062] The control circuit causing the light source to emit the pulsed light; causing the photodetector to detect and output, as a brain activity signal, an internally scattered component of the reflected pulsed light that is scattered within the brain of the subject; The photodetector detects and outputs a surface reflection component of the reflected pulsed light that is reflected on the surface of the subject's skin as a biological signal different from the brain activity signal.

[0063] The signal processing circuit generates and outputs a signal indicating the state of the subject based on the biological signal and the brain activity signal.

[0064] [Item 30] Item 29. The bioinstrumentation device according to Item 29, The control circuitry controls the photodetector to: detecting the internal scattering component by detecting a portion of the reflected pulsed light after the intensity of the reflected pulsed light starts to decrease; The surface reflection component may be detected by detecting at least a portion of the reflected pulsed light before the intensity of the reflected pulsed light starts to decrease.

[0065] [Item 31] Item 29. The bioinstrumentation device according to Item 29, The biological signal may include at least one piece of information selected from the group consisting of pulse rate, sweat rate, respiratory rate, and body temperature.

[0066] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, and component placement positions shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0067] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). An LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A field programmable gate array (FPGA), which is programmable after LSI fabrication, or a reconfigurable logic device, which can reconfigure connections within an LSI or set up circuit partitions within an LSI, may also be used for the same purpose.

[0068] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and required hardware devices, such as interfaces.

[0069] Hereinafter, embodiments will be described in detail with reference to the drawings. In the following description, the same or similar components are designated by the same reference numerals.

[0070] (Embodiment) [1. Biometric measurement device] The configuration of a bioinstrumentation device 100 according to an exemplary embodiment of the present disclosure will be described with reference to FIGS. 1A to 3B.

[0071] 1A is a schematic diagram showing a bioinstrumentation device 100 according to this embodiment. The bioinstrumentation device 100 includes a light source 10, an image sensor 20, a control circuit 30, and a signal processing circuit 40. The image sensor 20 includes a plurality of photoelectric conversion elements 22 and a plurality of charge accumulation units 24. The image sensor 20 is an example of the first detector, the second detector, and the photodetector of the present disclosure. Instead of the image sensor 20, it is also possible to use another type of photodetector including at least one photoelectric conversion element 22 and at least one charge accumulation unit 24.

[0072] The light source 10 emits pulsed light that is irradiated onto the head of the subject 500. The image sensor 20 detects at least a portion of the reflected pulsed light that is the pulsed light that has returned from the head of the subject 500. The control circuit 30 controls the light source 10 and the image sensor 20. The signal processing circuit 40 processes the signal output from the image sensor 20.

[0073] In this embodiment, the control circuit 30 includes a light source control unit 32 that controls the light source 10, and a sensor control unit 34 that controls the image sensor 20. The light source control unit 32 controls the intensity, pulse width, emission timing, and / or wavelength of the pulsed light emitted from the light source 10. The sensor control unit 34 controls the timing of signal accumulation in each pixel of the image sensor 20.

[0074] Each component will be described in more detail below.

[0075] [1-1.Light source 10] The light source 10 irradiates light onto the head (e.g., the forehead) of the subject 500. The light emitted from the light source 10 and reaching the subject 500 is divided into a surface reflection component I1 that is reflected on the surface of the subject 500 and an internal scattering component I2 that is scattered inside the subject 500. The internal scattering component I2 is a component that is reflected or scattered once or multiple times inside the living body. When light is irradiated onto the forehead of the subject 500, the internal scattering component I2 refers to a component that reaches a region about 8 mm to 16 mm inside the surface of the forehead, such as the brain, and then returns to the biometric measurement device 100. The surface reflection component I1 includes three components: a direct reflection component, a diffuse reflection component, and a scattered reflection component. The direct reflection component is a reflection component whose angle of incidence and angle of reflection are equal. The diffuse reflection component is a component that is diffusely reflected due to the unevenness of the surface. The scattered reflection component is a component that is scattered and reflected by internal tissue near the surface. When light is irradiated onto the forehead of the subject 500, the scattered reflection component is a component that is scattered and reflected within the epidermis. Hereinafter, in this disclosure, the surface reflection component I1 reflected on the surface of the subject 500 will be considered to include these three components. The surface reflection component I1 and the internal scattered component I2 change their traveling direction due to reflection or scattering, and some of them reach the image sensor 20.

[0076] First, a method for acquiring the internally scattered component I2 will be described. The light source 10 repeatedly generates pulsed light multiple times at predetermined time intervals or at predetermined timing in accordance with instructions from the control circuit 30. The pulsed light generated by the light source 10 may be, for example, a rectangular wave with a fall time close to zero. The fall time refers to the time from when the intensity of the pulsed light begins to decrease from its peak value until it reaches almost zero. Generally, light incident on the subject 500 propagates through the subject 500 via various paths and exits the surface of the subject 500 with a time difference. Therefore, the rear end of the internally scattered component I2 of the pulsed light has a spread. When the subject is the forehead, the spread of the rear end of the internally scattered component I2 is approximately 4 ns. Taking this into consideration, the fall time of the pulsed light can be set to, for example, half that, or less, 2 ns or less. The fall time may even be half that, or less, 1 ns or less. The rise time of the pulsed light generated by the light source 10 is arbitrary. This is because the detection of the internally scattered component I2 in this embodiment uses the falling portion of the pulsed light, not the rising portion. The rising portion of the pulsed light can be used to detect the surface reflection component I1. The light source 10 can be, for example, a laser such as an LD, whose falling portion of the pulsed light is nearly perpendicular to the time axis, that is, whose time response characteristics are rapid.

[0077] The wavelength of the light emitted by the light source 10 may be any wavelength within the wavelength range of, for example, 650 nm to 950 nm. This wavelength range includes the red to near-infrared wavelength range. In this specification, the term "light" is used to refer to not only visible light but also infrared light. This wavelength range is called the "biological window" and has the property of being relatively difficult to absorb by water and skin in a living body. When detecting a living body, using light within this wavelength range can improve detection sensitivity. When detecting changes in blood flow in the skin and brain of the subject 500, as in this embodiment, the light used is considered to be absorbed mainly by oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb). Oxygenated hemoglobin and deoxygenated hemoglobin have different wavelength dependences of light absorption. Generally, when blood flow changes, the concentrations of oxygenated hemoglobin and deoxygenated hemoglobin change, and therefore the degree of light absorption also changes. Therefore, when blood flow changes, the detected light amount also changes over time.

[0078] The light source 10 may emit light of two or more wavelengths included in the above wavelength range. Such light of multiple wavelengths may be emitted from multiple light sources, respectively.

[0079] In the bioinstrumentation device 100 of this embodiment, a light source 10 designed with consideration given to the effect on the retina can be used to measure the subject 500 in a non-contact manner. For example, a light source 10 that satisfies Class 1 of the laser safety standards established in various countries can be used. When Class 1 is satisfied, the subject 500 is irradiated with light of such low illuminance that the accessible emission limit (AEL) is below 1 mW. Note that the light source 10 itself does not have to satisfy Class 1. For example, a diffuser or ND filter may be placed in front of the light source 10 to diffuse or attenuate the light, thereby satisfying Class 1 of the laser safety standards.

[0080] Conventionally, streak cameras have been used to distinguish and detect information such as absorption coefficients or scattering coefficients at different depths within a living body. For example, Japanese Patent Laid-Open Publication No. 4-189349 discloses an example of such a streak camera. These streak cameras use ultrashort pulsed light with a pulse width of femtoseconds or picoseconds to achieve the desired spatial resolution.

[0081] In contrast, the bioinstrumentation device 100 of the present disclosure can distinguish between and detect the surface reflection component I1 and the internal scattering component I2. Therefore, the pulsed light emitted by the light source 10 does not need to be ultrashort pulsed light, and the pulse width can be selected arbitrarily.

[0082] When irradiating the forehead with light to measure cerebral blood flow, the amount of light from the internally scattered component I2 can be extremely small, ranging from several thousandths to several tens of thousandsths of the amount of light from the surface-reflected component I1. Furthermore, considering laser safety standards, the amount of light that can be irradiated becomes extremely small, making it very difficult to detect the internally scattered component I2. Even in this case, if the light source 10 generates pulsed light with a relatively large pulse width, it is possible to increase the accumulated amount of the internally scattered component I2, which involves a time delay. This increases the amount of detected light and improves the signal-to-noise ratio.

[0083] The light source 10 emits pulsed light with a pulse width of, for example, 3 ns or more. Generally, the temporal spread of light scattered within biological tissues, such as the brain, is approximately 4 ns. Figure 1B shows examples of the temporal change in the amount of light reaching the image sensor 20 when the width of the input pulsed light is 0 ns, 3 ns, and 10 ns. As shown in Figure 1B, as the width of the pulsed light from the light source 10 increases, the amount of internally scattered light component I2 that appears at the trailing edge of the pulsed light returning from the subject 500 increases. Figure 1C is a graph plotting the width of the input pulsed light on the horizontal axis and the amount of light detected by the sensor on the vertical axis. The image sensor 20 is equipped with an electronic shutter. The results in Figure 1C were obtained under the condition that the electronic shutter was opened 1 ns after the trailing edge of the pulsed light reflected from the surface of the subject 500 reached the image sensor 20. The reason for selecting this condition is that the ratio of the surface-reflected component I1 to the internally scattered component I2 is high immediately after the trailing edge of the pulsed light arrives. As shown in FIG. 1C, when the pulse width of the pulsed light emitted by the light source 10 is set to 3 ns or more, the amount of light detected by the sensor can be maximized.

[0084] The light source 10 may emit pulsed light with a pulse width of 5 ns or more, or even 10 ns or more. On the other hand, if the pulse width is too large, the amount of unused light increases and is wasted. Therefore, the light source 10 generates pulsed light with a pulse width of, for example, 50 ns or less. Alternatively, the light source 10 may emit pulsed light with a pulse width of 30 ns or less, or even 20 ns or less.

[0085] The illumination pattern of the light source 10 may be, for example, a pattern with a uniform intensity distribution within the illumination region. In this respect, this embodiment differs from conventional bioinstrumentation devices such as those disclosed in Japanese Patent Application Laid-Open No. 11-164826. In the device disclosed in Japanese Patent Application Laid-Open No. 11-164826, the detector and light source are separated by approximately 3 cm to spatially separate surface reflection components from internal scattering components. This necessitates discrete light illumination. In contrast, the bioinstrumentation device 100 of this embodiment can temporally separate and reduce the surface reflection component I1 from the internal scattering component I2. This allows the use of a light source 10 with an illumination pattern with a uniform intensity distribution. An illumination pattern with a uniform intensity distribution may also be formed by diffusing the light emitted by the light source 10 using a diffuser.

[0086] In this embodiment, unlike the conventional technology, the internal scattering component I2 can be detected even directly below the irradiation point of the pulsed light on the subject 500. By irradiating the subject 500 with light over a spatially wide range, the measurement resolution can also be improved.

[0087] [1-2. Image Sensor 20] The image sensor 20 receives light emitted from the light source 10 and reflected or scattered by the subject 500. The image sensor 20 has a plurality of photodetector cells arranged two-dimensionally, and simultaneously acquires two-dimensional information of the subject 500. In this specification, the photodetector cells are also referred to as "pixels." The image sensor 20 is, for example, any imaging element such as a CCD image sensor or a CMOS image sensor.

[0088] The image sensor 20 has an electronic shutter. The electronic shutter is a circuit that controls the timing of imaging. In this embodiment, the sensor control unit 34 in the control circuit 30 has the function of the electronic shutter. The electronic shutter controls the period of one signal accumulation, during which received light is converted into an effective electrical signal and accumulated, and the period during which signal accumulation is stopped. The signal accumulation period can also be referred to as the "exposure period" or "shooting period." In the following description, the width of the exposure period may be referred to as the "shutter width." The time from the end of one exposure period to the start of the next exposure period may be referred to as the "non-exposure period." Hereinafter, the exposed state may be referred to as "OPEN," and the stopped exposure state may be referred to as "CLOSE."

[0089] The image sensor 20 uses an electronic shutter to adjust the exposure and non-exposure periods to subnanoseconds, e.g., in the range of 30 ps to 1 ns. Conventional TOF (Time of Flight) cameras for distance measurement detect all light emitted from the light source 10 and reflected back from the subject to compensate for the effect of the subject's brightness. Therefore, conventional TOF cameras require a shutter width greater than the pulse width of the light. In contrast, the bioinstrumentation device 100 of this embodiment does not require compensation for the amount of light from the subject. Therefore, the shutter width does not need to be greater than the pulse width. Therefore, the shutter width can be set to a value between 1 ns and 30 ns. The bioinstrumentation device 100 of this embodiment can reduce the shutter width, thereby reducing the effect of dark current contained in the detection signal.

[0090] When detecting information such as cerebral blood flow by irradiating the forehead of the subject 500 with light, the internal light attenuation rate is extremely large. For example, the outgoing light may attenuate to approximately one millionth of the incident light. As a result, the amount of light emitted by a single pulse may be insufficient to detect the internal scattering component I2. The amount of light emitted is particularly weak in Class 1 laser safety standard irradiation. In this case, the light source 10 emits pulsed light multiple times, and the image sensor 20 also exposes the light multiple times using an electronic shutter, thereby integrating the detection signals and improving sensitivity.

[0091] An example of the configuration of the image sensor 20 will be described below.

[0092] The image sensor 20 may include a plurality of pixels arranged two-dimensionally on an imaging surface. Each pixel may include a photoelectric conversion element, such as a photodiode, and one or more charge accumulation units. Hereinafter, an example will be described in which each pixel includes a photoelectric conversion element that generates a signal charge according to the amount of received light through photoelectric conversion, a charge accumulation unit that accumulates a signal charge generated by a surface reflection component I1 of the pulsed light, and a charge accumulation unit that accumulates a signal charge generated by an internal scattering component I2 of the pulsed light. In the following example, the control circuit 30 causes the image sensor 20 to detect the surface reflection component I1 by detecting the portion of the pulsed light returned from the head of the subject 500 before the start of the fall. The control circuit 30 also causes the image sensor 20 to detect the portion of the pulsed light returned from the head of the subject 500 after the start of the fall, thereby detecting the internal scattering component I2. The light source 10 emits light of two wavelengths.

[0093] 1D is a diagram showing a schematic configuration example of one pixel 201 of the image sensor 20. Note that FIG. 1D shows the configuration of one pixel 201 in a schematic manner and does not necessarily reflect the actual structure. In this example, the pixel 201 includes a photodiode 203 that performs photoelectric conversion, a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207 that are charge accumulation sections, and a drain 202 that discharges signal charges.

[0094] Photons incident on each pixel due to the emission of one pulse of light are converted into signal electrons, which are signal charges, by the photodiode 203. The converted signal electrons are either discharged to the drain 202 or distributed to any of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, in accordance with a control signal input from the control circuit 30.

[0095] The emission of pulsed light from the light source 10, the accumulation of signal charges in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, and the discharge of the signal charges to the drain 202 are repeated in this order. This repetitive operation is fast and can be repeated tens of thousands to hundreds of millions of times within the time of one frame of a moving image (e.g., approximately 1 / 30 seconds). The pixel 201 ultimately generates and outputs four image signals based on the signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207.

[0096] In this example, the control circuit 30 causes the light source 10 to repeatedly emit a first pulsed light having a first wavelength and a second pulsed light having a second wavelength in sequence. By selecting two wavelengths with different absorption rates for the internal tissue of the subject 500 as the first and second wavelengths, the condition of the subject 500 can be analyzed. For example, a wavelength longer than 805 nm may be selected as the first wavelength, and a wavelength shorter than 805 nm may be selected as the second wavelength. This makes it possible to detect changes in the oxygenated hemoglobin concentration and the deoxygenated hemoglobin concentration in the blood of the subject 500.

[0097] The control circuit 30 first causes the light source 10 to emit a first pulsed light. The control circuit 30 causes the first floating diffusion layer 204 to accumulate signal charge during a first period during which the surface-reflected component I1 of the first pulsed light is incident on the photodiode 203. Next, the control circuit 30 causes the second floating diffusion layer 205 to accumulate signal charge during a second period during which the internally scattered component I2 of the first pulsed light is incident on the photodiode 203. Next, the control circuit 30 causes the light source 10 to emit a second pulsed light. The control circuit 30 causes the third floating diffusion layer 206 to accumulate signal charge during a third period during which the surface-reflected component I1 of the second pulsed light is incident on the photodiode 203. Next, the control circuit 30 causes the fourth floating diffusion layer 207 to accumulate signal charge during a fourth period during which the internally scattered component I2 of the second pulsed light is incident on the photodiode 203.

[0098] In this manner, the control circuit 30 sequentially accumulates the signal charge from the photodiode 203 in the first floating diffusion layer 204 and the second floating diffusion layer 205 after a predetermined time interval after starting the emission of the first pulsed light. Then, the control circuit 30 sequentially accumulates the signal charge from the photodiode 203 in the third floating diffusion layer 206 and the fourth floating diffusion layer 207 after a predetermined time interval after starting the emission of the second pulsed light. This operation is repeated multiple times. To estimate the amount of disturbance light and ambient light, a period may be set during which signal charge is accumulated in other floating diffusion layers (not shown) while the light source 10 is turned off. By subtracting the signal charge amount in the other floating diffusion layers from the signal charge amount in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, a signal from which disturbance light and ambient light components have been removed can be obtained.

[0099] In this embodiment, the number of charge accumulation units is four, but may be two or more depending on the purpose. For example, when only one type of wavelength is used, the number of charge accumulation units may be two. Furthermore, when only one type of wavelength is used and the surface reflection component I1 is not detected, the number of charge accumulation units per pixel may be one. Furthermore, even when two or more types of wavelengths are used, the number of charge accumulation units may be one if imaging using each wavelength is performed in a different frame. Furthermore, as will be described later, the number of charge accumulation units may be one if detection of the surface reflection component I1 and detection of the internal scattering component I2 are performed in different frames.

[0100] FIG. 1E is a diagram showing an example of the configuration of image sensor 20. In FIG. 1E, the area surrounded by a two-dot chain line corresponds to one pixel 201. Each pixel 201 includes one photodiode. While FIG. 1E shows only four pixels arranged in two rows and two columns, many more pixels may actually be arranged. Each pixel 201 includes a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207. The signals accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are treated as if they were four-pixel signals in a typical CMOS image sensor and are output from image sensor 20.

[0101] Each pixel 201 has four signal detection circuits. Each signal detection circuit includes a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. In this example, the reset transistor 310 corresponds to the drain 202 shown in FIG. 1D, and a pulse input to the gate of the reset transistor 310 corresponds to a drain discharge pulse. Each transistor is, for example, but not limited to, a field-effect transistor formed on a semiconductor substrate. As shown in the figure, one of the input terminal and output terminal (typically the source) of the source follower transistor 309 is connected to one of the input terminal and output terminal (typically the drain) of the row selection transistor 308. The gate, which is the control terminal of the source follower transistor 309, is connected to the photodiode 203. Signal charge (i.e., holes or electrons) generated by the photodiode 203 is accumulated in a floating diffusion layer, which is a charge storage portion between the photodiode 203 and the source follower transistor 309.

[0102] 1E, the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are connected to the photodiode 203. A switch may be provided between the photodiode 203 and the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The switch switches the conduction state between the photodiode 203 and each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 in response to a signal accumulation pulse from the control circuit 30. This controls the start and stop of accumulation of signal charges in each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The electronic shutter in this embodiment has a mechanism for such exposure control.

[0103] The signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are read out when the row selection circuit 302 turns on the gate of the row selection transistor 308. At this time, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified according to the signal potentials of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. An analog signal resulting from this current read out from the vertical signal line 304 is converted into digital signal data by an analog-to-digital (AD) conversion circuit 307 connected to each column. This digital signal data is read out for each column by the column selection circuit 303 and output from the image sensor 20. After reading out one row, the row selection circuit 302 and the column selection circuit 303 read out the signal charge information from the floating diffusion layers of all rows, and so on. After reading out all the signal charges, the control circuit 30 turns on the gate of the reset transistor 310 to reset all the floating diffusion layers. This completes the capture of one frame. By repeating this high-speed frame capture in the same manner, the image sensor 20 completes the capture of a series of frames.

[0104] In this embodiment, an example of a CMOS type image sensor 20 has been described, but the image sensor 20 may be of other types of imaging elements. The image sensor 20 may be, for example, a CCD type, a single-photon counting element, or an amplification type image sensor (for example, an EMCCD or an ICCD).

[0105] 1F is a diagram showing an example of operation within one frame in this embodiment. As shown in FIG. 1F, the emission of the first pulsed light and the emission of the second pulsed light may be alternately switched multiple times within one frame. In this way, the time difference between the acquisition timing of the detection images using the two wavelengths can be reduced, and even if the subject 500 is moving, it is possible to capture images using the first and second pulsed light almost simultaneously.

[0106] In this embodiment, the image sensor 20 detects both the surface reflection component I1 and the internal scattering component I2 of the pulsed light. First biological information of the subject 500 can be acquired from temporal or spatial changes in the surface reflection component I1. The first biological information can be, for example, the pulse rate of the subject 500. Meanwhile, second biological information of the subject 500, i.e., brain activity information, can be acquired from temporal or spatial changes in the internal scattering component I2.

[0107] The first biological information may be acquired by a method other than the method of detecting the surface reflection component I1. For example, the first biological information may be acquired using a detector of a type other than the image sensor 20. In this case, the image sensor 20 detects only the internal scattering component I2. The detector of the type other than the image sensor 20 may be, for example, a radar or a thermograph.

[0108] The first biological information may be, for example, at least one of the subject's pulse rate, sweat rate, respiratory rate, and body temperature. The first biological information may be any biological information other than brain activity information obtained by detecting the internal scattering component I2 of pulsed light irradiated onto the subject's head. Here, "other than brain activity information" does not mean that the first biological information does not contain any information attributable to brain activity. The first biological information may include any biological information attributable to biological activity other than brain activity. The first biological information may be, for example, biological information attributable to autonomous or reflex biological activity.

[0109] In this specification, a signal representing first biological information may be referred to as a “first biological signal” or simply as a “biological signal.” Also, a signal representing brain activity information may be referred to as a “brain activity signal.”

[0110] [1-3. Control circuit 30 and signal processing circuit 40] The control circuit 30 adjusts the time difference between the pulsed light emission timing of the light source 10 and the shutter timing of the image sensor 20. Hereinafter, this time difference may be referred to as "phase" or "phase delay." The "light emission timing" of the light source 10 is the timing at which the pulsed light emitted by the light source 10 starts to rise. The "shutter timing" is the timing at which exposure starts. The control circuit 30 may adjust the phase by changing the light emission timing, or may adjust the phase by changing the shutter timing.

[0111] The control circuit 30 may be configured to remove an offset component from the signal detected by each pixel of the image sensor 20. The offset component is a signal component caused by ambient light such as sunlight or fluorescent light, or disturbance light. The offset component caused by ambient light and disturbance light can be estimated by having the image sensor 20 detect a signal when the light source 10 is not emitting light, that is, when the light source 10 is turned off.

[0112] The control circuit 30 may be, for example, a combination of a processor and memory, or an integrated circuit such as a microcontroller having a built-in processor and memory. The control circuit 30 adjusts the light emission timing and shutter timing, estimates offset components, and removes offset components, for example, by having the processor execute a program recorded in the memory.

[0113] The signal processing circuit 40 processes the image signal output from the image sensor 20. The signal processing circuit 40 performs arithmetic processing such as image processing. The signal processing circuit 40 can be realized, for example, by a programmable logic device (PLD) such as a digital signal processor (DSP) or a field programmable gate array (FPGA), or by a combination of a central processing unit (CPU) or a graphics processing unit (GPU) and a computer program. The control circuit 30 and the signal processing circuit 40 may be integrated into a single circuit or may be separate, individual circuits. The signal processing circuit 40 may also be a component of an external device, such as a server located in a remote location. In this case, the external device, such as the server, is equipped with communication means for transmitting and receiving data to and from the light source 10, the image sensor 20, and the control circuit 30.

[0114] The signal processing circuit 40 in this embodiment can generate moving image data showing changes in blood flow on the skin surface and cerebral blood flow over time based on the signal output from the image sensor 20. The signal processing circuit 40 is not limited to such moving image data and may generate other information. For example, by synchronizing with other devices, the signal processing circuit 40 may generate biological information such as blood flow in the brain, blood pressure, blood oxygen saturation, or heart rate.

[0115] It is known that changes in cerebral blood flow or blood components such as hemoglobin are closely related to human neural activity. For example, changes in neuronal activity in response to changes in human emotions can lead to changes in cerebral blood flow or blood components. Therefore, measuring biological information such as changes in cerebral blood flow or blood components can estimate a subject's psychological state. The subject's psychological state refers to, for example, mood (e.g., pleasant, unpleasant), emotion (e.g., relief, anxiety, sadness, anger), health status (e.g., vitality, fatigue), and temperature sensation (e.g., hot, cold, muggy). Derived indicators of brain activity, such as proficiency, mastery, and concentration, are also included in the psychological state. The signal processing circuit 40 may estimate the subject's psychological state, such as concentration, based on changes in cerebral blood flow and output a signal indicating the estimation result.

[0116] FIG. 1G is a flowchart showing an outline of the operation of the control circuit 30. The control circuit 30 generally performs the operation shown in FIG. 1G. Note that, here, the operation when only detecting the internal scattering component I2 is described. The control circuit 30 first causes the light source 10 to emit pulsed light for a predetermined time (step S101). At this time, the electronic shutter of the image sensor 20 is in a state where exposure is stopped. The control circuit 30 causes the electronic shutter to stop exposure until a period in which a portion of the pulsed light is reflected by the surface of the subject 500 and reaches the image sensor 20 is completed. Next, the control circuit 30 causes the electronic shutter to start exposure at the timing when another portion of the pulsed light is scattered inside the subject 500 and reaches the image sensor 20 (step S102). After a predetermined time has elapsed, the control circuit 30 causes the electronic shutter to stop exposure (step S103). Next, the control circuit 30 determines whether the number of times the above signal accumulation has been performed has reached a predetermined number (step S104). If this determination is No, steps S101 to S103 are repeated until a Yes determination is made. If a Yes determination is made in step S104, the control circuit 30 causes the image sensor 20 to generate and output a signal representing an image based on the signal charges accumulated in each floating diffusion layer (step S105).

[0117] By the above operation, the components of light scattered inside the object to be measured can be detected with high sensitivity. Note that multiple light emission and exposure are not essential and are performed as needed.

[0118] [1-4.Other] The biomeasurement device 100 may include an imaging optical system that forms a two-dimensional image of the subject 500 on the light receiving surface of the image sensor 20. The optical axis of the imaging optical system is approximately perpendicular to the light receiving surface of the image sensor 20. The imaging optical system may include a zoom lens. When the position of the zoom lens is changed, the magnification ratio of the two-dimensional image of the subject 500 changes, and the resolution of the two-dimensional image on the image sensor 20 changes. Therefore, even if the distance to the subject 500 is long, it is possible to enlarge and observe the area to be measured in detail.

[0119] The bioinstrumentation device 100 may also include a bandpass filter between the subject 500 and the image sensor 20 that passes only light in or near the wavelength band emitted by the light source 10. This reduces the influence of disturbance components such as ambient light. The bandpass filter may be configured with a multilayer film filter or an absorption filter. Taking into account the temperature of the light source 10 and band shifts caused by oblique incidence on the filter, the bandwidth of the bandpass filter may be set to approximately 20 to 100 nm.

[0120] Furthermore, the biomeasurement device 100 may include a polarizing plate between the light source 10 and the subject 500, and between the image sensor 20 and the subject 500. In this case, the polarization directions of the polarizing plate arranged on the light source 10 side and the polarizing plate arranged on the image sensor side are in a crossed Nicol relationship. This makes it possible to prevent the specular reflection component of the surface reflection component I1 of the subject 500, i.e., the component having the same angle of incidence and reflection angle, from reaching the image sensor 20. In other words, the amount of light from the surface reflection component I1 reaching the image sensor 20 can be reduced.

[0121] [2. Operation] The bioinstrumentation device 100 of the present disclosure can distinguish and detect the internally scattered component I2 from the surface-reflected component I1. If the subject 500 is a human and the area being examined is the forehead, the signal intensity due to the internally scattered component I2 to be detected is very small. As mentioned above, this is because the light emitted is very small enough to meet laser safety standards, and light is significantly scattered and absorbed by the scalp, cerebrospinal fluid, skull, gray matter, white matter, and blood flow. Furthermore, changes in signal intensity due to changes in blood flow or blood flow components during brain activity are even smaller, equivalent to a few tens of times smaller. Therefore, in this embodiment, imaging is performed with as little contamination as possible with the surface-reflected component I1, which is several thousand to several tens of thousands times larger than the signal component to be detected.

[0122] An example of the operation of the bioinstrumentation device 100 according to this embodiment will now be described.

[0123] As shown in FIG. 1A, when the light source 10 irradiates the subject 500 with pulsed light, a surface reflection component I1 and an internal scattering component I2 are generated. Portions of the surface reflection component I1 and the internal scattering component I2 reach the image sensor 20. The internal scattering component I2 passes through the inside of the subject 500 before reaching the image sensor 20 from the light source 10, resulting in a longer optical path length than the surface reflection component I1. Therefore, the internal scattering component I2 reaches the image sensor 20 later on average than the surface reflection component I1. FIG. 2 shows an optical signal obtained when the light source 10 emits rectangular pulsed light, returns from the subject 500, and reaches the image sensor 20. The horizontal axis represents time (t) in all waveforms (a) to (d), and the vertical axis represents intensity in waveforms (a) to (c) and the open or closed state of the electronic shutter in waveform (d). Waveform (a) shows the surface reflection component I1. Waveform (b) shows the internal scattering component I2. Waveform (c) shows the sum of the surface reflection component I1 and the internal scattering component I2. As shown in waveform (a), the surface reflection component I1 maintains a rectangular shape. On the other hand, as shown in waveform (b), the internal scattering component I2 is the sum of light beams that have traveled various optical path lengths, and therefore exhibits a characteristic of trailing a tail at the trailing end of the pulsed light (i.e., a longer fall time than the surface reflection component I1). To extract a higher proportion of the internal scattering component I2 from the optical signal of waveform (c), the electronic shutter simply starts exposure after the trailing end of the surface reflection component I1 (i.e., when or after the surface reflection component I1 falls), as shown in waveform (d). This shutter timing is adjusted by the control circuit 30. As described above, the bioinstrumentation device 100 of the present disclosure only needs to distinguish and detect the surface reflection component I1 from the internal scattering component I2 that has reached the depth of the target, so the light emission pulse width and shutter width are arbitrary. Therefore, unlike conventional methods using streak cameras, this method can be implemented with a simple configuration and significantly reduces costs.

[0124] In the waveform (a) in Figure 2, the trailing edge of the surface reflection component I1 falls vertically. In other words, the time from the start to the end of the fall of the surface reflection component I1 is zero. However, in reality, the trailing edge of the surface reflection component I1 may not fall vertically due to the rising and falling edges of the waveform of the pulsed light emitted by the light source 10 not being perfectly vertical, the surface of the subject 500 having minute irregularities, or scattering within the epidermis. Furthermore, because the subject 500 is an opaque object, the surface reflection component I1 has a much greater amount of light than the internal scattering component I2. Therefore, even if the trailing edge of the surface reflection component I1 slightly deviates from the vertical falling position, the internal scattering component I2 may be obscured. Furthermore, due to the time delay associated with electron movement during the electronic shutter readout period, an ideal binary readout such as that shown in waveform (d) in Figure 2 may not be achieved. Therefore, the control circuit 30 may delay the shutter timing of the electronic shutter slightly from immediately after the trailing edge of the surface reflection component I1. For example, the delay may be about 0.5 ns to 5 ns. Instead of adjusting the shutter timing of the electronic shutter, the control circuit 30 may adjust the light emission timing of the light source 10. The control circuit 30 may adjust the time difference between the shutter timing of the electronic shutter and the light emission timing of the light source 10. When measuring changes in blood flow or blood flow components during brain activity in a non-contact manner, if the shutter timing is delayed too much, the internal scattering component I2, which is already small, will be further reduced. For this reason, the shutter timing may be kept near the rear end of the surface reflection component I1. Since the time delay due to scattering by the subject 500 is 4 ns, the maximum amount of delay in the shutter timing is about 4 ns.

[0125] The light source 10 may emit pulsed light multiple times, and exposure may be performed multiple times with each pulsed light at the same shutter timing, thereby amplifying the detected light amount of the internal scattering component I2.

[0126] Instead of or in addition to placing a bandpass filter between the subject 500 and the image sensor 20, the control circuit 30 may estimate the offset component by capturing an image with the same exposure time without emitting light from the light source 10. The estimated offset component is subtracted from the signal detected by each pixel of the image sensor 20. This makes it possible to remove the dark current component generated in the image sensor 20.

[0127] The internal scattering component I2 includes internal characteristic information of the subject 500, such as cerebral blood flow information. The amount of light absorbed by the blood changes according to temporal fluctuations in the cerebral blood flow of the subject 500, and the amount of light detected by the image sensor 20 also increases or decreases accordingly. Therefore, by monitoring the internal scattering component I2, it is possible to estimate the brain activity state from changes in the cerebral blood flow of the subject 500. In this specification, a signal indicating the internal scattering component I2, among the signals output from the image sensor 20, may be referred to as a "brain activity signal." The brain activity signal may include information on increases or decreases in the cerebral blood flow of the subject.

[0128] Next, an example of a method for detecting the surface reflection component I1 will be described. The surface reflection component I1 includes surface characteristic information of the subject 500, such as blood flow information on the face and scalp. The image sensor 20 detects the surface reflection component I1 from the optical signal that is generated when pulsed light emitted by the light source 10 reaches the subject 500 and returns to the image sensor 20.

[0129] FIG. 3A shows an example of a timing chart for detecting the surface reflection component I1. To detect the surface reflection component I1, for example, as shown in FIG. 3A, the shutter may be opened before the pulsed light reaches the image sensor 20 and closed before the trailing edge of the pulsed light arrives. Controlling the shutter in this manner reduces the amount of internally scattered light I2. This increases the proportion of light passing through the vicinity of the surface of the subject 500. In particular, the shutter may be closed immediately after the light reaches the image sensor 20. This enables signal detection that increases the proportion of the surface reflection component I1, which has a relatively short optical path length. Acquiring the signal of the surface reflection component I1 makes it possible to detect the pulse rate or the degree of facial blood oxygenation of the subject 500. Other methods for acquiring the surface reflection component I1 include having the image sensor 20 acquire the entire pulsed light or irradiating continuous light from the light source 10 for detection.

[0130] 3B shows an example of a timing chart for detecting the internal scattering component I2. By opening the shutter during the period when the trailing edge of the pulse reaches the image sensor 20, the signal of the internal scattering component I2 can be acquired.

[0131] The surface reflection component I1 may be detected by a device other than the bioinstrumentation device 100 that acquires the internal scattering component I2. Alternatively, a separate device, such as a pulse wave meter or Doppler blood flow meter, may be used. When using a separate device, consideration must be given to timing synchronization between the devices, optical interference, and the alignment of the detection points. Performing time-division imaging using the same camera or sensor, as in this embodiment, reduces temporal and spatial discrepancies. When acquiring signals for both the surface reflection component I1 and the internal scattering component I2 using the same sensor, the components to be acquired may be switched for each frame, as shown in FIGS. 3A and 3B. Alternatively, as described with reference to FIGS. 1D to 1F, the components to be acquired may be alternately switched at high speed within each frame. In this case, the detection time difference between the surface reflection component I1 and the internal scattering component I2 can be reduced.

[0132] Furthermore, the signals for the surface reflection component I1 and the internal scattering component I2 may be acquired using light of two wavelengths. For example, pulsed light of two wavelengths, 750 nm and 850 nm, may be used. In this way, the changes in the oxygenated hemoglobin concentration and the deoxygenated hemoglobin concentration can be calculated from the changes in the amount of detected light at each wavelength. When the surface reflection component I1 and the internal scattering component I2 are acquired using two wavelengths, a method of rapidly switching between four types of charge accumulation within one frame can be used, as described with reference to Figures 1D to 1F. This method can reduce the time lag in the detection signals.

[0133] The bioinstrumentation device 100 irradiates pulsed near-infrared or visible light toward the forehead of the subject 500, and can detect changes in the amount of oxygenated hemoglobin on the scalp or face or pulse rate from the temporal change in the surface reflection component I1. The light source 10 for acquiring the surface reflection component I1 emits near-infrared or visible light. Near-infrared light allows measurement day or night. When measuring pulse rate, visible light, which has higher sensitivity, may be used. During the day, ambient light such as sunlight or an indoor light source may be used instead of lighting. If the amount of light is insufficient, a dedicated light source may be used to supplement the light. The internal scattering component I2 includes light components that reach the brain. By measuring the temporal change in the internal scattering component I2, the temporal increase or decrease in cerebral blood flow can be measured.

[0134] Because the light reaching the brain also passes through the scalp and facial surface, fluctuations in scalp and facial blood flow are also detected as superimposed signals. To eliminate or reduce this effect, the signal processing circuit 40 may subtract the surface reflection component I1 from the internal scattering component I2 detected by the image sensor 20. This allows for the acquisition of pure cerebral blood flow information excluding scalp and facial blood flow information. For example, a subtraction method may be used in which the signal of the surface reflection component I1 is multiplied by a coefficient (one or more) determined by taking into account the optical path length difference, and then subtracted from the signal of the internal scattering component I2 a value obtained by multiplying the signal of the surface reflection component I1. This coefficient may be calculated, for example, by simulation or experiment based on the average optical constants of a typical human head. This subtraction process is particularly easy to perform when measurements are taken using the same camera or sensor and light of the same wavelength. This is because it is easy to reduce temporal and spatial discrepancies and to match the characteristics of the scalp blood flow component contained in the internal scattering component I2 with the characteristics of the surface reflection component I1.

[0135] The skull exists between the brain and the scalp. Therefore, the two-dimensional distribution of cerebral blood flow is independent of the two-dimensional distribution of scalp and facial blood flow. Therefore, based on the signal detected by the image sensor 20, the two-dimensional distribution of the internal scattering component I2 and the two-dimensional distribution of the surface reflection component I1 may be separated using a statistical method such as independent component analysis or principal component analysis.

[0136] Next, an example of performing vital sensing of the subject 500 using the bioinstrumentation device 100 will be described.

[0137] FIG. 4A is a diagram showing a schematic diagram of the relationship between a person's pulse rate, brain activity, and physical condition. In FIG. 4A, the horizontal axis represents the pulse rate per minute, and the vertical axis represents the brain activity. The brain activity can be estimated by the magnitude of change in the signal strength of the internal scattering component I2. For example, the brain activity can be estimated by an index such as the difference between the maximum and minimum values ​​of the signal strength of the internal scattering component I2, or the time rate of change.

[0138] When the subject is awake, both the pulse rate and the amount of brain activity are relatively large. On the other hand, when the subject is asleep, the amount of brain activity does not change much, but the pulse rate is lower than when the subject is awake. Therefore, by calculating the pulse rate from the measurement data of the surface reflection component I1, it is possible to determine whether the subject 500 is awake or asleep.

[0139] During sleep, the cerebral cortex is highly active to organize and consolidate memories. In contrast, brain activity decreases during impaired consciousness. Therefore, by measuring changes in cerebral blood flow from measurement data of the internal scattering component I2, it is possible to determine whether the subject 500 is asleep or in a state of impaired consciousness. In particular, with regard to pulse rate, there are overlapping areas among wakefulness, sleep, and impaired consciousness, making it difficult to distinguish between these states using pulse rate information alone. Impaired consciousness can be classified as bradycardia or tachycardia. Bradycardia is a state in which the pulse rate is slower than normal, leading to impaired consciousness due to insufficient blood supply to the brain. A pulse rate of 40 beats per minute or less is considered a dangerous state. On the other hand, tachycardia is a state in which the pulse rate is abnormally high. A pulse rate of 140 beats per minute or more is considered a dangerous state. Impaired consciousness in the case of bradycardia is difficult to distinguish from sleep. Impaired consciousness in the case of tachycardia is difficult to distinguish from wakefulness. By utilizing information on changes in cerebral blood flow, as in this embodiment, more accurate determination is possible.

[0140] In addition, because brain activity decreases in both mindfulness and impaired consciousness, it is difficult to distinguish between mindfulness and impaired consciousness by measuring cerebral blood flow alone. However, because the pulse rate is closer to normal in mindfulness than in impaired consciousness, measuring the pulse rate in addition to measuring cerebral blood flow makes it possible to distinguish between mindfulness and impaired consciousness.

[0141] The brain's oxygen consumption accounts for approximately 20% of the total body's oxygen consumption, and cerebral blood flow accounts for approximately 15% of the total body's blood flow. When this cerebral blood flow decreases, brain activity decreases and a loss of consciousness occurs. The cognitive center is located in the cerebral cortex. For this reason, the bioinstrumentation device 100 observes changes in cerebral blood flow in the cerebral cortex through the exposed skin of the forehead on the head. This makes it possible to determine whether the patient is in a state of consciousness loss, wakefulness, or sleep.

[0142] The biological information that can be used in this embodiment is not limited to pulse rate, but may also be other information such as sweat rate, body temperature, respiratory rate, etc. Physical conditions that are difficult to determine using only sweat rate, body temperature, or respiratory rate can be accurately determined by using brain activity information.

[0143] Figure 4B is a diagram that shows a schematic representation of the relationship between a person's sweating rate, brain activity, and physical condition. Figure 4C is a diagram that shows a schematic representation of the relationship between a person's body temperature, brain activity, and physical condition. During sleep, both in REM and non-REM sleep, sweating is greater and body temperature is lower than during wakefulness. Generally, at the onset of sleep, a large amount of sweating occurs, followed by a rapid drop in body temperature and the person entering non-REM sleep. Approximately one hour later, body temperature rises and the brain enters REM sleep, which is similar to wakefulness. After that, sweating begins again, body temperature drops, and the person transitions to non-REM sleep. This cycle is repeated multiple times during sleep.

[0144] Figure 4D is a diagram showing the relationship between a person's breathing rate, brain activity, and physical state. During REM sleep, the breathing rate is lower than when awake. During non-REM sleep, the breathing rate decreases further by approximately 10 to 0%. Furthermore, during bradycardia, the breathing rate is lower than when calm, and during tachycardia, the breathing rate tends to increase to compensate for the lack of oxygen.

[0145] As shown in Figures 4B to 4D, similar to pulse rate, sweat rate, body temperature, or respiratory rate can also be used to determine whether a person is asleep or awake. However, it is difficult to determine whether a person is in a state of consciousness disorder or asleep using only these biological information. As in this embodiment, by using brain activity information in addition to the above biological information, it is possible to determine whether a person is in a state of consciousness disorder or not.

[0146] When the biomeasurement device 100 measures the amount of sweating as bioinformation, the biomeasurement device 100 may include a sensor that measures the amount of sweating. When the biomeasurement device 100 measures body temperature as bioinformation, the biomeasurement device 100 may include a sensor that measures the body temperature. When the biomeasurement device 100 measures the respiratory rate as bioinformation, the biomeasurement device 100 may include a sensor that measures the respiratory rate. These sensors may be, for example, image sensors that detect infrared rays. The amount of sweating, the respiratory rate, or the body temperature may be measured by an image sensor 20 that acquires brain activity information. The amount of sweating, the body temperature, or the respiratory rate may also be estimated using image processing. In this way, the biosignal may include information on at least one of the pulse rate, the amount of sweating, the respiratory rate, and the body temperature.

[0147] Next, an application example of the bioinstrumentation device 100 of this embodiment will be described.

[0148] FIG. 5 shows how the biometric device 100 detects the vital signs of a passenger in an autonomously driven taxi. The biometric device 100 is installed inside the vehicle. The biometric device 100 senses the passenger's vital signs while or after the passenger is being picked up or dropped off by autonomous driving. If the biometric device 100 detects an abnormality in the passenger, it notifies an external device, such as a server computer at a medical institution, of the abnormality by, for example, wireless communication.

[0149] 6 is a diagram showing an example of the configuration of a bioinstrumentation device 100 in this application example. In this example, the bioinstrumentation device 100 includes a communication circuit 60 and a speaker 70 connected to a control circuit 30. The communication circuit 60 performs communication in accordance with, for example, a known wireless communication standard. When the signal processing circuit 40 detects an abnormality in a passenger, i.e., a subject 500, the control circuit 30 notifies an external device 200 in a medical institution, etc., of the abnormality via the communication circuit 60.

[0150] FIG. 7 is a flowchart showing an example of the operation of the biomeasurement device 100. The control circuit 30 of the biomeasurement device 100 determines whether there is anything abnormal about the passenger during or after the vehicle has automatically driven to pick up or drop off the passenger (step S601). For example, if the passenger's behavior is strange or if the passenger does not get off even after a certain time has passed since the vehicle arrived at the destination, it is determined that there is an abnormality. This determination may be made based on the output of the image sensor 20 or another sensor (not shown). If an abnormality in the passenger is detected, an error sound or a voice call is first made to check the passenger's condition (step S602). This call is made via the speaker 70. Note that the speaker 70 may be installed in the vehicle instead of in the biomeasurement device 100.

[0151] Next, the control circuit 30 determines whether there is a response from the passenger (step S603). This determination can be made based on information such as an image acquired by the image sensor 20 or the passenger's voice detected by a microphone (not shown). If there is a response and there are no particular problems, after arriving at the destination, the passenger completes the payment procedure by automatic payment or electronic payment and gets off the bus (step S604).

[0152] If the passenger does not respond to the voice call or is groaning, it is possible that the passenger is asleep or unconscious. Therefore, the biomeasurement device 100 performs an initial observation (step S605). First, the biomeasurement device 100 automatically adjusts the light source 10 and image sensor 20 of the biomeasurement device 100 so that they are facing the passenger. For example, the control circuit 30 adjusts the orientation of the light source 10 and image sensor 20 so that pulsed light from the light source 10 is incident on the passenger's forehead. The biomeasurement device 100 is equipped with a drive mechanism such as a motor that enables this adjustment. In this state, the biomeasurement device 100 performs, for example, checking the passenger's breathing, blood oxygen saturation (SpO2) based on the surface blood flow of the face and scalp, and pulse rate. SpO2 is the ratio of the amount of hemoglobin bound to oxygen to the total amount of hemoglobin in the blood. The biomeasurement device 100 also has a camera function that captures images. Therefore, breathing can be confirmed from the passenger's body movements captured in the camera image. To calculate SpO2, measurements can be taken at two wavelengths of near-infrared light, or at two wavelengths of red and near-infrared light. SpO2 can be estimated by calculating the ratio of the detected light levels or the ratio of HbO2 to Hb. Pulse rate can be measured from changes in blood flow in the passenger's face or forehead. Measurements can be taken over a short period of time, such as 10 or 20 seconds. Basic vital signs can be obtained through such a short screening.

[0153] Next, the control circuit 30 determines whether the measured pulse rate of the passenger is within a reference range (step S606). If the measured pulse rate is within the reference range, the control circuit 30 determines that the passenger is asleep and in a healthy state, and urges the passenger to disembark (step S607). For example, the control circuit 30 urges the passenger to disembark by voice via the speaker 70. The reference range for the pulse rate may be, for example, between 50 and 100 beats per minute.

[0154] If the measured pulse rate is outside the reference range, the pulse rate may be extremely low due to sleep, but it may also be due to impaired consciousness. Therefore, the bioinstrumentation device 100 immediately contacts a medical institution (step S608). For example, the control circuit 30 transmits a signal or alert to a computer at the medical institution via the communication circuit 60, indicating that the passenger's pulse rate is at an abnormal value. At this time, information such as the vehicle ID and location may also be transmitted. The bioinstrumentation device 100 begins measuring cerebral blood flow (step S609).

[0155] In particular, in mountainous areas and rural areas, where it takes time for emergency teams to arrive, an environment where passengers' vital signs can be checked immediately is required to make a quick decision. Cerebral blood flow can be measured by having the bioinstrumentation device 100 shine light onto the passenger's head and detect the internal scattering component I2 scattered in the frontal lobe. If a passenger is unconscious, there is a high possibility that the fluctuations in cerebral blood flow will be significantly reduced compared to when they are asleep. Therefore, it is possible to determine whether the passenger is asleep or unconscious based on the fluctuations in the measured cerebral blood flow.

[0156] The measurement time of cerebral blood flow in step S609 is longer than the measurement time of pulse rate in step S605. This is because the period of change in cerebral blood flow is relatively long compared to the pulse, which has a relatively short period. The measurement of cerebral blood flow may be performed for a period of, for example, one minute or more and five minutes or less. The control circuit 30 transfers the data obtained by this measurement, for example, in real time, to an external device 200, such as a computer in a medical institution such as a hospital (step S610). This allows a doctor to immediately check the passenger's vital signs. Since the bioinstrumentation device 100 can also acquire near-infrared images or visible light images, these images may also be transmitted. This allows the passenger's condition to be observed through images. In addition, a pulse signal may also be transmitted in step S610.

[0157] Next, the signal processing circuit 40 of the bioinstrumentation device 100 determines whether the amount of change in cerebral blood flow is equal to or greater than a reference value (step S611). The amount of change in cerebral blood flow may be, for example, the difference between the maximum and minimum values ​​of cerebral blood flow per cycle, the amplitude, or the time rate of change. If this value is smaller than a predetermined reference value, it is determined that the fluctuation in cerebral blood flow is small, and it is determined that the patient has a disturbance of consciousness (step S613). In this case, the control circuit 30 transmits a signal to the external device 200 via the communication circuit 60 requesting emergency medical assistance (step S614).

[0158] As described above, in this embodiment, the signal processing circuit 40 sequentially performs the initial observation, which is the first diagnosis, and the measurement of cerebral blood flow, which is the second diagnosis. In the initial observation, the signal processing circuit 40 estimates the state of the subject 500 based on a biological signal different from the brain activity signal, such as a pulse, and outputs a first signal indicating the estimation result. In the measurement of cerebral blood flow, the signal processing circuit 40 estimates the state of the subject 500 based on the brain activity signal and outputs a second signal indicating the estimation result. In this embodiment, the time required for the second diagnosis is longer than the time required for the first diagnosis. In other words, the time from the start of pulsed light irradiation to the output of a brain activity signal is longer than the time from the start of pulsed light irradiation to the acquisition of the biological signal. Furthermore, the time from the acquisition of a brain activity signal to the output of a second signal is longer than the time from the acquisition of a biological signal such as a pulse to the output of a first signal. When the signal processing circuit 40 determines that the subject 500 is not in a healthy state based on the above-mentioned biological signals and / or brain activity signals, the communication circuit 60 notifies the external device 200 at the medical institution. This notification may include the first and second signals.

[0159] With the above operations, even if a passenger in an autonomous vehicle loses consciousness, it is possible to quickly take action such as dispatching an ambulance to the scene. As in the above example, when a passenger is sleeping or unconscious, body movement is small. Therefore, the necessary brain activity signals can be obtained without correcting the positions of the light source and detector due to body movement.

[0160] In the example of FIG. 7, even if the passenger regains consciousness during the observation process, there may still be health problems. For this reason, a remote interview or full-body observation may be conducted. The remote interview or full-body observation may be achieved by a doctor contacting the passenger through the camera of the biometric device 100. For example, if the doctor strongly suspects that the passenger has hypoglycemia, the doctor may check whether the passenger can hold hands or check the eyes (e.g., pupils, eye position and movement, or nystagmus). Alternatively, if the doctor strongly suspects that the passenger has a stroke, the doctor may check for severe headache, nausea, dizziness, visual impairment, or aphasia.

[0161] The biomeasurement device 100 may transfer video data from the early stages of abnormality occurrence to an external device 200. This allows a doctor to check the time course of the impaired consciousness (e.g., whether it persists or recovers). The biomeasurement device 100 is also effective for checking the vital signs of passengers in the event of an accident caused by autonomous driving, not just during autonomous driving and when passengers disembark. Unlike conventional cerebral blood flow measurement devices such as functional near-infrared encephalography (fNIRS), the biomeasurement device 100 can perform non-contact measurements using a camera including an image sensor 20. This eliminates the need to attach a device to the subject 500, and allows measurements to be performed even when there are no other people present other than the subject.

[0162] Vital signs checks using the biometric measurement device 100 are not limited to being performed inside a vehicle, but may be performed in other environments. For example, as shown in FIG. 8, biometric information of a patient in bed may be measured. Also, as shown in FIG. 9, biometric information of a person taking a bath may be measured. The biometric measurement device 100 can be deployed in any environment where it is desired to check vital signs.

[0163] FIG. 10 is a diagram showing how the bioinstrumentation device 100 detects impaired consciousness or drowsiness in a vehicle driver, who is a subject 500. The bioinstrumentation device 100 measures the driver's pulse rate based on the surface reflection component I1 and measures changes in the driver's cerebral blood flow based on the internal scattering component I2. Based on the acquired pulse rate data and cerebral blood flow data, it is determined whether the driver is in an awake state, drowsy state, or impaired consciousness state. In measuring cerebral blood flow, the measurement data is compared with a database of cerebral blood flow volume or cerebral blood flow distribution of drivers during normal driving, or a database of cerebral blood flow volume or cerebral blood flow distribution of drivers in an awake state, which is accumulated daily, to determine whether the measured data is abnormal.

[0164] In addition to the pulse rate, the surface reflection component I1 may also be used to measure the amount of sweating. The amount of sweating can be measured using the specular reflection component from the skin surface. When sweating occurs, the component of light emitted from the biometric device 100 that is specularly reflected from the driver's skin surface increases, appearing as a luminance component. The luminance component is detected from the image as an area with high contrast compared to the surrounding area. This makes it possible to detect sweating due to tachycardia. By comprehensively evaluating cerebral blood flow data and sweating amount data, it is possible to improve the accuracy of determining whether the driver is unconscious or drowsy.

[0165] By sensing the room temperature, direct sunlight conditions, or the driver's body shape in parallel with the amount of sweating, it is possible to improve the accuracy of determining whether the sweating is due to the vehicle interior environment or the driver's poor physical condition. The room temperature, direct sunlight conditions, and the driver's body shape may be detected by a separate sensor installed in the vehicle interior. The direct sunlight conditions and the driver's body shape may also be detected by using the image sensor 20 included in the biometric measurement device 100 as a camera. Furthermore, by taking into account the traveling path detected by another sensing system installed in the vehicle body, the speed of the vehicle relative to surrounding vehicles, etc., it is possible to detect the driver's condition with even higher accuracy.

[0166] If the driver is suspected to have impaired consciousness based on biological information such as skin blood flow information, pulse rate information, or sweating information, and cerebral blood flow information, the bioinstrumentation device 100 will make an audio call to the driver. After making the audio call, the bioinstrumentation device 100 will again detect the driver's condition. If the driver is again suspected to have impaired consciousness, a medical institution or the like will be contacted. If the vehicle is equipped with an autonomous driving system, the autonomous driving system may take action to pull the vehicle over to the side of the road and stop it.

[0167] FIG. 11A shows an example of a biometric measurement device 100 integrated with a head-mounted display 400. FIG. 11B shows a subject using the biometric measurement device 100 shown in FIG. 11A. The head-mounted display 400 is used for entertainment purposes such as watching movies or playing games. An increase in pulse rate detected from pulse information acquired by detecting the surface reflection component I1 may indicate a state of tension or fear, while a decrease in pulse rate detected may indicate a state of relaxation or boredom. Sweating may also indicate a state of tension or excitement. Meanwhile, cerebral blood flow information acquired by detecting the internal scattering component I2 can be used to evaluate the level of understanding, concentration, or relaxation of the user (subject 500). Integrating this information can improve the accuracy of estimating the user's psychological state. A method for estimating the psychological state involves applying machine learning to comprehensive information that combines biometric information acquired by detecting the surface reflection component I1 and brain activity information acquired by detecting the internal scattering component I2. Based on the estimated psychological state, the intensity of the stimulation given to the user may be adjusted by switching the image displayed on the head-mounted display 400, switching the story, or adjusting the volume. As an example of volume adjustment, if it is estimated that the content is not scary enough, the contrast of the volume may be increased.

[0168] The head mounted display 400 can also be applied to virtual reality or a fusion of the real world and the virtual world, such as VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality). For example, in route guidance using AR or instructions for factory work, the user's level of understanding may be determined based on biological information obtained by detecting the surface reflection component I1 and brain activity information obtained by detecting the internal scattering component I2, and repeat playback or adjustment of the speed of the announcement may be performed depending on the level of understanding.

[0169] FIG. 12 shows an example of a biometric measurement device 100 installed in a home or office for daily health checks. Because the biometric measurement device 100 can acquire biometric information and brain activity information non-contact, it can be installed, for example, behind a bathroom mirror. The mirror is coated with an interference film that reflects visible light and transmits near-infrared light (e.g., light with a wavelength of 700 nm or more). The visible-light-reflecting interference film coating reflects the image of the subject 500 (the user), but the camera installed behind the mirror is invisible to the user. Meanwhile, the transmission of near-infrared light allows measurement light to be directed toward the user through the mirror. Therefore, the user is not aware of the presence of the camera, allowing the measurement of biometric information and cerebral blood flow to be performed in a natural mental state. Furthermore, measurements can be taken from the front of the user, enabling efficient measurement of the forehead. Furthermore, since people often use hairbands or other devices to hold their bangs in place when washing their face, there is no need to instruct them to lift their bangs, which can otherwise obstruct their view, thus reducing the user's effort. Furthermore, since the user stands in front of the mirror at the washstand, measurements can be taken at a fixed and close distance, which increases the stability of the measurements.

[0170] A human presence sensor such as a pyroelectric sensor may be installed near the biomeasurement device 100. When the human presence sensor detects a person, it sends a signal to the biomeasurement device 100, which starts measuring the person. By linking with the human presence sensor, it is possible to stop driving the light source and electrical circuits in the biomeasurement device 100 when no person is present, thereby reducing power consumption.

[0171] It is possible to obtain physical health data such as lack of sleep, fatigue, or anemia based on the surface blood flow or pulse rate measured by the biomeasurement device 100. In addition, it is possible to detect mental health data such as concentration state, mental clarity, or depression based on the cerebral blood flow information acquired at the same time. In this way, by evaluating both the bioinformation acquired by detecting the surface reflection component I1 and the brain activity information acquired by detecting the internal scattering component I2, it is possible to estimate both physical and mental health conditions.

[0172] The biometric device 100 may perform measurements each time the user stands in front of a mirror, and the acquired data may be accumulated. Relative change information obtained by comparing accumulated daily data may detect abnormalities in the user's physical condition on a given day relative to their normal daily state. Furthermore, by accumulating daily time-series data measurements and applying a moving average to the time-series data, high-frequency components representing temporary irregular changes in physical condition can be removed, enabling monitoring of low-frequency components representing long-term changes in physical condition over several years. This may lead to early detection of serious cardiovascular or neurological diseases such as stroke, brain tumor, or dementia. In this case, calibration may be performed by estimating the influence of seasonal fluctuations in temperature or physical condition in advance. Removing the influence of seasonal fluctuations in temperature or physical condition can improve detection accuracy.

[0173] If the system detects a change in the user's physical condition, it will display the estimated health status on a display installed on the sink or on the reflective mirror. Furthermore, by outputting daily measurement results as a graph, it is possible to visualize the progress of health improvement and increase the user's motivation to improve their health. Another method of displaying data is to send the measurement data to the user's smartphone or tablet via Wi-Fi (registered trademark), Wi-Fi Direct (registered trademark), BLUETOOTH (registered trademark), or the cloud, and display it on the smartphone or tablet.

[0174] FIG. 13 shows an example of the application of the bioinstrumentation device 100 to alcohol consumption detection for a driver, who is a subject 500. Brain activity information acquired based on the internal scattering component I2 is used to evaluate whether the driver is able to think normally, whether their mind is functioning, and whether they are able to concentrate on driving. Additionally, surface blood flow, heart rate, or sweat rate is detected based on the surface reflection component I1. By integrating and evaluating this information, the possibility that the driver has been drinking is determined, and if a high possibility is determined, a notification to that effect is given by voice or on a display. Additionally, vehicle control is performed, such as by not starting the engine or shutting it down.

[0175] FIG. 14 is a diagram showing an example of adjusting indoor air conditioning using the biometric measurement device 100. The user's sweating state may be detected from both information on the surface reflection component I1 and the internal scattering component I2. Since the surface reflection component I1 contains a large amount of specular reflection components from the user's skin surface, when the user sweats, the surface reflectance increases and can be detected as a glare component. In addition, the internal scattering component I2 is used to measure whether the user feels uncomfortable about the heat. When sweating is detected from the surface reflection component I1 and discomfort from the heat is detected from the internal scattering component I2, the temperature, airflow, or direction of the air conditioning device 600 may be controlled.

[0176] As described above, the biomeasurement device 100 can output a notification to the subject 500, control devices around the subject 500, or communicate with an external device based on the result of the determination of the state of the subject 500. Notification to the subject includes providing the subject with a stimulus such as a visual, auditory, or tactile stimulus. Devices around the subject include devices that are placed around the subject and can provide the subject with a stimulus such as a visual, auditory, or tactile stimulus. Devices around the subject also include devices operated by the subject. Communication with an external device includes transmitting a signal or data to a device such as a server located away from the biomeasurement device 100, as described above.

[0177] In this embodiment, as shown in FIG. 6, the signal processing circuit 40 includes a memory 42, and pre-stores a plurality of reference values ​​determined based on the data shown in FIGS. 4A to 4D in the memory 42. The signal processing circuit 40 compares the signal representing the first biological information with the first reference value and compares the signal representing the brain activity information with the second reference value. This allows the subject's condition to be determined based on both the first biological information and the signal representing the brain activity information. In this manner, the subject's condition can be determined by comparing the signal representing the first biological information and the signal representing the brain activity information with the two reference values ​​stored in the memory 42, respectively.

[0178] The signal processing circuit 40 may store in the memory 42 a data table indicating the relationship between brain activity amounts, activity amounts related to the first biological information, and physical conditions. The signal processing circuit 40 may determine the subject's condition by evaluating the acquired signals representing the brain activity information and the first biological information with reference to the data table. In this way, the subject's condition can be determined by evaluating the biological signals and brain activity signals with reference to the data table stored in the memory 42. The data table indicates the relationship between the brain activity signals, the biological signals, and physical conditions.

[0179] The signal processing circuit 40 may acquire time-series changes in a signal representing brain activity information and a signal representing first biological information. The signal processing circuit 40 may then compare the change pattern or feature of the time-series changes with a reference pattern or reference feature stored in the memory 42. Alternatively, the subject's condition may be determined by calculating statistical values ​​such as correlation. The feature of the time-series changes may be calculated by previously reading a large dataset of subjects and learning the features contained therein using supervised machine learning techniques such as deep learning, support vector machines, regression trees, or Bayesian estimation. The dataset is a set of detected signals and correct values ​​representing the subject's condition. The previously learned feature values ​​may be used as training data, and the current condition of the subject may be determined based on this learning data using machine learning. The training data may be updated using new subject data each time the subject is measured. A detailed examination by a specialized institution such as a medical institution may result in a determination result obtained by executing the flow shown in FIG. 7, for example, that differs from the true value. In this case, the correct value of the subject's condition may be corrected and resubmitted to machine learning.

[0180] In addition to using supervised learning, unsupervised machine learning such as hierarchical clustering, k-means clustering, self-organizing maps, or deep learning may also be used to determine the subject's condition.

[0181] In addition to the time-series changes of the signal representing brain activity information and the signal representing the first biological information, when a photodetector functioning as a camera or an image sensor is used, two-dimensional data may be used as a data set. For example, based on the distribution, bias, or high intensity positions of the cerebral blood flow signal or the first biological signal in the two-dimensional image data, the subject's condition may be determined, patterns or correlations may be compared, feature extraction may be performed, or machine learning may be performed.

[0182] As described above, the condition of the subject can be determined by comparing the time series data or the two-dimensional pattern with reference patterns or reference features stored in memory 42, calculating statistical values, and / or making a determination using machine learning.

[0183] The signal processing circuit 40 may determine the age, sex, or health condition of the subject based on information about the subject acquired by the biometric measurement device 100 or other sensors, or information about the subject that has been registered in advance. The signal processing circuit 40 may then update the reference values, data tables, and reference patterns based on the determination results.

[0184] As described above, according to the embodiments of the present disclosure, information regarding the brain activity state of a subject can be measured without contact with the subject and while suppressing noise due to reflected components from the surface of the subject. Furthermore, a stable vital sign check of the subject can be performed in an inexpensive manner.

[0185] (Other embodiments) The biometric measurement device 100 may acquire biometric information other than brain activity information from a detector other than a photodetector, such as the image sensor 20. FIG. 15 is a diagram illustrating an example of such a biometric measurement device 100. In this example, the biometric measurement device 100 includes a detector 50 that measures biometric information such as pulse rate in addition to the image sensor 20. If, for example, a millimeter-wave radar is used as the detector 50, the pulse wave or respiration of the subject 500 can be detected. If, for example, a thermograph is used as the detector 50, the amount of sweat and body temperature of the subject 500 can be measured. When the detector 50 is used, the image sensor 20 does not need to detect the surface reflection component I1 of the subject 500. A vital sign check of the subject 500 can be performed based on the biometric information of the subject 500 detected by the detector 50.

[0186] In the above-described embodiment, an example in which the photodetector is an image sensor 20 has been described. However, the present disclosure is not limited to such an example. For example, a combination of an avalanche photodiode and a memory, or a combination of a PIN photodiode and a memory, may be used as the photodetector. Even when a combination of an avalanche photodiode and a memory, or a combination of a PIN photodiode and a memory, is used as the photodetector, a sufficient signal amount can be obtained by, for example, detecting the rising and falling components of the reflected pulsed light for each pulse and repeatedly storing the detected components in the memory. By performing calculations using the signal amount stored in the memory, it is possible to separate the signal indicating skin blood flow from the signal indicating cerebral blood flow.

[0187] The present disclosure includes not only the biometric measurement device 100 but also a method for acquiring biometric information using the biometric measurement device 100 and a vehicle equipped with the biometric measurement device 100. [Industrial Applicability]

[0188] The bioinstrumentation device according to the present disclosure is useful for cameras or measuring instruments that acquire internal information of a subject in a non-contact manner. The bioinstrumentation device can be applied to biological or medical sensing. [Explanation of symbols]

[0189] 10 light source 20 Image Sensor 22 Photoelectric conversion element 24 Charge storage section 30 Control circuit 32 Light source control unit 34 Sensor control unit 40 Signal Processing Circuit 42 memory 50 detectors 60 Communication Circuit 70 Speaker 100 Biometrics 200 External device 400 Head-Mounted Display 500 subjects 600 Air Conditioning Unit

Claims

1. An information processing method in a computer, comprising: acquiring a brain activity signal indicative of a state of brain activity of the subject; acquiring a biological signal of the subject that is different from the brain activity signal; When determining that the biological information indicated by the biological signal is abnormal, outputting a signal indicating that the biological information is abnormal to a computer located outside the space where the subject is present; outputting the signal indicating that the biological information is abnormal to the computer outside the space, and then determining whether the subject is in a state of consciousness disorder, i.e., unconsciousness, based on the brain activity signal; When the subject is determined to be in the state of impaired consciousness, outputting a signal to the computer outside the space requesting emergency dispatch. method.

2. The method of claim 1, further comprising, when it is determined that the biological information is abnormal, outputting information indicating a location of the space to the computer outside the space in addition to the signal indicating that the biological information is abnormal.

3. the subject is a driver or passenger of a vehicle; When it is determined that the biometric information is abnormal, the method further includes outputting, to the computer outside the space, information for identifying the vehicle in addition to the signal indicating that the biometric information is abnormal and the information indicating the location of the space. The method of claim 2.

4. The method according to claim 1 , wherein the computer outside the space is a computer in a medical institution.

5. The subject is a driver or a passenger of an autonomously driven vehicle, The method of claim 1 , further comprising pulling over and stopping the vehicle if the subject is determined to be in the impaired state.

6. Determining whether or not the consciousness disorder is present includes: a result of comparing the brain activity signal with first reference information stored in a memory; and The method according to claim 1 , further comprising making a determination based on a result of comparing the biological signal with second reference information stored in the memory.

7. the first reference information includes at least one of a reference value corresponding to the brain activity signal, a data table corresponding to the brain activity signal, and a reference pattern corresponding to a time-series change in the brain activity signal; The method according to claim 6 , wherein the second reference information includes at least one of a reference value corresponding to the biological signal, a data table corresponding to the biological signal, and a reference pattern corresponding to a time-series change in the biological signal.

8. acquiring subject information including at least one of the subject's age, sex, and health condition; The method of claim 6 or 7, further comprising updating the first reference information and the second reference information based on the subject information.

9. acquiring the biological signal includes acquiring it from a first sensor; The method of claim 1 , wherein obtaining the brain activity signal includes obtaining the brain activity signal from a second sensor.

10. emitting pulsed light from a light source to irradiate the subject's head; acquiring, from an image sensor, an internal scattering component of the reflected pulsed light returned from the head of the subject, the internal scattering component being scattered within the brain of the subject, as the brain activity signal; detecting a surface reflection component reflected by a surface of the skin of the subject from the reflected pulsed light, and acquiring a signal indicating a variation in the surface reflection component as the biological signal from the image sensor; The method of claim 1 , comprising:

11. The method of claim 1 , wherein the biological information corresponds to at least one selected from the group consisting of pulse rate, sweat rate, respiratory rate, and body temperature.

12. An information processing system including a signal processing circuit, The signal processing circuit Acquiring a brain activity signal indicating the state of brain activity of the subject; acquiring a biological signal of the subject that is different from the brain activity signal; When determining that the biological information indicated by the biological signal is abnormal, outputting a signal indicating that the biological information is abnormal to a computer located outside the space where the subject is present; outputting the signal indicating that the biological information is abnormal to the computer outside the space, and then determining whether the subject is in a state of impaired consciousness, i.e., unconsciousness, based on the brain activity signal; When the subject is determined to be in the state of impaired consciousness, a signal requesting emergency dispatch is output to the computer located outside the space. Information processing system.

Images