Information processing method in a computer and bioinstrumentation system

Abstract

To provide a novel method to estimate the condition of an object person.SOLUTION: The information processing method in a computer comprises: causing a light source to emit an optical pulse to irradiate a vehicle driver therewith; causing an image sensor to output a first image showing the appearance of the face of the driver based on a reflected light pulse generated by irradiating the driver with the optical pulse and a second image corresponding to the light quantity distribution of a partial component of the reflected light pulse; and determining the consciousness state of the driver based on the first image and the second image.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] The present disclosure relates to an information processing method in a computer and a biometric measurement system. [Background technology]

[0002] Various methods are known for measuring biological signals resulting from brain activity of a subject.

[0003] For example, Patent Document 1 discloses a technology for presenting visual stimuli to a consumer while acquiring gaze data and biometric data unrelated to the eyes from the consumer in an unconstrained manner, and evaluating the consumer's response based on the data.

[0004] Patent Document 2 discloses an example of an imaging device that obtains information indicating changes over time in cerebral blood flow of a subject without coming into contact with the subject. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2008 / 030542 [Patent Document 2] JP 2017-009584 A [Non-patent literature]

[0006] [Non-Patent Document 1] ASHIT TALUKDER et al., "A Real-time Non-Intrusive Eyetracking and Gaze-point Determination for Human-Computer Interaction and Biomedicine", SPIE Defense and Security Symposium, Optical Patter Recognition XV, Orlando, FL, April 12-16, 2004 [Non-Patent Document 2] Hirokazu Doi et al., "NIRS as a tool for assaying emotional function in the prefrontal cortex", Front Hum Neurosci. 2013 [Non-Patent Document 3] Suda M et al., "Decreased cortical reactivity underlies subjective daytime light sleepiness in healthy subjects: a multichannel near-infrared spectroscopy study", Neurosci Res. 60: 319-326, 2008 [Non-Patent Document 4] Suda M et al., "Subjective feeling of psychological fatigue is related to decreased reactivity in ventrolateral prefrontal cortex", Brain Res. 1252 152-160, 2009 [Non-Patent Document 5] Mototaka Yoshioka et al., "Brain signal pattern of engrossed subjects using near infrared spectroscopy (NIRS) and its application to TV commercial evaluation", IJCNN 2012: 1-6 Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure provides a technology for non-contact acquisition of information indicating a subject's facial appearance and information indicating the state of cerebral blood flow using a single device, and estimating the subject's condition based on that information. [Means for solving the problem]

[0008] An information processing method in a computer according to one embodiment of the present disclosure includes causing a light source to emit a light pulse to be irradiated to a driver of a vehicle, causing an image sensor to output a first image showing the appearance of the driver's face based on a reflected light pulse generated by the light pulse being irradiated to the driver, and a second image corresponding to the light intensity distribution of a portion of the reflected light pulse, and determining the driver's state of consciousness based on the first image and the second image.

[0009] A comprehensive or specific aspect of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. A computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be arranged in one device, or may be arranged separately in two or more separate devices. In this specification and the claims, "apparatus" may mean not only one device, but also a system consisting of multiple devices. Effect of the Invention

[0010] According to the technology disclosed herein, it is possible to obtain information indicating the facial appearance of a subject and information indicating the state of cerebral blood flow in a non-contact manner using a single device, and to estimate the subject's condition based on that information. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1A is a diagram showing a schematic configuration of a bioinstrumentation device according to an exemplary embodiment of the present disclosure. [Figure 1B]FIG. 1B is a schematic diagram illustrating a bioinstrumentation system according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram showing an example of the change over time in the intensity of light reaching an image sensor. [Diagram 3] FIG. 3 is a graph showing the dependency of the amount of light detected by an image sensor on the width of an input light pulse. [Figure 4A] FIG. 4A is a diagram showing an example of a schematic configuration of one pixel of an image sensor. [Figure 4B] FIG. 4B is a diagram showing an example of the configuration of an image sensor. [Diagram 5] FIG. 5 is a diagram showing an example of an operation within one frame. [Figure 6A] FIG. 6A is a flowchart showing an overview of the control of the light source and the image sensor. [Figure 6B] FIG. 6B is a flowchart showing an example in which the signal processing circuit processes image data acquired from the image sensor while changing the resolution. [Figure 6C] FIG. 6C is a flowchart showing an example of a process in which the image sensor outputs image data by changing the frame rate, and the signal processing circuit generates moving image data. [Figure 7] FIG. 7 is a diagram showing an example of an optical signal in which a rectangular optical pulse is emitted and the light returned from the user reaches an image sensor. [Figure 8] FIG. 8 is a diagram showing another example of an optical signal in which a rectangular optical pulse is emitted and the light returned from the user reaches the image sensor. [Figure 9A] FIG. 9A is a diagram showing an example of a timing chart when detecting a surface reflection component. [Figure 9B] FIG. 9B is a diagram showing an example of a timing chart when the internal scattering component is detected. [Figure 10A] FIG. 10A is a diagram showing a schematic example of the change over time in cerebral blood flow. [Figure 10B] FIG. 10B is a diagram illustrating an example in which measurements are simultaneously performed at multiple locations within a target part of a user. [Figure 11A] FIG. 11A is a diagram illustrating an example of a light irradiation region. [Figure 11B] FIG. 11B is a diagram showing a schematic diagram of a change in a signal when the target part of the user is shifted in the lateral direction. [Figure 12A] FIG. 12A is a flowchart showing an example of the operation of the interest determination process. [Figure 12B] FIG. 12B is a flowchart showing an example of the process of step S205 in FIG. 12A. [Figure 12C] FIG. 12C is a flowchart showing another example of the process of step S205 in FIG. 12A. [Figure 13] FIG. 13 is a graph showing an example of changes over time in the concentrations of oxygenated hemoglobin and deoxygenated hemoglobin in cerebral blood. [Figure 14A] FIG. 14A is a flowchart showing an example of a process for identifying an object in which a user is interested within content. [Figure 14B] FIG. 14B is a flowchart showing details of the operation of step S403 in FIG. 14A. [Figure 15] FIG. 15 is a flowchart showing an example of a process for changing the details of content depending on the user's interest. [Figure 16] FIG. 16 is a diagram showing an example of how the bioinstrumentation device is used. [Figure 17] FIG. 17 is a diagram showing an example of a system that changes the content presented to a user depending on the user's subject of interest. [Figure 18] FIG. 18 is a diagram showing an embodiment of a head mounted display equipped with a biometric device. [Figure 19] FIG. 19 shows an embodiment of a smartphone equipped with a biometric measurement device. [Figure 20] FIG. 20 is a diagram illustrating a situation in which a user uses a bioinstrumentation device mounted on a vehicle. [Figure 21]FIG. 21 is a diagram showing an example in which a patient uses the bioinstrumentation device on the bed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present disclosure includes a biomeasurement device, a biomeasurement method, a computer-readable recording medium, and a program described in the following items.

[0013] [Item 1] The bioinstrumentation device according to the first aspect of the present invention includes a light source that emits light pulses to be irradiated to a target part including a head of a subject; an image sensor that receives a reflected light pulse generated by irradiating the target portion with the light pulse, and outputs first image data indicating the appearance of the face of the target portion and second image data corresponding to a light amount distribution of a portion of the reflected light pulse; a control circuit for controlling the light source and the image sensor; A signal processing circuit; Equipped with.

[0014] The control circuit includes: causing the light source to repeatedly emit the light pulses; causing the image sensor to output the first image data; causing the image sensor to output the second image data; The signal processing circuit detects the change over time of the first image data and the change over time of the second image data. Based on the changes over time, data indicative of the subject's condition is generated and output.

[0015] [Item 2] In the biomeasurement device relating to the first item, the control circuit may generate the second image data by causing the image sensor to detect components of the reflected light pulse after the start of a fall period, which is the period from when the intensity of the reflected light pulse starts to decrease to when the decrease ends, and during a period that includes at least a portion of the fall period.

[0016] [Item 3] In the biomeasurement device relating to the second item, the control circuit may generate the first image data by causing the image sensor to detect components of the reflected light pulse in a period including at least a portion of a period prior to the start of the falling period of the reflected light pulse.

[0017] [Item 4] In the bioinstrumentation device according to any one of the first to third aspects, a resolution of the first image data and a resolution of the second image data may be different.

[0018] [Item 5] In the bioinstrumentation device according to any one of the first to fourth aspects, a resolution of the first image data may be higher than a resolution of the second image data.

[0019] [Item 6] In the bioinstrumentation device according to any one of the first to third items, the signal processing circuit further performs a process of changing at least one selected from the group consisting of a resolution of at least a part of an image represented by the first image data and a resolution of at least a part of an image represented by the second image data; The signal processing circuit may generate the data indicating a condition of the subject based on the change over time of the first image data and the change over time of the second image data after the processing.

[0020] [Item 7] In the bioinstrumentation device according to any one of items 1 to 6, the image sensor outputs the first image data at a first frame rate; the image sensor outputs the second image data at a second frame rate; The first frame rate and the second frame rate may be different.

[0021] [Item 8] In the bioinstrumentation device according to any one of items 1 to 7, the image sensor outputs the first image data at a first frame rate; the image sensor outputs the second image data at a second frame rate; The first frame rate may be higher than the second frame rate.

[0022] [Item 9] In the bioinstrumentation device according to any one of items 1 to 8, The image sensor includes a plurality of photodetection cells arranged two-dimensionally; Each of the plurality of photodetection cells includes a photoelectric conversion element, a first charge accumulation unit, and a second charge accumulation unit; The control circuit includes: causing the first charge storage unit to store a first charge that is a source of the first image data; The second charge storage section may store second charges that are the source of the second image data.

[0023] [Item 10] In the bioinstrumentation device according to any one of items 1 to 9, The signal processing circuit includes: detect, based on the change over time of the first image data, a change over time in appearance information indicating at least one selected from the group consisting of a gaze of the subject, a pupil size of the subject, a blink frequency of the subject, a blink time interval of the subject, and a facial expression of the subject; The data indicating a condition of the subject may be generated based on the change over time of the appearance information and the change over time of the second image data.

[0024] [Item 11] In the bioinstrumentation device according to any one of items 1 to 10, The control circuit causes the light source to emit the light pulses and causes the image sensor to generate the first image data and the second image data while a stimulus is being applied to the subject; The data indicating the state of the subject may indicate at least one state selected from the group consisting of the subject's interest in the stimulus, the subject's pleasure, the subject's sleepiness, and the subject's concentration.

[0025] [Item 12] In the bioinstrumentation device according to any one of the first to eleventh items, the signal processing circuit may present the data indicating the state of the subject to the subject via an information device.

[0026] [Item 13] The bioinstrumentation method according to the thirteenth item is as follows: causing a light source to repeatedly emit light pulses that are irradiated onto a target portion including the head of the subject; providing an image sensor with a reflected light pulse resulting from irradiating the target portion with the light pulse; causing the image sensor to output first image data indicative of a facial appearance of the subject; causing the image sensor to output second image data corresponding to a light amount distribution of a portion of the reflected light pulse; generating and outputting data indicative of a state of the subject based on the change over time of the first image data and the change over time of the second image data; Includes.

[0027] [Item 14] A computer-readable recording medium according to the fourteenth aspect of the present invention comprises: A computer-readable recording medium storing a program for measuring a condition of a subject, When the program is executed by the computer, causing a light source to repeatedly emit light pulses that are irradiated onto a target portion including the subject's head; providing an image sensor with a reflected light pulse resulting from irradiating the target portion with the light pulse; causing the image sensor to output first image data indicative of a facial appearance of the subject; causing the image sensor to output second image data corresponding to a light amount distribution of a portion of the reflected light pulse; generating and outputting data indicative of a state of the subject based on the change over time of the first image data and the change over time of the second image data; is executed.

[0028] [Item 15] The program related to item 15 is as follows: causing a light source to repeatedly emit light pulses that are irradiated onto a target portion including the head of the subject; providing an image sensor with a reflected light pulse resulting from irradiating the target portion with the light pulse; causing the image sensor to output first image data indicative of a facial appearance of the subject; causing the image sensor to output second image data corresponding to a light amount distribution of a portion of the reflected light pulse; generating and outputting data indicative of a state of the subject based on the change over time of the first image data and the change over time of the second image data; to be executed by the computer.

[0029] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection forms of the components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components. Each figure is a schematic diagram and is not necessarily illustrated strictly. Furthermore, in each figure, substantially the same or similar components are given the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0030] In the present disclosure, all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram may be implemented by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Here, although it is called an LSI or an IC, the name may change depending on the degree of integration, and it may be called a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the manufacture of the LSI, or a reconfigurable logic device, which can reconfigure the junction relationship inside the LSI or set up the circuit partition inside the LSI, can also be used for the same purpose.

[0031] Furthermore, all or part of the functions or operations of a circuit, unit, device, member, or section can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device includes one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces. This is also fine.

[0032] First, an example of a basic configuration of a bioinstrumentation device according to an embodiment of the present disclosure will be described.

[0033] Fig. 1A is a diagram showing a schematic configuration of a biometric measurement device 100 according to an exemplary embodiment of the present disclosure. Fig. 1A also shows a subject of biometric measurement, i.e., a user 400 of the biometric measurement device 100.

[0034] The bioinstrumentation device 100 includes a light source 20, an image sensor 30, a control circuit 60, and a signal processing circuit 70. The light source 20 emits a light pulse that is irradiated onto a target portion including the subject's head. The image sensor 30 receives a reflected light pulse generated by irradiating the target portion with the light pulse, and outputs image data. The control circuit 60 controls the light source 20 and the image sensor 30. The signal processing circuit 70 processes the image data output from the image sensor 30, and generates and outputs a signal related to the subject's brain activity. The control circuit 60 and the signal processing circuit 70 may be realized by a single integrated electric circuit.

[0035] The control circuit 60 performs the following operations. (1) The light source 20 is caused to repeatedly emit a light pulse. (2) The image sensor 30 is caused to output first image data indicating the appearance of the subject's face, and (3) The image sensor 30 is caused to output second image data corresponding to the light amount distribution of some components of the reflected light pulse.

[0036] The signal processing circuit 70 generates and outputs data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data. The data indicating the state of the subject reflects, for example, the psychological or physical state of the subject, as will be described in detail later. The data may indicate, for example, at least one state selected from the group consisting of interest, pleasure, drowsiness, and concentration with respect to the stimulus given to the subject. The data indicating the state of the subject output from the signal processing circuit 70 may be used, for example, to control other equipment.

[0037] With the above configuration, the bioinstrumentation device 100 can obtain information on the subject's facial appearance and information on the state of cerebral blood flow in a non-contact manner using a single device. Furthermore, based on the obtained information, the bioinstrumentation device 100 can estimate the subject's psychological or physical state.

[0038] The first image data and the second image data can be generated, for example, by the following method.

[0039] The first image data may be generated based on the components of the reflected light pulse in a period including at least a part of the period before the start of the falling period of the reflected light pulse. Here, the "falling period" refers to the period from when the intensity of the light pulse starts to decrease to when the decrease ends at the position of the light receiving surface of the image sensor 30. The control circuit 60 can generate the first image data by causing the image sensor 30 to detect the components of the reflected light pulse in a period including at least a part of the period before the start of the falling period of the reflected light pulse. The "period including at least a part of the period before the start of the falling period of the reflected light pulse" may include the entire period during which the reflected light pulse is incident on the image sensor 30.

[0040] The first image data can also be generated based on light other than the light pulse emitted from the light source 20. For example, the first image data may be data of a face image captured under light from a lighting device other than the light source 20 or under background light such as sunlight.

[0041] The second image data may be generated based on, for example, a component of the reflected light pulse included in the falling period of the reflected light pulse. The control circuit 60 can cause the image sensor 30 to generate the second image data by detecting a component of the reflected light pulse in a period after the start of the falling period and including at least a part of the falling period.

[0042] As will be described in detail later, the intensity of the component after the start of the falling period in the reflected light pulse, i.e., the trailing component of the pulse, fluctuates due to the subject's brain activity. Based on the fluctuating component, the psychological or physical state of the subject can be estimated.

[0043] According to the above configuration, the first image data and the second image data can be generated in a non-contact manner using one image sensor 30. There is no need to provide multiple image sensors, and a low-cost, space-saving bioinstrumentation device 100 can be configured. By using this image sensor, it is possible to eliminate the need for synchronous control of multiple image sensors. Also, when the light pulse emitted from the light source 20 is, for example, an infrared pulse, it is possible to suppress signal interference caused by the reflected infrared light pulse being incident on another image sensor for generating a face image.

[0044] The image sensor 30 may be controlled to output the first image data at a first frame rate and the second image data at a second frame rate. The first frame rate may be higher than, the same as, or lower than the second frame rate. Changes in facial appearance are often faster than changes in cerebral blood flow. Thus, when the first frame rate is higher than the second frame rate, more frames can be devoted to relatively fast-changing appearance information. Also, even if the second frame rate is lower than the first frame rate, it often does not affect processing because changes in cerebral blood flow are relatively slow.

[0045] The first image data and the second image data may have different resolutions. By increasing the resolution of the image that is of particular interest, data capacity can be effectively utilized.

[0046] The resolution of the first image data may be higher than the resolution of the data of the second image. The first image data having a high resolution makes it easier to capture changes in the appearance of the face. The first image data may be image data in which edges are emphasized or edges are extracted. The first image data may show an image in which only the appearance information to be focused on is extracted. For example, when focusing on gaze or blinking, the first image data may show an image of only one eye, or may show an image of both eyes. By limiting the area to be focused on, the amount of data can be reduced and the data processing speed can be increased. When capturing an image, the subject may be wearing vision correction devices or eyewear such as glasses or contact lenses.

[0047] The first image data may be generated based on at least one of the light pulses irradiated to the target portion. The second image data may represent an image with a lower resolution than the first image data. By lowering the resolution of the second image data, the amount of data can be reduced and the data processing speed can be increased. In order to reduce the resolution, the second image data may be smoothed using a spatial filter of 50×50 size or the like. By smoothing, noise contained in weak cerebral blood flow signals can be reduced. In addition, the amount of data may be reduced by processing to reduce the number of gradations (i.e., the number of bits) of the image. As another method, the amount of data may be reduced by generating an image with a lower resolution by spatially thinning out pixels from the image or by resizing the image. The data processing speed can be increased by reducing the number of gradations or the number of pixels.

[0048] The resolution of each of the first image data and the second image data may be changed by the signal processing circuit 70 during the signal processing, or may be changed by the control circuit 60 controlling the light source 20 and / or the image sensor 22. This may be changed by adjusting the operation or output of the sensor 30 .

[0049] The image sensor may include a plurality of photodetection cells arranged two-dimensionally. Each of the plurality of photodetection cells may include a photoelectric conversion element, a first charge accumulation unit, and a second charge accumulation unit. The control circuit 60 causes the light source 20 to perform, for example, the following operations. (a) A light pulse is emitted. (b) A first charge generated by a component of the reflected light pulse being incident on a photoelectric conversion element during a period including at least a portion of a period before the start of a falling period of the reflected light pulse is accumulated in a first charge accumulation section in at least a portion of the light detection cells among the plurality of light detection cells. (c) causing a second charge accumulation unit in at least some of the light detection cells to accumulate a second charge generated when a component of the reflected light pulse is incident on a photoelectric conversion element during a period after start of the falling period of the reflected light pulse and including at least a portion of the falling period. (d) causing the image sensor to generate first image data based on the first charges accumulated in the first charge accumulation portions of the at least some of the light detection cells; (e) causing the image sensor to generate second image data based on the second charges accumulated in the second charge accumulation portions of the at least some of the light detection cells;

[0050] The above operations (a) to (c) may be repeated multiple times. In this case, in the above operations (d) and (e), the image sensor generates one frame of first image data based on the first charge accumulated multiple times in the first charge accumulation section, and generates one frame of second image data based on the second charge accumulated multiple times in the second charge accumulation section.

[0051] According to this configuration, the first image data and the second image data can be generated efficiently.

[0052] The signal processing circuit 70 can detect a change in the subject's appearance based on the change over time of the first image data. A known recognition algorithm, for example, can be used to detect the change in appearance. The signal processing circuit 70 may detect a change over time of appearance information indicating at least one selected from the group consisting of the subject's gaze, pupil size, blink frequency, blink time interval, and facial expression. The signal processing circuit 70 can generate data indicating the subject's condition based on the change over time of the appearance information and the change over time of the second image data.

[0053] The control circuit 60 may cause the light source 20 to emit light pulses and the image sensor 30 to generate the first image data and the second image data in a state where, for example, a visual or auditory stimulus is given to the subject. The data indicating the state of the subject may indicate the degree of at least one state selected from the group consisting of interest, pleasant feeling, sleepiness, and concentration level of the subject with respect to the stimulus.

[0054] The signal processing circuit 70 or the control circuit 60 may present information determined according to the subject's condition to the subject via an information device.

[0055] Hereinafter, the embodiments of the present disclosure will be described in more detail. In the following description, the same or similar components are denoted by the same reference numerals.

[0056] (Embodiment) [1. Biometrics system] FIG. 1B is a schematic diagram illustrating a bioinstrumentation system according to an exemplary embodiment of the present disclosure. The body measurement system includes a biomeasurement device 100 and a server 200. The biomeasurement device 100 includes a stimulation device 10, a light source 20, an image sensor 30, a control circuit 60, a signal processing circuit 70, a communication circuit 80, and a recording medium 90. The server 200 is a computer located at a location different from that of the biomeasurement device 100. The server 200 can be connected to the biomeasurement device 100 via a network such as a local area network (LAN) or the Internet.

[0057] The stimulation device 10 is a device that provides a stimulus, such as a visual or auditory stimulus, to a user who is a subject. The stimulation device 10 may be, for example, a display, a speaker, or other electronic device. The stimulation device 10 may be an external element of the biomeasurement device 100. The light source 20 emits light pulses that are irradiated onto a target portion including the head and face of the user. The light source 20 is not limited to a single light-emitting device, and may be realized by a combination of multiple light-emitting devices. The image sensor 30 detects at least a portion of the light pulses returned from the target portion of the user, and outputs image data. The image sensor 30 includes multiple pixels. Each of the multiple pixels includes a photoelectric conversion element 32 and one or more charge storage units 34. The signal processing circuit 70 performs various processes based on the image data output from the image sensor 30. The communication circuit 80 includes, for example, a network interface controller, and communicates with an external device, for example, a server 200. The recording medium 90 includes memories such as a RAM and a ROM. The recording medium 90 stores a program that defines the processing executed by the control circuit 60 and the signal processing circuit 70, and various data generated during the processing. The control circuit 60 is connected to the stimulation device 10, the light source 20, the image sensor 30, the signal processing circuit 70, the communication circuit 80, and the recording medium 90. The control circuit 60 controls the overall operation of the biomeasurement device 100.

[0058] The control circuit 60 in this embodiment includes a light source control unit 61 that controls the light source 20, a sensor control unit 62 that controls the image sensor 30, and a stimulus control unit 63 that controls the stimulator 10. The light source control unit 61, the sensor control unit 62, and the stimulus control unit 63 may be realized by three separate circuits, or may be realized by a single circuit. Each of the light source control unit 61, the sensor control unit 62, and the stimulus control unit 63 may be realized by the control circuit 60 executing a control program stored in the recording medium 90, for example, a memory.

[0059] The light source control unit 61 controls the intensity, pulse width, emission timing, and / or wavelength of the light pulse emitted from the light source 20. The sensor control unit 62 controls the timing of signal accumulation in each pixel of the image sensor 30. The stimulus control unit 63 controls the content and timing of the stimulus applied by the stimulus device 10. The stimulus control unit 63 controls, for example, at least one of the hue, saturation, and luminance of the image applied as the stimulus, or at least one of the sound quality and volume of the sound.

[0060] The signal processing circuit 70 generates data indicating the state of the subject based on the image data output from the image sensor 30. The signal processing circuit 70 can transmit the data to the server 200 via the communication circuit 80. The signal processing circuit 70 can also read data stored in the server 200 via the communication circuit 80. The control circuit 60 can determine the content of the stimulation to be applied by the stimulation device 10 based on the data generated by the signal processing circuit 70.

[0061] The server 200 accumulates data generated by the signal processing circuit 70 and data of candidate contents to be presented to the user. The candidate contents may include at least one of information of text, video, and audio, for example.

[0062] In this specification, "biological information" refers to a measurable quantity of a living body that changes in response to a stimulus. Biological information includes various quantities, such as blood flow, blood pressure, heart rate, pulse rate, respiratory rate, body temperature, brain waves, oxygenated hemoglobin concentration in blood, deoxygenated hemoglobin concentration in blood, blood oxygen saturation, and skin reflectance spectrum. Some biological information is sometimes called vital signs.

[0063] Each component of the bioinstrumentation device 100 will be described in more detail below.

[0064] [1-1. Stimulator 10] The stimulation device 10 applies a stimulus to a user. The stimulation device 10 may be configured to apply a stimulus to multiple users. The stimulus applied from the stimulation device 10 induces a biological response in the user. The stimulation device 10 may present information, such as content determined based on the biological response of the user, to the user or a person other than the user. In the example of FIG. 1B, the bioinstrumentation device 100 includes the stimulation device 10, but a part or the whole of the stimulation device 10 may be provided outside the bioinstrumentation device 100.

[0065] The stimulation device 10 may be, for example, a head-mounted device, a goggle and headset device, or a device with a display such as a smartphone. The stimulation device 10 may be an audio device, a lighting device, or an air conditioning device. The stimulation device 10 may include a plurality of devices that provide different stimuli. The stimulation device 10 may provide the user with at least one of the following stimuli: an image, a character, a sound such as music or a voice, brightness, heat, a cold sensation, wetness, dryness, vibration, or wind. The image and the character are stimuli for the sense of sight. The sound is a stimulus for the sense of hearing. The stimulation device 10 with a display may provide the user with image, video, or audio content as a stimulus. The visual stimulus may be, for example, a web advertisement, a video, or a game. Various tasks such as calculation problems, language problems, puzzles, or quizzes may be provided as visual stimuli. The tasks may be specially created to diagnose the state of the subject's brain activity. The stimulation device 10 may output a sound associated with the task simultaneously with the presentation of the task. Besides video or audio content, the visual stimulus may be a change in brightness or color of the lighting in a room.

[0066] In addition to visual or auditory stimuli, tactile, olfactory, or gustatory stimuli may be provided. The stimulator 10 has different structures and functions depending on the type of stimuli to be provided to the user. For example, when providing a tactile stimulus to the user, the stimulator 10 may be a device that generates vibration or heat. When providing an olfactory stimulus to the user, the stimulator 10 may be a device that generates an odor.

[0067] [1-2.Light source 20] The light source 20 irradiates light on a target part including the user's head, for example, the forehead. The light emitted from the light source 20 and reaching the user is divided into a surface reflection component I1 reflected on the surface of the user and an internal scattering component I2 scattered inside the user. The internal scattering component I2 is a component that is reflected or scattered once or multiple-scattered inside the living body. When light is emitted toward the user's head, the internal scattering component I2 refers to a component that reaches a part about 8 mm to 16 mm deep from the surface of the user's head, for example, the brain, and returns to the bioinstrumentation device 100 again. The surface reflection component I1 includes three components: a direct reflection component, a diffuse reflection component, and a diffuse reflection component. The direct reflection component is a reflection component whose incidence angle and reflection angle are equal. The diffuse reflection component is a component that is diffusely reflected due to the uneven shape of the surface. The diffuse reflection component is a component that is scattered and reflected by internal tissue near the surface. When light is emitted toward the user's head, the diffuse reflection component is a component that is scattered and reflected inside the epidermis. The surface reflection component I1 reflected by the surface of the user may include these three components. The surface reflection component I1 and the internal scattering component I2 change direction due to reflection or scattering, and some of them reach the image sensor 30.

[0068] In this embodiment, a surface reflection component I1 and an internal scattering component I2 are detected from the reflected light returning from the user's head. The surface reflection component I1 reflects the appearance of the user's face. Therefore, by analyzing the change over time of the surface reflection component I1, it is possible to estimate the change in the appearance of the user's face. On the other hand, the intensity of the internal scattering component I2 fluctuates, reflecting the user's brain activity. Therefore, by analyzing the change over time of the internal scattering component I2, it is possible to estimate the state of the user's brain activity.

[0069] First, a method for acquiring the internal scattering component I2 will be described. The light source 20 repeatedly emits a light pulse multiple times at a predetermined time interval or at a predetermined timing according to an instruction from the control circuit 60. The light pulse emitted from the light source 20 may be, for example, a rectangular wave with a fall period close to zero. In this specification, the "fall period" means the period from when the intensity of the light pulse starts to decrease until the decrease ends. In general, light incident on a user propagates through the user via various paths and is emitted from the surface of the user with a time difference. For this reason, the rear end of the internal scattering component I2 of the light pulse has a spread. When the target part of the user is the forehead, the spread of the rear end of the internal scattering component I2 is about 4 ns. In consideration of this, the fall period of the light pulse can be set to, for example, 2 ns or less, which is half of that. The fall period may be further half that, 1 ns or less. The length of the rise period of the light pulse emitted from the light source 20 is arbitrary. The "rise period" is the period from when the intensity of the light pulse starts to increase until the increase ends. In the present embodiment, the internal scattering component I2 is detected using the falling edge of the light pulse, and not the rising edge. The rising edge of the light pulse can be used to detect the surface reflection component I1. The light source 20 can be, for example, a laser such as an LD. The light emitted from the laser has a steep time response characteristic in which the falling edge of the light pulse is approximately perpendicular to the time axis.

[0070] The wavelength of the light emitted from the light source 20 may be any wavelength within the wavelength range of, for example, 650 nm or more and 950 nm or less. This wavelength range is included in the wavelength range from red to near-infrared. The above wavelength range is called the "biological window" and has the property that light is relatively difficult to absorb by moisture and skin in a living body. When a living body is to be detected, the detection sensitivity can be increased by using light within the above wavelength range. When detecting changes in blood flow in the user's brain as in this embodiment, the light used is mainly oxygenated hemoglobin (HbO 2It is believed that light is absorbed by oxygenated hemoglobin (Hb) and deoxygenated hemoglobin (Hb). The wavelength dependence of light absorption differs between oxygenated hemoglobin and deoxygenated hemoglobin. In general, when there is a change in blood flow, the concentrations of oxygenated hemoglobin and deoxygenated hemoglobin change. This change also changes the degree of light absorption. Therefore, when blood flow changes, the amount of light detected also changes over time.

[0071] The light source 20 may emit light of a single wavelength included in the above wavelength range, or may emit light of two or more wavelengths. Light of multiple wavelengths may be emitted from multiple light sources, respectively.

[0072] In general, biological tissues have different absorption and scattering characteristics depending on the wavelength. Therefore, by detecting the wavelength dependence of the optical signal due to the internal scattering component I2, more detailed component analysis of the measured object is possible. For example, in biological tissues, oxygenated hemoglobin (HbO 2 ) is greater than the absorbance due to deoxyhemoglobin (Hb). On the other hand, the opposite characteristics are observed at wavelengths of 805 nm or less. Therefore, for example, the light source 20 may be configured to emit light with a wavelength near 750 nm and light with a wavelength near 850 nm. In this case, the light intensity of the internal scattering component I2 due to the light with a wavelength near 750 nm and the light intensity of the internal scattering component I2 due to the light with a wavelength near 850 nm are measured. The signal processing circuit 70 calculates a signal value of the light intensity input for each pixel by calculating a predetermined simultaneous equation. By solving the equation, HbO in blood 2 The amount of change from the initial value for each of the concentrations of Hb and Ig can be calculated.

[0073] In the bioinstrumentation device 100 of this embodiment, the user's cerebral blood flow is measured in a non-contact manner. For this reason, a light source 20 designed in consideration of the effect on the retina may be used. For example, a light source 20 that satisfies Class 1 of the laser safety standards established in various countries may be used. When Class 1 is satisfied, light with such low illuminance that the exposed emission limit (AEL) is below 1 mW is irradiated onto the user. Note that the light source 20 itself does not have to satisfy Class 1. For example, the laser safety standard Class 1 may be satisfied by disposing a diffusion plate or an ND filter in front of the light source 20 to diffuse or attenuate the light.

[0074] Conventionally, streak cameras have been used to distinguish and detect information such as absorption coefficients or scattering coefficients at different locations in the depth direction inside a living body. For example, Japanese Patent Application Laid-Open No. 4-189349 discloses an example of such a streak camera. In these streak cameras, ultrashort light pulses with a pulse width of femtoseconds or picoseconds are used to measure with a desired spatial resolution.

[0075] In contrast, the bioinstrumentation device 100 of this embodiment can distinguish and detect the surface reflection component I1 and the internal scattering component I2. Therefore, the light pulse emitted by the light source 20 does not need to be an ultrashort light pulse, and the pulse width can be selected arbitrarily.

[0076] When illuminating the user's head with light to measure cerebral blood flow, the amount of light of the internal scattering component I2 can be an extremely small value, about one thousandth to one tens of thousandsth of the amount of light of the surface reflection component I1. Furthermore, considering the safety standards of lasers, the amount of light that can be irradiated becomes extremely small. Therefore, it is very difficult to detect the internal scattering component I2. Even in this case, if the light source 20 emits a light pulse with a relatively large pulse width, it is possible to increase the accumulated amount of the internal scattering component I2, which is accompanied by a time delay. This increases the amount of detected light and improves the signal-to-noise ratio.

[0077] The light source 20 emits a light pulse with a pulse width of, for example, 3 ns or more. In general, the time spread of light scattered in biological tissues such as the brain is about 4 ns. FIG. 2 is a diagram showing an example of the time change in the intensity of light reaching the image sensor 30. FIG. 2 shows three examples in which the width of the input light pulse emitted from the light source 20 is 0 ns, 3 ns, and 10 ns. As shown in FIG. 2, as the width of the light pulse from the light source 20 is widened, the amount of light of the internal scattering component I2 that appears at the rear end of the light pulse returned from the user increases.

[0078] FIG. 3 is a diagram in which the width of the input light pulse is plotted on the horizontal axis and the amount of light detected by the image sensor 30 is plotted on the vertical axis. The image sensor 30 is equipped with an electronic shutter. The results in FIG. 3 were obtained under the condition that the electronic shutter was opened 1 ns after the trailing end of the light pulse was reflected by the surface of the user and reached the image sensor 30. The reason for selecting this condition is that immediately after the trailing end of the light pulse arrives, the ratio of the surface reflection component I1 is high compared to the internal scattering component I2. As shown in FIG. 3, when the pulse width of the light pulse emitted from the light source 20 is set to 3 ns or more, the amount of detected light can be maximized.

[0079] The light source 20 may emit a light pulse with a pulse width of 5 ns or more, or even 10 ns or more. On the other hand, if the pulse width is too large, the amount of unused light increases and is wasted. For this reason, the light source 20 emits a light pulse with a pulse width of, for example, 50 ns or less. Alternatively, the light source 20 may emit a light pulse with a pulse width of 30 ns or less, or even 20 ns or less.

[0080] The irradiation pattern of the light source 20 is, for example, a pattern having a uniform intensity distribution within the irradiation area. In this respect, the present embodiment is different from the conventional bioinstrumentation device disclosed in, for example, Japanese Patent Application Laid-Open No. 11-164826. In the device disclosed in Japanese Patent Application Laid-Open No. 11-164826, the image sensor and the light source are separated by about 3 cm, and the surface reflection component is spatially separated from the internal scattering component. For this reason, discrete light irradiation is unavoidable. In contrast, the bioinstrumentation device 100 of the present embodiment can temporally separate and reduce the surface reflection component I1 from the internal scattering component I2. For this reason, a light source 20 with an irradiation pattern having a uniform intensity distribution can be used. The irradiation pattern having a uniform intensity distribution may be formed by diffusing the light emitted by the light source 20 with a diffusion plate.

[0081] In this embodiment, unlike the conventional technology, the internal scattering component I2 can be detected even directly under the irradiation point of the user. By irradiating the user with light over a spatially wide range, the measurement resolution can be improved.

[0082] [1-3.Image sensor 30] The image sensor 30 detects at least a portion of the reflected light pulse returning from the user's head for each pixel. The image sensor 30 outputs a plurality of signals corresponding to the intensity of the detected light for each pixel. The plurality of signals include a signal corresponding to the intensity included in at least a portion of the rising period of the reflected light pulse and a signal corresponding to the intensity included in at least a portion of the falling period.

[0083] The image sensor 30 includes a plurality of photodetection cells arranged two-dimensionally, and can obtain two-dimensional information of a user at once. Each photodetection cell includes a photoelectric conversion element and one or more charge storage units. In this specification, the photodetection cells are also referred to as "pixels." The image sensor 30 can be any imaging element, such as a CCD image sensor or a CMOS image sensor.

[0084] The image sensor 30 includes an electronic shutter. The electronic shutter is a circuit that controls the timing of imaging. In this embodiment, the sensor control unit 62 in the control circuit 60 has the function of the electronic shutter. The electronic shutter controls the period of one signal accumulation in which the received light is converted into an effective electric signal and accumulated, and the period in which the signal accumulation is stopped. The signal accumulation period can also be called an "exposure period." In the following description, the width of the exposure period may be called a "shutter width." The time from the end of one exposure period to the start of the next exposure period may be called a "non-exposure period." Hereinafter, the exposed state may be called "OPEN," and the stopped exposure state may be called "CLOSE."

[0085] The image sensor 30 can adjust the exposure period and non-exposure period by an electronic shutter in the range of sub-nanoseconds, for example, 30 ps to 1 ns. A conventional TOF camera for distance measurement detects all light emitted from the light source 20 and reflected by the subject and returned in order to measure the distance regardless of the brightness of the subject. Therefore, in the conventional TOF camera, the shutter width needs to be larger than the pulse width of the light. In contrast, in the bioinstrumentation device 100 of this embodiment, it is not necessary to correct the amount of light of the subject. Therefore, it is not necessary for the shutter width to be larger than the pulse width. The shutter width can be set to a value of, for example, 1 ns or more and 30 ns or less. According to the bioinstrumentation device 100 of this embodiment, the shutter width can be reduced, so that the influence of the dark current included in the detection signal can be reduced.

[0086] When detecting information such as cerebral blood flow by irradiating a user's forehead with light, the attenuation rate of light inside the living body is very large. For example, the outgoing light may attenuate to about one millionth of the incoming light. For this reason, the amount of light may be insufficient to detect the internal scattering component I2 with only one pulse of irradiation. The amount of light is particularly weak in irradiation under Class 1 laser safety standards. In this case, the light source 20 emits multiple light pulses, and the image sensor 30 also electronically shuts down in response. By exposing the detector multiple times, the detection signals can be integrated to improve sensitivity.

[0087] An example of the configuration of the image sensor 30 will now be described.

[0088] The image sensor 30 may include a plurality of pixels arranged two-dimensionally on the imaging surface. Each pixel may include a photoelectric conversion element such as a photodiode, and one or more charge accumulation units. Hereinafter, an example will be described in which each pixel includes a photoelectric conversion element that generates a signal charge according to the amount of received light by photoelectric conversion, a charge accumulation unit that accumulates a signal charge generated by a surface reflection component I1 of the light pulse, and a charge accumulation unit that accumulates a signal charge generated by an internal scattering component I2 of the light pulse. In the following example, the control circuit 60 causes the image sensor 30 to detect the surface reflection component I1 by detecting a portion of the light pulse returned from the user's head before the start of the fall. The control circuit 60 also causes the image sensor 30 to detect the internal scattering component I2 by detecting a portion of the light pulse returned from the user's head after the start of the fall. The light source 20 in this example emits light of two types of wavelengths.

[0089] Fig. 4A is a diagram showing a schematic configuration example of one pixel 201 of the image sensor 30. Note that Fig. 4A shows the configuration of one pixel 201 in a schematic manner and does not necessarily reflect an actual structure. In this example, the pixel 201 includes a photodiode 203 that performs photoelectric conversion, a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207 that are charge storage sections, and a drain 202 that discharges signal charges.

[0090] Photons incident on each pixel due to the emission of one light pulse are converted into signal electrons, which are signal charges, by the photodiode 203. The converted signal electrons are either discharged to the drain 202 or distributed to any one of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, in accordance with a control signal input from the control circuit 60.

[0091] The emission of a light pulse from the light source 20, the accumulation of signal charges in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, and the discharge of the signal charges to the drain 202 are repeated in this order. This repetitive operation is fast and can be repeated tens of thousands to hundreds of millions of times within the time of one frame of a moving image (for example, about 1 / 30 seconds). The pixel 201 ultimately generates and outputs four image signals based on the signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207.

[0092] In this example, the control circuit 60 causes the light source 20 to repeatedly emit a first light pulse having a first wavelength and a second light pulse having a second wavelength in sequence. The user's condition can be analyzed by selecting two wavelengths having different absorption rates for the user's internal tissue as the first wavelength and the second wavelength. For example, a wavelength longer than 805 nm may be selected as the first wavelength, and a wavelength shorter than 805 nm may be selected as the second wavelength. This makes it possible to detect changes in the oxygenated hemoglobin concentration and the deoxygenated hemoglobin concentration in the user's blood.

[0093] As shown in Fig. 5, the control circuit 60 first causes the light source 20 to emit a first optical pulse. The control circuit 60 causes the first floating diffusion layer 204 to accumulate signal charge during a first period in which a surface reflection component I1 of the first optical pulse is incident on the photodiode 203. Next, the control circuit 60 causes the second floating diffusion layer 205 to accumulate signal charge during a second period in which an internal scattering component I2 of the first optical pulse is incident on the photodiode 203. Next, the control circuit 60 causes light source 20 to emit a second optical pulse. Control circuit 60 causes third floating diffusion layer 206 to accumulate signal charge during a third period in which surface reflection component I1 of the second optical pulse is incident on photodiode 203. Next, control circuit 60 causes fourth floating diffusion layer 207 to accumulate signal charge during a fourth period in which internal scattering component I2 of the second optical pulse is incident on photodiode 203.

[0094] In this way, the control circuit 60 sequentially accumulates the signal charge from the photodiode 203 in the first floating diffusion layer 204 and the second floating diffusion layer 205 with a predetermined time difference after starting the emission of the first light pulse. After that, the control circuit 60 sequentially accumulates the signal charge from the photodiode 203 in the third floating diffusion layer 206 and the fourth floating diffusion layer 207 with the predetermined time difference after starting the emission of the second light pulse. The above operation is repeated multiple times. In order to estimate the amount of disturbance light and environmental light, a period may be provided in which signal charge is accumulated in other floating diffusion layers (not shown) with the light source 20 turned off. By subtracting the signal charge amount of the other floating diffusion layers from the signal charge amount of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, a signal from which disturbance light and environmental light components have been removed can be obtained.

[0095] In this embodiment, the number of charge accumulation units is set to four, but may be set to two or more depending on the purpose. For example, when only one type of wavelength is used, the number of charge accumulation units may be two. In addition, when only one type of wavelength is used and the surface reflection component I1 is not detected, the number of charge accumulation units per pixel may be one. Even when two or more types of wavelengths are used, the number of charge accumulation units may be one if imaging using each wavelength is performed in a different frame. When the detection of the surface reflection component I1 and the detection of the internal scattering component I2 are performed in different frames, the number of charge accumulation units may be one.

[0096] FIG. 4B is a diagram showing an example of the configuration of the image sensor 30. In FIG. 4B, an area surrounded by a two-dot chain line frame corresponds to one pixel 201. The pixel 201 includes one photodiode. Although FIG. 4B shows only four pixels arranged in two rows and two columns, a larger number of pixels may be arranged in practice. The pixel 201 includes a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207. The signals accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are treated as if they were signals of four pixels of a general CMOS image sensor, and are output from the image sensor 30.

[0097] Each pixel 201 has four signal detection circuits. Each signal detection circuit includes a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. In this example, the reset transistor 310 corresponds to the drain 202 shown in FIG. 4A, and a pulse input to the gate of the reset transistor 310 corresponds to a drain discharge pulse. Each transistor is, for example, but not limited to, a field effect transistor formed on a semiconductor substrate. As shown in the figure, one of the input terminal and output terminal (typically a source) of the source follower transistor 309 is connected to one of the input terminal and output terminal (typically a drain) of the row selection transistor 308. The gate, which is the control terminal of the source follower transistor 309, is connected to the photodiode 203. The signal charge (i.e., holes or electrons) generated by the photodiode 203 is accumulated in a floating diffusion layer, which is a charge accumulation portion between the photodiode 203 and the source follower transistor 309.

[0098] The first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are connected to the photodiode 203. A switch may be provided between the photodiode 203 and each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. This switch switches the conduction state between the photodiode 203 and each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 in response to a signal accumulation pulse from the control circuit 60. This controls the start and stop of accumulation of signal charges in each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The electronic shutter in this embodiment has a mechanism for such exposure control.

[0099] The signal charges stored in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are read out by turning on the gate of the row selection transistor 308 by the row selection circuit 302. At this time, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified according to the signal potential of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The analog signal due to this current read out from the vertical signal line 304 is converted into digital signal data by an analog-digital (AD) conversion circuit 307 connected for each column. This digital signal data is read out for each column by the column selection circuit 303 and output from the image sensor 30. After reading out one row, the row selection circuit 302 and the column selection circuit 303 read out the next row, and so on, reading out information on the signal charges of the floating diffusion layers of all rows. After reading out all the signal charges, the control circuit 60 resets all the floating diffusion layers by turning on the gate of the reset transistor 310. This completes the imaging of one frame. Similarly, high-speed imaging of frames is repeated thereafter, completing the imaging of a series of frames by the image sensor 30.

[0100] In this embodiment, an example of the CMOS type image sensor 30 has been described, but the image sensor 30 may be another type of imaging element. The image sensor 30 may be, for example, a CCD type, a single photon counting element, or an amplified image sensor (for example, an EMCCD or ICCD).

[0101] Fig. 5 is a diagram showing an example of the operation in one frame in this embodiment. As shown in Fig. 5, the emission of the first light pulse and the emission of the second light pulse may be alternately switched multiple times in one frame. In this way, the time difference between the acquisition timing of the detection images by the two types of wavelengths can be reduced, and even if a user is moving, it is possible to capture images by the first and second light pulses almost simultaneously.

[0102] In this embodiment, the image sensor 30 detects both the surface reflection component I1 and the internal scattering component I2 of the light pulse. From the temporal or spatial change of the surface reflection component I1, the first biometric information of the user can be obtained. The first biometric information indicates the appearance of the user's face, and may be information on, for example, gaze, pupil diameter, blinking, or facial expression. Meanwhile, from the temporal or spatial change of the internal scattering component I2, the second biometric information of the user, that is, brain activity information, can be obtained.

[0103] In this specification, a signal representing the first biological information may be referred to as a "first biological signal." Also, a signal representing brain activity information may be referred to as a "brain activity signal."

[0104] [1-4. Control circuit 60 and signal processing circuit 70] The control circuit 60 adjusts the time difference between the emission timing of the light pulse of the light source 20 and the shutter timing of the image sensor 30. In this specification, this time difference is sometimes referred to as a "phase difference." The "emission timing" of the light source 20 is the timing at which the light pulse emitted from the light source 20 starts to rise. The "shutter timing" is the timing at which exposure starts. The control circuit 60 may adjust the phase difference by changing the emission timing, or may adjust the phase difference by changing the shutter timing.

[0105] The control circuit 60 may be configured to remove an offset component from a signal detected by each pixel of the image sensor 30. The offset component is a signal component due to ambient light such as sunlight or fluorescent light, or disturbance light. The offset component due to ambient light or disturbance light is estimated by detecting a signal by the image sensor 30 in a state in which the driving of the light source 20 is turned off and no light is emitted from the light source 20.

[0106] The control circuit 60 may be, for example, a combination of a processor and a memory, or an integrated circuit such as a microcontroller having a built-in processor and memory. The control circuit 60 adjusts, for example, the emission timing and the shutter timing by, for example, the processor executing a program recorded in the memory.

[0107] The signal processing circuit 70 is a circuit that processes the image signal output from the image sensor 30. The signal processing circuit 70 performs arithmetic processing such as image processing. The signal processing circuit 70 can be realized by, for example, a digital signal processor (DSP), a programmable logic device (PLD) such as a field programmable gate array (FPGA), or a combination of a central processing unit (CPU) or an image processing arithmetic processor (GPU) and a computer program. The control circuit 60 and the signal processing circuit 70 may be an integrated single circuit, or may be separate individual circuits. The signal processing circuit 70 may be a component of an external device such as a server provided in a remote location. In this case, the external device such as the server transmits and receives data to and from the light source 20, the image sensor 30, and the control circuit 60 by wireless communication or wired communication.

[0108] The signal processing circuit 70 in this embodiment can generate moving image data showing a time-dependent change in cerebral blood flow and a time-dependent change in facial appearance based on image data output from the image sensor 30 for each frame. The signal processing circuit 70 is not limited to such moving image data, and may generate other information. For example, by synchronizing with other devices, biological information such as blood flow rate, blood pressure, blood oxygen saturation, or heart rate in the brain may be generated. The signal processing circuit 70 may estimate an offset component due to ambient light and remove the offset component.

[0109] It is known that there is a close relationship between changes in blood components such as cerebral blood flow or hemoglobin and human neural activity. For example, changes in neural cell activity according to a person's interest level cause changes in cerebral blood flow or blood components. Therefore, if biological information such as cerebral blood flow or facial appearance information can be measured, the user's psychological state or physical state can be estimated. The user's psychological state can be, for example, mood, emotion, health state, or temperature sensation. The mood can include, for example, a pleasant or unpleasant mood. The emotion can include, for example, a feeling of relief, anxiety, sadness, or anger. The health state can include, for example, a state of vitality or fatigue. The temperature sensation can include, for example, a feeling of hot, cold, or muggy. Derived from these, indicators that represent the degree of brain activity, such as interest level, proficiency level, mastery level, and concentration level, can also be included in the psychological state. In addition, physical states such as fatigue level, drowsiness, or the degree of intoxication due to drinking are also included in the subject of estimation by the signal processing circuit 70. The signal processing circuit 70 can estimate the user's psychological or physical state based on changes in cerebral blood flow state and changes in facial appearance, and output a signal indicating the estimation result.

[0110] 6A is a flowchart showing an outline of the control of the light source 20 and the image sensor 30 by the control circuit 60. For simplicity, the operation will be described here when light of one wavelength is used to detect the surface reflection component I1 and the internal scattering component I2 of the reflected light for each pixel. In this example, each pixel of the image sensor 30 includes a first charge storage section that stores charge due to the surface reflection component I1, and a second charge storage section that stores charge due to the internal scattering component I2.

[0111] In step S101, the control circuit 60 first causes the light source 20 to emit a light pulse for a predetermined period of time. At this time, the electronic shutter of the image sensor 30 is in a state where exposure is stopped. The control circuit 60 causes the electronic shutter to stop exposure until the light pulse is reflected by the surface of the user and begins to reach the image sensor 30.

[0112] Next, in step S102, the control circuit 60 causes the electronic shutter to start exposure at a predetermined timing between when the reflected light pulse starts to reach the image sensor 30 and when the falling period starts. This exposure is called the "first exposure". The timing to start the first exposure can be appropriately set for each pixel by measuring the distance to the target part for each pixel in advance. The start timing of the first exposure may be different for each pixel depending on the degree of curvature of the surface of the target part, or may be the same for all pixels. The light detected by the first exposure is mainly light that is scattered on the surface of the skin of the target part and reaches the image sensor 30.

[0113] After a predetermined time has elapsed, in step S103, the control circuit 60 stops the first exposure with the electronic shutter. The timing of stopping the exposure can be, for example, before the falling period of the reflected light pulse starts.

[0114] Next, in step S104, the control circuit 60 causes the electronic shutter to start a second exposure at the timing when a part of the light pulse is scattered inside the user and reaches the image sensor 30. More specifically, the control circuit 60 causes the second exposure to start after the start of the falling period of the reflected light pulse. The start timing of the second exposure can also be calculated based on the distance to the target portion measured in advance for each pixel. The start timing of the second exposure can also be different for each pixel depending on the degree of curvature of the surface of the target portion, or can be the same for all pixels.

[0115] After a predetermined time has elapsed, in step S105, the control circuit 60 stops the second exposure with the electronic shutter. The time length of the first exposure and the time length of the second exposure may be the same or different. In general, the amount of light of the surface reflection component I1 detected in the first exposure is greater than the amount of light of the internal scattering component I2 detected in the second exposure. Therefore, the time length of the first exposure may be set shorter than the time length of the second exposure.

[0116] Next, in step S106, the control circuit 60 judges whether the number of times the signal accumulation has been performed reaches a predetermined number. If this judgment is No, steps S101 to S105 are repeated until it is judged as Yes. This number is set to an appropriate number depending on the detection sensitivity of the internal scattering component I2. If it is judged as Yes in step S106, in step S107, the control circuit 60 causes the image sensor 30 to generate an image signal based on the signal charge accumulated in each charge accumulation section. The image sensor 30 outputs first image data based on the charge accumulated in the first charge accumulation section of each pixel and second image data based on the charge accumulated in the second charge accumulation section of each pixel.

[0117] By the above operation, it is possible to detect with high sensitivity the components of light scattered near the surface of the target part and the components of light scattered inside the target part. Note that multiple light emission and exposure are not essential and are performed as needed.

[0118] The signal processing circuit 70 performs necessary image processing such as color correction, pixel interpolation, or frame interpolation on the first image data to generate first moving image data showing changes in the appearance of the user's face. The signal processing circuit 70 also performs necessary image processing on the second image data to generate second moving image data showing changes in the state of the user's cerebral blood flow. The signal processing circuit 70 further estimates the user's psychological state or physical state based on the first moving image data and the second moving image data. For example, the user's state of interest or concentration can be estimated based on changes in facial expression or gaze estimated from the first moving image data and changes in brain activity estimated from the second moving image data. Details of these processes will be described later.

[0119] The signal processing circuit 70 may perform a process to change the resolution of at least one of the first image data and the second image data. For example, the first image data may be processed so as to have a higher resolution than the second image data. The process to change the resolution may be performed only on a part of each image. That is, the signal processing circuit may perform a process to change the resolution of at least a part of the image represented by the first image data and / or the resolution of at least a part of the image represented by the second image data. In this case, the signal processing circuit 70 generates data indicating the state of the subject based on the time-dependent changes of the first image data and the second image data after the process. An example of such a process will be described below.

[0120] FIG. 6B is a flowchart showing an example of a process for changing the resolution of the first image data and the second image data. In this example, the signal processing circuit 70 changes the resolution of the first and second image data by executing the processes of steps S108 to S114 shown in FIG. 6B. Through the above processes, the signal processing circuit 70 can make the first image data have a higher resolution than the second image data. The operation of each step will be described below.

[0121] In step S108, the signal processing circuit 70 acquires the first image data and the second image data generated by the image sensor 30.

[0122] In step S109, the signal processing circuit 70 selects a necessary region from the first image data. For example, when acquiring gaze data, a region near the eyeball is selected.

[0123] In step S110, the signal processing circuit 70 performs high resolution processing on the data of the selected region of the first image data. For example, by performing a known super-resolution processing, the first image data is processed so as to have a higher resolution than the second image data. Note that the signal processing circuit 70 may perform processing to reduce the resolution of the first image data. In this case, the first image data may have a higher resolution than the second image data by suppressing the rate of reduction in the resolution of the first image data.

[0124] In step S111, the signal processing circuit 70 selects a necessary region from the second image data. For example, a part of the subject's forehead is selected.

[0125] In step S112, the signal processing circuit 70 performs low-resolution processing on the second image data. To reduce the resolution, signal values ​​of neighboring pixels may be averaged. By performing the average processing, noise contained in weak cerebral blood flow signals can be reduced.

[0126] In step S113, the signal processing circuit 70 outputs the first image data and the second image data after the processing. For example, the signal processing circuit 70 records the first image data and the second image data after the processing on the recording medium 90.

[0127] In step S114, the signal processing circuit 70 judges whether the processing is completed. If the judgment is No, the signal processing circuit 70 repeats step S108 until it judges Yes. The process is repeated from S113. The determination as to whether the process is completed may be made based on, for example, whether the output of image data by the image sensor 30 is completed or whether a stop command has been received from the user. Alternatively, the determination as to whether the process is completed may be made based on, for example, whether the elapsed time from the start of measurement has reached a predetermined time or whether the amount of accumulated data from the start of measurement has reached a predetermined amount of data.

[0128] The signal processing circuit 70 may change the resolution of only one of the first image data and the second image data. When the resolution of the first image data is increased and the resolution of the second image data is decreased, the amount of data can be reduced and high-resolution facial appearance information can be obtained at the same time.

[0129] The first frame rate at which the image sensor 30 outputs the first image data and the second frame rate at which the image sensor 30 outputs the second image data may be different.

[0130] FIG. 6C is a flowchart showing an outline of a process in which the image sensor 30 changes the frame rate to output image data, and the signal processing circuit 70 generates moving image data.

[0131] In step S115, the image sensor 30 calculates a first Image data is output at a high frame rate. In this case, in order to increase the frame rate, the control circuit 60 adjusts the shutter timing of the image sensor 30 to shorten the integrated exposure time within one frame. In order to increase the frame rate, the control circuit 60 adjusts the light emission timing of the light source 20 to lengthen the duration of one light emission and reduce the number of times light is emitted within one frame. By performing continuous light emission within one frame, the number of light emissions within one frame can be reduced to 1. It may be set to 10 times.

[0132] In order to obtain facial appearance information, the light source 20 may be an LED. To obtain facial appearance information, a laser light such as an LD in which the trailing edge of the optical pulse is approximately perpendicular to the time axis is not necessarily required, and a steep time response characteristic is not essential.

[0133] In step S116, signal processing circuit 70 generates and outputs facial appearance data based on the first image data output in step S115.

[0134] In step S117, the image sensor 30 calculates a second scattering component I2 based on the internal scattering component I2. The image data is output at a low frame rate. In this case, in order to lower the frame rate, the control circuit 60 adjusts the shutter timing of the image sensor 30 to make the integrated exposure time within one frame longer than when the first image data was acquired.

[0135] In step S118, the signal processing circuit 70 generates cerebral blood flow data based on the second image data acquired in step S117.

[0136] In addition, instead of the image sensor 30 outputting the second image data at a low frame rate, the image sensor 30 may output all frames at a high frame rate, and then the signal processing circuit 70 may subsequently accumulate multiple sets of image data and output them as a single set of image data.

[0137] The image sensor 30 and the signal processing circuit 70 repeat the processes of steps S115 to S118 until receiving an instruction to end the measurement in step S119. Note that the number of repetitions may differ between the first image and the second image.

[0138] [1-5. Control of the server 200 and the stimulator 10] The biomeasurement device 100 in this embodiment may be used in cooperation with an external server 200. The server 200 includes a storage device that stores content such as video or audio, and data of games, tests, or tasks. The server 200 also includes a communication circuit that communicates with the communication circuit 80 of the biomeasurement device 100. The server 200 also stores video data and application data that may be provided to a user, as well as diagnostic data of brain activity generated by the signal processing circuit 70. Some or all of the functions of the server 200 may be built into the biomeasurement device 100. Conversely, some of the functions of the signal processing circuit 70 in the biomeasurement device 100 may be performed by the server 200.

[0139] The control circuit 60 includes a stimulus control unit 63. The stimulus control unit 63 can control the stimulator 10 to provide a stimulus such as a video or sound to the user. The stimulus control unit 63 can control, for example, the hue, saturation, or brightness of the content of the video provided as the stimulus, or the type, sound quality, or volume of the sound.

[0140] The control circuit 60 can determine the next stimulation to be provided to the user based on the user's psychological or physical state estimated by the signal processing circuit 70. For example, when it is determined that a user viewing a certain content is losing interest or concentration, it can determine to display a different content. This determination process may be performed by a processor included in the server 200. The control circuit 60 can obtain necessary data such as video or audio from the server 200 and cause the stimulation device 10 to provide a stimulation based on the data.

[0141] [1-6.Other] The biomeasurement device 100 may include an imaging optical system that forms a two-dimensional image of the user on the light receiving surface of the image sensor 30. The optical axis of the imaging optical system is approximately perpendicular to the light receiving surface of the image sensor 30. The imaging optical system may include a zoom lens. When the position of the zoom lens is changed, the magnification ratio of the two-dimensional image of the user changes, and the resolution of the two-dimensional image on the image sensor 30 changes. Therefore, even if the user is far away, it is possible to enlarge the area to be measured and observe it in detail.

[0142] The bioinstrumentation device 100 may include a bandpass filter between the user and the image sensor 30 that passes only light in or near the band of wavelengths emitted from the light source 20. This can reduce the influence of disturbance components such as ambient light. The bandpass filter is made of a multilayer film filter or an absorption filter. Taking into account the temperature of the light source 20 and band shifts caused by oblique incidence on the filter, the bandwidth of the bandpass filter may be about 20 nm to 100 nm.

[0143] The biomeasurement device 100 may include a polarizing plate between the light source 20 and the user, and between the image sensor 30 and the user. In this case, the polarization directions of the polarizing plate arranged on the light source 20 side and the polarizing plate arranged on the image sensor 30 side are in a crossed Nicol relationship. This makes it possible to prevent the specular reflection component of the surface reflection component I1 of the user, i.e., the component having the same angle of incidence and reflection angle, from reaching the image sensor 30. In other words, the amount of light that the surface reflection component I1 reaches the image sensor 30 can be reduced.

[0144] [2. Time-resolved imaging operation] As described above, the bioinstrumentation device 100 in this embodiment can distinguish and detect the surface reflection component I1 and the internal scattering component I2 of the light irradiated to the target area. This type of imaging is referred to as "time-resolved imaging" in this specification.

[0145] An example of the operation of the bioinstrumentation device 100 in this embodiment will now be described.

[0146] As shown in FIG. 1B, when the light source 20 irradiates a light pulse on the user, a surface reflection component I1 and an internal scattering component I2 are generated. A part of the surface reflection component I1 and the internal scattering component I2 reach the image sensor 30. The internal scattering component I2 passes through the inside of the user before being emitted from the light source 20 and reaching the image sensor 30. That is, the optical path length of the internal scattering component I2 is longer than the optical path length of the surface reflection component I1. Therefore, the internal scattering component I2 reaches the image sensor 30 later on average than the surface reflection component I1.

[0147] The surface reflection component I1 can be detected, for example, by the following operations.

[0148] FIG. 7 is a diagram showing an example of an optical signal in which a rectangular light pulse is emitted from the light source 20 and the light returned from the user reaches the image sensor 30. The horizontal axis represents time (t) in all parts (a) to (d). The vertical axis represents intensity in parts (a) to (c), and the OPEN or CLOSE state of the electronic shutter in part (d). Part (a) shows the surface reflection component I1. Part (b) shows the internal scattering component I2. Part (c) shows the combined component of part (a) showing the surface reflection component I1 and part (b) showing the internal scattering component I2. Part (d) shows the shutter timing for acquiring the user's surface reflection component I1.

[0149] By opening the shutter as shown in part (d) of FIG. 7, it is possible to efficiently collect components of the reflected light incident on the image sensor 30 that arrive early. The components that arrive early mean that there is less scattering at the target part, and contain surface information of the target part. The time during which light is substantially accumulated is a short time at the front end of the pulse wave, but the shutter does not necessarily have to be open for that period. As shown in part (d) of FIG. 7, the shutter may be opened at a stage earlier than the front end of the pulse wave reaches the image sensor 30. In the example shown in part (d) of FIG. 7, the front end part of the reflected light pulse is detected just before the exposure period ends. When such shutter control is adopted, it is not necessary to use an expensive image sensor capable of picosecond-order exposure. The bioinstrumentation device 100 in this embodiment can be configured with an inexpensive image sensor 30.

[0150] 7, the light source 20 emits a rectangular pulse wave. At this time, the pulse width does not need to be on the order of ps, but can be several ns. Therefore, an inexpensive light source can be used.

[0151] In this example, only the leading edge of the reflected light pulse is detected, but the detection method is not limited to this. For example, the exposure period may include the period from the end of the rising period to the start of the falling period. Even with such a detection method, image data showing the appearance of the user's face can be obtained.

[0152] Next, an example of a method for detecting the internal scattering component I2 will be described.

[0153] Fig. 8 is a diagram showing another example of an optical signal in which a rectangular optical pulse is emitted from the light source 20 and the light returned from the user reaches the image sensor 30. Parts (a) to (c) in Fig. 8 show the same time changes as parts (a) to (c) in Fig. 7. Part (d) in Fig. 8 shows the shutter timing for acquiring the internal scattering component I2.

[0154] As shown in part (a) of FIG. 8, the surface reflection component I1 maintains a rectangular shape. On the other hand, as shown in part (b) of FIG. 8, the internal scattering component I2 is a sum of light of various optical path lengths, and therefore exhibits a characteristic of a trailing tail at the rear end of the optical pulse. In other words, the fall period is longer than that of the surface reflection component I1. From the optical signal in part (c) of FIG. 8, the internal scattering component I In order to extract a high proportion of 2, as shown in part (d) of FIG. 8, the electronic shutter starts charge accumulation after the rear end of the surface reflection component I1. After the rear end of the surface reflection component I1 means when or after the surface reflection component I1 falls. This shutter timing is adjusted by the control circuit 60. As described above, the bioinstrumentation device 100 in this embodiment detects the surface reflection component I1 and the internal scattering component I2 separately. Therefore, the light emission pulse width and the shutter width are arbitrary. Therefore, unlike the conventional method using a streak camera, the internal scattering component I2 can be obtained with a simple configuration, and the cost can be reduced.

[0155] 8, the control circuit 60 causes the image sensor 30 to detect at least a part of the component in the falling period of the reflected light pulse, and output a signal indicating a two-dimensional image of the user. In this embodiment, the signal output from the image sensor 30 may include a signal indicating the amount of light of at least a part of the component in the falling period of the reflected light pulse.

[0156] In part (a) of FIG. 8, the rear end of the surface reflection component I1 falls vertically. In other words, the time from when the surface reflection component I1 starts to fall until it ends is zero. However, in reality, the light pulse itself irradiated by the light source 20 may not be completely vertical, the surface of the user may have fine irregularities, or the surface may be scattered within the epidermis, so the rear end of the surface reflection component I1 may not fall vertically. In addition, since the user is an opaque object, the light amount of the surface reflection component I1 is much greater than the light amount of the internal scattering component I2. Therefore, even if the rear end of the surface reflection component I1 slightly protrudes from the vertical falling position, the internal scattering component I2 may be buried. Furthermore, during the readout period of the electronic shutter, a time delay may occur due to the movement of electrons. For the above reasons, it may not be possible to realize an ideal binary readout as shown in part (d) of FIG. 8. In that case, the control circuit 60 may delay the shutter timing of the electronic shutter slightly from immediately after the surface reflection component I1 starts to fall. For example, the shutter timing may be delayed by about 0.5 ns to 5 ns. Instead of adjusting the shutter timing of the electronic shutter, the control circuit 60 may adjust the light emission timing of the light source 20. In other words, the control circuit 60 may adjust the time difference between the shutter timing of the electronic shutter and the light emission timing of the light source 20. If the shutter timing is delayed too much, the internal scattering component I2, which is originally small, will be further reduced. For this reason, the shutter timing may be kept near the rear end of the surface reflection component I1. As described above, since the time delay due to scattering inside the forehead is 4 ns, the maximum amount of delay in the shutter timing may be about 4 ns.

[0157] A signal may be accumulated by exposing each of a plurality of light pulses emitted from the light source 20 at shutter timings with the same time difference, thereby amplifying the detected light amount of the internal scattering component I2.

[0158] Instead of or in addition to placing a bandpass filter between the user and the image sensor 30, the offset component may be estimated by capturing an image for the same exposure period without emitting light from the light source 20. The estimated offset component is removed by subtraction from the signal detected by each pixel of the image sensor 30. This makes it possible to remove the effects of dark current components and / or ambient light generated on the image sensor 30.

[0159] Next, an example of a method for detecting the surface reflection component I1 and the internal scattering component I2 per frame will be described.

[0160] FIG. 9A shows an example of a timing chart for detecting the surface reflection component I1. 9A, for example, the shutter may be opened before the light pulse reaches the image sensor 30, and closed before the trailing edge of the light pulse reaches the image sensor 30. By controlling the shutter in this way, the mixing of the internal scattering component I2 can be reduced. The proportion of light passing through the vicinity of the surface of the user can be increased. In particular, the timing of closing the shutter may be immediately after the light reaches the image sensor 30. This enables signal detection with a higher proportion of the surface reflection component I1, which has a relatively short optical path length. As another method of acquiring the surface reflection component I1, the image sensor 30 may acquire the entire light pulse, or continuous light may be irradiated from the light source 20 to detect it.

[0161] 9B shows an example of a timing chart for detecting the internal scattering component I2. In this example, the shutter is opened after the trailing end of the pulse starts to reach the image sensor 30. By such control, the signal of the internal scattering component I2 can be acquired.

[0162] As in this embodiment, when time-division imaging is performed using the same image sensor, temporal and spatial deviations are unlikely to occur. When acquiring signals of both the surface reflection component I1 and the internal scattering component I2 using the same image sensor, the components to be acquired may be switched for each frame as shown in Figs. 9A and 9B. Alternatively, as described with reference to Figs. 5 and 6A, the components to be acquired within one frame may be alternately switched at high speed. In this case, the detection time difference between the surface reflection component I1 and the internal scattering component I2 can be reduced.

[0163] Furthermore, the signals of the surface reflection component I1 and the internal scattering component I2 may be acquired using light of two wavelengths. When acquiring the surface reflection component I1 and the internal scattering component I2 using two wavelengths, a method of quickly switching between four types of charge accumulation within one frame may be used, as described with reference to Figures 4A to 5. Such a method can reduce the time lag of the detection signals.

[0164] [3. Detection of changes in cerebral blood flow] Next, an example of a method for detecting a change in a user's cerebral blood flow will be described.

[0165] FIG. 10A is a diagram showing a schematic example of the change in cerebral blood flow over time. As shown in FIG. 10A, a target area of ​​a user is irradiated with light from a light source 20, and the return light is detected. In this case, the surface reflection component I1 is much larger than the internal scattering component I2. However, by adjusting the shutter as described above, it is possible to extract only the internal scattering component I2. The graph shown in FIG. 10A shows the change in cerebral blood flow over time as a function of oxygenated hemoglobin (HbO 210A shows an example of the change over time in the concentration of deoxygenated hemoglobin (Hb) and deoxygenated hemoglobin (Hb). In this example, the internal scattering component I2 is acquired using light of two wavelengths. The concentration shown in FIG. 10A shows the change amount based on the amount at normal times. This change amount is calculated by the signal processing circuit 70 based on the light intensity signal. The cerebral blood flow rate changes depending on the brain activity state, such as normal state, concentration state, or relaxation state. For example, there is a difference in brain activity or a difference in absorption coefficient or scattering coefficient for each location in the target area. Therefore, the change over time in the cerebral blood flow is measured at the same position in the target area of ​​the user. When detecting the change over time in the brain activity, it is possible to estimate the user's state from the relative change over time in the cerebral blood flow even if the absolute amount of the cerebral blood flow is not known.

[0166] FIG. 10B is a schematic diagram showing an example of a case where measurements are simultaneously performed at multiple locations in a target area of ​​a user. In this example, a two-dimensional area is imaged, so that a two-dimensional distribution of cerebral blood flow can be obtained. In this case, the irradiation pattern of the light source 20 may be, for example, a uniform distribution of uniform intensity, a dot-shaped distribution, or a doughnut-shaped distribution. With a uniform distribution of uniform intensity, adjustment of the irradiation position of the target area is unnecessary or can be easily performed. With a uniform distribution of irradiation, If the illumination is partial, light is incident on the target part of the user from a wide range. Therefore, the signal detected by the image sensor 30 can be amplified. Furthermore, measurement can be performed at any position within the illumination area. If the illumination is partial, such as a dot-shaped distribution or a doughnut-shaped distribution, the influence of the surface reflection component I1 can be reduced by simply removing the target part from the illumination area.

[0167] FIG. 11A is a diagram showing a schematic example of a light irradiation area 22. In non-contact cerebral blood flow measurement, the amount of detected light attenuates inversely proportional to the square of the distance from the device to the target area. Therefore, the signal of each pixel detected by the image sensor 30 may be amplified by integrating the signals of multiple nearby pixels. In this way, the number of integrated pulses can be reduced while maintaining the signal-to-noise ratio. This allows the frame rate to be improved.

[0168] 11A shows an example in which light is irradiated only on the user's head, but if the same light is used to capture a facial image of the user, the face is also included in the irradiated area. If a facial image is captured using a different light, it is not necessary for the light from light source 20 to irradiate areas other than the head.

[0169] FIG. 11B is a diagram showing a schematic diagram of a change in a signal when the target part of a user is shifted laterally. As described above, a change in brain activity is read by detecting the difference between the cerebral blood flow when the brain activity state changes from the normal state and the cerebral blood flow in the normal state. When an image sensor 30 having a plurality of photoelectric conversion elements arranged two-dimensionally is used, a two-dimensional brain activity distribution can be obtained as shown in the upper part of FIG. 11B. In this case, even if a signal in the normal state is not obtained in advance, a part where brain activity is active can be detected from a relative intensity distribution in the two-dimensional distribution. Since the measurement is performed in a non-contact manner in this embodiment, the position of the target part may change during measurement as shown in the lower part of FIG. 11B. This may occur, for example, when the user moves slightly to breathe. In general, the two-dimensional distribution of cerebral blood flow does not change suddenly within a short time. For this reason, for example, the position shift of the target part can be corrected by pattern matching between frames of the detected two-dimensional distribution. Alternatively, if the movement is periodic, such as breathing, only the frequency component may be extracted and corrected or removed. The target portion does not have to be a single region, but may be multiple regions. The multiple regions may be, for example, one on each side, or a 2×6 matrix of dots.

[0170] [4. Estimating user state] Next, an example of a method for estimating a user's state using the biometric device 100 will be described. The biometric device 100 may be used in a system that provides video or audio content via a network such as the Internet. Such a system may include, for example, a server operated by a business operator and various computers owned by users, such as personal computers (PCs), smartphones, or tablets. The server 200 shown in FIG. 1B may be a server in such a system.

[0171] Such a system can be used by multiple users. Each user uses a biometric device 100. Each user can view content such as applications, videos, or games distributed from a server through the stimulation device 10 using a computer such as a smartphone. The biometric device 100 may be built into or attached externally to a computer owned by the user.

[0172] [4-1. Determining user interests] As an example, a method for determining a user's interest will be described. In this example, the user is viewing video content displayed on a display, while image data showing the appearance of the user's face and image data showing the state of the user's cerebral blood flow are repeatedly generated by the above-mentioned method. In this example, as described with reference to Figs. 4A to 5, two wavelengths of light are used. Using the above, a total of four types of image data based on the surface reflection component I1 and the internal scattering component I2 of the light of each wavelength are generated for each frame. Hereinafter, the two types of image data based on the surface reflection component I1 are referred to as "first image data", and the two types of image data based on the internal scattering component I2 are referred to as "second image data". The signal processing circuit 70 detects changes in the user's line of sight based on the changes over time in the first image data, and detects changes in the user's brain activity based on the second image data. Based on these detection results, the signal processing circuit 70 identifies the timing when the user's interest is high and the location the user is gazing at at that time. This makes it possible to estimate which part of the content the user is watching is of strong interest to the user.

[0173] 12A is a flowchart showing an example of the operation of the interest determination process in this embodiment. In this example, the signal processing circuit 70 executes the processes of steps S201 to S206 shown in FIG.

[0174] In step S201, the signal processing circuit 70 acquires first image data based on the surface reflection component I1. As described above, the first image data is repeatedly output from the image sensor 30 at the first frame rate.

[0175] In step S202, the signal processing circuit 70 generates gaze data J1 based on the first image data. The gaze data J1 is data indicating the direction of the user's gaze. The gaze data J1 may be, for example, data indicating the coordinates of the center of the user's pupil. The coordinates of the center of the user's pupil may be calculated from the positional relationship between the center position of the user's pupil and the Purkinje image, which is a corneal reflection image, using, for example, a known corneal reflection method. There are various methods for calculating the eyeball position using the corneal reflection method. For example, as disclosed in Non-Patent Document 1, a method can be used in which the amount of movement in the horizontal and vertical directions of the center of the pupil as seen from the camera is mapped onto the display plane. In this embodiment, the first image data includes image data using light with a wavelength shorter than 805 nm and image data using light with a wavelength longer than 805 nm. The gaze data may be generated based on only one of these two types of image data, or may be generated based on both.

[0176] In step S203, the signal processing circuit 70 acquires second image data based on the internal scattering component I1. The second image data is repeatedly output from the image sensor 30 at the second frame rate, as described above.

[0177] In step S204, the signal processing circuit 70 generates cerebral blood flow data J2 based on the second image data. The cerebral blood flow data J2 is data indicating the state of the user's cerebral blood flow. The cerebral blood flow data J2 is, for example, oxygenated hemoglobin (HbO 2 In the present embodiment, the second image data includes image data using light with a wavelength shorter than 805 nm and image data using light with a wavelength longer than 805 nm. As described above, for light with a wavelength longer than 805 nm, HbO 2 The absorbance of light by HbO is greater than that by Hb. Conversely, for light with wavelengths shorter than 805 nm, the absorbance of light by Hb is greater than that by HbO. 2Therefore, the amount of HbO in blood can be calculated by solving a predetermined simultaneous equation using the detected light amount values ​​at each pixel. 2 The amount of change from the reference value of each concentration of HbO can be calculated. The data of the amount of change can be used as cerebral blood flow data J2. 2 Alternatively, data indicating the concentration of either Hb or Hb may be used as the cerebral blood flow data J2.

[0178] In step S205, the signal processing circuit 70 determines the user's interest based on the line-of-sight data J1 and the cerebral blood flow data J2, and records data indicating the determination result. The details will be described later with reference to FIG. 12B.

[0179] The signal processing circuit 70 repeats the processes of steps S201 to S205 until an instruction to end the measurement is received in step S206.

[0180] Fig. 12B is a flow chart showing a specific example of the process of step S205. The signal processing circuit 70 executes the processes of steps S301 to S317 shown in Fig. 12B.

[0181] In step S301, the signal processing circuit 70 calculates the movement of the gaze coordinates based on the gaze data J1. The movement of the gaze coordinates may be, for example, the difference between the gaze coordinates and the gaze coordinates at the previous sampling timing.

[0182] In step S303, the signal processing circuit 70 determines whether the difference is less than a threshold value. For example, assuming that the horizontal direction is the X-axis direction and the vertical direction is the Y-axis direction, the signal processing circuit 70 determines whether the difference between the gaze coordinates for each of the X coordinate and the Y coordinate is less than a predetermined threshold value such as ±10.

[0183] If the difference in the gaze position is equal to or greater than the threshold, it is determined that the gaze is moving. In this case, the subsequent processes are omitted and the process proceeds to step S206. On the other hand, if the difference in the gaze position is less than the threshold, the process proceeds to step S305.

[0184] In step S305, the signal processing circuit 70 calculates the dwell time at that gaze position. The dwell time can be calculated by adding the sampling time interval to the dwell time calculated at the previous sampling timing.

[0185] In step S307, the signal processing circuit 70 judges whether the calculated dwell time exceeds a predetermined threshold. If the dwell time is less than the threshold, the subsequent processes are omitted and the process proceeds to step S206. On the other hand, if the dwell time is equal to or greater than the threshold, the process judges that the user is gazing at the location and proceeds to step S313.

[0186] In step S313, the signal processing circuit 70 judges whether the amount of change from the reference value of the cerebral blood flow is equal to or greater than a threshold based on the cerebral blood flow data J2 generated in step S204. For example, a threshold value of the amount of change from the reference value may be set in advance for each of the oxygenated hemoglobin concentration and the deoxygenated hemoglobin concentration in the cerebral blood. The signal processing circuit 70 judges whether the amount of change from the reference value for each of the oxygenated hemoglobin concentration and the deoxygenated hemoglobin concentration in the cerebral blood is equal to or greater than each threshold. If this judgment is No, it is estimated that the user's interest level is low, so the process proceeds to step S206 without performing the subsequent process. If this judgment is Yes, it is estimated that the user's interest level is high, so the process proceeds to step S315.

[0187] Here, an example of the determination process based on the cerebral blood flow data J2 will be described with reference to FIG. 13. FIG. 13 shows an example of the change over time in the concentration of oxygenated hemoglobin (Oxy-Hb) and deoxygenated hemoglobin (Deoxy-Hb) in the cerebral blood. The horizontal axis shows time, and the vertical axis shows the change in each concentration from the reference value. This data was obtained by measuring the blood flow in the frontal lobe using the bioinstrumentation device 100 of this embodiment with the gaze fixed. An experiment was conducted in which an object of interest and an object of no interest were sequentially presented in front of the gaze without moving the gaze. From the results shown in FIG. 13, it can be seen that the tendency of the change over time in the cerebral blood flow is different when looking at an object of interest and when looking at an object of no interest. According to the results shown in FIG. 13, when looking at an object of interest, the concentration of Oxy-Hb tends to increase and the concentration of Deoxy-Hb tends to decrease. Therefore, a user's interest can be determined based on the change from the reference value in one or both of the concentration of Oxy-Hb and Deoxy-Hb. It is possible to estimate the user's level of interest.

[0188] In this embodiment, the presence or absence of interest of the user is estimated, but other psychological or physical states such as fear, sleepiness, pleasure, or fatigue can also be estimated in a similar manner. For example, Non-Patent Document 2 reports that Oxy-Hb increases with unpleasant stimuli, and Oxy-Hb in the right prefrontal cortex (PFC) increases with anxiety due to fear stimuli. Furthermore, it has been reported that right oxy-Hb increases in response to cognitive load due to mental arithmetic tasks, and cerebral blood flow of Oxy-Hb in the orbitofrontal cortex (OFC) increases when a smile is seen. Non-Patent Document 3 discloses that the stronger the subjective sleepiness, the smaller the increase in Oxy-Hb in the dorsolateral frontal lobe (DLPFC) during a verbal fluency task, or the Oxy-Hb decreases, compared to when the subject does not feel sleepy. Non-Patent Document 4 discloses that when activation of the frontal lobe by a verbal fluency task is examined by near-infrared spectroscopy (NIRS), the more fatigued a person is, the smaller the increase in Oxy-Hb in the ventrolateral frontal lobe (VLPFC) on both sides is. Furthermore, Non-Patent Document 5 discloses that when a person is absorbed in a task and concentrates, Oxy-Hb in the frontal cortex decreases. Therefore, the biomeasurement method of this embodiment can be used not only to estimate the degree of interest of the user, but also to estimate other psychological or physical states.

[0189] 12B again, if the determination in step S313 is Yes, the process proceeds to step S315. In step S315, the signal processing circuit 70 acquires time information of the content currently being viewed by the user.

[0190] In step S317, the signal processing circuit 70 integrates the gaze coordinates and the time at that time and stores them in the server 200. As a result, the position of the object in the content that the user is gazing at is associated with the time and stored.

[0191] Note that the operation shown in FIG. 12B is only an example, and various modifications are possible. For example, as shown in FIG. 12C, the process of steps S301 to S307 and the process of step S313 may be interchanged. In the example of FIG. 12C, first, in step S313, it is determined whether the amount of change from the reference value of the cerebral blood flow is equal to or greater than a threshold value. If this determination is No, the subsequent processes are omitted and the process proceeds to step S206. Only if this determination is Yes, the process proceeds to the processes after step S301. Even with such an operation, the same effect can be obtained. In the example of FIG. 12A, the operation of step S205 is performed in real time while the user is watching the content. The operation of step S205 may be performed after all the operations of steps S201 to S204 during watching are completed. In that case, the signal processing circuit 70 may repeatedly perform the operation of FIG. 12A or FIG. 12B at a predetermined sampling interval.

[0192] Through the above operations, it is possible to identify which object the user is interested in among the contents the user is viewing. The server 200 may accumulate data associating the gaze coordinates with the time for each user and for each content. The control circuit 60 in the biometric device 100 of each user may perform control such as changing the stimuli or contents presented to the user based on the data accumulated in the server 200.

[0193] [4-2. Application process after determining user interests 1] Next, an example of the process after determining the user's interests will be described.

[0194] FIG. 14A is a flowchart showing an example of a process for identifying an object in which a user is interested within content.

[0195] In step S401, the signal processing circuit 70 receives content data from the server 200. The data D1 is data of content such as video, audio, applications, games, and assignments to be presented to the user.

[0196] In step S402, the signal processing circuit 70 reads data D2 of the line of sight coordinates and time from the server 200. The data D2 is recorded in advance in the above-mentioned step S317.

[0197] In step S403, the signal processing circuit 70 identifies one or more objects in the content that the user is interested in based on the data D1 and D2. The object may be, for example, a specific person, animal, plant, machine, building, or scenery. The process of step S403 will be described in detail later with reference to FIG. 14B.

[0198] In step S404, the signal processing circuit 70 records in the server 200 the identification of one or more objects in the content in which the user is interested.

[0199] The operation in step S403 will now be described in detail.

[0200] Fig. 14B is a flowchart showing details of the operation of step S403. The signal processing circuit 70 repeats the processes of steps S511 to S514 shown in Fig. 14B for all scenes of the content. After that, the signal processing circuit 70 executes the process of step S521.

[0201] In step S511, the signal processing circuit 70 acquires position information of each object in the content. The object may be, for example, a person, an animal, a plant, or other object appearing in an application or a video. The position information of each object may be included in the data D1 of the content, or may be generated by the signal processing circuit 70 by analyzing the data D1.

[0202] In step S512, the signal processing circuit 70 acquires information on the presentation time of each object. Here, the presentation time indicates the timing at which the object appears in the content. This time information may also be included in the data D1 beforehand, or may be generated by the signal processing circuit 70 by analyzing the data D1.

[0203] In step S513, the signal processing circuit 70 records the object's position information and time information in a unified manner in memory or in the server 200.

[0204] The process of steps S511 to S513 is repeated until it is determined in step S514 that the process has been completed for all scenes in the content. When the process of steps S511 to S513 has been completed for all scenes in the content, the process proceeds to step S521.

[0205] In step S521, the signal processing circuit 70 identifies an object in which the user is interested by comparing the gaze coordinates and time indicated by the data D2 with the position and presentation time of each object. For example, when the gaze coordinates and the object position coordinates are both expressed in the same display coordinate system, the coordinates can be matched by simply comparing them. After step S521 is completed, the process proceeds to step S404 shown in FIG. 14A, where the identification result is recorded.

[0206] [4-3. Application process after determining user interests 2] Next, an example of a process for changing the content presented to a user in accordance with the user's interests will be described.

[0207] In this embodiment, while content such as an application or a video is presented to the user, the biometric device 100 starts generating data showing the appearance of the user's face and data showing the state of cerebral blood flow. If the user's interest is identified based on the generated data, the content of the next application or video to be presented can be appropriately changed according to the interest. For example, if it is found that the user is interested in accommodation while watching a travel introduction video, the content of the video can be changed to focus on accommodation information. On the other hand, if the user does not show any interest while watching the application or video, it is also possible to present the content of a predetermined template.

[0208] In this way, the stimulation given to the user can be controlled based on the video data showing the appearance of the user's face and the video data showing the state of the user's cerebral blood flow. At that time, the stimulation control unit 63 in the control circuit 60 can cause the stimulation device 10 to output at least one of an image and a sound associated with the classification of the biological reaction. The classification of the biological reaction can be, for example, a binary classification such as "interested" and "not interested." Alternatively, the degree of interest or other state can be classified into three or more stages.

[0209] FIG. 15 is a flowchart showing an example of a process for changing the details of content depending on the user's interest.

[0210] In step S601, the signal processing circuit 70 reads information on the content being presented to the user from the server 200. In step S602, the signal processing circuit 70 reads information on the object in which the user is interested, which was recorded in step S404 of Fig. 14A.

[0211] In step S603, the signal processing circuit 70 accesses the server 200 and searches for content of interest. For example, if it is found that the user is interested in real estate, the signal processing circuit 70 searches for content containing real estate information.

[0212] In step S604, the signal processing circuit 70 judges whether or not there is content that corresponds to the user's interest. If there is no content that corresponds to the user's interest, the process proceeds to step S606, where a content of a predefined template is presented. If there is content that corresponds to the user's interest, the process proceeds to step S605, where the content of the user's interest is presented.

[0213] The above operations are repeated until it is determined in step S607 that the content has ended. Through the above operations, appropriate content according to the user's interests can be presented to the user.

[0214] The operations shown in FIG. 15 may be executed after the operations shown in FIG. 14A are completed, or may be executed simultaneously in parallel with the operations shown in FIG. 14A.

[0215] 16 is a schematic diagram showing a situation in which a stimulating device 10 equipped with a network-connectable display provides a user with a video presented by a server 200 as a stimulus. The user is provided with a stimulus such as a video from the stimulating device 10 such as a PC or a television. While the user is watching the video, a bioinstrumentation device 100 built into or externally attached to the PC or television acquires cerebral blood flow information and appearance information of the user.

[0216] FIG. 17 is a diagram showing an example of a system that changes the content presented to a user depending on the user's interests. In this example, the system changes the content presented to a user on a PC, a TV, a tablet, a smartphone, and the like. The biomeasurement device 100, which is built into or attached to an information device such as a head mounted display, the signal processing circuit 70, and the server 200 are disposed in different locations and connected to each other via a network 500. As in this example, the signal processing circuit 70 shown in FIG. 1B may be provided outside the biomeasurement device 100.

[0217] In this example, the brain blood flow information and appearance information acquired from the user by the bio-measurement device 100 are transmitted to the signal processing circuit 70 via the network 500. The signal processing circuit 70 uses the received information to determine the user's interest. Based on the determined interest, the scene in the video to be presented to the user next is determined, and data showing the scene is transmitted to the stimulation device 10 such as a display. The video stimulation presented to the user may change in real time or at regular intervals depending on the user's interest. For example, the video is composed of multiple scenes that are played continuously. For scene 2 and onwards, the content can be determined depending on the classification of the user's interest in the previous scene.

[0218] 5. Other embodiments The above embodiment is merely an example, and various modifications may be made. Below, the following description will focus on the differences from the above-mentioned example of configuration and operation, and a description of the common parts will be omitted.

[0219] [5-1. Understanding assessment based on pupil diameter] For example, the pupil diameter of the user may be detected based on the signal output from the image sensor 30. When the brain is working hard, the pupil diameter expands due to the action of the autonomic nervous system, and when not, it shrinks. Information on the pupil diameter of the user may be acquired by a signal processing circuit 70 equipped with pupil recognition technology. The signal processing circuit 70 may also determine the level of understanding of the user during learning based on the detected pupil diameter and the information indicating the change in cerebral blood flow generated in step S204 shown in FIG. 12A. For example, the biomeasurement device 100 may determine whether the user is keeping up with the conversation by detecting the pupil diameter and the change in cerebral blood flow of the user during a foreign language lesson. If the pupil diameter becomes smaller and no change in cerebral blood flow is detected, it is possible that the user does not understand the content even if he or she nods. In that case, a system can be constructed to inform the instructor of this fact. Alternatively, an information transmission terminal or an AI robot equipped with the biomeasurement device 100 may read the user's level of understanding based on the information on the pupil diameter and the change in cerebral blood flow of the user detected from the image data acquired by the image sensor 30. The technology disclosed herein can be applied to a human-machine interface that dynamically changes the information or conversation content presented to the user depending on the degree of comprehension read. If the user's pupil diameter becomes large, the user may have difficulty comprehending. In that case, the information or conversation may be repeated or slowed down.

[0220] [5-2. Application examples using head-mounted displays or smartphones] The bioinstrumentation device 100 may be incorporated into a device such as a head-mounted display or a smartphone.

[0221] Fig. 18 shows an embodiment of a head mounted display including the biometric device 100. Fig. 19 shows an embodiment of a smartphone including the biometric device 100.

[0222] In the embodiment of the head mounted display or the smartphone, a camera built into or attached to the head mounted display or the smartphone acquires cerebral blood flow information and facial appearance information. The embodiment of the head mounted display or the smartphone can be used as follows.

[0223] For example, based on the gaze data J1 in step S202 of FIG. 12A and the cerebral blood flow data J2 in step S204, a position on the display in which the user is interested may be determined. Based on the information on the identified position, the following control, for example, is possible. Highlighting locations on the display that are of interest to the user. Increase the resolution around the location. - Display a pointer or cursor at that location. Identify an object of interest, either real or virtual, and view detailed information about that object. Feedback information about objects of interest to the information source, for example by recording data indicating the feedback to the server. · Record interest information on the server in conjunction with user information. - Turn off the smartphone or put it into sleep mode if the user is distracted for an extended period of time.

[0224] Alternatively, instead of the gaze data J1 in step S202 in Fig. 12A, information indicating the frequency or time interval of the user's blinking may be used as information indicating a change in facial appearance. The user's fatigue level or concentration level can be estimated based on the information indicating the frequency or time interval of the blinking and the information indicating a change in cerebral blood flow. Based on the estimated fatigue level or concentration level information, for example, the following control is possible. -Display a message encouraging you to take a break. -Display images that have a relaxing effect. -Monitor the user's fatigue level, workload, task difficulty, or task proficiency. - Reduce the brightness of your display. Monitor your proficiency with an application or game. -Turn your smartphone off and on.

[0225] In addition, the drowsiness of the user can be estimated based on information on the frequency or time interval of blinking and information on changes in cerebral blood flow. Based on the information on the estimated drowsiness, for example, the following control is possible. -Turn the head-mounted device OFF or ON. -Display a message prompting you to refresh. -Display images with an awakening effect.

[0226] Furthermore, based on the pupil diameter information and the cerebral blood flow change information, it is possible to obtain information on the degree of interest in the content or on the proper evaluation of the illuminance. Based on the degree of interest in the content or the information on the proper evaluation of the illuminance, it is possible to determine whether to switch the content or not, and to control the illuminance by increasing or decreasing it.

[0227] In the embodiment of the present disclosure, data showing changes over time in face images and data showing changes over time in cerebral blood flow can be obtained. Based on these data, information related to user identification or authentication can also be obtained. Based on the information related to user identification or authentication, for example, the following control is possible. · Select the content you want to display. -If the user is not eligible, the device will not start. - Set a limit on usage time to prevent fatigue accumulation.

[0228] By using not only facial images but also cerebral blood flow information, it is possible to prevent impersonation using silicone masks used in special makeup, photographs, or mannequins.

[0229] [5-3. Examples of in-vehicle applications] The bioinstrumentation device 100 can also be mounted on a vehicle and used. Figure 20 is a diagram showing a schematic diagram of a situation in which a user uses the bioinstrumentation device 100 mounted on a vehicle.

[0230] In this example, the biomeasurement device 100 may be built into the vehicle or may be externally attached. The biomeasurement device 100 may be a device specialized for biomeasurement, or may be built into other devices such as a driving recorder or a car navigation system. The stimulus may be presented to a person other than the user.

[0231] When the biomeasurement device 100 is mounted on a vehicle, the following uses are possible. For example, information on gaze or blinking can be acquired as information indicating a change in facial appearance. Based on the information on gaze or blinking and information on changes in cerebral blood flow, it is possible to acquire information indicating, for example, a sudden illness such as myocardial infarction or stroke, or an abnormal state of consciousness such as drinking alcohol. Based on the information indicating a sudden illness or an abnormal state of consciousness, for example, the following control is possible. Stop the vehicle. · Flashing hazard lights or blue lights on the bus. - Change the bus electronic display to read "Emergency Situation." Switch to autonomous driving and bring the car to a safe stop. -Issue audio messages to call for breaks or attention -Limit the top speed. -Do not run the engine.

[0232] In such an example, the stimulator 10 may be, for example, a car light or an electronic billboard, in which case the stimuli may be presented to a person other than the user.

[0233] Furthermore, information indicating interest in the gaze point can be obtained based on information on pupil diameter or line of sight and information on changes in cerebral blood flow. Based on the information indicating interest in the gaze point, for example, detailed information on the object of the gaze point, such as recommendation information or parking availability information, or a message regarding the object can be displayed.

[0234] In addition, information related to the identification or authentication of the user can be obtained based on the information on changes in the face image and the information on changes in the cerebral blood flow. Based on the information related to the identification or authentication of the user, if an unintended user is trying to drive the vehicle, the device can be prevented from starting or the engine can be stopped.

[0235] [5-4. Examples of application during nursing care or hospitalization] Furthermore, the bioinstrumentation device 100 may be installed on a bed during nursing care or hospitalization. Fig. 21 shows an example in which a subject, i.e., a patient, uses the bioinstrumentation device 100 on the bed.

[0236] In such an example, information on the patient's gaze or facial expression may be obtained as the facial appearance change information. Based on the information on the gaze or facial expression and the cerebral blood flow change information, information indicating the patient's physical condition, such as cognitive state, pain, urge to urinate, or urge to defecate, can be obtained. Based on the information indicating the patient's physical condition, control such as sending information to a caregiver, nurse, or doctor is possible.

[0237] Information on pupil diameter can also be acquired as information on changes in facial appearance. Information on the level of consciousness or vital status can be acquired based on the acquired information on pupil diameter and information on changes in cerebral blood flow. Based on the information on the level of consciousness or vital status, control such as sending information to a caregiver, nurse, or doctor is possible. [Industrial Applicability]

[0238] The bioinstrumentation device according to the present disclosure can be used in various devices such as cameras, measuring devices, or information devices that obtain internal information of a user in a non-contact manner. [Explanation of symbols]

[0239] 10 Stimulator 20 light source 30 Image Sensor 60 Control circuit 70 Signal Processing Circuit 80 Communication Circuit 90 Recording media 100 Biometrics 200 Servers

Claims

1. An information processing method in a computer, comprising: causing a light source to repeatedly emit light pulses that are directed at a driver of the vehicle; outputting, to an image sensor, first image data based on a reflected light pulse generated by irradiating the driver with the light pulse, and second image data corresponding to a light amount distribution of a portion of the reflected light pulse; generating appearance information corresponding to an appearance of the driver's face based on the change over time of the first image data; determining a state of consciousness of the driver based on the appearance information and a time-dependent change in the second image data; A method comprising:

2. 10. The method of claim 1, further comprising flashing hazard lights of the vehicle if the driver's state of consciousness is determined to be abnormal.

3. The method of claim 1 , further comprising limiting a speed of travel of the vehicle if the state of consciousness of the driver is determined to be abnormal.

4. the vehicle is a bus, 2. The method of claim 1, further comprising, when it is determined that the driver's state of consciousness is not normal, displaying a message indicating that an emergency is occurring on an electronic display board of the bus.

5. The method of claim 1 , wherein a resolution of the first image data is greater than a resolution of the second image data.

6. performing a process of reducing the resolution of the second image data; The state of consciousness of the driver is determined based on the first image data and the second image data after the processing. The method of claim 1.

7. A system for performing biometric measurements, comprising: A light source that repeatedly emits light pulses that are directed at the driver of the vehicle; an image sensor that outputs first image data based on a reflected light pulse generated by irradiating the driver with the light pulse, and second image data according to a light amount distribution of a portion of the reflected light pulse; a processing circuit for generating appearance information corresponding to an appearance of the face of the driver based on the change over time of the first image data, and for determining a state of consciousness of the driver based on the appearance information and the change over time of the second image data; Including, the system.

8. An information processing method in a computer, comprising: causing a light source to repeatedly emit light pulses that are directed at a driver of the vehicle; outputting, to an image sensor, first image data showing an appearance of the face of the driver based on a reflected light pulse generated by irradiating the driver with the light pulse, and second image data corresponding to a light amount distribution of a portion of the component of the reflected light pulse; determining a state of consciousness of the driver based on a time-varying appearance shown in a moving image based on the first image data and a time-varying appearance of the second image data; A method comprising:

9. A system for performing biometric measurements, comprising: A light source that repeatedly emits light pulses that are directed at the driver of the vehicle; an image sensor that outputs first image data showing an appearance of the face of the driver based on a reflected light pulse generated by irradiating the driver with the light pulse, and second image data corresponding to a light amount distribution of a portion of the component of the reflected light pulse; a processing circuit for determining a state of consciousness of the driver based on a time-varying appearance shown in a moving image based on the first image data and a time-varying appearance of the second image data; Including, the system.

Images