Information processing method and biometric measurement system in computer

Abstract

To provide a new method for estimating the condition of a subject.SOLUTION: An information processing method in a computer includes: causing a light source to repeatedly emit a light pulse to be irradiated to a user; causing an image sensor to output first image data based on a reflected light pulse generated by irradiating the user with the light pulse and second image data corresponding to a light amount distribution of a part of the reflected light pulse; generating appearance information corresponding to the appearance of the face of the user on the basis of the first image data; and determining the state of the user on the basis of the appearance information and the change over time of the second image data.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present disclosure relates to an information processing method in a computer and a biological measurement system.

Background Art

[0002] Various methods for measuring biological signals resulting from the brain activity of a subject are known.

[0003] For example, Patent Document 1 discloses a technique for acquiring gaze data and biometric data unrelated to the eyeball from a consumer in an unconstrained manner while presenting a visual stimulus to the consumer, and evaluating the consumer's reaction based on the data.

[0004] Patent Document 2 discloses an example of an imaging device that acquires information indicating the change over time of the cerebral blood flow of a subject without contacting the object.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Summary of the Invention

Problems to be Solved by the Invention

[0007] The present disclosure provides a technique for non-contact acquisition, using a single device, of information indicating the appearance of a subject's face and information indicating the state of cerebral blood flow, and for estimating the state of the subject based on such information.

Means for Solving the Problems

[0008] An information processing method in a computer according to one aspect of the present disclosure includes causing a light source to emit an optical pulse that irradiates a driver of a vehicle, and a first image showing an appearance of the face of the driver based on a reflected optical pulse generated when the driver is irradiated with the optical pulse, and causing an image sensor to output a second image corresponding to a light amount distribution of a part of the components of the reflected optical pulse, and determining a conscious state of the driver based on the first image and the second image.

[0009] The comprehensive or specific aspects of the present disclosure may be implemented by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or may be implemented by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). The apparatus may be composed of one or more apparatuses. When the apparatus is composed of two or more apparatuses, the two or more apparatuses may be arranged in one device, or may be divided and arranged in two or more separate devices. In this specification and the claims, the "apparatus" may mean not only one apparatus but also a system composed of a plurality of apparatuses.

Effect of the Invention

[0010] According to the technology of the present disclosure, it is possible to non-contact acquire information showing the appearance of a subject's face and information showing the state of cerebral blood flow using a single device, and estimate the state of the subject based on the information.

Brief Description of the Drawings

[0011]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 10A

Figure 10B

Figure 11A

Figure 11B

Figure 12A

Figure 12B

Figure 12C

Figure 13

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0012] The present disclosure includes a biological measurement device, a biological measurement method, a computer-readable recording medium, and a program described in each of the following items.

[0013] [Item 1] The biological measurement device according to the first item includes a light source that emits a light pulse irradiated onto a target part including the head of a subject, an image sensor that receives a reflected light pulse generated by irradiating the target part with the light pulse, and outputs first image data showing the appearance of the face of the image data, and second image data corresponding to the light quantity distribution of a part of the components of the reflected light pulse, a control circuit that controls the light source and the image sensor, a signal processing circuit, and is provided with.

[0014] The control circuit causes the light source to repeatedly emit the light pulse, causes the image sensor to output the first image data, causes the image sensor to output the second image data, The signal processing circuit generates and outputs data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data.

[0015] [Item 2] In the biological measurement device according to the first item, the control circuit causes the image sensor to detect a component of the reflected light pulse in a period after the start of the decrease in the intensity of the reflected light pulse until the decrease ends, which is a fall period, and the second image data may be generated by detecting the component of the reflected light pulse in a period including at least a part of the fall period after the start of the period.

[0016] [Item 3] In the biological measurement device according to the second item, the control circuit may generate the first image data by causing the image sensor to detect a component of the reflected light pulse in a period including at least a part of the period before the start of the fall period of the reflected light pulse.

[0017] [Item 4] In the biological measurement device according to any one of the first to third items, the resolution of the first image data and the resolution of the second image data may be different.

[0018] [Item 5] In the biological measurement device according to any one of the first to fourth items, the resolution of the first image data may be higher than the resolution of the second image data.

[0019] [Item 6] In the biological measurement device according to any one of the first to third items, the signal processing circuit further performs a process of changing at least one selected from the group consisting of at least a part of the resolution of the image indicated by the first image data and at least a part of the resolution of the image indicated by the second image data, and the signal processing circuit may generate the data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data after performing the process.

[0020] [Item 7] In the biological measurement device according to any one of the first to sixth items, the image sensor outputs the first image data at a first frame rate, the image sensor outputs the second image data at a second frame rate, and the first frame rate and the second frame rate may be different.

[0021] [Item 8] In the biological measurement device according to any one of Items 1 to 7, the image sensor outputs the first image data at a first frame rate, the image sensor outputs the second image data at a second frame rate, the first frame rate may be higher than the second frame rate.

[0022] [Item 9] In the biological measurement device according to any one of Items 1 to 8, the image sensor includes a plurality of light detection cells arranged two-dimensionally, each of the plurality of light detection cells includes a photoelectric conversion element, a first charge storage unit, and a second charge storage unit, the control circuit, accumulates a first charge that is the source of the first image data in the first charge storage unit, and may accumulate a second charge that is the source of the second image data in the second charge storage unit.

[0023] [Item 10] In the biological measurement device according to any one of Items 1 to 9, the signal processing circuit, detects a change over time of appearance information indicating at least one selected from the group consisting of the line of sight of the subject, the size of the pupil of the subject, the frequency of blinking of the subject, the time interval between blinks of the subject, and the expression of the subject based on the change over time of the first image data, and may generate the data indicating the state of the subject based on the change over time of the appearance information and the change over time of the second image data.

[0024] [Item 11] In the biological measurement device according to any one of Items 1 to 10, the control circuit causes the light source to emit the light pulse and causes the image sensor to generate the first image data and the second image data in a state where the subject is being stimulated. The data indicating the state of the subject may indicate at least one state selected from the group consisting of the subject's interest in the stimulus, the subject's pleasure, the subject's drowsiness, and the subject's concentration.

[0025] [Item 12] In the biological measurement device according to any one of Items 1 to 11, the signal processing circuit may present the data indicating the state of the subject to the subject via an information device.

[0026] [Item 13] The biological measurement method according to Item 13 causing a light source to repeatedly emit light pulses irradiated onto a target part including the head of a subject; causing an image sensor to receive reflected light pulses generated by irradiating the light pulses onto the target part; causing the image sensor to output first image data indicating the appearance of the subject's face; causing the image sensor to output second image data corresponding to the light quantity distribution of a part of the components of the reflected light pulses; generating and outputting data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data; and includes.

[0027] [Item 14] The computer-readable recording medium according to Item 14 is a computer-readable recording medium storing a program for measuring the state of a subject, when the program is executed by the computer, causing a light source to repeatedly emit light pulses irradiated onto a target part including the head of the subject; causing an image sensor to receive reflected light pulses generated by irradiating the light pulses onto the target part; Causing the image sensor to output first image data indicating the appearance of the face of the subject; Causing the image sensor to output second image data corresponding to the light quantity distribution of a part of the components of the reflected light pulse; Generating and outputting data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data; are executed.

[0028] [Item 15] The program according to the 15th item Causing a light source to repeatedly emit light pulses that irradiate a target part including the head of a subject; Causing an image sensor to receive a reflected light pulse generated by irradiating the target part with the light pulse; Causing the image sensor to output first image data indicating the appearance of the face of the subject; Causing the image sensor to output second image data corresponding to the light quantity distribution of a part of the components of the reflected light pulse; Generating and outputting data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data; is executed by a computer.

[0029] The embodiments described below all show comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, and the order of the steps shown in the following embodiments are examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, the components not described in the independent claims indicating the most general concept are described as optional components. Each figure is a schematic diagram and is not necessarily drawn precisely. Further, in each figure, substantially the same or similar components are denoted by the same reference numerals. Redundant descriptions may be omitted or simplified.

[0030] In the present disclosure, all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram can be executed by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or a large scale integration (LSI). The LSI or IC may be integrated on one chip or may be configured by combining a plurality of chips. For example, functional blocks other than memory elements may be integrated on one chip. Here, although it is called an LSI or IC, the name may change depending on the degree of integration, and it may be called a system LSI, a very large scale integration (VLSI), or an ultra large scale integration (ULSI). A Field Programmable Gate Array (FPGA) programmed after the manufacture of the LSI, or a reconfigurable logic device capable of reconfiguring the bonding relationship inside the LSI or setting up circuit sections inside the LSI can also be used for the same purpose.

[0031] Furthermore, the functions or operations of all or part of a circuit, unit, device, member or part can be executed by software processing. In this case, the software is recorded on a non-transitory recording medium such as one or more ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified by the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and required hardware devices such as an interface.

[0032] First, an example of the basic configuration of a biological measurement device according to an embodiment of the present disclosure will be described.

[0033] FIG. 1A shows a schematic configuration of a biological measurement device 100 according to an exemplary embodiment of the present disclosure It is a figure. Figure 1A also shows a subject for biological measurement, that is, a user 400 of the biological measurement device 100.

[0034] The biological measurement device 100 includes a light source 20, an image sensor 30, a control circuit 60, and a signal processing circuit 70. The light source 20 emits an optical pulse that irradiates a target part including the head of the subject. The image sensor 30 receives the reflected optical pulse generated by irradiating the target part with the optical pulse and outputs image data. The control circuit 60 controls the light source 20 and the image sensor 30. The signal processing circuit 70 processes the image data output from the image sensor 30 and generates and outputs a signal related to the brain activity of the subject. The control circuit 60 and the signal processing circuit 70 may be realized by an integrated single electric circuit.

[0035] The control circuit 60 executes the following operations. (1) Cause the light source 20 to repeatedly emit an optical pulse. (2) Cause the image sensor 30 to output first image data showing the appearance of the subject's face. (3) Cause the image sensor 30 to output second image data according to the light amount distribution of a part of the components of the reflected optical pulse.

[0036] The signal processing circuit 70 generates and outputs data indicating the state of the subject based on the change over time of the first image data and the change over time of the second image data. The data indicating the state of the subject reflects, for example, the psychological or physical state of the subject as will be described in detail later. The data may indicate at least one state selected from the group consisting of, for example, the interest, pleasure, drowsiness, and concentration of the subject in response to a stimulus given to the subject. The data indicating the state of the subject output from the signal processing circuit 70 can be used, for example, to control other devices.

[0037] With the above configuration, the biological measurement device 100 can non - contactedly acquire information showing the appearance of the subject's face and information showing the state of cerebral blood flow using a single device. Furthermore, based on the acquired information, the psychological or physical state of the subject can be estimated.

[0038] The first image data and the second image data can be generated, for example, by the following method.

[0039] The first image data can be generated, for example, based on the components of the reflected light pulse in a period including at least a part of the period before the start of the falling period of the reflected light pulse. Here, the "falling period" means the period from when the intensity of the light pulse starts to decrease until the decrease ends at the position of the light receiving surface of the image sensor 30. The control circuit 60 can cause the image sensor 30 to generate the first image data by detecting the components of the reflected light pulse in a period including at least a part of the period before the start of the falling period of the reflected light pulse. The "period including at least a part of the period before the start of the falling period of the reflected light pulse" may include the entire period during which the reflected light pulse is incident on the image sensor 30.

[0040] The first image data can also be generated based on light different from the light pulse emitted from the light source 20. For example, the data of a face image taken under light from an illumination device different from the light source 20 or background light such as sunlight may be used as the first image data.

[0041] The second image data can be generated, for example, based on the components of some of the reflected light pulses included in the falling period of the reflected light pulse. The control circuit 60 can cause the image sensor 30 to generate the second image data by detecting the components of the reflected light pulse in a period after the start of the falling period and including at least a part of the falling period.

[0042] As will be described in detail later, the components after the start of the falling period in the reflected light pulse, that is, the trailing edge components of the pulse, vary in intensity due to the brain activity of the subject. Based on the varying components, the mental or physical state of the subject can be estimated.

[0043] According to the above configuration, the first image data and the second image data can be generated non - contact using one image sensor 30. There is no need to provide a plurality of image sensors, and a low - cost and space - saving biological measurement device 100 can be configured. Also, by using one image sensor, synchronization control of a plurality of image sensors can be made unnecessary. Further, when the light pulse emitted from the light source 20 is, for example, an infrared pulse, interference of signals due to the reflected light pulse of the infrared light entering another image sensor for generating a face image can be suppressed.

[0044] The image sensor 30 can be controlled to output the first image data at a first frame rate and the second image data at a second frame rate. The first frame rate may be higher than, the same as, or lower than the second frame rate. Changes in the appearance of the face are often faster than changes in cerebral blood flow. Therefore, when the first frame rate is higher than the second frame rate, more frames can be allocated for appearance information with relatively fast changes. Also, even if the second frame rate is lower than the first frame rate, since changes in cerebral blood flow are relatively slow, it often does not affect the processing.

[0045] The resolutions of the first image data and the second image data may be different. By increasing the resolution of the image that one particularly wants to focus on, effective utilization of the data capacity becomes possible.

[0046] The resolution of the first image data may be higher than that of the second image data. By having the first image data with a high resolution, it becomes easier to capture changes in the appearance of the face. The first image data may be image data in which edges are emphasized or edges are extracted. The first image data may show an image in which only the appearance information to be focused on is extracted. For example, when focusing on the line of sight or blinking, the first image data may show an image of only one eye or an image of both eyes. By limiting the area to be focused on, the amount of data can be reduced and the data processing speed can be increased. At the time of imaging, the subject may wear a vision correction device or eyewear such as glasses or contact lenses.

[0047] The first image data can be generated based on at least one of the light pulses irradiated to the target part. The second image data may represent an image with a lower resolution than the first image data. By reducing the resolution of the second image data, the amount of data can be reduced and the data processing speed can be increased. To reduce the resolution, the second image data may be smoothed using a spatial filter such as a 50×50 size. By smoothing, the noise included in the weak cerebral blood flow signal can be reduced. Also, the amount of data may be reduced by performing a process of reducing the number of gray levels (i.e., the number of bits) of the image. As another method, an image with a low resolution may be generated by thinning out pixels spatially from the image or by performing a resizing process, thereby reducing the amount of data. The data processing speed can be increased by reducing the number of gray levels or the number of pixels.

[0048] The resolution of each of the first image data and the second image data may be changed by the signal processing circuit 70 during the signal processing, or may be changed by the control circuit 60 adjusting the operation or output of the light source 20 and / or the image sensor 30.

[0049] The image sensor may include a plurality of photodetection cells arranged in a two-dimensional array. Each of the plurality of photodetection cells may include a photoelectric conversion element, a first charge storage unit, and a second charge storage unit. The control circuit 60 causes the light source 20 to execute, for example, the following operations. (a) Emit a light pulse. (b) Accumulate a first charge generated by the component of the reflected light pulse incident on the photoelectric conversion element in at least a part of the first charge storage unit in at least a part of the photodetection cells among the plurality of photodetection cells during a period including at least a part of the period before the start of the fall period of the reflected light pulse. (c) Accumulate a second charge generated by the component of the reflected light pulse incident on the photoelectric conversion element in the second charge storage unit in the at least a part of the photodetection cells during a period after the start of the fall period of the reflected light pulse and including at least a part of the fall period. (d) Cause the image sensor to generate first image data based on the first charge accumulated in the first charge storage unit in the at least a part of the photodetection cells. (e) Cause the image sensor to generate second image data based on the second charge accumulated in the second charge storage unit in the at least a part of the photodetection cells.

[0050] The operations (a) to (c) may be repeatedly executed a plurality of times. In that case, in the operations (d) and (e), the image sensor generates first image data for one frame based on the first charge accumulated in the first charge storage unit a plurality of times, and generates second image data for one frame based on the second charge accumulated in the second charge storage unit a plurality of times.

[0051] According to such a configuration, the first image data and the second image data can be efficiently generated.

[0052] The signal processing circuit 70 can detect changes in the appearance of the subject based on the temporal changes in the first image data. For detecting changes in appearance, for example, a known recognition algorithm can be used. The signal processing circuit 70 may detect, for example, temporal changes in appearance information indicating at least one selected from the group consisting of the subject's line of sight, pupil size, blink frequency, blink time interval, and facial expression. The signal processing circuit 70 can generate data indicating the state of the subject based on the temporal changes in the appearance information and the temporal changes in the second image data.

[0053] The control circuit 60 may cause the light source 20 to emit an optical pulse and cause the image sensor 30 to generate the first image data and the second image data in a state where the subject is given a visual or auditory stimulus, for example. The data indicating the state of the subject may indicate the degree of at least one state selected from the group consisting of the subject's interest, pleasure, drowsiness, and concentration with respect to the stimulus.

[0054] The signal processing circuit 70 or the control circuit 60 may present information determined according to the state of the subject to the subject via an information device.

[0055] Hereinafter, embodiments of the present disclosure will be described more specifically. In the following description, the same or similar components are denoted by the same reference numerals.

[0056] (Embodiment) [1. Biometric measurement system] FIG. 1B is a schematic diagram showing a biometric measurement system according to an exemplary embodiment of the present disclosure. The biometric measurement system includes a biometric measurement device 100 and a server 200. The biometric measurement device 100 includes a stimulation device 10, a light source 20, an image sensor 30, a control circuit 60, a signal processing circuit 70, a communication circuit 80, and a recording medium 90. The server 200 is a computer located at a location different from the biometric measurement device 100. The server 200 can be connected to the biometric measurement device 100 via a network such as, for example, a local area network (LAN) or the Internet. The biometric measurement device 100 can be connected.

[0057] The stimulation device 10 is a device that applies stimulation such as visual or auditory stimulation to a user who is the subject. The stimulation device 10 can be, for example, a display, a speaker, or other electronic devices. The stimulation device 10 may be an external element of the biological measurement device 100. The light source 20 emits light pulses that irradiate a target part including the head and face of the user. The light source 20 is not limited to a single light emitting device and may be realized by a combination of a plurality of light emitting devices. The image sensor 30 detects at least a part of the light pulses that have returned from the target part of the user and outputs image data. The image sensor 30 includes a plurality of pixels. Each of the plurality of pixels includes a photoelectric conversion element 32 and one or more charge storage parts 34. The signal processing circuit 70 performs various processes based on the image data output from the image sensor 30. The communication circuit 80 includes, for example, a network interface controller and communicates with an external device, such as the server 200. The recording medium 90 includes memories such as RAM and ROM. The recording medium 90 stores a program that defines the processes executed by the control circuit 60 and the signal processing circuit 70, and various data generated during the process. The control circuit 60 is connected to the stimulation device 10, the light source 20, the image sensor 30, the signal processing circuit 70, the communication circuit 80, and the recording medium 90. The control circuit 60 controls the overall operation of the biological measurement device 100.

[0058] The control circuit 60 in the present embodiment includes a light source control unit 61 that controls the light source 20, a sensor control unit 62 that controls the image sensor 30, and a stimulation control unit 63 that controls the stimulation device 10. The light source control unit 61, the sensor control unit 62, and the stimulation control unit 63 may be realized by three separate circuits or may be realized by a single circuit. Each of the light source control unit 61, the sensor control unit 62, and the stimulation control unit 63 may be realized by the control circuit 60 executing a control program stored in the recording medium 90, for example, a memory.

[0059] The light source control unit 61 controls the intensity, pulse width, emission timing, and / or wavelength of the light pulses emitted from the light source 20. The sensor control unit 62 controls the timing of signal accumulation in each pixel of the image sensor 30. The stimulation control unit 63 controls the content and timing of the stimulation to be applied to the stimulation device 10. The stimulation control unit 63 controls, for example, at least one of the hue, saturation, and brightness of the video image to be applied as the stimulation, or at least one of the sound quality and volume of the audio.

[0060] The signal processing circuit 70 generates data indicating the state of the subject based on the image data output from the image sensor 30. The signal processing circuit 70 can transmit the data to the server 200 via the communication circuit 80. The signal processing circuit 70 can also read out the data stored in the server 200 via the communication circuit 80. The control circuit 60 can determine the content of the stimulation to be applied to the stimulation device 10 based on the data generated by the signal processing circuit 70.

[0061] The server 200 stores the data generated by the signal processing circuit 70 and the data of the candidate content to be presented to the user. The candidate content can include, for example, at least one of information such as characters, video, and audio.

[0062] In this specification, "biological information" means a measurable quantity of a living body that changes due to stimulation. Biological information includes various quantities such as, for example, blood flow, blood pressure, heart rate, pulse rate, respiratory rate, body temperature, electroencephalogram, concentration of oxygenated hemoglobin in the blood, concentration of deoxygenated hemoglobin in the blood, blood oxygen saturation, and skin reflection spectrum. A part of the biological information may be called vital signs.

[0063] Each component of the biological measurement device 100 will be described in more detail below.

[0064] [1-1. Stimulation device 10] The stimulation device 10 applies stimulation to the user. The stimulation device 10 may be configured to apply stimulation to a plurality of users. The stimulation applied from the stimulation device 10 causes a biological reaction of the user. The stimulation device 10 may present information such as content determined based on the biological reaction of the user to the user or a person other than the user. In the example of FIG. 1B, the biometric measurement device 100 includes the stimulation device 10, but a part or all of the stimulation device 10 may be provided outside the biometric measurement device 100.

[0065] The stimulation device 10 may be, for example, a head-mounted device, goggles and headset device, or a device equipped with a display such as a smartphone. The stimulation device 10 may be an audio device, a lighting device, or an air conditioning device. The stimulation device 10 may include a plurality of devices that apply different stimulations. The stimulation device 10 can apply at least one stimulation of, for example, video, text, sound such as music or voice, brightness, heat, cold feeling, wetness, dryness, vibration, or wind to the user. Video and text are stimulations to vision. Sound is a stimulation to hearing. The stimulation device 10 equipped with a display may apply image, video, or audio content to the user as a stimulation. Visual stimulations can be, for example, web advertisements, videos, games. Various tasks such as calculation problems, language problems, puzzles, or quizzes may be given as visual stimulations. The tasks may be specially created to diagnose the state of the subject's brain activity. The stimulation device 10 may output a sound associated with the task simultaneously with the presentation of the task. In addition to video or audio content, visual stimulations may be the brightness of the lighting in the room or a change in color.

[0066] In addition to stimulations to vision or hearing, stimulations to touch, smell, or taste may be applied. The stimulation device 10 has different structures and functions depending on the type of stimulation given to the user. For example, when giving a tactile stimulation to the user, the stimulation device 10 may be a device that generates vibration or heat. When giving an olfactory stimulation to the user, the stimulation device 10 may be a device that generates an odor.

[0067] [1-2. Light source 20] The light source 20 irradiates light onto a target part including the user's head, for example, the forehead. The light emitted from the light source 20 and reaching the user is divided into a surface reflection component I1 reflected on the user's surface and an internal scattering component I2 scattered inside the user. The internal scattering component I2 is a component that undergoes single reflection or scattering, or multiple scattering inside the living body. When emitting light toward the user's head, the internal scattering component I2 refers to a component that reaches a part about 8 mm to 16 mm deep from the surface of the user's head, for example, the brain, and then returns to the biological measurement device 100 again. The surface reflection component I1 includes three components: a direct reflection component, a diffuse reflection component, and a scattered reflection component. The direct reflection component is a reflection component where the incident angle and the reflection angle are equal. The diffuse reflection component is a component that diffuses and reflects due to the uneven shape of the surface. The scattered reflection component is a component that scatters and reflects due to the internal tissue near the surface. When emitting light toward the user's head, the scattered reflection component is a component that scatters and reflects inside the epidermis. The surface reflection component I1 reflected on the user's surface may include these three components. The surface reflection component I1 and the internal scattering component I2 change their traveling directions by reflection or scattering, and a part of them reaches the image sensor 30.

[0068] In this embodiment, among the reflected light returning from the user's head, the surface reflection component I1 and the internal scattering component I2 are detected. The surface reflection component I1 reflects the appearance of the user's face. Therefore, by analyzing the change over time of the surface reflection component I1, the change in the appearance of the user's face can be estimated. On the other hand, the internal scattering component I2 reflects the brain activity of the user and its intensity fluctuates. Therefore, by analyzing the change over time of the internal scattering component I2, the state of the user's brain activity can be estimated.

[0069] First, a method for obtaining the internal scattering component I2 will be described. The light source 20 repeatedly emits optical pulses a plurality of times at a predetermined time interval or at a predetermined timing according to an instruction from the control circuit 60. The optical pulse emitted from the light source 20 can be, for example, a rectangular wave with a fall period close to zero. In this specification, the "fall period" means the period from when the intensity of the optical pulse starts to decrease until the decrease ends. Generally, the light incident on the user propagates inside the user through various paths and exits from the surface of the user with a time difference. Therefore, the trailing edge of the internal scattering component I2 of the optical pulse has a spread. When the target part of the user is the forehead, the spread of the trailing edge of the internal scattering component I2 is about 4 ns. Considering this, the fall period of the optical pulse can be set to 2 ns or less, which is, for example, half or less of that value. The fall period may be further 1 ns or less, which is half of that value. The length of the rise period of the optical pulse emitted from the light source 20 is arbitrary. The "rise period" is the period from when the intensity of the optical pulse starts to increase until the increase ends. In the detection of the internal scattering component I2 in this embodiment, the falling part of the optical pulse is used, and the rising part is not used. The rising part of the optical pulse can be used for detecting the surface reflection component I1. The light source 20 can be, for example, a laser such as an LD. The light emitted from the laser has a steep time response characteristic in which the falling part of the optical pulse is substantially perpendicular to the time axis.

[0070] The wavelength of the light emitted from the light source 20 can be any wavelength included in the wavelength range of, for example, 650 nm or more and 950 nm or less. This wavelength range is included in the wavelength range from red to near-infrared. The above wavelength range is called the "biological window" and has the property that light is relatively difficult to be absorbed by moisture and skin in the living body. When the living body is the detection target, the detection sensitivity can be increased by using light in the above wavelength range. As in this embodiment, when detecting the blood flow change in the user's brain, the light used is considered to be mainly absorbed by oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb). Oxygenated hemoglobin and deoxygenated hemoglobin have different wavelength dependencies of light absorption. Generally, when a change occurs in the blood flow, the concentrations of oxygenated hemoglobin and deoxygenated hemoglobin change. Along with this change, the degree of light absorption also changes. Therefore, when the blood flow changes, the amount of detected light also changes over time.

[0071] The light source 20 may emit light of a single wavelength included in the above wavelength range, or may emit light of two or more wavelengths. The light of a plurality of wavelengths may be emitted from a plurality of light sources respectively.

[0072] Generally, biological tissues have different absorption characteristics and scattering characteristics depending on the wavelength. For this reason, by detecting the wavelength dependence of the optical signal by the internal scattering component I2, more detailed component analysis of the measurement target becomes possible. For example, in biological tissues, at wavelengths of 805 nm or more, the absorbance by oxygenated hemoglobin (HbO2) is larger than the absorbance by deoxygenated hemoglobin (Hb). On the other hand, at wavelengths of 805 nm or less, the reverse characteristic is shown. Therefore, for example, the light source 20 may be configured to emit light having a wavelength near 750 nm and light having a wavelength near 850 nm. In this case, the light intensity of the internal scattering component I2 by the light having a wavelength near 750 nm and the light intensity of the internal scattering component I2 by the light having a wavelength near 850 nm are measured. The signal processing circuit 70 can obtain the amount of change of each concentration of HbO2 and Hb in the blood from the initial value by solving a predetermined set of simultaneous equations based on the signal values of the light intensity input for each pixel.

[0073] In the biological measurement device 100 according to this embodiment, the cerebral blood flow of the user is measured non - contact. Therefore, the light source 20 designed considering the influence on the retina can be used. For example, in various countries a light source 20 that satisfies Class 1 of the laser safety standard established can be used. When Class 1 is satisfied, light with a low illuminance such that the accessible emission limit (AEL) is less than 1 mW is irradiated to the user. Note that the light source 20 itself may not satisfy Class 1. For example, by installing a diffuser plate or an ND filter in front of the light source 20 to diffuse or attenuate the light, Class 1 of the laser safety standard may be satisfied.

[0074] Conventionally, a streak camera has been used to separately detect information such as absorption coefficients or scattering coefficients at different locations in the depth direction inside a living body. For example, Japanese Patent Laid - Open No. 4 - 189349 discloses an example of such a streak camera. In these streak cameras, in order to measure with a desired spatial resolution, an extremely ultrashort optical pulse with a pulse width in femtoseconds or picoseconds has been used.

[0075] On the other hand, the biological measurement device 100 of this embodiment can separately detect the surface reflection component I1 and the internal scattering component I2. Therefore, the optical pulse emitted by the light source 20 does not need to be an extremely ultrashort optical pulse, and the pulse width can be arbitrarily selected.

[0076] When irradiating the user's head with light to measure cerebral blood flow, the light amount of the internal scattering component I2 can be a very small value, about one - thousandth to one - ten - thousandth of the light amount of the surface reflection component I1. Furthermore, considering the laser safety standard, the light amount of the light that can be irradiated becomes extremely small. Therefore, the detection of the internal scattering component I2 is very difficult. Even in that case, if the light source 20 emits an optical pulse with a relatively large pulse width, the integrated amount of the internal scattering component I2 with a time delay can be increased. Thereby, the detected light amount can be increased and the SN ratio can be improved.

[0077] The light source 20 emits, for example, an optical pulse with a pulse width of 3 ns or more. Generally, the temporal spread of light scattered within a living tissue such as the brain is about 4 ns. FIG. 2 is a diagram showing an example of the temporal change in the intensity of light reaching the image sensor 30. FIG. 2 shows examples of three cases where the widths of the input optical pulses emitted from the light source 20 are 0 ns, 3 ns, and 10 ns. As shown in FIG. 2, as the width of the optical pulse from the light source 20 is increased, the amount of light of the internal scattering component I2 that appears at the rear end of the optical pulse returned from the user increases.

[0078] FIG. 3 is a diagram showing the width of the input optical pulse on the horizontal axis and the detected light amount at the image sensor 30 on the vertical axis. The image sensor 30 includes an electronic shutter. The results in FIG. 3 were obtained under the condition that the electronic shutter was opened 1 ns after the time when the rear end of the optical pulse was reflected from the surface of the user and reached the image sensor 30. The reason for selecting this condition is that immediately after the rear end of the optical pulse arrives, the ratio of the surface reflection component I1 is higher than that of the internal scattering component I2. As shown in FIG. 3, when the pulse width of the optical pulse emitted from the light source 20 is 3 ns or more, the detected light amount can be maximized.

[0079] The light source 20 may emit an optical pulse with a pulse width of 5 ns or more, and further 10 ns or more. On the other hand, if the pulse width is too large, the amount of unused light increases and it becomes wasteful. For this reason, the light source 20 emits, for example, an optical pulse with a pulse width of 50 ns or less. Alternatively, the light source 20 may emit an optical pulse with a pulse width of 30 ns or less, and further 20 ns or less.

[0080] The irradiation pattern of the light source 20 may be, for example, a pattern having a uniform intensity distribution within the irradiation region. In this regard, the present embodiment is different from the conventional biological measurement apparatuses disclosed in, for example, Japanese Patent Application Laid-Open No. 11-164826. In the apparatus disclosed in Japanese Patent Application Laid-Open No. 11-164826, the image sensor and the light source are separated by about 3 cm to spatially separate the surface reflection component from the internal scattering component. For this reason, discrete light irradiation is inevitable. In contrast, the biological measurement apparatus 100 of the present embodiment separates the surface reflection component I1 from the internal scattering component I2 temporally. It can be separated and reduced. Therefore, the light source 20 with an irradiation pattern having a uniform intensity distribution can be used. The irradiation pattern having a uniform intensity distribution may be formed by diffusing the light emitted from the light source 20 with a diffusion plate.

[0081] In this embodiment, different from the prior art, the internal scattering component I2 can be detected even directly below the irradiation point of the user. By irradiating the user with light over a spatially wide range, the measurement resolution can also be increased.

[0082] [1-3. Image sensor 30] The image sensor 30 detects at least a part of the reflected light pulse returning from the user's head for each pixel. The image sensor 30 outputs a plurality of signals corresponding to the intensity of the detected light for each pixel. The plurality of signals include a signal corresponding to the intensity included in at least a part of the rising period and a signal corresponding to the intensity included in at least a part of the falling period among the reflected light pulses.

[0083] The image sensor 30 includes a plurality of two-dimensionally arranged light detection cells and can acquire the two-dimensional information of the user at once. Each light detection cell includes a photoelectric conversion element and one or more charge storage parts. In this specification, the light detection cell is also referred to as a "pixel". The image sensor 30 can be any imaging device such as a CCD image sensor or a CMOS image sensor, for example.

[0084] The image sensor 30 is provided with an electronic shutter. The electronic shutter is a circuit that controls the imaging timing. In the present embodiment, the sensor control unit 62 in the control circuit 60 has the function of the electronic shutter. The electronic shutter controls a period of one signal accumulation in which received light is converted into an effective electrical signal and accumulated, and a period in which signal accumulation is stopped. The signal accumulation period can also be referred to as the "exposure period". In the following description, the width of the exposure period may be referred to as the "shutter width". The time from the end of one exposure period to the start of the next exposure period may be referred to as the "non-exposure period". Hereinafter, the exposed state may be referred to as "OPEN", and the state in which exposure is stopped may be referred to as "CLOSE".

[0085] The image sensor 30 can adjust the exposure period and the non-exposure period in the range of sub-nanoseconds, for example, from 30 ps to 1 ns, by the electronic shutter. A conventional TOF camera for the purpose of distance measurement detects all of the light emitted from the light source 20, reflected by the subject, and returned in order to measure the distance regardless of the brightness of the subject. Therefore, in a conventional TOF camera, the shutter width needs to be larger than the light pulse width. On the other hand, in the biological measurement device 100 of the present embodiment, it is not necessary to correct the light amount of the subject. For this reason, the shutter width does not need to be larger than the pulse width. The shutter width can be set to a value of, for example, 1 ns or more and 30 ns or less. According to the biological measurement device 100 of the present embodiment, since the shutter width can be reduced, the influence of dark current included in the detection signal can be reduced.

[0086] When irradiating the user's forehead with light to detect information such as cerebral blood flow, the light attenuation rate inside the living body is very large. For example, the emitted light can be attenuated to about one millionth of the incident light. Therefore, in order to detect the internal scattering component I2, the light amount may be insufficient with only one pulse irradiation. In particular, the light amount is weak in the irradiation of Class 1 of the laser safety standard. In this case, the light source 20 emits light pulses a plurality of times, and accordingly, the image sensor 30 also exposes a plurality of times by the electronic shutter, so that the detection signals can be integrated to improve the sensitivity.

[0087] Hereinafter, a configuration example of the image sensor 30 will be described.

[0088] The image sensor 30 may include a plurality of pixels two-dimensionally arranged on the imaging surface. Each pixel may include a photoelectric conversion element such as a photodiode, and one or more charge storage units. Hereinafter, an example will be described in which each pixel includes a photoelectric conversion element that generates signal charges corresponding to the amount of received light by photoelectric conversion, a charge storage unit that stores the signal charges generated by the surface reflection component I1 of the light pulse, and a charge storage unit that stores the signal charges generated by the internal scattering component I2 of the light pulse. In the following example, the control circuit 60 causes the image sensor 30 to detect the surface reflection component I1 by detecting a portion before the fall start in the light pulse that has returned from the user's head. The control circuit 60 also causes the image sensor 30 to detect the internal scattering component I2 by detecting a portion after the fall start in the light pulse that has returned from the user's head. The light source 20 in this example emits light of two types of wavelengths.

[0089] FIG. 4A is a diagram showing a schematic configuration example of one pixel 201 of the image sensor 30. Note that FIG. 4A schematically shows the configuration of one pixel 201 and does not necessarily reflect the actual structure. The pixel 201 in this example includes a photodiode 203 that performs photoelectric conversion, a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207 that are charge storage units, and a drain 202 that discharges signal charges.

[0090] Photons incident on each pixel due to the emission of one light pulse are converted into signal electrons, which are signal charges, by the photodiode 203. The converted signal electrons are either discharged to the drain 202 or distributed to any one of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 according to a control signal input from the control circuit 60.

[0091] The emission of optical pulses from the light source 20, the accumulation of signal charges in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, and the discharge of signal charges to the drain 202 are repeatedly performed in this order. This repetitive operation is fast and can be repeated tens of thousands to hundreds of millions of times, for example, within the time of one frame of a moving image (e.g., about 1 / 30 second). The pixel 201 finally generates and outputs four image signals based on the signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207.

[0092] In this example, the control circuit 60 causes the light source 20 to repeatedly emit, in order, a first optical pulse having a first wavelength and a second optical pulse having a second wavelength. By selecting two wavelengths with different absorption rates for the user's internal tissues as the first wavelength and the second wavelength, the user's state can be analyzed. For example, a wavelength longer than 805 nm may be selected as the first wavelength, and a wavelength shorter than 805 nm may be selected as the second wavelength. Thereby, it becomes possible to detect changes in the concentrations of oxygenated hemoglobin and deoxygenated hemoglobin in the user's blood, respectively.

[0093] As shown in FIG. 5, the control circuit 60 first causes the light source 20 to emit a first optical pulse. The control circuit 60 accumulates signal charges in the first floating diffusion layer 204 during a first period in which the surface reflection component I1 of the first optical pulse is incident on the photodiode 203. Subsequently, the control circuit 60 accumulates signal charges in the second floating diffusion layer 205 during a second period in which the internal scattering component I2 of the first optical pulse is incident on the photodiode 203. Next, the control circuit 60 causes the light source 20 to emit a second optical pulse. The control circuit 60 accumulates signal charges in the third floating diffusion layer 206 during a third period in which the surface reflection component I1 of the second optical pulse is incident on the photodiode 203. Subsequently, the control circuit 60 accumulates signal charges in the fourth floating diffusion layer 207 during a fourth period in which the internal scattering component I2 of the second optical pulse is incident on the photodiode 203.

[0094] In this way, after starting the emission of the first optical pulse, the control circuit 60 sequentially accumulates the signal charges from the photodiode 203 in the first floating diffusion layer 204 and the second floating diffusion layer 205 with a predetermined time difference. Then, after starting the emission of the second optical pulse, the control circuit 60 sequentially accumulates the signal charges from the photodiode 203 in the third floating diffusion layer 206 and the fourth floating diffusion layer 207 with the above-mentioned predetermined time difference. The above operations are repeated multiple times. In order to estimate the light amounts of the ambient light and the environmental light, a period for accumulating signal charges in other floating diffusion layers (not shown) may be provided with the light source 20 turned off. By subtracting the signal charge amount of the above-mentioned other floating diffusion layer from the signal charge amounts of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, a signal with the ambient light and the environmental light components removed can be obtained.

[0095] In the present embodiment, the number of charge accumulation parts is set to 4, but it may be designed to be a plurality of numbers of 2 or more according to the purpose. For example, when only one type of wavelength is used, the number of charge accumulation parts may be 2. Also, in an application where only one type of wavelength is used and the surface reflection component I1 is not detected, the number of charge accumulation parts per pixel may be 1. Further, even when two or more types of wavelengths are used, if imaging using each wavelength is performed in a different frame, the number of charge accumulation parts may be 1. When detecting the surface reflection component I1 and the internal scattering component I2 in different frames respectively, the number of charge accumulation parts may be 1.

[0096] Figure 4B is a diagram showing an example of the configuration of the image sensor 30. In Figure 4B, the region surrounded by the dashed double-dotted line frame corresponds to one pixel 201. The pixel 201 includes one photodiode. Although Figure 4B shows only 4 pixels arranged in 2 rows and 2 columns, actually, a larger number of pixels can be arranged. The pixel 201 includes a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207. The signals accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are treated as if they were the signals of 4 pixels of a general CMOS image sensor and are output from the image sensor 30.

[0097] Each pixel 201 has four signal detection circuits. Each signal detection circuit includes a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. In this example, the reset transistor 310 corresponds to the drain 202 shown in Figure 4A, and the pulse input to the gate of the reset transistor 310 corresponds to the drain discharge pulse. Each transistor is, for example, a field effect transistor formed on a semiconductor substrate, but is not limited thereto. As shown, one of the input terminal and the output terminal (typically the source) of the source follower transistor 309 and one of the input terminal and the output terminal (typically the drain) of the row selection transistor 308 are connected. The gate, which is the control terminal of the source follower transistor 309, is connected to the photodiode 203. The signal charges (i.e., holes or electrons) generated by the photodiode 203 are accumulated in the floating diffusion layer, which is a charge accumulation portion between the photodiode 203 and the source follower transistor 309.

[0098] The first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are connected to the photodiode 203. A switch may be provided between the photodiode 203 and the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. This switch switches the conduction state between the photodiode 203 and each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 in response to a signal accumulation pulse from the control circuit 60. Thereby, the start and stop of the accumulation of signal charges in each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are controlled. The electronic shutter in this embodiment has such a mechanism for exposure control.

[0099] The signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are read out when the gate of the row selection transistor 308 is turned on by the row selection circuit 302. At this time, according to the signal potentials of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified. The analog signal due to this current read out from the vertical signal line 304 is converted into digital signal data by the analog-digital (AD) conversion circuit 307 connected for each column. This digital signal data is read out for each column by the column selection circuit 303 and output from the image sensor 30. After reading one row, the row selection circuit 302 and the column selection circuit 303 read the next row, and in the same manner, the information on the signal charges in the floating diffusion layers of all rows is read out. After reading all the signal charges, the control circuit 60 resets all the floating diffusion layers by turning on the gate of the reset transistor 310. Thereby, the imaging of one frame is completed. Similarly, by repeating the high-speed imaging of frames, the imaging of a series of frames by the image sensor 30 is completed.

[0100] In this embodiment, an example of the CMOS type image sensor 30 has been described, but the image sensor 30 may be other types of imaging elements. The image sensor 30 may be, for example, of the CCD type, a single photon counting type element, or an amplified type image sensor (e.g., EMCCD or ICCD).

[0101] FIG. 5 is a diagram showing an example of the operation within one frame in this embodiment. As shown in FIG. 5, within one frame, the emission of the first optical pulse and the emission of the second optical pulse may be alternately switched a plurality of times. By doing so, the time difference in the acquisition timing of the detection images with two types of wavelengths can be reduced, and even for a moving user, it is possible to capture images with the first and second optical pulses almost simultaneously.

[0102] In this embodiment, the image sensor 30 detects both the surface reflection component I1 and the internal scattering component I2 of the optical pulse. From the temporal or spatial changes in the surface reflection component I1, the first biometric information of the user can be obtained. The first biometric information indicates the appearance of the user's face and can be, for example, information related to the line of sight, pupil diameter, blinking, or facial expression. On the other hand, from the temporal or spatial changes in the internal scattering component I2, the brain activity information, which is the second biometric information of the user, can be obtained.

[0103] In this specification, a signal indicating the first biometric information may be referred to as the "first biometric signal". Also, a signal indicating the brain activity information may be referred to as the "brain activity signal".

[0104] [1-4. Control Circuit 60 and Signal Processing Circuit 70] The control circuit 60 adjusts the time difference between the emission timing of the optical pulse of the light source 20 and the shutter timing of the image sensor 30. In this specification, this time difference may be referred to as the "phase difference". The "emission timing" of the light source 20 is the timing at which the optical pulse emitted from the light source 20 starts to rise. The "shutter timing" is the timing at which exposure starts. The control circuit 60 may adjust the phase difference by changing the emission timing, or may adjust the phase difference by changing the shutter timing.

[0105] The control circuit 60 may be configured to remove the offset component from the signals detected by each pixel of the image sensor 30. The offset component is a signal component due to ambient light such as sunlight or fluorescent light or disturbance light. By detecting signals with the image sensor 30 in a state where the driving of the light source 20 is turned off and no light is emitted from the light source 20, the offset component due to ambient light or disturbance light can be estimated.

[0106] The control circuit 60 can be an integrated circuit such as, for example, a combination of a processor and a memory, or a microcontroller incorporating a processor and a memory. The control circuit 60, for example, adjusts the emission timing and the shutter timing by, for example, the processor executing a program recorded in the memory.

[0107] The signal processing circuit 70 is a circuit that processes the image signal output from the image sensor 30. The signal processing circuit 70 performs arithmetic processing such as image processing. The signal processing circuit 70 can be realized, for example, by a programmable logic device (PLD) such as a digital signal processor (DSP), a field programmable gate array (FPGA), or a combination of a central processing unit (CPU) or a graphics processing unit (GPU) for image processing and a computer program. The control circuit 60 and the signal processing circuit 70 may be an integrated single circuit or may be separate individual circuits. The signal processing circuit 70 may be a component of an external device such as, for example, a server provided at a remote location. In this case, an external device such as a server performs data transmission and reception with the light source 20, the image sensor 30, and the control circuit 60 by wireless communication or wired communication.

[0108] The signal processing circuit 70 in the present embodiment can generate moving image data indicating the temporal change of cerebral blood flow and moving image data indicating the temporal change of the facial appearance based on the image data output from the image sensor 30 for each frame. The signal processing circuit 70 is not limited to such moving image data and may generate other information. For example, by synchronizing with other devices, biological information such as blood flow volume, blood pressure, blood oxygen saturation, or heart rate in the brain may be generated. The signal processing circuit 70 may estimate the offset component due to ambient light and remove the offset component.

[0109] It is known that there is a close relationship between changes in cerebral blood flow or blood components such as hemoglobin and human neural activities. For example, as the activities of nerve cells change according to the degree of human interest, the cerebral blood flow or the components in the blood change. Therefore, if biological information such as cerebral blood flow or facial appearance information can be measured, the psychological state or physical state of the user can be estimated. The psychological state of the user can be, for example, mood, emotion, health state, or temperature sensation. The mood can include, for example, moods such as pleasant or unpleasant. Emotions can include, for example, emotions such as reassurance, anxiety, sadness, or anger. The health state can include, for example, states such as energetic or listless. The temperature sensation can include, for example, sensations such as hot, cold, or stuffy. Derived from these, indices representing the degree of brain activity, such as the degree of interest, proficiency, familiarity, and concentration, can also be included in the psychological state. Furthermore, physical states such as the degree of fatigue, sleepiness, or drunkenness due to drinking are also included in the targets of estimation by the signal processing circuit 70. The signal processing circuit 70 can estimate the psychological state or physical state of the user based on changes in the cerebral blood flow state and changes in the facial appearance, and output a signal indicating the estimation result.

[0110] FIG. 6A is a flowchart showing an outline of the control of the light source 20 and the image sensor 30 by the control circuit 60. Here, for simplicity, the operation in the case of detecting the surface reflection component I1 and the internal scattering component I2 of the reflected light for each pixel using light of one wavelength will be described. In this example, each pixel of the image sensor 30 includes a first charge accumulation unit that accumulates charges due to the surface reflection component I1 and a second charge accumulation unit that accumulates charges due to the internal scattering component I2.

[0111] In step S101, the control circuit 60 first causes the light source 20 to emit a light pulse for a predetermined time. At this time, the electronic shutter of the image sensor 30 is in a state where the exposure is stopped. The control circuit 60 keeps the exposure of the electronic shutter stopped until the light pulse is reflected from the surface of the user and starts to reach the image sensor 30.

[0112] Next, in step S102, the control circuit 60 starts the exposure of the electronic shutter at a predetermined timing from when the reflected light pulse starts to reach the image sensor 30 until the start of the fall period. This exposure is referred to as the "first exposure". The timing to start the first exposure can be appropriately set for each pixel by measuring in advance the distance to the target part for each pixel. The start timing of the first exposure may vary for each pixel according to the degree of curvature of the surface of the target part, or may be the same for all pixels. The light detected by the first exposure is mainly the light scattered on the surface of the skin of the target part and reaching the image sensor 30.

[0113] After a predetermined time has elapsed, in step S103, the control circuit 60 stops the first exposure of the electronic shutter. The timing of stopping the exposure can be, for example, before the start of the fall period of the reflected light pulse.

[0114] Next, in step S104, the control circuit 60 starts the second exposure of the electronic shutter at the timing when a part of the light pulse is scattered inside the user and reaches the image sensor 30. More specifically, the control circuit 60 starts the second exposure after the start of the fall period of the reflected light pulse. The start timing of the second exposure can also be calculated based on the distance to the target part measured in advance for each pixel. The start timing of the second exposure may vary for each pixel according to the degree of curvature of the surface of the target part, or may be the same for all pixels.

[0115] After a predetermined time has elapsed, in step S105, the control circuit 60 stops the second exposure of the electronic shutter. The time length of the first exposure and the time length of the second exposure may be the same or different. Generally, the amount of light of the surface reflection component I1 detected by the first exposure is larger than the amount of light of the internal scattering component I2 detected by the second exposure. Therefore, the time length of the first exposure may be set shorter than the time length of the second exposure.

[0116] Subsequently, in step S106, the control circuit 60 determines whether or not the number of times the above signal accumulation has been executed has reached a predetermined number of times. If this determination is No, steps S101 to S105 are repeated until it is determined as Yes. This number of times is set to an appropriate number depending on the detection sensitivity of the internal scattering component I2. When it is determined as Yes in step S106, in step S107, the control circuit 60 causes the image sensor 30 to generate an image signal based on the signal charges accumulated in each charge accumulation unit. The image sensor 30 outputs first image data based on the charges accumulated in the first charge accumulation unit of each pixel and second image data based on the charges accumulated in the second charge accumulation unit of each pixel.

[0117] By the above operations, the component of the light scattered near the surface of the target part and the component of the light scattered inside the target part can be detected with high sensitivity. Note that multiple light emissions and exposures are not essential and are performed as necessary.

[0118] The signal processing circuit 70 generates first moving image data indicating a change in the appearance of the user's face by performing necessary image processing such as color correction, pixel interpolation, or frame interpolation on the first image data. The signal processing circuit 70 also generates second moving image data indicating a change in the state of the user's cerebral blood flow by performing necessary image processing on the second image data. The signal processing circuit 70 further estimates the mental state or physical state of the user based on the first moving image data and the second moving image data. For example, based on the change in expression or gaze estimated from the first moving image data and the change in the cerebral activity state estimated from the second moving image data, a state such as the user's interest or concentration can be estimated. Details of these processes will be described later.

[0119] The signal processing circuit 70 may perform a process of changing the resolution of at least one of the first image data and the second image data. For example, the process may be performed such that the first image data has a higher resolution than the second image data. The process of changing the resolution may be performed only on a part of each image. That is, the signal processing circuit may perform a process of changing the resolution of at least a part of the image indicated by the first image data and / or at least a part of the image indicated by the second image data. In that case, the signal processing circuit 70 generates data indicating the state of the subject based on the respective temporal changes of the first image data and the second image data after the process. An example of such a process will be described below.

[0120] FIG. 6B is a flowchart showing an example of a process of changing the resolution of the first image data and the second image data. The signal processing circuit 70 in this example changes the resolution of the first and second image data by executing the processes of steps S108 to S114 shown in FIG. 6B. By the above process, the signal processing circuit 70 can make the first image data have a higher resolution than the second image data. The operations of each step will be described below.

[0121] In step S108, the signal processing circuit 70 acquires the first image data and the second image data generated by the image sensor 30.

[0122] In step S109, the signal processing circuit 70 selects a necessary region from the first image data. For example, when acquiring gaze data, a region near the eyeball is selected.

[0123] In step S110, the signal processing circuit 70 performs high-resolution processing on the data of the selected region of the first image data. For example, by performing known super-resolution processing, the first image data is processed to have a higher resolution than the second image data. Note that the signal processing circuit 70 may perform processing to reduce the resolution of the first image data. In that case, by suppressing the reduction rate of the resolution of the first image data, the first image data may have a higher resolution than the second image data.

[0124] In step S111, the signal processing circuit 70 selects a necessary region from the second image data. For example, a region of a part of the forehead of the subject is selected.

[0125] In step S112, the signal processing circuit 70 performs low-resolution processing on the second image data. To reduce the resolution, an arithmetic mean of signal values may be performed between adjacent pixels. By performing the arithmetic mean processing, noise included in the weak cerebral blood flow signal can be reduced.

[0126] In step S113, the signal processing circuit 70 outputs the processed first image data and second image data. For example, the signal processing circuit 70 records the processed first image data and second image data on the recording medium 90.

[0127] In step S114, the signal processing circuit 70 determines whether the processing has ended. If this determination is No, the signal processing circuit 70 repeats steps S108 to S113 until it is determined to be Yes. The determination of whether the processing has ended can be made based on, for example, whether the output of the image data by the image sensor 30 has ended, or whether there is a stop instruction from the user. Alternatively, the determination of the end may be made based on whether the elapsed time from the start of measurement has reached a predetermined time, or whether the accumulated amount of data from the start of measurement has reached a predetermined data amount, etc.

[0128] The signal processing circuit 70 may perform a process of changing the resolution for only one of the first image data and the second image data. When increasing the resolution of the first image data and decreasing the resolution of the second image data, the data amount can be reduced, and at the same time, high-resolution face appearance information can be obtained.

[0129] The first frame rate at which the image sensor 30 outputs the first image data and the second frame rate at which the image sensor 30 outputs the second image data may be different.

[0130] FIG. 6C is a flowchart showing an outline of a process in which the image sensor 30 changes the frame rate to output image data and the signal processing circuit 70 generates moving image data.

[0131] In step S115, the image sensor 30 outputs the first image data based on the surface reflection component I1 at a high frame rate. In this case, in order to increase the frame rate, the control circuit 60 adjusts the shutter timing of the image sensor 30 to shorten the integration exposure time within one frame. In order to increase the frame rate, the control circuit 60 may adjust the light emission timing of the light source 20 to increase the light emission time per time and reduce the number of light emission times within one frame by making the number of continuous light emissions within one frame 1 time.

[0132] Note that in order to acquire face appearance information, the light source 20 may be an LED. For acquiring face appearance information, laser light such as an LD in which the falling edge portion of the optical pulse is substantially perpendicular to the time axis is not necessarily required, and steep time response characteristics are not essential.

[0133] In step S116, the signal processing circuit 70 generates and outputs face appearance data based on the first image data output in step S115.

[0134] In step S117, the image sensor 30 outputs second image data based on the internal scattering component I2 at a low frame rate. In this case, in order to lower the frame rate, the control circuit 60 adjusts the shutter timing of the image sensor 30 to make the integration exposure time within one frame longer than when the first image data was acquired.

[0135] In step S118, the signal processing circuit 70 generates cerebral blood flow data based on the second image data acquired in step S117.

[0136] Note that instead of the image sensor 30 outputting the second image data at a low frame rate, after the image sensor 30 outputs all frames at a high frame rate, the signal processing circuit 70 may subsequently integrate the image data for a plurality of times and output it as the image data for one time.

[0137] The image sensor 30 and the signal processing circuit 70 repeat the processes of steps S115 to S118 until an instruction to end the measurement is received in step S119. Note that the number of repetitions may be different between the first image and the second image.

[0138] [1-5. Control of Server 200 and Stimulation Device 10] The biological measurement device 100 in the present embodiment can be used in cooperation with an external server 200. The server 200 includes a storage device that stores content such as video or audio, games, tests, or task data. The server 200 also includes a communication circuit that communicates with the communication circuit 80 of the biological measurement device 100. In addition to data such as videos or applications that can be provided to the user, the server 200 also stores the moving image data generated by the signal processing circuit 70 and the diagnostic data of brain activity. Part or all of the functions of the server 200 may be incorporated in the biological measurement device 100. Conversely, the server 200 may perform some of the functions of the signal processing circuit 70 in the biological measurement device 100.

[0139] ​The control circuit 60 includes a stimulus control unit 63. The stimulus control unit 63 can control the stimulus device 10 to provide stimuli such as video or audio to the user. The stimulus control unit 63 can control, for example, the hue, saturation, or brightness of the content of the video provided as a stimulus, or the type, sound quality, or volume of the audio.

[0140] The control circuit 60 can determine the next stimulus to be provided to the user based on the mental or physical state of the user estimated by the signal processing circuit 70. For example, when it is determined that a user watching a certain content has lost interest or concentration, it can be determined to display different content. This determination process may be performed by a processor provided in the server 200. The control circuit 60 can acquire necessary data such as video or audio from the server 200 and cause the stimulus device 10 to provide a stimulus based on the data.

[0141] [1-6. Others] The biometric measurement device 100 may include an imaging optical system that forms a two-dimensional image of the user on the light-receiving surface of the image sensor 30. The optical axis of the imaging optical system is substantially orthogonal to the light-receiving surface of the image sensor 30. The imaging optical system may include a zoom lens. When the position of the zoom lens changes, the magnification of the two-dimensional image of the user changes, and the resolution of the two-dimensional image on the image sensor 30 changes. Therefore, even if the distance to the user is far, it is possible to expand and observe in detail the area to be measured.

[0142] The biometric measurement device 100 may include a band-pass filter that allows only light in the band of the wavelength emitted from the light source 20 or in the vicinity thereof to pass between the user and the image sensor 30. Thereby, the influence of disturbance components such as ambient light can be reduced. The band-pass filter is composed of a multilayer film filter or an absorption filter. Considering the temperature of the light source 20 and the band shift due to the oblique incidence on the filter, the bandwidth of the band-pass filter may have a width of about 20 nm to 100 nm.

[0143] The biological measurement device 100 may include polarizing plates between the light source 20 and the user, and between the image sensor 30 and the user, respectively. In this case, the polarization directions of the polarizing plate disposed on the light source 20 side and the polarizing plate disposed on the image sensor 30 side are in a crossed Nicol relationship. Thereby, it is possible to prevent the specular reflection component, that is, the component with the same incident angle and reflection angle, among the surface reflection components I1 of the user from reaching the image sensor 30. That is, the amount of light of the surface reflection component I1 reaching the image sensor 30 can be reduced.

[0144] [2. Operation of time-resolved imaging] As described above, the biological measurement device 100 in the present embodiment can distinguish and detect the surface reflection component I1 and the internal scattering component I2 of the light irradiated on the target part. Such imaging is referred to as "time-resolved imaging" in this specification.

[0145] Hereinafter, an example of the operation of the biological measurement device 100 in the present embodiment will be described.

[0146] As shown in FIG. 1B, when the light source 20 irradiates the user with a light pulse, the surface reflection component I1 and the internal scattering component I2 are generated. A part of the surface reflection component I1 and the internal scattering component I2 reaches the image sensor 30. The internal scattering component I2 passes through the inside of the user before being emitted from the light source 20 and reaching the image sensor 30. That is, the light of the internal scattering component I2 has a longer optical path length than that of the surface reflection component I1. Therefore, the internal scattering component I2 is on average delayed in time to reach the image sensor 30 compared to the surface reflection component I1.

[0147] The surface reflection component I1 can be detected, for example, by the following operation.

[0148] FIG. 7 is a diagram showing an example of an optical signal in which a rectangular optical pulse is emitted from the light source 20 and the light returning from the user reaches the image sensor 30. The horizontal axis represents time (t) in all of parts (a) to (d). The vertical axis represents intensity in parts (a) to (c), and represents the OPEN or CLOSE state of the electronic shutter in part (d). Part (a) shows the surface reflection component I1. Part (b) shows the internal scattering component I2. Part (c) shows the combined component of part (a) showing the surface reflection component I1 and part (b) showing the internal scattering component I2. Part (d) shows the shutter timing for acquiring the surface reflection component I1 of the user.

[0149] As shown in part (d) of FIG. 7, by opening the shutter, among the reflected light incident on the image sensor 30, the components that reach earlier can be efficiently collected. The components that reach earlier mean that there is little scattering in the target part and include the surface information of the target part. Although the time during which light is substantially accumulated is a very short time at the front end of the pulse wave, the shutter does not necessarily have to be only during that period. As shown in part (d) of FIG. 7, the shutter may be opened at a stage earlier than when the front end of the pulse wave reaches the image sensor 30. In the example shown in part (d) of FIG. 7, just before the exposure period ends, the front end part of the reflected light pulse is detected. When such shutter control is adopted, it is not necessary to use an expensive image sensor capable of picosecond-order exposure. The biological measurement device 100 in the present embodiment can be configured by an inexpensive image sensor 30.

[0150] In the example of FIG. 7, the light source 20 irradiates a rectangular pulse wave. At this time, the pulse width does not have to be on the order of ps and may be several ns. Therefore, an inexpensive light source can be used.

[0151] In this example, only the front end part of the reflected light pulse is detected, but the detection method is not limited to such a method. For example, the period from the end of the rising period to the start of the falling period may be included in the exposure period. Even with such a detection method, image data showing the appearance of the user's face can be acquired.

[0152] Next, an example of a method for detecting the internal scattering component I2 will be described.

[0153] FIG. 8 is a diagram showing another example of an optical signal in which a rectangular optical pulse is emitted from the light source 20 and the light returned from the user reaches the image sensor 30. Parts (a) to (c) in FIG. 8 show the same time changes as parts (a) to (c) in FIG. 7, respectively. Part (d) in FIG. 8 shows the shutter timing for acquiring the internal scattering component I2.

[0154] As shown in part (a) of FIG. 8, the surface reflection component I1 maintains a rectangle. On the other hand, as shown in part (b) of FIG. 8, since the internal scattering component I2 is the sum of lights with different optical path lengths, it shows a characteristic like trailing at the rear end of the optical pulse. That is, the fall period becomes longer than that of the surface reflection component I1. In order to extract the internal scattering component I2 with a higher ratio from the optical signal in part (c) of FIG. 8, as shown in part (d) of FIG. 8, the electronic shutter starts charge accumulation after the rear end of the surface reflection component I1. After the rear end of the surface reflection component I1 means when the surface reflection component I1 has fallen or later. This shutter timing is adjusted by the control circuit 60. As described above, the biological measurement device 100 in the present embodiment detects the surface reflection component I1 and the internal scattering component I2 separately. Therefore, the emission pulse width and the shutter width are arbitrary. Therefore, different from the method using a conventional streak camera, the acquisition of the internal scattering component I2 can be realized with a simple configuration, and the cost can be reduced.

[0155] In order to execute the operation shown in part (d) of FIG. 8, the control circuit 60 causes the image sensor 30 to detect at least a part of the components in the fall period of the reflected optical pulse and output a signal indicating a two-dimensional image of the user. In the present embodiment, the signal output from the image sensor 30 may include a signal indicating the amount of light of at least a part of the components in the fall period of the reflected optical pulse.

[0156] In part (a) of FIG. 8, the trailing edge of the surface reflection component I1 rises vertically. In other words, the time from when the surface reflection component I1 starts to fall until it ends is zero. However, in reality, the light pulse itself emitted by the light source 20 may not be perfectly vertical, there may be minute irregularities on the user's surface, or scattering within the epidermis may cause the trailing edge of the surface reflection component I1 not to fall vertically. Also, since the user is an opaque object, the light amount of the surface reflection component I1 is much larger than that of the internal scattering component I2. Therefore, even if the trailing edge of the surface reflection component I1 slightly protrudes from the vertically falling position, the internal scattering component I2 may be buried. Furthermore, a time delay may occur due to the movement of electrons during the readout period of the electronic shutter. For the above reasons, an ideal binary readout as shown in part (d) of FIG. 8 may not be achievable. In that case, the control circuit 60 may delay the shutter timing of the electronic shutter slightly more than immediately after the start of the fall of the surface reflection component I1. For example, it may be delayed by about 0.5 ns to 5 ns. Instead of adjusting the shutter timing of the electronic shutter, the control circuit 60 may adjust the light emission timing of the light source 20. In other words, the control circuit 60 may adjust the time difference between the shutter timing of the electronic shutter and the light emission timing of the light source 20. If the shutter timing is delayed too much, the originally small internal scattering component I2 will further decrease. Therefore, the shutter timing may be set near the trailing edge of the surface reflection component I1. As described above, since the time delay due to scattering inside the forehead is 4 ns, the maximum amount of delay of the shutter timing can be about 4 ns.

[0157] Each of the plurality of light pulses emitted from the light source 20 may be exposed with a shutter timing having the same time difference to accumulate signals. Thereby, the detected light amount of the internal scattering component I2 is amplified.

[0158] Instead of, or in addition to, disposing a band-pass filter between the user and the image sensor 30, the offset component may be estimated by taking a photograph with the same exposure period while the light source 20 does not emit light. The estimated offset component is removed by difference from the signals detected by each pixel of the image sensor 30. Thereby, the influence of the dark current component and / or ambient light generated on the image sensor 30 can be removed.

[0159] Next, an example of a method for detecting the surface reflection component I1 and the internal scattering component I2 per frame will be described.

[0160] FIG. 9A shows an example of a timing chart when detecting the surface reflection component I1. For detecting the surface reflection component I1, for example, as shown in FIG. 9A, the shutter may be opened before the light pulse reaches the image sensor 30, and the shutter may be closed before the trailing edge of the light pulse reaches. By controlling the shutter in this way, the mixing of the internal scattering component I2 can be reduced. The ratio of the light passing near the surface of the user can be increased. In particular, the timing of shutter closing may be set immediately after the light reaches the image sensor 30. Thereby, signal detection with a higher ratio of the surface reflection component I1 having a relatively short optical path length becomes possible. As another method for obtaining the surface reflection component I1, the image sensor 30 may acquire the entire light pulse, or may irradiate continuous light from the light source 20 and detect it. FIG. 9B shows an example of a timing chart when detecting the internal scattering component I2. In this example, after the trailing edge portion of the pulse starts to reach the image sensor 30, the shutter is opened. By such control, the signal of the internal scattering component I2 can be acquired.

[0161]

[0162] ​When performing time-division imaging using the same image sensor as in this embodiment, temporal and spatial misalignments are less likely to occur. When acquiring signals of both the surface reflection component I1 and the internal scattering component I2 with the same image sensor, as shown in FIGS. 9A and 9B, the components acquired for each frame may be switched. Alternatively, as described with reference to FIGS. 5 and 6A, the components acquired within one frame may be switched alternately at high speed. In that case, the detection time difference between the surface reflection component I1 and the internal scattering component I2 can be reduced.

[0163] Furthermore, the signals of the surface reflection component I1 and the internal scattering component I2 may be acquired using light of two wavelengths. When the surface reflection component I1 and the internal scattering component I2 are acquired at two wavelengths respectively, for example, as described with reference to FIGS. 4A to 5, a method of switching four types of charge accumulations at high speed within one frame can be used. By such a method, the temporal misalignment of the detection signals can be reduced.

[0164] [3. Example of Detection of Changes in Cerebral Blood Flow] Next, an example of a method for detecting changes in the cerebral blood flow of a user will be described.

[0165] FIG. 10A is a diagram schematically showing an example of the temporal change in cerebral blood flow. As shown in FIG. 10A, the target part of the user is irradiated with light from the light source 20, and the returning light is detected. In this case, the surface reflection component I1 is much larger than the internal scattering component I2. However, by the shutter adjustment described above, only the internal scattering component I2 can be extracted. The graph shown in FIG. 10A shows an example of the change over time in the respective concentrations of oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb) in cerebral blood. The internal scattering component I2 in this example is acquired using light of two wavelengths. The concentrations shown in FIG. 10A indicate the amount of change based on the amount in the normal state. This amount of change is calculated by the signal processing circuit 70 based on the light intensity signal. Depending on the brain activity state such as the normal state, the concentrated state, or the relaxed state, a change is seen in the cerebral blood flow. For each location within the target part, there are differences in, for example, brain activity, or absorption coefficient or scattering coefficient. For this reason, the temporal change in cerebral blood flow is measured at the same position within the target part of the user. When detecting the temporal change in brain activity, even if the absolute amount of cerebral blood flow is unknown, it is possible to estimate the state of the user from the temporal relative change in cerebral blood flow.

[0166] FIG. 10B is a diagram schematically showing an example of the case where measurements are simultaneously performed at a plurality of locations within the target part of the user. In this example, since a two-dimensional region is imaged, it is possible to acquire the two-dimensional distribution of cerebral blood flow. In this case, the irradiation pattern of the light source 20 may be, for example, a uniform distribution of uniform intensity, a dot-like distribution, or a donut-like distribution. If it is irradiation with a uniform distribution of uniform intensity, adjustment of the irradiation position of the target part is unnecessary or can be made simple. If it is irradiation with a uniform distribution, light enters the target part of the user from a wide range. For this reason, the signal detected by the image sensor 30 can be enhanced. Furthermore, measurement can be performed at an arbitrary position within the irradiation region. If it is partial irradiation such as a dot-like distribution or a donut-like distribution, the influence of the surface reflection component I1 can be reduced simply by removing the target part from the irradiation region.

[0167] FIG. 11A is a diagram schematically showing an example of the light irradiation region 22. In non-contact cerebral blood flow measurement, the amount of detected light attenuates in inverse proportion to the square of the distance from the device to the target part. Therefore, the signal of each pixel detected by the image sensor 30 may be enhanced by integrating the signals of a plurality of adjacent pixels. By doing so, the number of integration pulses can be reduced while maintaining the SNR. As a result, the frame rate can be improved.

[0168] FIG. 11A shows an example in which only the user's head is irradiated with light. However, when acquiring the user's face image using the same light, the face is also included in the irradiation region. When acquiring the face image using other light, it is not necessary for the light from the light source 20 to irradiate regions other than the head.

[0169] FIG. 11B is a diagram schematically showing the change in signal when the target part of the user shifts laterally. As described above, by detecting the difference between the cerebral blood flow when the cerebral activity state changes from the normal state and the cerebral blood flow in the normal state, the change in cerebral activity can be read. When using an image sensor 30 including a plurality of two-dimensionally arranged photoelectric conversion elements, as shown in the upper part of FIG. 11B, a two-dimensional cerebral activity distribution can be acquired. In this case, even if the signal in the normal state is not acquired in advance, the site where the cerebral activity is active can be detected from the relative intensity distribution within the two-dimensional distribution. Since the measurement is performed non-contact in this embodiment, as shown in the lower part of FIG. 11B, the position of the target part may change during the measurement. This may occur, for example, when the user moves slightly due to breathing. Generally, the two-dimensional distribution of cerebral blood flow does not change rapidly within a short time. Therefore, for example, the positional deviation of the target part can be corrected by pattern matching between the detected two-dimensional distributions of frames. Alternatively, for periodic movements such as breathing, only the frequency component thereof may be extracted and corrected or removed. The target part does not have to be a single region and may be a plurality of regions. The plurality of regions may be, for example, one on each of the left and right, or a 2×6 matrix-like dot distribution.

[0170] [4. Estimation of User's State] Next, an example of a method for estimating the user's state using the above-described biological measurement device 100 will be described. The biological measurement device 100 can be used in a system that provides content such as video or audio via a network such as the Internet. Such a system can include, for example, a server operated by a service provider and various computers such as a personal computer (PC), a smartphone, or a tablet owned by the user. The server 200 shown in FIG. 1B may be a server in such a system.

[0171] Such a system can be used by a plurality of users. Each user uses the biological measurement device 100. Each user can browse content such as an application, a video, or a game distributed from the server through the stimulation device 10 using a computer such as a smartphone. The biological measurement device 100 may be built-in or externally attached to the computer owned by the user.

[0172] [4-1. Determination of User's Interest] As an example, an example of a method for determining the user's interest will be described. In this example, the user is viewing the content of the video displayed on the display, and during that time, image data showing the appearance of the user's face and image data showing the state of the user's cerebral blood flow are repeatedly generated by the above-described method. In this example, as described with reference to FIGS. 4A to 5, using light of two wavelengths, a total of four types of image data based on the surface reflection component I1 and the internal scattering component I2 by the light of each wavelength are generated for each frame. Hereinafter, the two types of image data based on the surface reflection component I1 are referred to as "first image data", and the two types of image data based on the internal scattering component I2 are referred to as "second image data". The signal processing circuit 70 detects the change in the user's line of sight based on the change over time of the first image data, and based on the second image data, the user's brain activity Detect changes. Based on those detection results, the signal processing circuit 70 identifies the timing when the user has a high level of interest and the location the user is gazing at at that time. Thereby, it is possible to estimate which part of the content the user is watching the user has a strong interest in.

[0173] FIG. 12A is a flowchart showing an example of the operation of the interest determination process in the present embodiment. In this example, the signal processing circuit 70 executes the processes of steps S201 to S206 shown in FIG. 12A for each frame.

[0174] In step S201, the signal processing circuit 70 acquires first image data based on the surface reflection component I1. The first image data is repeatedly output from the image sensor 30 at the first frame rate as described above.

[0175] In step S202, the signal processing circuit 70 generates line-of-sight data J1 based on the first image data. The line-of-sight data J1 is data indicating the direction of the user's line of sight. The line-of-sight data J1 can be, for example, data indicating the coordinates of the center of the user's pupil. The coordinates of the center of the user's pupil can be calculated, for example, from the positional relationship between the center position of the user's pupil and the Purkinje image, which is a corneal reflection image, using a known corneal reflection method. There are various methods for calculating the eye position by the corneal reflection method. For example, as disclosed in Non-Patent Document 1, a method of mapping the amount of movement of the center of the pupil in the horizontal and vertical directions as seen from the camera onto the display plane can be used. In the present embodiment, the first image data includes image data by light having a wavelength shorter than 805 nm and image data by light having a wavelength longer than 805 nm. The generation of the line-of-sight data may be generated based on only one of these two types of image data or based on both.

[0176] In step S203, the signal processing circuit 70 acquires second image data based on the internal scattering component I1. The second image data is repeatedly output from the image sensor 30 at the second frame rate as described above.

[0177] In step S204, the signal processing circuit 70 generates cerebral blood flow data J2 based on the second image data. The cerebral blood flow data J2 is data indicating the state of the user's cerebral blood flow. The cerebral blood flow data J2 can be, for example, data indicating the respective concentrations of oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb) in cerebral blood. In the present embodiment, the second image data includes image data obtained by light having a wavelength shorter than 805 nm and image data obtained by light having a wavelength longer than 805 nm. As described above, for light having a wavelength longer than 805 nm, the absorbance by HbO2 is larger than the absorbance by Hb. Conversely, for light having a wavelength shorter than 805 nm, the absorbance by Hb is larger than the absorbance by HbO2. Therefore, by using the values of the detected light amounts at each pixel and solving a predetermined set of simultaneous equations, it is possible to obtain the amounts of change of the respective concentrations of HbO2 and Hb in the blood from the reference values. The data of those amounts of change can be used as the cerebral blood flow data J2. Alternatively, data indicating the concentration of either HbO2 or Hb may be used as the cerebral blood flow data J2.

[0178] In step S205, the signal processing circuit 70 determines the user's interest based on the gaze data J1 and the cerebral blood flow data J2, and records data indicating the determination result. The details of this determination process will be described later with reference to FIG. 12B.

[0179] The signal processing circuit 70 repeats the processes of steps S201 to S205 until it receives an instruction to end the measurement in step S206.

[0180] FIG. 12B is a flowchart showing a specific example of the process of step S205. The signal processing circuit 70 executes the processes of steps S301 to S317 shown in FIG. 12B.

[0181] In step S301, the signal processing circuit 70 calculates the movement of the gaze coordinates based on the gaze data J1. The movement of the gaze coordinates can be, for example, the difference between the gaze coordinates and the gaze coordinates at the timing of the previous sampling.

[0182] In step S303, the signal processing circuit 70 determines whether the difference is less than the threshold value. For example, taking the horizontal direction as the X-axis direction and the vertical direction as the Y-axis direction, the signal processing circuit 70 determines whether the difference in the gaze coordinates is less than a predetermined threshold value such as ±10 for each of the X coordinate and the Y coordinate.

[0183] If the difference in the gaze position is equal to or greater than the threshold value, it can be determined that the gaze has moved. In that case, the subsequent processing is omitted and the process proceeds to step S206. On the other hand, if the difference in the gaze position is less than the threshold value, the process proceeds to step S305.

[0184] In step S305, the signal processing circuit 70 calculates the dwell time at that gaze position. The dwell time can be calculated by adding the time interval of sampling to the dwell time calculated at the timing of the previous sampling.

[0185] In step S307, the signal processing circuit 70 determines whether the calculated dwell time exceeds a predetermined threshold value. If the dwell time is less than the threshold value, the subsequent processing is omitted and the process proceeds to step S206. On the other hand, if the dwell time is equal to or greater than the threshold value, it is determined that the user is gazing at that location, and the process proceeds to step S313.

[0186] In step S313, the signal processing circuit 70 determines whether the amount of change from the reference value of the cerebral blood flow is equal to or greater than a threshold value based on the cerebral blood flow data J2 generated in step S204. For example, a threshold value for the amount of change from the reference value can be set in advance for each of the oxygenated hemoglobin concentration and deoxygenated hemoglobin concentration in cerebral blood. The signal processing circuit 70 determines whether the amount of change from the reference value is equal to or greater than the respective threshold value for each of the oxygenated hemoglobin concentration and deoxygenated hemoglobin concentration in cerebral blood. If this determination is No, since it is estimated that the user's level of interest is low, subsequent processing is omitted and the process proceeds to step S206. If this determination is Yes, it is estimated that the user's level of interest is high, and the process proceeds to step S315.

[0187] Here, with reference to FIG. 13, an example of the determination process based on the cerebral blood flow data J2 will be described. FIG. 13 shows an example of the change over time in the concentration of each of oxygenated hemoglobin (Oxy-Hb) and deoxygenated hemoglobin (Deoxy-Hb) in cerebral blood. The horizontal axis represents time, and the vertical axis represents the amount of change in each concentration from the reference value. This data was obtained by measuring the blood flow in the frontal lobe using the biological measurement device 100 of the present embodiment while keeping the line of sight fixed. An experiment was conducted in which objects of interest and non-interesting objects were sequentially presented in front of the fixed line of sight without moving the line of sight. From the results shown in FIG. 13, it can be seen that the tendency of the change over time in cerebral blood flow is different when looking at something interesting and when looking at something uninteresting. According to the results shown in FIG. 13, when looking at something interesting, the concentration of Oxy-Hb tends to increase and the concentration of Deoxy-Hb tends to decrease. Therefore, the user's level of interest can be estimated based on the amount of change from the reference value of one or both of the concentration of Oxy-Hb and the concentration of Deoxy-Hb.

[0188] Note that in the present embodiment, the presence or absence of the user's interest is estimated, but other mental states or physical states such as fear, drowsiness, pleasure, or fatigue can also be estimated by the same method. For example, in Non-Patent Document 2, Oxy-Hb increases with an unpleasant stimulus, and with the anxiety caused by a fear stimulus, the right It has been reported that the Oxy-Hb in the prefrontal cortex (PFC) increases. Furthermore, there are reports that the oxy-Hb on the right side increases in response to the cognitive load imposed by a mental arithmetic task, and that the cerebral blood flow of Oxy-Hb in the orbitofrontal cortex (OFC) region increases when seeing a smiling face. Non-Patent Document 3 discloses that the stronger the subjective drowsiness, the smaller the increase or the decrease in Oxy-Hb in the dorsolateral prefrontal cortex (DLPFC) when performing a verbal fluency task compared to when not feeling drowsy. Non-Patent Document 4 discloses that when examining the activation of the prefrontal lobe by a verbal fluency task using near-infrared spectroscopy (NIRS), the more the user feels fatigue, the smaller the increase in Oxy-Hb in the ventrolateral prefrontal cortex (VLPFC) on both sides. Furthermore, Non-Patent Document 5 discloses that the Oxy-Hb in the prefrontal cortex decreases when the user is engrossed in and concentrating on a task. Therefore, the biological measurement method of the present embodiment can be used not only for estimating the degree of the user's interest but also for the purpose of estimating other mental states or physical states.

[0189] Referring again to FIG. 12B. In step S313, if it is determined to be Yes, the process proceeds to step S315. In step S315, the signal processing circuit 70 acquires the time information at that time in the content being viewed by the user.

[0190] In step S317, the signal processing circuit 70 integrates the line-of-sight coordinates and the time at that time and stores them in the server 200. Thereby, the position of the object in the content being gazed at by the user and the time are stored in association with each other.

[0191] Note that the operation shown in FIG. 12B is merely an example, and various modifications are possible. For example, as shown in FIG. 12C, the processes of steps S301 to S307 and the process of step S313 may be interchanged. In the example of FIG. 12C, first, in step S313, it is determined whether the amount of change from the reference value of cerebral blood flow is equal to or greater than a threshold value. If this determination is No, the subsequent processes are omitted and the process proceeds to step S206. Only when this determination is Yes, the process proceeds to the processes after step S301. Even with such an operation, similar effects can be obtained. In the example of FIG. 12A, the operation of step S205 is performed in real time while the user is viewing the content. The operation of step S205 may be performed after all the operations of steps S201 to S204 during viewing are completed. In that case, the signal processing circuit 70 may repeatedly perform the operation of FIG. 12A or FIG. 12B at a predetermined sampling interval.

[0192] By the above operations, it is possible to identify which object the user is interested in among the content being viewed by the user. The server 200 may accumulate data associating the gaze coordinates and the time for each user and for each content. The control circuit 60 in the biometric measurement device 100 of each user may perform control such as changing the stimulus or content presented to the user based on the data accumulated in the server 200.

[0193] [4-2. Application Processing 1 after Determination of User Interest] Next, an example of the processing after the determination of the user's interest will be described.

[0194] FIG. 14A is a flowchart showing an example of a process for identifying an object in the content in which the user is interested.

[0195] In step S401, the signal processing circuit 70 reads the content data D1 from the server 200. The data D1 is content data such as video, audio, application, game, task, etc. presented to the user.

[0196] In step S402, the signal processing circuit 70 reads the data D2 of the line-of-sight coordinates and time from the server 200. The data D2 is pre-recorded in the aforementioned step S317. It is recorded.

[0197] In step S403, the signal processing circuit 70 identifies one or more objects in the content that the user is interested in based on the data D1 and D2. The object can be, for example, a specific person, animal, plant, machine, building, or scenery. For details of the processing in step S403, refer to FIG. 14B and it will be described later.

[0198] In step S404, the signal processing circuit 70 records the identification result of one or more objects in the content that the user is interested in in the server 200.

[0199] Hereinafter, the details of the operation in step S403 will be described.

[0200] FIG. 14B is a flowchart showing the details of the operation in step S403. The signal processing circuit 70 repeats the processing from steps S511 to S514 shown in FIG. 14B for all scenes of the content. Thereafter, the signal processing circuit 70 executes the processing in step S521.

[0201] In step S511, the signal processing circuit 70 acquires the position information of each object in the content. The object can be, for example, an object such as a person, animal, or plant appearing in an application or video. The position information of each object may be included in the content data D1, or may be generated by the signal processing circuit 70 analyzing the data D1.

[0202] In step S512, the signal processing circuit 70 acquires information on the presentation time of each object. Here, the presentation time indicates at what timing in the content the object appears. This time information may also be included in the data D1 in advance, or may be generated by the signal processing circuit 70 analyzing the data D1.

[0203] In step S513, the signal processing circuit 70 integrates the position information and time information of the object and records them in the memory or the server 200.

[0204] The processes of steps S511 to S513 are repeated until it is determined in step S514 that the processing for all scenes of the content is completed. When the processing of steps S511 to S513 is completed for all scenes of the content, the process proceeds to step S521.

[0205] In step S521, the signal processing circuit 70 identifies the objects that the user is interested in by comparing the gaze coordinates and time indicated by the data D2 with the positions and presentation times of the respective objects. For example, when the gaze coordinates and the position coordinates of the object are both expressed in the same display coordinate system, the coordinates can be compared simply by comparison. When step S521 ends, the process proceeds to step S404 shown in FIG. 14A, and the identification result is recorded.

[0206] [4-3. Application processing 2 after determination of user interest] Next, an example of a process of changing the content presented to the user according to the user's object of interest will be described.

[0207] In the present embodiment, while content such as an application or a video is being presented to the user, the biological measurement device 100 starts generating data indicating the appearance of the user's face and data indicating the state of cerebral blood flow. If the object of the user's interest becomes clear based on the generated data, the content such as the application or video to be presented next is The content can be appropriately changed. For example, when it is found that the user is interested in the accommodation during the viewing of the travel introduction video, control such as changing the content of the video to the content focused on the accommodation information is possible. On the other hand, when the user shows no reaction of interest during the viewing of the application or video, it is also possible to present the content of the pre-determined template.

[0208] In this way, based on the moving image data showing the appearance of the user's face and the moving image data showing the state of the user's cerebral blood flow, the stimulus given to the user can be controlled. At this time, the stimulus control unit 63 in the control circuit 60 can cause the stimulus device 10 to output at least one of video and sound associated with the classification of the biological reaction. The classification of the biological reaction can be, for example, a binary classification such as "interested" and "not interested". Alternatively, it may be a classification in which the degree of interest or other states is classified into three or more levels.

[0209] FIG. 15 is a flowchart showing an example of a process of changing the content of the content according to the user's object of interest.

[0210] In step S601, the signal processing circuit 70 reads the information of the content presented to the user from the server 200. In step S602, the signal processing circuit 70 reads the information of the object that the user is interested in, which was recorded in step S404 of FIG. 14A.

[0211] In step S603, the signal processing circuit 70 accesses the server 200 and searches for the content of the object of interest. For example, when it is found that the user is interested in real estate, the signal processing circuit 70 searches for the content including real estate information.

[0212] In step S604, the signal processing circuit 70 determines whether there is content corresponding to the user's object of interest. If there is no corresponding content, the process proceeds to step S606, and the content of a pre-determined template is presented. If there is corresponding content, the process proceeds to step S605, and the content of the object of interest is presented.

[0213] The above operations are repeated until it is determined in step S607 that the content has ended. By the above operations, appropriate content corresponding to the user's object of interest can be presented to the user.

[0214] The operation shown in FIG. 15 may be executed after the operation shown in FIG. 14A is completed, or may be executed simultaneously and in parallel with the operation shown in FIG. 14A.

[0215] FIG. 16 schematically shows a situation in which a stimulation device 10 equipped with a display connectable to a network is applying a video presented by a server 200 as a stimulus to a user. The user is given a stimulus such as a video from a stimulation device 10 such as a PC or a television. While the user is watching the video, a biological measurement device 100 built-in or externally attached to the PC or television acquires the user's cerebral blood flow information and appearance information.

[0216] FIG. 17 is a diagram showing an example of a system that changes the content of the content presented to the user according to the user's object of interest. In this example, a biological measurement device 100 built-in or externally attached to an information device such as a PC, a television, a tablet, a smartphone, or a head-mounted display, a signal processing circuit 70, and a server 200 are arranged in different locations and are connected to each other via a network 500. As in this example, the signal processing circuit 70 shown in FIG. 1B may be provided outside the biological measurement device 100.

[0217] In this example, the cerebral blood flow information and appearance information acquired from the user by the biological measurement device 100 are transmitted to the signal processing circuit 70 via the network 500. The signal processing circuit 70 determines the user's object of interest using the received information. Based on the determined object of interest, the scene in the video to be presented to the user next is determined, and the data indicating the scene is transmitted to the stimulation device 10 such as a display. The stimulation of the video presented to the user can change in real time or at regular intervals according to the user's object of interest. For example, the video is composed of a plurality of scenes that are played continuously. For scenes after scene 2, the content can be determined according to the classification of the user's interest in the previous scene.

[0218] [5. Other Embodiments] The above-described embodiments are merely examples, and various modifications may be made. Hereinafter, the description will focus on the points different from the above-described examples of the configuration and operation, and the description of the common parts will be omitted.

[0219] [5-1. Understanding Degree Judgment Based on Pupil Diameter] For example, based on the signal output from the image sensor 30, the pupil diameter of the user may be detected. When the brain tries to work desperately, the pupil diameter is dilated by the action of the autonomic nervous system, and it is constricted otherwise. The information on the pupil diameter of the user can be obtained by the signal processing circuit 70 equipped with pupil recognition technology. The signal processing circuit 70 can also determine the degree of understanding of the user during learning based on the detected pupil diameter and the information indicating the change in cerebral blood flow generated in step S204 shown in FIG. 12A. For example, the biometric measurement device 100 can determine whether the user is following the conversation by detecting the pupil diameter and the change in cerebral blood flow of the user during a foreign language lesson. If the pupil diameter becomes smaller and no change in cerebral blood flow is detected, there is a possibility that the user does not understand the content even if nodding. In that case, a system can be constructed to notify the instructor to that effect. Alternatively, an information transmission terminal or an AI robot equipped with the biometric measurement device 100 may read the degree of understanding of the user based on the information on the pupil diameter and the change in cerebral blood flow of the user detected from the image data obtained by the image sensor 30. The technology of the present disclosure can be applied to a human-machine interface that flexibly changes the information presented to the user or the content of the conversation according to the read degree of understanding. When the pupil diameter of the user becomes larger, there is a possibility that the user is at a loss to understand. In that case, the provision of information or the conversation may be repeated or slowed down.

[0220] [5-2. Application Example Using a Head-Mounted Display or a Smartphone] The biometric measurement device 100 may be incorporated into a device such as a head-mounted display or a smartphone.

[0221] FIG. 18 shows an embodiment of a head-mounted display equipped with the biometric measurement device 100. FIG. 19 shows an embodiment of a smartphone equipped with the biometric measurement device 100.

[0222] In an embodiment of a head-mounted display or a smartphone, a camera built in or externally attached to the head-mounted display or the smartphone acquires cerebral blood flow information and facial appearance information. In an embodiment of a head-mounted display or a smartphone, the following uses are possible.

[0223] For example, based on the gaze data J1 in step S202 of FIG. 12A and the cerebral blood flow data J2 in step S204, the position on the display that the user is interested in can be specified. Based on the information of the specified position, for example, the following controls are possible. · Emphasize the position on the display that the user is interested in. · Increase the resolution near the position. · Display a pointer or cursor at the position. · Identify an object of interest and display detailed information about the object. Note that the object may be real or virtual. · Feedback the information of the object of interest to the information source. For example, record data indicating the content fed back to the server. · Record interest information in the server in association with user information. · Turn off the power of the smartphone or set it to the sleep state when the user is looking away for a long time.

[0224] Alternatively, instead of the gaze data J1 in step S202 of FIG. 12A, information indicating the frequency or time interval of the user's blinks may be used as information indicating changes in the facial appearance. Based on the information indicating the frequency or time interval of blinks and the information indicating changes in cerebral blood flow, the fatigue level or concentration of the user can be estimated. Based on the estimated fatigue level or concentration information, for example, the following controls are possible. · Display a message prompting the user to take a break. · Display an image with a relaxing effect. · Monitor the user's fatigue level, workload, difficulty of the task, or proficiency in the task. · Reduce the illuminance of the display. · Monitor the user's proficiency with an application or game. · Turn the smartphone on or off.

[0225] In addition, based on the information on the blink frequency or time interval and the information on the change in cerebral blood flow, it is also possible to estimate the user's drowsiness. Based on the estimated drowsiness information, for example, the following controls can be performed. · Turn the power of the head-mounted device on or off. · Display a message to prompt refreshing. · Display an image with an awakening effect.

[0226] Furthermore, based on the information on the pupil diameter and the information on the change in cerebral blood flow, it is possible to obtain information related to the degree of interest in the content or the proper evaluation of the illuminance. Based on the information related to the degree of interest in the content or the proper evaluation of the illuminance, controls such as determining whether to switch the content, increasing the illuminance, or decreasing the illuminance are possible.

[0227] In an embodiment of the present disclosure, it is possible to obtain data indicating the change over time of the face image and data indicating the change over time of the cerebral blood flow. Based on these data, it is also possible to obtain information related to the identification or authentication of the user. Based on the information related to the identification or authentication of the user, for example, the following controls can be performed. · Select the content to be displayed. · Do not start the device in the case of an unauthorized user. · Set an upper limit on the usage time to suppress fatigue accumulation.

[0228] By using not only the face image but also the cerebral blood flow information, it is possible to prevent impersonation using a silicon mask used in special makeup, a photo, or a mannequin.

[0229] [5-3. Application Example for In-Vehicle Use] The biological measurement device 100 can also be mounted on a vehicle and used. FIG. 20 is a diagram schematically showing a situation where a user is using the biological measurement device 100 mounted on a vehicle.

[0230] In this example, the biological measurement device 100 may be built into the vehicle or externally attached. The biological measurement device 100 may be a device specialized for biological measurement, or may be built into other devices such as a driving recorder or car navigation. The stimulus may be presented to a person other than the user.

[0231] When the biological measurement device 100 is mounted on a vehicle, the following uses can be considered. For example, information regarding the line of sight or blinking can be acquired as information indicating a change in the appearance of the face. Based on the information regarding the line of sight or blinking and the change information of cerebral blood flow, for example, information indicating an acute illness such as myocardial infarction or stroke, or an abnormal conscious state such as drinking alcohol can be acquired. Based on the information indicating an acute illness or an abnormal conscious state, for example, the following controls can be performed. · Stop the vehicle. · Flash the hazard lamp or the blue lamp of the bus. · Switch the electronic display board of the bus to "Emergency occurred". · Switch to automatic driving and safely stop the vehicle. · Emit an audio message prompting rest or attention. · Set a limit on the maximum speed. · Do not operate the engine.

[0232] In such an example, the stimulating device 10 may be, for example, a car light or an electronic display board. In that case, the stimulus may be presented to a person other than the user.

[0233] Furthermore, information indicating interest in the fixation point can also be obtained based on information on the pupil diameter or line of sight and information on changes in cerebral blood flow. Based on the information indicating interest in the fixation point, for example, control such as displaying detailed information about the object at the fixation point, such as recommendation information or parking space availability information, or issuing a message regarding the object can be performed.

[0234] In addition, information related to the identification or authentication of the user can be obtained based on the change information of the face image and the change information of the cerebral blood flow. Based on the information related to the identification or authentication of the user, when an unauthorized user attempts to drive, control such as not starting the device or stopping the engine operation can be performed.

[0235] [5-4. Application Examples at the Time of Nursing or Hospitalization] Furthermore, the biological measurement device 100 may be installed and used on the bed at the time of nursing or hospitalization. FIG. 21 shows an example in which the subject, that is, the patient, uses the biological measurement device 100 on the bed.

[0236] In such an example, as the change information of the facial appearance, information regarding the patient's line of sight or expression can be obtained. Based on the information regarding the line of sight or expression and the change information of the cerebral blood flow, information indicating the patient's physical condition such as cognitive state, pain, urinary urge, or bowel movement urge can be obtained. Based on the information indicating the patient's physical condition, control such as transmitting information to the caregiver, nurse, or doctor is possible.

[0237] As the change information of the facial appearance, information regarding the pupil diameter can also be obtained. Based on the obtained information regarding the pupil diameter and the change information of the cerebral blood flow, information indicating the level of consciousness or vital state can be obtained. Based on the information indicating the level of consciousness or vital state, control such as transmitting information to the caregiver, nurse, or doctor is possible.

Industrial Applicability

[0238] The biological measurement device in the present disclosure can be used in various devices such as a camera, a measuring instrument, or an information device that acquires internal information of a user non - contact.

Explanation of symbols

[0239] 10 Stimulation device 20 Light source 30 Image sensor 60 Control circuit 70 Signal processing circuit 80 Communication circuit 90 Recording medium 100 Biological measurement device 200 Server

Claims

1. An information processing method in a computer, comprising: causing a light source to repeatedly emit light pulses irradiated to a user; causing an image sensor to output first image data based on a reflected light pulse generated by irradiating the user with the light pulse, and second image data corresponding to a light quantity distribution of a part of components of the reflected light pulse; generating appearance information corresponding to the appearance of the user's face based on the first image data; determining the state of the user based on the appearance information and a change over time of the second image data; A method comprising the above.

2. The state includes at least one of a cognitive state and a conscious state. The method according to claim 1.

3. The image sensor is provided in a vehicle. The method according to claim 1 or 2.

4. Further comprising controlling the vehicle based on the determined state. The method according to claim 3.

5. The resolution of the first image data is higher than the resolution of the second image data. The method according to claim 1 or 2.

6. Further comprising performing a process of reducing the resolution of the second image data, wherein the state of the user is determined based on the first image data and the second image data after the process is performed. The method according to claim 1 or 2.

7. A light source that repeatedly emits light pulses irradiated to a user, an image sensor that outputs first image data based on a reflected light pulse generated by irradiating the user with the light pulse, and second image data corresponding to a light quantity distribution of a part of components of the reflected light pulse, a processing circuit, wherein the processing circuit: generates appearance information corresponding to the appearance of the user's face based on the first image data; determines the state of the user based on the appearance information and a change over time of the second image data. A system.

Images