Control methods, programs, systems, and vehicles

Abstract

We provide technology that can further improve authentication accuracy. [Solution] The control method performed by a computer includes: emitting illumination light from a light source provided in the vehicle to project a dot pattern onto the user's face; having an image sensor provided in the vehicle detect the light from the face and generate an image signal; performing user authentication based on the image signal; and controlling the vehicle based on the result of the user authentication.

Description

Technical Field

[0001] The present disclosure relates to a control method, a program, a system, and a vehicle.

Background Art

[0002] In the field of personal identification, authentication has been transitioning from password authentication to biometric authentication. In password authentication, authentication is performed based on the password entered by the user. On the other hand, in biometric authentication, authentication is performed based on information related to human physical characteristics. Biometric authentication has the advantage of having less risk of forgetting, leakage, or brute-force attacks. In biometric authentication, for example, a part of the user's body is irradiated with light, and reflected light is observed to obtain personal identification information.

[0003] Patent Document 1 discloses a method for improving the accuracy of user biometric authentication. In the method described in Patent Document 1, fingerprint authentication and vein authentication are combined to perform user biometric authentication. Specifically, the surface of the user's finger is irradiated with light having a wavelength of 900 nm for fingerprint authentication and light having a wavelength of 750 nm for vein authentication. Authentication of the user is performed based on the reflected light.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure provides a technology that can further improve authentication accuracy.

Means for Solving the Problems

[0006] An identification device according to one aspect of the present disclosure comprises a light source, an image sensor, a memory storing biometric data indicating the physical characteristics of a user, and a processor. The processor causes the light source to emit pulsed light with a pulse width of 0.2 ns to 1 μs or less that is irradiated onto the user, causes the image sensor to detect at least a portion of the reflected pulsed light that has returned from the user, causes the processor to output a signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the reflected pulsed light, and identifies the user by comparing the signal with the biometric data. [Effects of the Invention]

[0007] The technology disclosed herein enables more accurate biometric authentication. [Brief explanation of the drawing]

[0008] [Figure 1A] Figure 1A is a schematic diagram showing an example of the identification device in this embodiment. [Figure 1B] Figure 1B shows an example of the time variation in the intensity of light reaching the image sensor. [Figure 1C] Figure 1C shows the width of the input pulse light on the horizontal axis and the amount of light detected by the sensor on the vertical axis. [Figure 1D] Figure 1D shows an example of a schematic configuration of a single pixel in an image sensor. [Figure 1E] Figure 1E shows an example of the configuration of an image sensor. [Figure 1F] Figure 1F shows an example of the actions within one frame. [Figure 1G] Figure 1G is a flowchart illustrating the general operation of the control circuit. [Figure 2A] Figure 2A is a diagram illustrating an example of an optical signal that reaches an image sensor when rectangular pulsed light is emitted from a light source and the light returns from the user. [Figure 2B] Figure 2B illustrates another example of an optical signal where a rectangular pulse of light is emitted from a light source and the light returns from the user to the image sensor. [Figure 3A] FIG. 3A is a diagram showing an example of a timing chart when detecting a surface reflection component. [Figure 3B] FIG. 3B is a diagram showing an example of a timing chart when detecting an internal scattering component. [Figure 4A] FIG. 4A is a diagram schematically showing an example of the distribution of two-dimensional pattern light projected from a light source to a user. [Figure 4B] FIG. 4B is a diagram schematically showing another example of the distribution of two-dimensional pattern light projected from a light source to a user. [Figure 4C] FIG. 4C is a diagram schematically showing still another example of the distribution of two-dimensional pattern light projected from a light source to a user. [Figure 5A] FIG. 5A is a diagram schematically showing a state where a user is being photographed by a general camera. [Figure 5B] FIG. 5B is a diagram schematically showing an example of photographing a user by a surface reflection component in the present embodiment. [Figure 5C] FIG. 5C is a diagram schematically showing an example of photographing a user by an internal scattering component in the present embodiment. [Figure 6] FIG. 6 is a diagram explaining the principle of measuring the round-trip time of pulsed light by an indirect TOF method. [Figure 7] FIG. 7 is a diagram schematically showing an example of photographing a user by a TOF method in the present embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of the processing executed by a control circuit. [Figure 9] FIG. 9 is a diagram schematically showing an example of mounting the identification device in the present embodiment in a vehicle interior. [Figure 10] FIG. 10 is a diagram schematically showing an example of applying the identification device in the present embodiment to a mobile terminal.

MODE FOR CARRYING OUT THE INVENTION

[0009] The embodiments described below all show comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions of the components, etc. shown in the following embodiments are just examples and not the gist of limiting the present disclosure. Also, among the components in the following embodiments, the components not described in the independent claims indicating the top-level concept are described as optional components.

[0010] In the present disclosure, all or part of a circuit, unit, device, member or part, or all or part of the functional blocks in a block diagram may be executed by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or a large scale integration (LSI). The LSI or IC may be integrated on one chip or may be configured by combining a plurality of chips. For example, the functional blocks other than the memory elements may be integrated on one chip. Here, although it is called LSI or IC, the name may change depending on the degree of integration and may be called system LSI, very large scale integration (VLSI), or ultra large scale integration (ULSI). A Field Programmable Gate Array (FPGA) programmed after the manufacture of the LSI, or a reconfigurable logic device capable of reconfiguring the bonding relationship inside the LSI or setting up the circuit sections inside the LSI can also be used for the same purpose.

[0011] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROM, optical disk, or hard disk drive, and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0012] The embodiments will be described in detail below with reference to the drawings.

[0013] (Embodiment) [1. Identification device] First, the configuration of the identification device 100 in the embodiment of this disclosure will be described using Figures 1A to 1G.

[0014] Figure 1A is a schematic diagram showing an example of the identification device 100 in this embodiment. The identification device 100 comprises a light source 10, an image sensor 20, a memory 25, a control circuit 30, and a signal processing circuit 40. The image sensor 20 includes a plurality of photoelectric conversion elements 22 and a plurality of charge storage units 24. In this specification, the control circuit 30 and the signal processing circuit 40 are sometimes collectively referred to as the "processor".

[0015] The light source 10 emits pulsed light that is shone onto the head of user 1. The image sensor 20 detects at least a portion of the pulsed light that returns from the head of user 1. The control circuit 30 controls the light source 10 and the image sensor 20. The signal processing circuit 40 processes the signal output from the image sensor 20.

[0016] In this embodiment, the control circuit 30 includes a light source control unit 32 that controls the light source 10 and a sensor control unit 34 that controls the image sensor 20. The light source control unit 32 controls the intensity, pulse width, emission timing, and / or wavelength of the pulsed light emitted from the light source 10. The sensor control unit 34 controls the timing of signal accumulation at each pixel of the image sensor 20.

[0017] The following provides a more detailed explanation of each component.

[0018] [1-1.Light source 10] Light source 10 irradiates light onto user 1's head, for example, the forehead. The light emitted from light source 10 and reaching user 1 is divided into a surface reflection component I1 that is reflected by the surface of user 1 and an internal scattering component I2 that is scattered inside user 1. The internal scattering component I2 is a component that is reflected once, scattered, or multiple times inside the living body. When light is irradiated onto user 1's forehead, the internal scattering component I2 refers to a component that reaches a part about 8 mm to 16 mm inward from the surface of the forehead, for example, the brain, and returns to the identification device 100. Surface reflection component I1 includes three components: a direct reflection component, a diffuse reflection component, and a scattered reflection component. The direct reflection component is a reflection component where the angle of incidence and the angle of reflection are equal. The diffuse reflection component is a component that is diffused and reflected by the uneven shape of the surface. The scattered reflection component is a component that is scattered and reflected by the internal tissue near the surface. When light is irradiated onto user 1's forehead, the scattered reflection component is a component that is scattered and reflected inside the epidermis. Hereafter, in this disclosure, the surface reflection component I1 reflected from the surface of user 1 includes these three components. The surface reflection component I1 and the internal scattering component I2 change their direction of propagation due to reflection or scattering, and a portion of them reaches the image sensor 20.

[0019] The light source 10 generates pulsed light multiple times at predetermined time intervals or timings, in accordance with instructions from the control circuit 30. The pulsed light generated by the light source 10 may be, for example, a square wave with a fall time close to zero. The fall time is the period from when the intensity of the pulsed light begins to decrease until the decrease ends. The component of the pulsed light during the fall time is also called the trailing edge of the pulsed light. Generally, light incident on user 1 propagates through user 1 via various paths and exits from the surface of user 1 with a time difference. For this reason, the trailing edge of the internal scattering component I2 of the pulsed light has a broadened shape. When the area under test is the forehead, the broadening of the trailing edge of the internal scattering component I2 is about 4 ns. Considering this, the fall time of the pulsed light can be set to, for example, less than half of that, which is 2 ns or less. The fall time may be even less than half of that, which is 1 ns or less. On the other hand, the component of the pulsed light during the rise time can be used to detect the surface reflection component I1. The rise time is the period from when the intensity of the pulsed light begins to increase until the increase ends. The component of pulsed light during its rise time is also called the leading edge of the pulsed light.

[0020] The light source 10 may be formed by a combination of a light source such as a laser (LD) and a diffuser plate. Using a laser enables optical output with high time response. Optical output with high time response has a waveform with sharp rise or fall characteristics. Rise characteristics and fall characteristics are also called Tr characteristics and Tf characteristics, respectively. By placing the diffuser plate in front of the light source 10, the user 1 can also be illuminated with light in two dimensions.

[0021] The wavelength of the light emitted by the light source 10 can be any wavelength within the wavelength range of, for example, 650 nm to 950 nm. This wavelength range includes the red to near-infrared wavelength range. In this specification, the term "light" is used not only for visible light but also for infrared light. The above wavelength range is called the "window of life" and has the property of being relatively unabsorbed by water and skin in living organisms. When detecting living organisms, the detection sensitivity can be increased by using light within the above wavelength range.

[0022] In the identification device 100 of this embodiment, a light source 10 designed with consideration for the impact on the retina may be used to measure the user 1 non-contactually. For example, a light source 10 that satisfies Class 1 of the laser safety standards established in various countries may be used. If Class 1 is satisfied, the user 1 is irradiated with light of such low intensity that the exposure limit (AEL) is less than 1 mW. Note that the light source 10 itself does not have to satisfy Class 1. For example, a diffuser or ND filter may be placed in front of the light source 10 to diffuse or attenuate the light, thereby satisfying Class 1 of the laser safety standards.

[0023] Conventionally, streak cameras have been used to distinguish and detect information such as absorption coefficients or scattering coefficients at different locations in the depth direction within living organisms. For example, Japanese Patent Application Publication No. 4-189349 discloses an example of such a streak camera. These streak cameras used ultrashort pulse light with a pulse width of femtoseconds or picoseconds to measure with a desired spatial resolution. In this specification, "pulse width" means the full width at half maximum of the pulse. Compared to conventional methods using streak cameras, the identification device 100 in this embodiment can distinguish and detect the surface reflection component I1 and the internal scattering component I2. Therefore, the pulse light emitted by the light source 10 does not need to be ultrashort pulse light, and the pulse width can be arbitrarily selected.

[0024] When the forehead is illuminated with light, the amount of light from the internal scattering component I2 can be extremely small, ranging from a few thousandth to a few tens of thousandsth of the amount of light from the surface reflection component I1. Furthermore, considering laser safety standards, the amount of light that can be irradiated becomes extremely small, making it very difficult to detect the internal scattering component I2. Even in this case, if the light source 10 generates pulsed light with a relatively large pulse width, the integrated amount of the internal scattering component I2 with a time delay can be increased. This makes detection possible. This can increase the amount of light and improve the signal-to-noise ratio.

[0025] The light source 10 emits pulsed light with a pulse width of, for example, 3 ns or more. Generally, the temporal spread of light scattered within biological tissues such as the brain is about 4 ns. Figure 1B shows an example of the temporal change in the amount of light reaching the image sensor 20 for input pulsed light widths of 0 ns, 3 ns, and 10 ns. As shown in Figure 1B, as the width of the pulsed light from the light source 10 is increased, the amount of internal scattering component I2 that appears at the trailing end of the pulsed light returning from user 1 increases. Figure 1C is a diagram in which the width of the input pulsed light is shown on the horizontal axis and the amount of light detected by the image sensor 20 is shown on the vertical axis. The image sensor 20 is equipped with an electronic shutter. The results in Figure 1C were obtained under the condition that the electronic shutter was opened 1 ns after the trailing end of the pulsed light was reflected from the surface of user 1 and reached the image sensor 20. The reason for selecting this condition is that immediately after the trailing end of the pulsed light arrives, the ratio of the surface reflection component I1 to the internal scattering component I2 is high. As shown in Figure 1C, by setting the pulse width of the pulsed light emitted by the light source 10 to 3 ns or more, the amount of light detected by the image sensor 20 can be maximized.

[0026] Furthermore, the timing control resolution of the drive circuits for the light source 10 and the electronic shutter is approximately 0.2 ns. Therefore, the pulse width of the light source 10 is set to, for example, 0.2 ns or more.

[0027] The light source 10 may emit pulsed light with a pulse width of 5 ns or more, and even 10 ns or more. On the other hand, if the pulse width is too large, an increase in unused light will result in waste. Therefore, the light source 10 generates pulsed light with a pulse width of 50 ns or less, for example. Alternatively, the light source 10 may emit pulsed light with a pulse width of 30 ns or less, and even 20 ns or less.

[0028] Furthermore, the irradiation pattern of the light source 10 may be a uniform intensity distribution within the irradiation area. This is for the following reason: In the identification device 100 of this embodiment, the surface reflection component I1 is separated and reduced over time. As a result, the internal scattering component I2 can be detected even at the Null point, which is directly below the irradiation point on the user 1. Therefore, in the identification device 100 of this embodiment, the measurement resolution can be increased by irradiating the part of the user 1 being examined over a wide spatial area.

[0029] [1-2. Image Sensor 20] The image sensor 20 receives light emitted from the light source 10 and reflected or scattered by the user 1. The image sensor 20 has multiple light detection cells arranged in two dimensions and acquires two-dimensional information of the user 1 at once. This allows for the acquisition of two-dimensional information of the user 1 in a relatively short time compared to a line sensor that detects by sliding the part of the user 1 being examined. In this specification, light detection cells are also referred to as "pixels". The image sensor 20 is any image sensor, such as a CCD image sensor or a CMOS image sensor.

[0030] The image sensor 20 has an electronic shutter. The electronic shutter is a circuit that controls the timing of image acquisition. In this embodiment, the sensor control unit 34 in the control circuit 30 has the function of an electronic shutter. The electronic shutter controls the period of one signal accumulation, during which the received light is converted into a valid electrical signal and stored, and the period during which signal accumulation is stopped. The signal accumulation period can also be called the "exposure period". In the following description, the width of the exposure period may be called the "shutter width". The time from the end of one exposure period to the start of the next exposure period may be called the "non-exposure period". Hereinafter, the state of exposure may be called "OPEN", and the state of exposure being stopped may be called "CLOSE".

[0031] The image sensor 20 uses an electronic shutter to define the exposure and non-exposure periods in sub-nanosecond increments. For example, it can be adjusted in the range of 30 ps to 1 ns. Conventional TOF (Time Of Flight) cameras, intended for distance measurement, detect all the light emitted from the light source 10, reflected by the subject, and returned to compensate for the effect of the subject's brightness. Therefore, in conventional TOF cameras, the shutter width had to be larger than the pulse width of the light. In contrast, the identification device 100 of this embodiment does not need to compensate for the light intensity of the subject. For this reason, the shutter width does not need to be larger than the pulse width. Therefore, the shutter width can be set to a value of, for example, 1 ns or more and 30 ns or less. According to the identification device 100 of this embodiment, since the shutter width can be reduced, the effect of dark current included in the detection signal can be reduced.

[0032] When light is shone onto the forehead of user 1, the internal attenuation rate of the light is very large. For example, the emitted light can be attenuated to about 1 / 1,000,000th of the incident light. Therefore, the amount of light may be insufficient with only one pulse of irradiation to detect the internal scattering component I2. The amount of light is especially weak when irradiating at Class 1 of the laser safety standards. In this case, the light source 10 emits pulsed light multiple times, and the image sensor 20 is exposed multiple times by the electronic shutter in response, thereby integrating the detection signal and improving sensitivity.

[0033] The following describes an example configuration of the image sensor 20.

[0034] The image sensor 20 may comprise a plurality of pixels arranged two-dimensionally on the imaging surface. Each pixel may comprise a photoelectric conversion element, such as a photodiode, and one or more charge storage units. Below, an example is described in which each pixel comprises a photoelectric conversion element that generates a signal charge corresponding to the amount of light received by photoelectric conversion, a charge storage unit that stores the signal charge generated by the surface reflection component I1 of the pulsed light, and a charge storage unit that stores the signal charge generated by the internal scattering component I2 of the pulsed light. In the following example, the control circuit 30 causes the image sensor 20 to detect the surface reflection component I1 by detecting the portion of the pulsed light that has returned from the user 1's head before the start of the falling edge. The control circuit 30 also causes the image sensor 20 to detect the internal scattering component I2 by detecting the portion of the pulsed light that has returned from the user 1's head after the start of the falling edge. The light source 10 emits light of two different wavelengths, for example.

[0035] Figure 1D shows an example of the schematic configuration of one pixel 201 of the image sensor 20. Note that Figure 1D schematically shows the configuration of one pixel 201 and does not necessarily reflect the actual structure. In this example, pixel 201 includes a photodiode 203 that performs photoelectric conversion, a first floating diffusion layer 204 which is a charge storage section, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207, and a drain 202 that discharges signal charge.

[0036] Photons incident on each pixel due to the emission of a single pulse of light are converted into signal electrons, which are signal charges, by the photodiode 203. The converted signal electrons are then discharged to the drain 202 or distributed to one of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, or the fourth floating diffusion layer 207, according to a control signal input from the control circuit 30.

[0037] The emission of pulsed light from the light source 10, the accumulation of signal charge in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, and the discharge of signal charge to the drain 202 are repeated in this order. This repetitive operation is fast and can be repeated, for example, tens of thousands to hundreds of millions of times within the time of one frame. The time of one frame is, for example, about 1 / 30 of a second. The pixel 201 ultimately generates and outputs four image signals based on the signal charge accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207.

[0038] In this example, the control circuit 30 repeatedly emits a first pulse light with a first wavelength and a second pulse light with a second wavelength to the light source 10 in sequence. By selecting two wavelengths with different absorption rates for the internal tissue of user 1 as the first and second wavelengths, the state of user 1 can be analyzed.

[0039] First, the control circuit 30 emits a first pulse of light from the light source 10. During the first period when the surface reflection component I1 of the first pulse of light is incident on the photodiode 203, the control circuit 30 accumulates a signal charge in the first floating diffusion layer 204. Next, during the second period when the internal scattering component I2 of the first pulse of light is incident on the photodiode 203, the control circuit 30 accumulates a signal charge in the second floating diffusion layer 205. Then, the control circuit 30 emits a second pulse of light from the light source 10. During the third period when the surface reflection component I1 of the second pulse of light is incident on the photodiode 203, the control circuit 30 accumulates a signal charge in the third floating diffusion layer 206. Next, during the fourth period when the internal scattering component I2 of the second pulse of light is incident on the photodiode 203, the control circuit 30 accumulates a signal charge in the fourth floating diffusion layer 207.

[0040] In this manner, after the control circuit 30 starts emitting the first pulse light, it sequentially accumulates the signal charge from the photodiode 203 in the first floating diffusion layer 204 and the second floating diffusion layer 205 with a predetermined time difference. Subsequently, after the control circuit 30 starts emitting the second pulse light, it sequentially accumulates the signal charge from the photodiode 203 in the third floating diffusion layer 206 and the fourth floating diffusion layer 207 with the same predetermined time difference. This operation is repeated multiple times. In order to estimate the amount of ambient light and turbulent light, a period may be provided during which signal charge is accumulated in other floating diffusion layers (not shown) with the light source 10 turned off. By subtracting the amount of signal charge in the other floating diffusion layers from the amount of signal charge accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, a signal with the ambient light and turbulent light components removed can be obtained.

[0041] In this embodiment, the number of charge storage units per pixel is set to 4, but it may be designed to any number of 1 or more depending on the purpose. For example, if only one wavelength is used, the number of charge storage units may be 2. Also, if only one wavelength is used and only the surface reflection component I1 or only the internal scattering component I2 is detected, the number of charge storage units per pixel may be 1. Furthermore, even if two or more wavelengths are used, if imaging using each wavelength is performed in separate frames, the number of charge storage units may be 1. As will be described later, if the detection of the surface reflection component I1 and the detection of the internal scattering component I2 are performed in separate frames, the number of charge storage units may be 1.

[0042] Figure 1E shows an example of the configuration of the image sensor 20. In Figure 1E, the area enclosed by the dashed-dot frame corresponds to one pixel 201. Pixel 201 contains one photodiode. Although Figure 1E shows only four pixels arranged in a 2x2 grid, many more pixels can actually be arranged. Pixel 201 includes a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207. The signals accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are treated as if they were the signals of four pixels in a typical CMOS image sensor and output from the image sensor 20.

[0043] Each pixel 201 has four signal detection circuits. Each signal detection circuit includes a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. In this example, the reset transistor 310 corresponds to the drain 202 shown in Figure 1D, and the pulse input to the gate of the reset transistor 310 corresponds to the drain discharge pulse. Each transistor is, for example, a field-effect transistor formed on a semiconductor substrate. This is not limited to the above. As shown in the figure, one of the input and output terminals of the source follower transistor 309 is connected to one of the input and output terminals of the row selection transistor 308. One of the input and output terminals of the source follower transistor 309 is typically the source. One of the input and output terminals of the row selection transistor 308 is typically the drain. The gate, which is the control terminal of the source follower transistor 309, is connected to the photodiode 203. The signal charge generated by the photodiode 203 is stored in a floating diffusion layer, which is a charge storage area between the photodiode 203 and the source follower transistor 309. The signal charge is either a hole or an electron.

[0044] Although not shown in Figure 1E, the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are connected to the photodiode 203. A switch may be provided between the photodiode 203 and the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. This switch switches the conductivity between the photodiode 203 and each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 in response to signal accumulation pulses from the control circuit 30. This controls the start and stop of signal charge accumulation in each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The electronic shutter in this embodiment has a mechanism for such exposure control.

[0045] The signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are read out by the row selection circuit 302, which turns on the gate of the row selection transistor 308. At this time, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified according to the signal potential of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The analog signal resulting from this current read out from the vertical signal line 304 is converted into digital signal data by the analog-to-digital (AD) conversion circuit 307 connected to each column. This digital signal data is read out column by column by the column selection circuit 303 and output from the image sensor 20. The row selection circuit 302 and column selection circuit 303 read out one row, then the next row, and so on, until the signal charge information of the floating diffusion layer for all rows is read out. After reading out all the signal charges, the control circuit 30 resets all the floating diffusion layers by turning on the gate of the reset transistor 310. This completes the imaging of one frame. Similarly, by repeating high-speed imaging of frames, the imaging of a series of frames by the image sensor 20 is completed.

[0046] In this embodiment, an example of a CMOS-type image sensor 20 has been described, but the image sensor 20 may be of other types. The image sensor 20 may be, for example, a CCD type, a single-photon counting type element, or an amplified image sensor such as an EMCCD or ICCD.

[0047] Figure 1F shows an example of the operation within one frame in this embodiment. As shown in Figure 1F, the emission of the first pulsed light and the emission of the second pulsed light may be switched alternately multiple times within one frame. In this way, the time difference in the timing of acquiring detection images with two different wavelengths can be reduced, and even if user 1 is moving, it is possible to capture images with the first and second pulsed lights almost simultaneously.

[0048] [1-3. Memory 25] Memory 25 contains biometric data representing the physical characteristics of user 1, which have been acquired and registered in advance. This is stored in the memory. This biometric data is called a template. The memory 25 may be built into the identification device 100 or provided externally. The memory 25 may be, for example, one or more ROMs, optical disks, or hard disk drives.

[0049] The biometric data stored in memory 25 may be, for example, the image of user 1 itself, or data characterizing that image. The data characterizing the image may include, for example, information indicating the texture of user 1's skin surface, information indicating the distribution of user 1's blood vessels, or information indicating the irregularities of user 1's feature parts. "Information indicating the texture of user 1's skin surface" may be, for example, a two-dimensional distribution of frequency components obtained by Fourier transforming the pixel values ​​of a two-dimensional distribution of wrinkles or minute irregularities in a certain area of ​​user 1's skin surface. This transformation is effective when the wrinkles or minute irregularities show repetition of similar two-dimensional patterns. "Information indicating the distribution of user 1's blood vessels" may be, for example, image information showing the distribution of user 1's veins. "Information indicating the irregularities of user 1's feature parts" may be, for example, image information showing a two-dimensional distribution of distances indicating irregularities in at least one of user 1's orbit, nose, cheek, cheekbone, mouth, jaw, and below the ear. By storing data that characterizes user 1's image, it is possible to remove unnecessary information and improve recognition accuracy, as well as reduce the amount of data that needs to be stored.

[0050] [1-4. Control circuit 30 and signal processing circuit 40] The control circuit 30 causes the light source 10 to emit pulsed light to be irradiated onto user 1. The control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected pulsed light that has returned from user 1. The control circuit 30 adjusts the time difference between the pulsed light emission timing of the light source 10 and the shutter timing of the image sensor 20. Hereinafter, this time difference may be referred to as "phase" or "phase delay". The "emission timing" of the light source 10 is the timing when the rising edge of the pulsed light emitted by the light source 10 begins. The "shutter timing" is the timing when exposure begins. The control circuit 30 may adjust the phase by changing the emission timing, or by changing the shutter timing.

[0051] The control circuit 30 causes the image sensor 20 to output a signal indicating a two-dimensional image of user 1 corresponding to the detected light intensity distribution. The control circuit 30 causes the signal processing circuit 40 to process the signal output from the image sensor 20. The control circuit 30 may be configured to remove an offset component from the signal detected by each pixel of the image sensor 20. The offset component is a signal component due to ambient light such as sunlight or fluorescent light, or ambient light. When the light source 10 is not emitting light, that is, when the drive of the light source 10 is turned OFF, the offset component due to ambient light and ambient light is estimated by the image sensor 20 detecting a signal.

[0052] The control circuit 30 may be, for example, a combination of a processor and memory, or an integrated circuit such as a microcontroller that incorporates a processor and memory. The control circuit 30 performs tasks such as adjusting the light emission timing and shutter timing, estimating the offset component, and removing the offset component by having the processor execute a program recorded in the memory.

[0053] The signal processing circuit 40 is a circuit that processes the image signal output from the image sensor 20. The signal processing circuit 40 performs calculations such as image processing. The signal processing circuit 40 can be implemented, for example, by a combination of a digital signal processor (DSP), a programmable logic device (PLD) such as a field-programmable gate array (FPGA), or a central processing unit (CPU) or a graphics processing unit (GPU) and a computer program. The control circuit 30 and the signal processing circuit 40 may be an integrated single circuit or separate individual circuits. Furthermore, the signal processing circuit 40 may be a component of an external device, such as a server located in a remote location. In this case, external devices such as servers are equipped with communication means and communicate with the light source 10, image sensor 20, and control circuit 30 to send and receive data.

[0054] Figure 1G is a flowchart illustrating the general operation of the control circuit 30. The control circuit 30 generally performs the operations shown in Figure 1G. Here, as an example, the operation when only the detection of the internal scattering component I2 is performed is described. First, the control circuit 30 causes the light source 10 to emit pulsed light for a predetermined time (step S101). At this time, the electronic shutter of the image sensor 20 is in a state where charge accumulation has stopped. The control circuit 30 stops charge accumulation in the electronic shutter until the period in which a portion of the pulsed light is reflected from the surface of the user 1 and reaches the image sensor 20 is completed. Next, the control circuit 30 starts charge accumulation in the electronic shutter at the timing when the other portion of the pulsed light is scattered inside the user 1 and reaches the image sensor 20 (step S102). After a predetermined time has elapsed, the control circuit 30 stops charge accumulation in the electronic shutter (step S103). Subsequently, the control circuit 30 determines whether the number of times the above charge accumulation has been performed has reached a predetermined number (step S104). The predetermined number may be, for example, between 1,000 and 100,000. If step S104 is determined to be No, the control circuit 30 repeats steps S101 to S103 until it is determined to be Yes. If step S104 is determined to be Yes, the control circuit 30 causes the image sensor 20 to generate and output a signal showing a two-dimensional distribution image based on the total amount of signal charge accumulated in each floating diffusion layer for each pixel (step S105).

[0055] The above operation allows for the detection of light components scattered within the object being measured with high sensitivity. Note that multiple flashes and exposures are not mandatory and can be performed as needed. Furthermore, the operation of steps S101 to S105 shown in Figure 1G can also be applied when detecting only the surface reflection component I1.

[0056] The control circuit 30 authenticates user 1 by comparing the signal output from the image sensor 20 with the biometric data stored in the memory 25. The operation of the control circuit 30 is as follows: The control circuit 30 causes the signal processing circuit 40 to generate image data from the information obtained by the image sensor 20. The control circuit 30 compares this image data with the biometric data stored in the memory 25. If the memory 25 contains data characterizing user 1's image, the signal processing circuit 40 extracts the data characterizing that image from the generated image of user 1, and the control circuit 30 compares the data characterizing user 1's image with each other. Alternatively, the control circuit 30 may simply compare the correlation between the information acquired by the image sensor 20 and pre-registered information. Furthermore, the control circuit 30 may authenticate user 1 based on user characteristics acquired by machine learning such as deep learning or support vector machines.

[0057] [1-5.Optical system] The identification device 100 may include an imaging optical system that forms a two-dimensional image of user 1 on the light-receiving surface of the image sensor 20. The optical axis of the imaging optical system is substantially perpendicular to the light-receiving surface of the image sensor 20. The imaging optical system may include a zoom lens. When the position of the zoom lens changes, the magnification of the two-dimensional image of user 1 changes, and the resolution of the two-dimensional image on the image sensor 20 changes. Therefore, even if the distance to user 1 is far, it becomes possible to magnify the area to be measured and observe it in detail.

[0058] Furthermore, the identification device 100 may include a band-pass filter between the user 1 and the image sensor 20 that allows only light within or near the wavelength range emitted from the light source 10 to pass through. This reduces the influence of disturbance components such as ambient light. The band-pass filter is composed of a multilayer filter or an absorption filter. Considering the temperature of the light source 10 and the band shift due to grazing incidence to the filter, the bandwidth of the band-pass filter is 20 to 100 nm. A range of degree is acceptable.

[0059] Furthermore, the identification device 100 may be equipped with polarizing plates between the light source 10 and the user 1, and between the image sensor 20 and the user 1. In this case, the polarization directions of the polarizing plate placed on the light source 10 side and the polarizing plate placed on the image sensor side are in a relationship of orthogonal nicols. This prevents the specular reflection component of the surface reflection component I1 of the user 1, i.e., the component with the same incident angle and reflection angle, from reaching the image sensor 20. In other words, the amount of light that reaches the image sensor 20 from the surface reflection component I1 can be reduced.

[0060] [2. Operation of time-resolved imaging] In this embodiment, the identification device 100 can distinguish and detect the surface reflection component I1 and the internal scattering component I2 by time-resolved imaging.

[0061] The following describes an example of the operation of the identification device 100 in this embodiment.

[0062] As shown in Figure 1A, when the light source 10 irradiates the user 1 with pulsed light, a surface reflection component I1 and an internal scattering component I2 are generated. A portion of the surface reflection component I1 and the internal scattering component I2 reach the image sensor 20. The internal scattering component I2 passes through the inside of the user 1 from the light source 10 to the image sensor 20. That is, the optical path length of the internal scattering component I2 is longer than the optical path length of the surface reflection component I1. Therefore, the time it takes for the internal scattering component I2 to reach the image sensor 20 is, on average, delayed compared to the surface reflection component I1.

[0063] In time-resolved imaging, the surface reflection component I1 is acquired by the following operation.

[0064] Figure 2A shows an example of an optical signal that reaches the image sensor 20 when rectangular pulsed light is emitted from the light source 10 and returns from user 1. The horizontal axis represents time (t) for signals (a) to (d), and the vertical axis represents intensity for signals (a) to (c), and the OPEN or CLOSE state of the electronic shutter for signal (d). Signal (a) shows the surface reflection component I1. Signal (b) shows the internal scattering component I2. Signal (c) shows the combined component of the surface reflection component I1(a) and the internal scattering component I2(b). Signal (d) is a diagram showing the electronic shutter timing for acquiring the surface reflection component I1 of user 1.

[0065] As shown in signal (d) in Figure 2A, by closing the shutter, the component of the reflected light that arrives earlier on the image sensor 20 can be efficiently collected. The component that arrives earlier means that there is less scattering at user 1 and it contains surface information of user 1. The time during which light is substantially accumulated is only a short time at the leading edge of the pulse wave, but the shutter does not necessarily have to be closed only for that period. As shown in signal (d) in Figure 2A, if charge is accumulated just before closing, the shutter may be opened earlier than when the leading edge of the pulse wave reaches the image sensor 20. This eliminates the need for expensive picosecond-order high-time-resolution imaging devices. The identification device 100 in this embodiment can be composed of an inexpensive image sensor 20.

[0066] To perform the operation shown in signal (d) in Figure 2A, the control circuit 30 causes the image sensor 20 to detect at least a portion of the rise time component of the reflected pulse light and output a signal indicating a two-dimensional image of user 1. In this embodiment, the signal output from the image sensor 20 may include a signal obtained from at least a portion of the rise time component of the reflected pulse light.

[0067] Light source 10 emits a rectangular pulse wave. In this case, the pulse width does not need to be on the order of ps, but can be a few ns. Therefore, an inexpensive light source can be used. At the leading edge of the pulse wave... If the transistor characteristics are rapid, when shutter imaging is performed only on the front end, the inclusion of unwanted internal scattering components I2 with time delay into the acquired image can be minimized.

[0068] In time-resolved imaging, the internal scattering component I2 is acquired by the following operation.

[0069] Figure 2B shows another example of an optical signal reaching the image sensor 20 when rectangular pulsed light is emitted from the light source 10 and returns from the user 1. Signals (a) to (c) in Figure 2B show the same time evolution as signals (a) to (c) in Figure 2A. Signal (d) in Figure 2B shows the electron shutter timing for acquiring the internal scattering component I2.

[0070] As shown in signal (a) in Figure 2B, the surface reflection component I1 maintains a rectangular shape. On the other hand, as shown in signal (b) in Figure 2B, the internal scattering component I2 is the sum of light that has passed through various optical path lengths, and therefore exhibits a characteristic of having a tail at the trailing end of the pulsed light. That is, its fall time is longer than that of the surface reflection component I1. In order to extract a higher proportion of the internal scattering component I2 from the optical signal in signal (c) in Figure 2B, the electron shutter starts accumulating charge after the trailing end of the surface reflection component I1, as shown in signal (d) in Figure 2B. "After the trailing end of the surface reflection component I1" means when the surface reflection component I1 falls or thereafter. This shutter timing is adjusted by the control circuit 30. As mentioned above, the identification device 100 in this embodiment distinguishes and detects the surface reflection component I1 and the internal scattering component I2 that has reached deep into the user 1, so the pulse width and shutter width are arbitrary. Therefore, unlike conventional methods using streak cameras, the acquisition of the internal scattering component I2 can be achieved with a simple configuration, and costs can be significantly reduced.

[0071] To perform the operation shown in signal (d) in Figure 2B, the control circuit 30 causes the image sensor 20 to detect at least a portion of the falling-end component of the reflected pulse light and output a signal indicating a two-dimensional image of user 1. In this embodiment, the signal output from the image sensor 20 may include a signal obtained from at least a portion of the falling-end component of the reflected pulse light.

[0072] In signal (a) in Figure 2B, the trailing edge of the surface reflection component I1 falls vertically. In other words, the time from when the surface reflection component I1 starts falling to when it finishes falling is zero. However, in reality, the pulsed light emitted by the light source 10 may not be perfectly vertical, or there may be fine irregularities on the surface of user 1, or scattering within the skin may cause the trailing edge of the surface reflection component I1 to not fall vertically. Also, since user 1 is an opaque object, the light intensity of the surface reflection component I1 is much greater than that of the internal scattering component I2. Therefore, even if the trailing edge of the surface reflection component I1 extends slightly beyond the vertical falling position, the internal scattering component I2 may be obscured. Furthermore, due to the time delay associated with electron movement during the readout period of the electronic shutter, it may not be possible to achieve an ideal binary readout as shown in signal (d) in Figure 2B. Therefore, the control circuit 30 may slightly delay the shutter timing of the electronic shutter from immediately after the falling edge of the surface reflection component I1. For example, it may be delayed by about 0.5 ns to 5 ns. Alternatively, instead of adjusting the shutter timing of the electronic shutter, the control circuit 30 may adjust the light emission timing of the light source 10. The control circuit 30 adjusts the time difference between the shutter timing of the electronic shutter and the light emission timing of the light source 10. If the shutter timing is delayed too much, the already small internal scattering component I2 will decrease further. For this reason, the shutter timing may be kept near the trailing end of the surface reflection component I1. Since the time delay due to scattering by user 1, which can be expected from the sensor sensitivity, is 4 ns, the maximum amount of shutter timing delay is about 4 ns.

[0073] The light source 10 emits pulsed light multiple times, and the shutter timing for each pulse of light is the same phase. The amount of light detected for the internal scattering component I2 may be amplified by exposing the sensor multiple times.

[0074] Alternatively, instead of placing a band-pass filter between User 1 and the image sensor 20, or in addition to that, the control circuit 30 may estimate the offset component by taking images with the same exposure time while the light source 10 is not emitting light. The estimated offset component is subtracted from the signal detected by each pixel of the image sensor 20. This makes it possible to remove the dark current component and / or ambient light generated on the image sensor 20.

[0075] Next, we will describe an example of a method for detecting the surface reflection component I1 and the internal scattering component I2 per frame.

[0076] Figure 3A shows an example of a timing chart for detecting the surface reflection component I1. To detect the surface reflection component I1, for example, as shown in Figure 3A, the shutter may be opened before the pulsed light reaches the image sensor 20 and closed before the trailing end of the pulsed light reaches it. By controlling the shutter in this way, the inclusion of the internal scattering component I2 can be reduced. The proportion of light that passes near the surface of user 1 can be increased. In particular, the timing of closing the shutter may be set to immediately after the light reaches the image sensor 20. This makes it possible to detect a signal with a higher proportion of the surface reflection component I1, which has a relatively short optical path length. Other methods for acquiring the surface reflection component I1 include the image sensor 20 acquiring the entire pulsed light, or detecting it by irradiating continuous light from the light source 10.

[0077] Figure 3B shows an example of a timing chart for detecting the internal scattering component I2. By opening the shutter during the period when the trailing end of the pulse reaches the image sensor 20, the signal of the internal scattering component I2 can be acquired.

[0078] As in this embodiment, time-resolved imaging using the same camera or sensor makes it less likely for temporal and spatial discrepancies to occur. When acquiring signals for both the surface reflection component I1 and the internal scattering component I2 with the same sensor, the components to be acquired may be switched every frame, as shown in Figures 3A and 3B. Alternatively, as explained with reference to Figures 1D to 1F, the components to be acquired may be rapidly switched alternately within a single frame. In that case, the detection time difference between the surface reflection component I1 and the internal scattering component I2 can be reduced.

[0079] Furthermore, the signals for the surface reflection component I1 and the internal scattering component I2 may be acquired using light of two different wavelengths. When acquiring the surface reflection component I1 and the internal scattering component I2 using two wavelengths each, a method of rapidly switching between four types of charge accumulation within a single frame can be used, for example, as explained with reference to Figures 1D to 1F. Such a method can reduce the temporal delay of the detected signal.

[0080] [3. Operation of spatially resolved imaging] The surface reflection component I1 and the internal scattering component I2 can be acquired not only by the time-resolved imaging described above, but also by spatially resolved imaging.

[0081] Figures 4A to 4C schematically show examples of the distribution of two-dimensional patterned light projected from light source 10 onto user 1. In the examples shown in Figures 4A to 4C, white areas represent bright areas where light is present, and black areas represent dark areas where light is absent. Figure 4A shows a checkerboard pattern in which multiple bright and dark areas are periodically distributed. Figure 4B shows a dot pattern in which multiple dark areas are randomly distributed within bright areas. Figure 4C shows a dot pattern in which multiple bright areas are randomly distributed within dark areas.

[0082] The control circuit 30 causes the light source 10 to emit two-dimensional pattern light that projects a two-dimensional pattern onto the user 1. As shown in Figures 4A to 4C, the two-dimensional pattern light includes at least one bright area and at least one dark area. The two-dimensional pattern light can be obtained, for example, by placing a light-shielding mask having a two-dimensional distribution in front of the light source 10. Alternatively, the two-dimensional pattern light may be formed using a digital micromirror device (DMD) or a spatial light modulator (SLM). The two-dimensional pattern light can be continuous light or pulsed light.

[0083] As shown in Figures 4A to 4C, the light source 10 emits two-dimensional patterned light with spatially modulated intensity, such as a checkerboard pattern or a dot pattern. When the two-dimensional patterned light is projected onto the user 1, most of the light returns from the bright areas, and a small amount of light returns from the dark areas. The reflected light returning from the dark areas contains an internal scattering component I2 scattered within the user 1, and contains almost no surface reflection component I1.

[0084] To detect the internal scattering component I2 using two-dimensional pattern light, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected light returning from at least one dark area of ​​the user 1 onto which the two-dimensional pattern light is projected, and to output a signal corresponding to the intensity distribution of the above-mentioned portion of the detected reflected light. As a result, if the two-dimensional pattern light is pulsed light, the portion of the internal scattering component I2 from the front end to the rear end, as shown in signal (b) in Figure 2A or Figure 2B, can be obtained. In this embodiment, the signal output from the image sensor 20 may include the above-mentioned signal obtained from at least one dark area.

[0085] On the other hand, the reflected light returning from the bright area contains both a surface reflection component I1 and an internal scattering component I2. Therefore, the surface reflection component I1 can be calculated by subtracting the detection data from the nearby dark area from the detection data from the bright area. In this case, the spatial resolution decreases. As a countermeasure, instead of acquiring the signal in a single shot, it is conceivable to acquire the signal multiple times by, for example, shifting the distribution of the 2D pattern light or changing the distribution of the 2D pattern light itself. This makes it possible to acquire the surface reflection component I1 without reducing the spatial resolution.

[0086] To detect the surface reflection component I1 using two-dimensional pattern light, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected light returning from at least one dark area and at least a portion of the reflected light returning from at least one bright area of ​​the user 1 onto which the two-dimensional pattern light is projected. The control circuit 30 causes the image sensor 20 to output a signal corresponding to the intensity distribution of at least a portion of the reflected light returning from at least one dark area and a signal corresponding to the intensity distribution of at least a portion of the reflected light returning from at least one bright area. The control circuit 30 causes the signal processing circuit 40 to calculate the surface reflection component I1 by subtracting the signal acquired from at least one dark area from the signal acquired from at least one bright area. As a result, if the two-dimensional pattern light is pulsed light, the portion of the surface reflection component I1 from the front end to the rear end, as shown in signal (a) in Figure 2A or Figure 2B, can be obtained. In this embodiment, the signal output from the image sensor 20 may include the signal acquired from at least one bright area in addition to the signal acquired from at least one dark area.

[0087] Spatially resolved imaging and time-resolved imaging may be combined to acquire the surface reflection component I1 and the internal scattering component I2, respectively.

[0088] If the two-dimensional pattern light is pulsed light, the surface reflection component I1 is obtained by detecting at least a portion of the rise time of the reflected pulse light returning from at least one bright area. Alternatively, the internal scattering component I2 may be obtained by detecting at least a portion of the fall period of the reflected pulse light returning from at least one bright area or at least one dark area. The operation of the control circuit 30 and the signal processing circuit 40 when detecting the rise period component and the fall period component of the reflected pulse light is as described above.

[0089] [4. Biometric authentication using surface reflection component I1, and biometric authentication using internal scattering component I2] A specific example of biometric authentication using the identification device 100 of this embodiment will be described in comparison to a method using a general camera.

[0090] Figure 5A schematically shows how a typical camera 90 captures user 1. The light illuminating user 1 penetrates several millimeters into the surface of user 1. Therefore, conventional cameras 90 detect both the surface reflection component I1 and the internal scattering component I2 contained in the reflected light returning from user 1. Images containing both the surface reflection component I1 and the internal scattering component I2 may be slightly blurred. As a result, facial recognition using a typical camera 90 may experience an increase in false acceptance or false rejection. For example, it may mistakenly identify a twin sibling as the same person, or mistakenly identify someone as someone else if they have changed their hairstyle. In other words, the accuracy of facial recognition may deteriorate when using a typical camera 90.

[0091] Figure 5B schematically shows an example of capturing user 1 using the surface reflection component I1 in this embodiment. In the identification device 100 in this embodiment, as described above, the surface reflection component I1 can be detected by time-resolved imaging or spatial-resolved imaging. This makes it possible to detect the texture of user 1's skin surface, such as wrinkles or minute irregularities, more clearly. Authentication accuracy can be improved by using the results of comparing the information obtained from the surface reflection component I1 with the information indicating the texture of user 1's skin surface contained in the biometric data in memory 25. As a result, the rate of false rejection or false acceptance is reduced.

[0092] Figure 5C schematically shows an example of imaging of user 1 using the internal scattering component I2 in this embodiment. In the identification device 100 in this embodiment, as described above, the internal scattering component I2 can be detected by time-resolved imaging or spatial-resolved imaging. This makes it possible to detect the distribution of blood vessels such as veins of user 1 more clearly. Authentication accuracy can be improved by using the results of comparing the information obtained from the internal scattering component I2 with the information showing the distribution of user 1's blood vessels contained in the biometric data of memory 25. As a result, the rate of false rejection or false acceptance is reduced.

[0093] Authentication using the surface reflection component I1 and authentication using the internal scattering component I2 may be combined. For example, depending on the application, authentication may be considered successful only after both are authenticated, or successful authentication may be considered successful if only one of the two is authenticated. When the internal scattering component I2 is used for vein authentication, near-infrared light, which penetrates biological tissue more easily, may be used. On the other hand, when the surface reflection component I1 is used for facial recognition, short-wavelength light such as blue light, which does not penetrate biological tissue easily, may be used. In this way, light of different wavelengths may be used to detect the surface reflection component I1 and the internal scattering component I2.

[0094] The internal scattering component I2 contains more biological information from deeper parts of user 1 than the surface reflection component I1. Therefore, it can acquire different information from the surface reflection component I1, such as vein information or internal cells. Thus, it becomes possible to perform multiple different types of authentication, such as primarily using facial recognition for surface reflection component I1 and primarily using vein recognition for internal scattering component I2. Even if one authentication fails, the other authentication can compensate for the failure. This leads to stable and highly accurate authentication, ensuring high security. Furthermore, if someone were to impersonate user 1, they would need to forge both authentications, making tampering difficult. This makes it extremely difficult. Thus, the identification device 100 in this embodiment makes it possible to realize an authentication system that is more difficult to forge or tamper with.

[0095] In addition to independently performing multiple types of authentication, authentication may also be performed using machine learning with data that integrates the surface reflection component I1 and the internal scattering component I2. Including different features in the data increases the amount of information, which can improve the accuracy of recognition by machine learning.

[0096] In addition to the above, user 1 may be authenticated using the TOF method, which calculates the distance of an object from the round-trip time of pulsed light. The TOF method can obtain a two-dimensional distribution of the distance between the image sensor 20 and the surface of user 1. The two-dimensional distribution of the distance between the image sensor 20 and the surface of user 1 can also be called the three-dimensional distribution of the surface of user 1.

[0097] The distance between the image sensor 20 and the surface of user 1 is calculated as follows: Let τ be the round-trip time from when the pulsed light emitted from the light source 10 is reflected off the surface of user 1 and detected by the image sensor 20, and let c be the speed of light in air. Then, the distance d between the image sensor 20 and the surface of user 1 is d = cτ / 2.

[0098] There are two methods for obtaining the round-trip time τ: the direct TOF method and the indirect TOF method.

[0099] In the direct TOF method, the distance d to an object is calculated by directly measuring the round-trip time τ of light. In the direct TOF method, the time resolution required to achieve a distance resolution of Δd = 1 mm is Δτ = 2Δd / c ≈ 6.6 picoseconds. Thus, in the direct TOF method, high-speed imaging with picosecond-level time resolution is used to achieve millimeter-level distance resolution.

[0100] On the other hand, in the commonly used indirect TOF method, the distance d to the object is calculated by measuring the round-trip time τ of the pulsed light from the phase difference of the reflected pulsed light. The phase difference of the reflected pulsed light corresponds to the time shift between the emitted pulsed light and the reflected pulsed light. In the indirect TOF method, it is not the case that the shorter the pulse width Δt, the higher the measurement accuracy. In the indirect TOF method, in order to detect the phase difference, it is common to modulate the light intensity of the emitted light with a rectangular wave or a sine wave.

[0101] FIG. 6 is a diagram for explaining the principle of measuring the round-trip time τ of pulsed light by the indirect TOF method. Signal (a) in FIG. 6 shows the temporal change in the light intensity of the emitted pulsed light, and signals (b) and (c) in FIG. 6 show the temporal change in the intensity of the reflected pulsed light.

[0102] In the example shown in signal (a) in FIG. 6, the light intensity of the emitted pulsed light is modulated by a rectangular wave. In order to accurately measure the phase difference, basically, the light intensity of the emitted pulsed light is modulated at a speed of the same order as the round-trip time τ of the light.

[0103] By opening the electronic shutter in the image sensor 20 and measuring the intensity, the phase difference of the light pulse can be detected. In the example shown in signal (b) in FIG. 6, the image sensor 20 opens the electronic shutter after a time t0 from emitting the pulsed light and starts detecting the reflected pulsed light. The start time t0 is longer than the pulse width Δt of the pulsed light and shorter than the round-trip time τ of the light. That is, Δt < t0 < τ. The image sensor 20 accumulates signal charges corresponding to the amount of the reflected pulsed light every time Δt after opening the electronic shutter. Let the amount of signal charges accumulated during the first time Δt be S1, and the amount of signal charges accumulated during the next time Δt be S2. The difference between the round-trip time τ of the light and the start time t0 is equal to [S2 / (S1 + S2)]Δt. Therefore, the round-trip time of the light is τ = t0 + [S2 / (S1 + S2)]Δt. Indirectly obtained from the accumulated signal charge amounts S1 and S2 From the round-trip time τ of the light, the distance d to the object is calculated.

[0104] In the example shown in signal (c) in Figure 6, the reflected pulsed light returns to the image sensor 20 before the emitted pulsed light has finished emitting. That is, τ < Δt. In this case, the image sensor 20 opens its electronic shutter simultaneously with the emission of pulsed light from the light source 10 and accumulates a signal charge corresponding to the amount of reflected pulsed light at time intervals Δt. The round-trip time of the light is τ = [S2 / (S1+S2)]Δt.

[0105] In the TOF method, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected pulsed light and output a signal indicating the three-dimensional distribution of the user's surface. In this embodiment, the signal output from the image sensor 20 may include a signal indicating the three-dimensional distribution of the user's surface.

[0106] By using the surface reflection component I1, which is included in at least a portion of the reflected pulsed light, in the TOF method, it is possible to detect the three-dimensional distribution of the user's surface, including wrinkles and minute irregularities, more clearly than with conventional TOF methods.

[0107] In the direct TOF method, the distance to an object can be calculated by detecting the surface reflection component I1 contained in the rise time of the reflected pulse light. In the indirect TOF method, the distance to an object is calculated by detecting the portion of the reflected pulse light from the front end to the rear end, as shown in signals b) and (c) in Figure 6. Using spatially resolved imaging, the portion of the surface reflection component I1 from the front end to the rear end can be separated from the reflected pulse light.

[0108] Furthermore, by using spatially resolved imaging, it is possible to acquire the portion of the internal scattering component I2 from its front end to its rear end from the reflected pulsed light. If the portion of the internal scattering component I2 from its front end to its rear end is used in the indirect TOF method, it is also possible to acquire the two-dimensional distribution of the distance between the image sensor 20 and the user 1's blood vessels, i.e., the three-dimensional distribution of the user 1's blood vessels.

[0109] Figure 7 is a schematic diagram showing an example of capturing user 1 using the TOF method in this embodiment. In the example shown in Figure 7, the three-dimensional distribution of user 1's surface is calculated from the surface reflection component I1. Authentication accuracy can be improved by comparing the information obtained from the three-dimensional distribution of user 1's surface with the information indicating the unevenness of the characteristic parts of user 1 contained in the biometric data in memory 25. As a result, the rate of false rejection or false acceptance is reduced. Alternatively, the three-dimensional distribution of user 1's surface may be calculated from reflected pulse light that includes both the surface reflection component I1 and the internal scattering component I2.

[0110] Next, a series of processes for biometric authentication of user 1 using the identification device 100 in this embodiment will be described.

[0111] Figure 8 is a flowchart showing an example of a process performed by the control circuit 30.

[0112] In step S201, the control circuit 30 causes the light source 10 to illuminate the user 1 with light. If time-resolved imaging is performed, the light is pulsed light; if spatially resolved imaging is performed, the light is a two-dimensional pattern light. If time-resolved imaging and spatially resolved imaging are combined, the light is both a two-dimensional pattern light and pulsed light.

[0113] In step S202, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected light returned from user 1 and to output a signal indicating a two-dimensional image of user 1 corresponding to the intensity distribution of the detected light. If the reflected light is pulsed light, at least a portion of the reflected light is, for example, the rising or falling phase component of the pulsed light. The surface reflection component I1 is obtained from the intermediate components, and the internal scattering component I2 is obtained from the components during the falling phase. When acquiring the 3D distribution of user 1's surface using the indirect TOF method, the portion of the reflected pulse light from the leading end to the trailing end of the surface reflection component I1, or the portion from the leading end to the trailing end of the combined component of surface reflection component I1 and internal scattering component I2, is detected.

[0114] In step S203, the control circuit 30 causes the signal processing circuit 40 to generate an image of user 1 from the output signal and extracts data characterizing that image. The data characterizing the image generated from the surface reflection component I1 includes information indicating the texture of user 1's skin surface. The data characterizing the image generated from the internal scattering component I2 includes information indicating the distribution of user 1's blood vessels. In the case of the TOF method, the data characterizing the image generated from the surface reflection component I1, or the image generated from a composite component of the surface reflection component I1 and the internal scattering component I2, includes information indicating the three-dimensional distribution of user 1's surface, i.e., the unevenness of the characteristic parts of user 1.

[0115] In step S204, the control circuit 30 compares the data characterizing the image of user 1 extracted from the generated image of user 1 with the data characterizing the image of user 1 stored in memory.

[0116] In step S205, the control circuit 30 determines whether the difference between the two data being compared is below a reference value. For example, when comparing information on the three-dimensional distribution of user 1's surface obtained by the TOF method with information showing the unevenness of characteristic parts of user 1 included in the biometric data, an example of the judgment criteria is as follows. The control circuit 30 evaluates whether the difference between the two data is below a reference value at all measurement points on user 1's surface, or at a certain percentage or more of measurement points, such as 80%.

[0117] If the above difference is below the reference value, in step S206, the control circuit 30 determines that user 1 is the real person. If the above difference is not below the reference value, in step S207, the control circuit 30 determines that user 1 is a different person.

[0118] The identification device 100 in this embodiment enables contactless biometric authentication of user 1. Therefore, the burden on user 1 during authentication is minimal.

[0119] In addition to the face, other parts of User 1's body, such as the hands, may also be used. For example, the surface reflection component I1 may be obtained from fingerprints, or the internal scattering component I2 may be obtained from the veins of the fingers. Fingerprints or veins, which are unique to each individual, tend to be clearly visible on the hands, improving recognition accuracy. When the hands are used as the part of User 1 being examined, body movement may be suppressed by placing the hands on glass. In this case, having a space between the glass and the identification device 100 has the advantage of allowing the camera to focus. Furthermore, in the case of hand authentication, the signal-to-noise ratio of the detection signal on the image sensor 20 can be improved by emitting a light intensity exceeding Class 1 from the light source 10.

[0120] Next, an example of the application of the identification device 100 in this embodiment will be described.

[0121] Figure 9 is a schematic diagram showing an example in which the identification device 100 in this embodiment is installed inside the passenger compartment of an automobile. As shown in Figure 9, the identification device 100 installed inside the passenger compartment may identify the driver, i.e., user 1. Instead of a car key, the identification device 100 may be used to identify user 1 and start the car. During autonomous driving, the identification device 100 may be used to identify passengers, and the identification result may be fed back into the autonomous driving system, which may be customized according to the passenger. Feedback may include, for example, applying gentler brakes if the passenger is elderly, or providing a more exhilarating driving experience if the passenger enjoys driving. This will enable personalized autonomous driving.

[0122] Figure 10 schematically shows an example of applying the identification device 100 in this embodiment to a mobile terminal. The mobile terminal is, for example, a smartphone, tablet, or personal computer. The image sensor 20 in the identification device 100 may be an image sensor built into the mobile terminal.

[0123] The identification device 100 in this embodiment may be applied not only to automobiles or mobile terminals, but also, for example, to ATMs installed in banks or convenience stores, or to the entrance of a house.

[0124] The identification device 100 in this embodiment enables highly accurate authentication through a combination of two or more authentication methods. The identification device 100 in this embodiment can be used not only for authentication of login to a terminal, but also for security authentication using a terminal. Security authentication using a terminal includes, for example, authentication of login to a bank account or various service accounts after logging into a terminal. Conventional password authentication may also be used in conjunction, with the authentication described in this disclosure serving as additional authentication. This reduces the risk of hacking and ensures even higher security compared to systems that only use conventional password authentication.

[0125] This disclosure also includes methods that include operations performed by the control circuit 30 and the signal processing circuit 40.

[0126] As described above, this disclosure includes identification devices and identification methods as described in the following items.

[0127] [Item 1] The identification device relating to the first item comprises a light source, an image sensor, a memory storing biometric data indicating the physical characteristics of the user, and a processor. The processor causes the light source to emit pulsed light with a pulse width of 0.2 ns to 1 μs or less that is irradiated onto the user, causes the image sensor to detect at least a portion of the reflected pulsed light that has returned from the user, causes the processor to output a signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the reflected pulsed light, and identifies the user by comparing the signal with the biometric data.

[0128] [Item 2] In the identification device relating to the first item, the pulse width of the pulsed light may be 3 ns or more and 50 ns or less.

[0129] [Item 3] In an identification device relating to the first or second item, the image sensor includes a plurality of pixels, and the processor causes the image sensor to accumulate charge in each of the plurality of pixels, corresponding to the intensity of at least a portion of the reflected pulse light, for a period of 1,000 to 100,000 times, and the signal may represent a two-dimensional distribution of the total amount of charge accumulated in the plurality of pixels.

[0130] [Item 4] In an identification device relating to any of the first to third items, the biometric data may include first information indicating the texture of the user's skin surface.

[0131] [Item 5] In the identification device relating to the fourth item, the processor provides the image sensor with the reflected pulse light from the time the intensity of the reflected pulse light starts to increase until the increase ends. A first signal obtained by detecting at least a portion of the rise time, which is the period of the signal, may be output, and the user may be identified by comparing the information obtained from the first signal with the first information.

[0132] [Item 6] In an identification device relating to any of the first to fifth items, the biometric data may include second information indicating the distribution of the user's blood vessels.

[0133] [Item 7] In the identification device relating to item 6, the processor may cause the image sensor to output a second signal obtained by detecting at least a portion of the fall time component of the reflected pulse light, which is the period from when the intensity of the reflected pulse light starts to decrease until the decrease ends, and identify the user by comparing the information obtained from the second signal with the second information.

[0134] [Item 8] In an identification device relating to any of the first to seventh items, the biometric data may include third information indicating the unevenness of the user's characteristic features. In an identification device relating to any of the first to seventh items, the biometric data may include fourth information indicating the shape of the user's characteristic portion. In an identification device relating to any of the first to seventh items, the biometric data may include fifth information indicating the location of the user's characteristic portion.

[0135] [Item 9] In the identification device relating to item eight, the processor may cause the image sensor to output a third signal indicating a two-dimensional distribution of the distance between the image sensor and the user's surface, and identify the user by comparing the information obtained from the third signal with the third information.

[0136] [Item 10] In the identification device relating to item 9, the processor may cause the image sensor to detect at least a portion of the rise period component of the reflected pulse light, which is the period from when the intensity of the reflected pulse light starts to increase until the increase ends, and calculate the two-dimensional distribution of the distance from the component.

[0137] [Item 11] The identification device relating to item 11 comprises a light source, an image sensor, a memory storing biometric data indicating the physical characteristics of the user, and a processor. The biometric data includes first information indicating the texture of the user's skin surface. The processor causes the light source to emit illumination light that projects a two-dimensional pattern including at least one bright area and at least one dark area onto the user, causes the image sensor to detect at least a portion of the reflected light returning from the user, causes the image sensor to output a signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the reflected light, and identifies the user by comparing the signal with the biometric data stored in the memory.

[0138] [Item 12] In the identification device relating to item 11, the biometric data may further include second information indicating the distribution of the user's blood vessels.

[0139] [Item 13] In the identification device relating to item 12, the processor may cause the image sensor to detect at least a portion of the first reflected light returning from the at least one dark area of ​​the user onto which the two-dimensional pattern is projected, output a first signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the first reflected light, and identify the user by comparing the information obtained from the first signal with the second information.

[0140] [Item 14] In the identification device relating to item 13, the processor may cause the image sensor to detect at least a portion of the second reflected light returning from the at least one bright area of ​​the user onto which the two-dimensional pattern is projected, output a second signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the second reflected light, and identify the user by comparing the information obtained by subtracting the first signal from the second signal with the first information.

[0141] [Item 15] In the identification device relating to item 15, the processor may cause the image sensor to detect at least a portion of the first reflected light returning from the at least one dark area of ​​the user onto which the two-dimensional pattern is projected, to detect at least a portion of the second reflected light returning from the at least one bright area, to output a first signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the first reflected light, to output a second signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the second reflected light, and to identify the user by comparing the information obtained by subtracting the first signal from the second signal with the first information.

[0142] [Item 16] In the identification device relating to item 11, the irradiated light may be pulsed light.

[0143] [Item 17] In the identification device relating to item 16, the pulse width of the pulsed light may be 3 ns or more and 50 ns or less.

[0144] [Item 18] In an identification device relating to item 16 or 17, the image sensor includes a plurality of pixels, and the processor causes the image sensor to repeatedly accumulate charge in each of the plurality of pixels, corresponding to the intensity of at least a portion of the reflected light, for 1,000 to 100,000 times, and the signal may represent a two-dimensional distribution of the total amount of charge accumulated in the plurality of pixels.

[0145] [Item 19] In an identification device relating to any of items 16 to 18, the biometric data may further include third information indicating the unevenness of the user's characteristic features.

[0146] [Item 20] In the identification device relating to item 19, the processor causes the image sensor to detect at least a portion of the first reflected pulse light returning from the at least one dark area of ​​the user onto which the two-dimensional pattern is projected, to detect at least a portion of the second reflected pulse light returning from the at least one bright area, to output a first signal corresponding to the two-dimensional distribution of the intensity of at least a portion of the first reflected pulse light, to output a second signal corresponding to the two-dimensional distribution of the intensity of at least a portion of the second reflected pulse light, and to compare the information obtained by subtracting the first signal from the second signal with the third information. The user may be identified by this method.

[0147] [Item 21] In an identification device relating to any of items 16 to 18, the biometric data may further include second information indicating the distribution of the user's blood vessels.

[0148] [Item 22] In the identification device relating to item 21, the processor may cause the image sensor to detect at least a portion of the fall time, which is the period from when the intensity of the first reflected pulse light returning from the at least one dark area of ​​the user onto which the two-dimensional pattern is projected starts to decrease until the decrease ends, and output a first signal corresponding to the two-dimensional distribution of the intensity of at least a portion of the fall time of the first reflected pulse light, and identify the user by comparing the information obtained from the first signal with the second information.

[0149] [Item 23] In an identification device relating to any of the items 16 to 18, the processor may cause the image sensor to detect at least a portion of the rise period, which is the period from when the intensity of the second reflected pulse light returning from the at least one bright area of ​​the user onto which the two-dimensional pattern is projected starts to increase until the increase stops; cause the image sensor to output a second signal corresponding to the two-dimensional distribution of the intensity of at least a portion of the rise period of the second reflected pulse light; and identify the user by comparing the information obtained from the second signal with the first information.

[0150] [Item 24] The identification method relating to item 24 is an identification method using an identification device comprising a light source and an image sensor, comprising: causing the light source to emit pulsed light with a pulse width of 0.2 ns or more and 1 μs or less that is irradiated onto a user; causing the image sensor to detect at least a portion of the reflected pulsed light returned from the user and to output a signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the reflected pulsed light; and identifying the user by comparing the signal with biometric data indicating the physical characteristics of the user.

[0151] [Item 25] The identification method relating to item 25 is an identification method using an identification device comprising a light source and an image sensor, comprising: causing the light source to emit illumination light that projects a two-dimensional pattern including at least one bright area and at least one dark area onto the user; causing the image sensor to detect at least a portion of the reflected light returned from the user and to output a signal corresponding to the two-dimensional distribution of the intensity of the at least portion of the reflected light; and identifying the user by comparing the signal with biometric data indicating the texture of the user's skin surface. [Industrial applicability]

[0152] The identification device described in this disclosure is useful for cameras or authentication devices that identify users personally without contact. The identification device can be applied to security services. [Explanation of Symbols]

[0153] 1 user 10 light source 20 Image Sensors 22 Photoelectric conversion unit 24 Charge storage section 25 memory 30 Control circuits 32 Light source control unit 34 Sensor Control Unit 40 Signal Processing Circuits 100 Identification device I1 Surface reflection component I2 internal scattering component

Claims

1. A control method performed by a computer, The system involves emitting light from a light source installed in the vehicle to project a dot pattern onto the user's face, The vehicle is equipped with an image sensor that detects light from the face and generates an image signal. Based on the aforementioned image signal, authentication of the user is performed, This includes controlling the vehicle based on the results of the user's authentication, Control method.

2. Performing the authentication includes performing a first identification process based on the internal scattering component of light from the face, based on the image signal. The control method according to claim 1.

3. Controlling the vehicle includes starting the vehicle if the authentication is successful. The control method according to claim 1 or 2.

4. Controlling the vehicle includes performing automated driving controls customized for the user based on the results of the authentication. The control method according to claim 1 or 2.

5. Performing the first identification process includes performing the first identification process based on a signal included in the image signal that corresponds to the intensity distribution of reflected light returned from at least one dark area of ​​the dot pattern, The control method according to claim 2.

6. A program for executing the control method described in claim 1.

7. A light source installed in the car, An image sensor provided in the aforementioned vehicle, Processor and Equipped with, The aforementioned processor, The aforementioned light source emits illumination light that projects a dot pattern onto the user's face. The image sensor is made to detect light from the face and generate an image signal. Based on the aforementioned image signal, the user is authenticated. Based on the results of the user's authentication, the vehicle is controlled. system.

8. A vehicle equipped with the system described in claim 7.

Images