Recognition device and recognition method

Abstract

The invention provides an identification apparatus and an identification method. The recognition device includes: a light source; an image sensor; and a processor; the processor causes the light source to emit irradiation light projecting a dot pattern onto the face of a user. The processor causes the image sensor to detect light from the face and generate an image signal. The processor executes a first recognition process based on a surface reflection component of light from the face on the basis of the image signal. The processor executes a second recognition process based on an internal scattering component of light from the face on the basis of the image signal. The processor authenticates the user on the basis of a result of the first recognition process and a result of the second recognition process.

Description

[0001] This application is a divisional application of the Chinese patent application with the application date of May 8, 2019, application number "201980022548.7", and invention name "Identification Device and Identification Method". Technical Field

[0002] The invention relates to an identification device and an identification method. Background Art

[0003] In the field of personal identification, authentication has shifted from password authentication to biometric authentication. In password authentication, authentication is performed based on a password input by a user. On the other hand, in biometric authentication, authentication is performed based on information about a person's physical features. In biometric authentication, there is an advantage that there is less risk of forgetfulness, leakage, or brute force attack. In biometric authentication, for example, personal identification information is obtained by illuminating a part of the user's body with light and observing the reflected light.

[0004] Patent document 1 discloses a method for improving the accuracy of biometric authentication of a user. In the method described in Patent document 1, fingerprint authentication and vein authentication are combined to perform biometric authentication of the user. Specifically, the surface of the user's finger is irradiated with light of a wavelength of 900 nm for fingerprint authentication and light of 750 nm for vein authentication. The user is authenticated based on the reflected light.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2008-020942 Summary of the invention

[0008] Problems to be solved by the invention

[0009] The present invention provides a technology capable of further improving authentication accuracy.

[0010] Means used to solve problems

[0011] A device of one technical solution of the present invention comprises: a light source; an image sensor; and a processor; the processor causes the light source to emit irradiation light that projects a dot pattern onto the face of a user; the processor causes the image sensor to detect light from the face to generate an image signal; the processor performs a first recognition process based on a surface reflection component of the light from the face based on the image signal; the processor performs a second recognition process based on an internal scattering component of the light from the face based on the image signal; the processor authenticates the user based on the results of the first recognition process and the results of the second recognition process.

[0012] Effects of the Invention

[0013] According to the technology of the present invention, it is possible to perform biometric authentication with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A This is a diagram schematically showing an example of a recognition device according to the present embodiment.

[0015] Figure 1B This is a diagram showing an example of temporal changes in the intensity of light reaching the image sensor.

[0016] Figure 1C The horizontal axis represents the width of input pulse light, and the vertical axis represents the amount of light detected by the sensor.

[0017] Figure 1D This is a diagram showing an example of a schematic structure of one pixel of an image sensor.

[0018] Figure 1E This is a diagram showing an example of the structure of an image sensor.

[0019] Figure 1F This is a diagram showing an example of the operation within one frame.

[0020] Figure 1G This is a flowchart schematically showing the operation performed by the control circuit.

[0021] Figure 2A This is a diagram showing an example of a light signal in which rectangular pulse light is emitted from a light source and light returned from a user reaches an image sensor.

[0022] Figure 2B This is a diagram showing another example of an optical signal in which rectangular pulse light is emitted from a light source and light returned from a user reaches an image sensor.

[0023] Figure 3A This is a diagram showing an example of a time chart when detecting a surface reflection component.

[0024] Figure 3B FIG. 2 is a diagram showing an example of a time chart when detecting an internal scattered component.

[0025] Figure 4A This is a diagram schematically showing an example of distribution of two-dimensional pattern light projected from a light source toward a user.

[0026] Figure 4B This is a diagram schematically showing another example of the distribution of two-dimensional pattern light projected from the light source toward the user.

[0027] Figure 4C This is a diagram schematically showing still another example of the distribution of the two-dimensional pattern light projected from the light source toward the user.

[0028] Figure 5A This is a diagram schematically showing a situation in which a user is photographed using a normal camera.

[0029] Figure 5B This is a diagram schematically showing an example of photographing a user based on a surface reflection component according to the present embodiment.

[0030] Figure 5C FIG. 1 is a diagram schematically showing an example of photographing a user based on an internal scattering component according to the present embodiment.

[0031] Figure 6 This is a diagram for explaining the principle of measuring the round trip time of pulse light by the indirect TOF method.

[0032] Figure 7 This is a diagram schematically showing an example of photographing a user using the TOF method according to the present embodiment.

[0033] Figure 8 This is a flowchart showing an example of processing executed by the control circuit.

[0034] Fig. 9 This is a diagram schematically showing an example in which the recognition device according to the present embodiment is mounted in the interior of a car.

[0035] Fig.10 This is a diagram schematically showing an example in which the recognition device according to the present embodiment is applied to a mobile terminal. DETAILED DESCRIPTION

[0036] The embodiments described below all represent general or specific examples. The numerical values, shapes, materials, constituent elements, and the configuration positions of the constituent elements shown in the following embodiments are examples and do not limit the meaning of the present invention. In addition, the constituent elements of the following embodiments that are not recorded in the independent claims representing the highest concept of the present invention are described as arbitrary constituent elements.

[0037] In the present invention, all or part of a circuit, unit, device, component or component, or all or part of a functional block of a block diagram can be performed by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC) or an LSI (large scale integration). LSI or IC can be integrated into one chip, or a plurality of chips can be combined to form it. For example, functional blocks other than storage elements can also be integrated into one chip. It is referred to as LSI or IC here, but the name changes according to the degree of integration, and it can also be referred to as system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA) that is programmed after the manufacture of the LSI, or a reconfigurable logic device that can reconstruct the bonding relationship within the LSI or set the circuit division within the LSI can also be used for the same purpose.

[0038] Furthermore, all or part of the functions or operations of a circuit, unit, device, component or component can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processor, the functions determined by the software are executed by the processor and peripheral devices. The system or device may have one or more non-transitory recording media with software recorded, a processor, and necessary hardware devices, such as interfaces.

[0039] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0040] (Implementation Method)

[0041] [1. Identification device]

[0042] First, use Figures 1A to 1G , the structure of the recognition device 100 according to the embodiment of the present invention is described.

[0043] Figure 1A 1 is a diagram schematically showing an example of a recognition device 100 according to the present embodiment. The recognition device 100 includes a light source 10, an image sensor 20, a memory 25, a control circuit 30, and a signal processing circuit 40. The image sensor 20 includes a plurality of photoelectric conversion elements 22 and a plurality of charge storage units 24. In this specification, the control circuit 30 and the signal processing circuit 40 are sometimes collectively referred to as a "processor".

[0044] The light source 10 emits pulse light to illuminate the head of the user 1. The image sensor 20 detects at least a portion of the pulse light returning from the head of the user 1. The control circuit 30 controls the light source 10 and the image sensor 20. The signal processing circuit 40 processes the signal output from the image sensor 20.

[0045] In this embodiment, the control circuit 30 includes a light source control unit 32 that controls the light source 10 and a sensor control unit 34 that controls the image sensor 20. The light source control unit 32 controls the intensity, pulse width, emission timing and / or wavelength of the pulse light emitted from the light source 10. The sensor control unit 34 controls the timing of signal accumulation in each pixel of the image sensor 20.

[0046] Hereinafter, each component will be described in more detail.

[0047] [1-1. Light source 10]

[0048] The light source 10 irradiates light to the head of the user 1, for example, the forehead. The light emitted from the light source 10 and reaching the user 1 is divided into a surface reflection component I1 reflected by the surface of the user 1 and an internal scattering component I2 scattered inside the user 1. The internal scattering component I2 is a component that is reflected or scattered once or multiple times inside the biological body. When the forehead of the user 1 is irradiated with light, the internal scattering component I2 refers to a component that reaches a part of about 8mm to 16mm from the surface of the forehead to the inside, such as the brain, and returns to the recognition device 100 again. The surface reflection component I1 includes three components: a direct reflection component, a diffuse reflection component, and a scattered reflection component. The direct reflection component is a reflection component with an incident angle equal to a reflection angle. The diffuse reflection component is a component that is diffused and reflected due to the uneven shape of the surface. The scattered reflection component is a component that is scattered and reflected due to the internal tissue near the surface. When the forehead of the user 1 is irradiated with light, the scattered reflection component is a component that is scattered and reflected inside the epidermis. Hereinafter, in the present invention, the surface reflection component I1 reflected on the surface of the user 1 is considered to include these three components. The surface reflection component I1 and the internal scattering component I2 change their traveling directions due to reflection or scattering, and part of them reaches the image sensor 20 .

[0049] The light source 10 generates pulsed light repeatedly multiple times at a predetermined time interval or at a predetermined timing according to instructions from the control circuit 30. The pulsed light generated by the light source 10 can be, for example, a rectangular wave whose falling period is close to zero. The falling period is the period from when the intensity of the pulsed light starts to decrease to when the decrease ends. The component in the falling period of the pulsed light is also called the rear end of the pulsed light. Usually, the light incident on the user 1 propagates in the user 1 in various paths and is emitted from the surface of the user 1 with a time difference. Therefore, the rear end of the internal scattering component I2 of the pulsed light has an extension. When the inspected part is the forehead, the extension of the rear end of the internal scattering component I2 is about 4ns. Considering this situation, the falling period of the pulsed light can be set to less than 2ns, which is less than half of it. The falling time can also be less than 1ns, which is another half of it. On the other hand, the component in the rising period of the pulsed light can be used for the detection of the surface reflection component I1. The rising period is the period from when the intensity of the pulsed light starts to increase to when the increase ends. The component in the rising period of the pulsed light is also called the front end of the pulsed light.

[0050] The light source 10 can be formed by a combination of a light source such as a laser such as an LD and a diffusion plate. By using a laser, a light output with high time responsiveness can be achieved. The light output with high time responsiveness has a waveform with a sharp rise characteristic or a fall characteristic. The rise characteristic and the fall characteristic are also referred to as Tr characteristics and Tf characteristics, respectively. It is also possible to irradiate the user 1 with light in two dimensions by setting a diffusion plate in front of the light source 10.

[0051] The wavelength of the light emitted by the light source 10 can be, for example, any wavelength included in the wavelength range of 650 nm to 950 nm. This wavelength range is included in the wavelength range from red to near infrared. In this specification, the term "light" is used not only for visible light but also for infrared light. The above-mentioned wavelength range is called the "window of the organism" and has the property of being difficult to be absorbed by the water and skin in the organism. When the organism is used as the detection object, the detection sensitivity can be improved by using light in the above-mentioned wavelength range.

[0052] In the recognition device 100 of the present embodiment, in order to measure the user 1 contactlessly, a light source 10 designed in consideration of the effect on the retina can be used. For example, a light source 10 that satisfies level 1 of the laser safety standards established by various countries can be used. When level 1 is satisfied, low-illuminance light with an achievable emission limit (AEL) of less than 1 mW is irradiated toward the user 1. In addition, the light source 10 itself may not satisfy level 1. For example, a diffuser plate or an ND filter may be set in front of the light source 10 to diffuse or attenuate the light, thereby satisfying level 1 of the laser safety standards.

[0053] In the past, in order to distinguishably detect information such as the absorption coefficient or scattering coefficient at different locations in the depth direction inside a biological body, an ultra-high-speed streak camera was used. For example, Japanese Patent Laid-Open No. 4-189349 discloses an example of such an ultra-high-speed streak camera. In these ultra-high-speed streak cameras, in order to perform measurements with a desired spatial resolution, ultra-short pulse light with a pulse width of femtoseconds or picoseconds is used. In addition, in this specification, the so-called "pulse width" refers to the half-value full width of the pulse. Compared with the previous method of using an ultra-high-speed streak camera, the recognition device 100 of this embodiment can distinguish and detect the surface reflection component I1 and the internal scattering component I2. Therefore, the pulse light emitted by the light source 10 does not need to be an ultra-short pulse light, and the pulse width can be arbitrarily selected.

[0054] When the forehead is irradiated with light, the amount of light of the internal scattered component I2 may become a very small value of about one thousandth to one tens of thousandsth of the amount of light of the surface reflected component I1. Furthermore, considering the safety standards of the laser, the amount of light that can be irradiated is extremely small, and the detection of the internal scattered component I2 becomes very difficult. In this case, as long as the light source 10 generates pulse light with a relatively large pulse width, the cumulative amount of the internal scattered component I2 accompanied by a time delay can be increased. In this way, the detection light amount can be increased, and the SN ratio can be improved.

[0055] The light source 10 emits pulse light having a pulse width of, for example, 3 ns or more. Generally, the temporal spread of light scattered in a living tissue such as the brain is about 4 ns. Figure 1B FIG. 2 shows an example of the time variation of the amount of light reaching the image sensor 20 when the width of the input pulse light is 0 ns, 3 ns, and 10 ns. Figure 1B As shown, as the width of the pulse light from the light source 10 is expanded, the amount of light of the internal scattered component I2 appearing at the rear end of the pulse light returning from the user 1 increases. Figure 1C The horizontal axis of the graph shows the width of input pulse light, and the vertical axis shows the amount of light detected by the image sensor 20. The image sensor 20 includes an electronic shutter. Figure 1C The result is obtained under the condition that the electronic shutter is opened 1ns after the rear end of the pulse light is reflected by the surface of the user 1 and reaches the image sensor 20. The reason for selecting this condition is that the ratio of the surface reflection component I1 to the internal scattering component I2 is high just after the rear end of the pulse light arrives. Figure 1C As shown, if the pulse width of the pulse light emitted by the light source 10 is set to 3 ns or more, the amount of light detected by the image sensor 20 can be maximized.

[0056] In addition, the resolution of the timing control of the light source 10 and the driving circuit by the electronic shutter is about 0.2 ns. Therefore, the pulse width of the light source 10 is set to be 0.2 ns or more, for example.

[0057] The light source 10 may also emit a pulse light with a pulse width of more than 5 ns, and further more than 10 ns. On the other hand, even if the pulse width is too large, unused light increases and becomes a waste. Thus, the light source 10, for example, generates a pulse light with a pulse width of less than 50 ns. Alternatively, the light source 10 may also emit a pulse light with a pulse width of less than 30 ns, and further less than 20 ns.

[0058] In addition, the irradiation pattern of the light source 10 may be a uniform intensity distribution within the irradiation area. This is based on the following reasons. In the recognition device 100 of the present embodiment, the surface reflection component I1 is temporally separated / reduced. As a result, the internal scattering component I2 can also be detected at the Null point directly below the irradiation point on the user 1. Therefore, in the recognition device 100 of the present embodiment, by irradiating the inspected part of the user 1 over a large range in space, the measurement resolution can be improved.

[0059] [1-2. Image sensor 20]

[0060] The image sensor 20 receives light emitted from the light source 10 and reflected or scattered by the user 1. The image sensor 20 has a plurality of light detection units arranged two-dimensionally, and acquires two-dimensional information of the user 1 at a time. Thus, compared with a line sensor that performs detection by sliding relative to the detected part of the user 1, it is possible to acquire two-dimensional information of the user 1 in a relatively short time. In this specification, the light detection unit is also referred to as a "pixel". The image sensor 20 is, for example, an arbitrary imaging element such as a CCD image sensor or a CMOS image sensor.

[0061] The image sensor 20 has an electronic shutter. The electronic shutter is a circuit that controls the timing of imaging. In the present embodiment, the sensor control unit 34 of the control circuit 30 has the function of an electronic shutter. The electronic shutter controls the period of signal accumulation in which the received light is converted into an effective electrical signal and accumulated, and the period of stopping signal accumulation. The signal accumulation period may also be referred to as an "exposure period". In the following description, the width of the exposure period is sometimes referred to as a "shutter width". The time from the end of one exposure period to the beginning of the next exposure period is sometimes referred to as a "non-exposure period". In the following, the exposure state is sometimes referred to as "OPEN" and the state of stopping exposure is sometimes referred to as "CLOSE".

[0062] The image sensor 20 can adjust the exposure period and the non-exposure period in the sub-nanosecond range, for example, 30ps to 1ns, through an electronic shutter. In order to correct the influence of the brightness of the subject, the previous TOF (Time Of Flight) camera for the purpose of distance measurement detects all the light emitted from the light source 10 and reflected by the subject. Therefore, in the previous TOF camera, it is necessary to make the shutter width larger than the pulse width of the light. In contrast, in the recognition device 100 of the present embodiment, it is not necessary to correct the amount of light of the subject. Therefore, it is not necessary to make the shutter width larger than the pulse width. Thus, the shutter width can be set to a value of, for example, more than 1ns and less than 30ns. According to the recognition device 100 of the present embodiment, since the shutter width can be reduced, the influence of the dark current contained in the detection signal can be reduced.

[0063] When the forehead of the user 1 is irradiated with light, the attenuation rate of the internal light is very large. For example, the emitted light may be attenuated to about 1 in 1,000,000 relative to the incident light. Therefore, there is a case where the amount of light is insufficient for detecting the internal scattered component I2 by irradiating only one pulse. In the irradiation of level 1 of the laser safety standard, the amount of light is particularly weak. In this case, the light source 10 emits multiple pulses of light, and accordingly, the image sensor 20 is also exposed multiple times through the electronic shutter, so that the detection signal can be accumulated to improve the sensitivity.

[0064] Hereinafter, a configuration example of the image sensor 20 will be described.

[0065] The image sensor 20 may include a plurality of pixels arranged two-dimensionally on an imaging surface. Each pixel may include, for example, a photoelectric conversion element such as a photodiode and one or more charge accumulation units. The following describes an example in which each pixel includes a photoelectric conversion element that generates a signal charge corresponding to the amount of light received by photoelectric conversion, a charge accumulation unit that accumulates a signal charge generated by the surface reflection component I1 of the pulse light, and a charge accumulation unit that accumulates a signal charge generated by the internal scattering component I2 of the pulse light. In the following example, the control circuit 30 detects the surface reflection component I1 by causing the image sensor 20 to detect the portion before the start of the descent of the pulse light returned from the head of the user 1. The control circuit 30 also detects the internal scattering component I2 by causing the image sensor 20 to detect the portion after the start of the descent of the pulse light returned from the head of the user 1. The light source 10 outputs, for example, two wavelengths of light.

[0066] Figure 1D 2 is a diagram showing an example of a schematic structure of one pixel 201 of the image sensor 20. Figure 1DThe structure of one pixel 201 is schematically shown, and does not necessarily reflect the actual structure. The pixel 201 in this example includes a photodiode 203 for photoelectric conversion, a first floating diffusion layer (Floating Diffusion) 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207 as a charge accumulation portion, and a drain 202 for discharging signal charges.

[0067] The photons incident on each pixel due to the emission of one pulse light are converted into signal electrons as signal charges by the photodiode 203. The converted signal electrons are discharged to the drain 202 or distributed to any of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 according to the control signal input from the control circuit 30.

[0068] The emission of pulse light from the light source 10, the accumulation of signal charges in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207, and the discharge of signal charges to the drain 202 are repeated in this order. This repetitive action is high-speed, for example, it can be repeated tens of thousands to hundreds of millions of times within the time of one frame. The time of one frame is, for example, about 1 / 30 seconds. The pixel 201 finally generates and outputs four image signals based on the signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207.

[0069] The control circuit 30 in this example causes the light source 10 to repeatedly emit a first pulse light having a first wavelength and a second pulse light having a second wavelength. By selecting two wavelengths having different absorption rates in the internal tissue of the user 1 as the first wavelength and the second wavelength, the state of the user 1 can be analyzed.

[0070] The control circuit 30 first causes the light source 10 to emit a first pulse of light. The control circuit 30 causes the first floating diffusion layer 204 to accumulate signal charges during a first period in which the surface reflection component I1 of the first pulse of light is incident on the photodiode 203. Next, the control circuit 30 causes the second floating diffusion layer 205 to accumulate signal charges during a second period in which the internal scattering component I2 of the first pulse of light is incident on the photodiode 203. Next, the control circuit 30 causes the light source 10 to emit a second pulse of light. The control circuit 30 causes the third floating diffusion layer 206 to accumulate signal charges during a third period in which the surface reflection component I1 of the second pulse of light is incident on the photodiode 203. Next, the control circuit 30 causes the fourth floating diffusion layer 207 to accumulate signal charges during a fourth period in which the internal scattering component I2 of the second pulse of light is incident on the photodiode 203.

[0071] In this way, after the control circuit 30 starts to emit the first pulse light, the first floating diffusion layer 204 and the second floating diffusion layer 205 are separated by a predetermined time difference, so that the signal charge from the photodiode 203 is accumulated in sequence. Then, after the control circuit 30 starts to emit the second pulse light, the third floating diffusion layer 206 and the fourth floating diffusion layer 207 are separated by the predetermined time difference, so that the signal charge from the photodiode 203 is accumulated in sequence. The above operation is repeated multiple times. In order to estimate the amount of interference light and ambient light, a period during which signal charge is accumulated in other floating diffusion layers not shown in the figure when the light source 10 is turned off can also be set. By subtracting the signal charge amount of the other floating diffusion layers from the signal charge amount accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206 and the fourth floating diffusion layer 207, a signal with interference light and ambient light components removed can be obtained.

[0072] In addition, in the present embodiment, the number of charge accumulation units per pixel is set to 4, but it can also be designed to be any number greater than 1 according to the purpose. For example, when only one wavelength is used, the number of charge accumulation units can be 2. In addition, when one wavelength is used and only the surface reflection component I1 is detected or only the internal scattering component I2 is detected, the number of charge accumulation units per pixel can be 1. In addition, even when two or more wavelengths are used, the number of charge accumulation units can be 1 when imaging using each wavelength is performed in different frames. As described later, the number of charge accumulation units can be 1 when the detection of the surface reflection component I1 and the detection of the internal scattering component I2 are performed in different frames, respectively.

[0073] Figure 1E FIG. 2 is a diagram showing an example of the structure of the image sensor 20. Figure 1E In FIG. 1 , the area surrounded by the double-dashed line frame corresponds to one pixel 201. The pixel 201 includes one photodiode. Figure 1E 2, but more pixels may be arranged. The pixel 201 includes a first floating diffusion layer 204, a second floating diffusion layer 205, a third floating diffusion layer 206, and a fourth floating diffusion layer 207. The signals accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are processed like the signals of the four pixels of a conventional CMOS image sensor and are output from the image sensor 20.

[0074] Each pixel 201 has four signal detection circuits. Each signal detection circuit includes a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. In this example, the reset transistor 310 is connected to the Figure 1DThe pulse input to the gate of the reset transistor 310 corresponds to the drain 202 shown, and the pulse discharged from the drain corresponds to the pulse discharged from the drain. Each transistor is, for example, a field effect transistor formed on a semiconductor substrate, but is not limited to this. As shown in the figure, one of the input terminal and the output terminal of the source follower transistor 309 is connected to one of the input terminal and the output terminal of the row selection transistor 308. Typically, one of the input terminal and the output terminal of the source follower transistor 309 is the source. Typically, one of the input terminal and the output terminal of the row selection transistor 308 is the drain. The gate of the source follower transistor 309, which serves as a control terminal, is connected to the photodiode 203. The signal charge generated by the photodiode 203 is accumulated in the floating diffusion layer serving as a charge accumulation unit between the photodiode 203 and the source follower transistor 309. The signal charge is a hole or an electron.

[0075] Although in Figure 1E Although not shown in the figure, the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are connected to the photodiode 203. A switch may be provided between the photodiode 203 and the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The switch switches the conduction state between the photodiode 203 and each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 according to the signal accumulation pulse from the control circuit 30. Thus, the start and stop of the accumulation of the signal charge to each of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are controlled. The electronic shutter in this embodiment has a mechanism for such exposure control.

[0076] The gate of the row selection transistor 308 is turned on by the row selection circuit 302, and the signal charges accumulated in the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207 are read out. At this time, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified according to the signal potential of the first floating diffusion layer 204, the second floating diffusion layer 205, the third floating diffusion layer 206, and the fourth floating diffusion layer 207. The analog signal based on the current read out from the vertical signal line 304 is converted into digital signal data by the analog-to-digital (AD) conversion circuit 307 connected to each column. The digital signal data is read out by the column selection circuit 303 for each column and output from the image sensor 20. After the row selection circuit 302 and the column selection circuit 303 read out one row, they read out the next row, and similarly, the information of the signal charge of the floating diffusion layers of all rows is read out. After all the signal charges are read out, the control circuit 30 turns on the gate of the reset transistor 310, thereby resetting all the floating diffusion layers. Thus, the imaging of one frame is completed. Similarly, by repeatedly performing high-speed imaging of frames, the imaging of a series of frames based on the image sensor 20 is completed.

[0077] In this embodiment, an example of a CMOS image sensor 20 is described, but the image sensor 20 may be another type of imaging element. The image sensor 20 may be, for example, a CCD type, a single photon counting element, or an amplification type image sensor such as an EMCCD or ICCD.

[0078] Figure 1F FIG. 1 is a diagram showing an example of an operation within one frame of the present embodiment. Figure 1F As shown, the emission of the first pulse light and the emission of the second pulse light can be switched alternately multiple times within one frame. In this way, the time difference between the acquisition timing of the detection images of the two wavelengths can be reduced, and even if the user 1 is moving, it is possible to perform photography based on the first and second pulse lights at approximately the same time.

[0079] [1-3. Memory 25]

[0080] In the memory 25, biometric data representing the physical characteristics of the user 1 acquired and registered in advance is stored. This biometric data is called a template. The memory 25 can be built into the recognition device 100 or provided outside. The memory 25 can be, for example, one or more ROMs, optical disks, or hard disk drives.

[0081] The biometric data stored in the memory 25 may be, for example, the image of the user 1 itself, or data characterizing the image. The data characterizing the image may include, for example, information indicating the texture of the skin surface of the user 1, information indicating the blood vessel distribution of the user 1, or information indicating the concavity and convexity of the characteristic part of the user 1. The "information indicating the texture of the skin surface of the user 1" may be, for example, a two-dimensional distribution of frequency components obtained by Fourier transforming the pixel values ​​of the two-dimensional distribution of wrinkles or micro-concavity and convexity at a certain part of the skin surface of the user 1. This transformation is effective when wrinkles or micro-concavity and convexity present a repetition of similar two-dimensional patterns. The "information indicating the blood vessel distribution of the user 1" may be, for example, image information indicating the vein distribution of the user 1. The "information indicating the concavity and convexity of the characteristic part of the user 1" may be, for example, image information indicating the two-dimensional distribution of the distance of the concavity and convexity in at least one of the eye sockets, nose, cheeks, cheekbones, mouth, jaw, and under the ears of the user 1. By storing the data characterizing the image of the user 1, it is possible not only to remove redundant information and improve the recognition accuracy, but also to reduce the amount of stored data.

[0082] [1-4. Control Circuit 30 and Signal Processing Circuit 40]

[0083] The control circuit 30 causes the light source 10 to emit pulse light to irradiate the user 1. The control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected pulse light returning from the user 1. The control circuit 30 adjusts the time difference between the pulse light emission timing of the light source 10 and the shutter timing of the image sensor 20. Hereinafter, this time difference is sometimes referred to as "phase" or "phase delay". The "emission timing" of the light source 10 is the timing when the pulse light emitted by the light source 10 starts to rise. The "shutter timing" is the timing when exposure starts. The control circuit 30 can change the emission timing to adjust the phase, and can also change the shutter timing to adjust the phase.

[0084] The control circuit 30 causes the image sensor 20 to output a signal representing a two-dimensional image of the user 1 corresponding to the intensity distribution of the detected light. The control circuit 30 causes the signal processing circuit 40 to process the signal output from the image sensor 20. The control circuit 30 may be configured to remove the offset component from the signal detected by each pixel of the image sensor 20. The offset component is a signal component of ambient light such as sunlight or fluorescent light, or interference light. By detecting a signal from the image sensor 20 in a state where the light source 10 does not emit light, that is, the drive of the light source 10 is turned off (OFF), the offset component based on the ambient light and interference light is estimated.

[0085] The control circuit 30 may be, for example, a combination of a processor and a memory, or an integrated circuit such as a microcontroller having a processor and a memory built therein. The control circuit 30 adjusts the light emission timing and shutter timing, estimates the offset component, removes the offset component, etc., by, for example, executing a program recorded in the memory by the processor.

[0086] The signal processing circuit 40 is a circuit for processing the image signal output from the image sensor 20. The signal processing circuit 40 performs arithmetic processing such as image processing. The signal processing circuit 40 can be implemented, for example, by a combination of a programmable logic device (PLD) such as a digital signal processor (DSP), a field programmable gate array (FPGA), or a central processing unit (CPU) or an image processing processor (GPU) and a computer program. In addition, the control circuit 30 and the signal processing circuit 40 can be a combined circuit or separate individual circuits. In addition, the signal processing circuit 40 can also be a component of an external device such as a server located remotely. In this case, the external device such as the server has a communication mechanism, and transmits and receives data to and from the light source 10, the image sensor 20 and the control circuit 30.

[0087] Figure 1G FIG. 2 is a flowchart schematically showing the operation of the control circuit 30. The control circuit 30 generally performs Figure 1G The operation shown. In addition, here, as an example, the operation in the case of detecting only the internal scattered component I2 is described. The control circuit 30 first causes the light source 10 to emit pulse light for a specified time (step S101). At this time, the electronic shutter of the image sensor 20 is in a state where the charge accumulation is stopped. The control circuit 30 stops the charge accumulation of the electronic shutter until the period in which a part of the pulse light is reflected by the surface of the user 1 and reaches the image sensor 20 is completed. Next, the control circuit 30 starts the charge accumulation of the electronic shutter at the timing when another part of the pulse light is scattered inside the user 1 and reaches the image sensor 20 (step S102). After the specified time has passed, the control circuit 30 stops the charge accumulation of the electronic shutter (step S103). Next, the control circuit 30 determines whether the number of times the above-mentioned charge accumulation has been performed has reached a specified number of times (step S104). The specified number of times can be, for example, more than one thousand times and less than 100,000 times. If it is determined to be no in step S104, the control circuit 30 repeats steps S101 to S103 until it is determined to be yes. If the determination is YES in step S104 , the control circuit 30 causes the image sensor 20 to generate and output a signal representing a two-dimensional distribution image based on the total amount of signal charges accumulated in each floating diffusion layer of each pixel (step S105 ).

[0088] Through the above operation, the components of light scattered inside the measured object can be detected with high sensitivity. In addition, multiple light emission and exposure are not necessary, but can be performed as needed. In addition, Figure 1G The operations of steps S101 to S105 shown above are also applicable when only the surface reflection component I1 is detected.

[0089] The control circuit 30 authenticates the user 1 by comparing the signal output from the image sensor 20 with the biometric data stored in the memory 25. The operation of the control circuit 30 is as follows. The control circuit 30 causes the signal processing circuit 40 to generate image data based on the information obtained by the image sensor 20. The control circuit 30 compares the image data with the biometric data stored in the memory 25. In the case where data that characterizes the image of the user 1 is stored in the memory 25, the signal processing circuit 40 extracts the data that characterizes the image from the generated image of the user 1, and the control circuit 30 compares the data that characterizes the image of the user 1 with each other. In addition, the control circuit 30 may simply compare the correlation between the information obtained by the image sensor 20 and the information registered in advance. In addition, the control circuit 30 may authenticate the user 1 based on the user's characteristics obtained by machine learning such as deep learning or support vector machine (SVM).

[0090] [1-5. Optical system]

[0091] The recognition device 100 may also include an imaging optical system that forms a two-dimensional image of the user 1 on the light receiving surface of the image sensor 20. The optical axis of the imaging optical system is substantially orthogonal to the light receiving surface of the image sensor 20. The imaging optical system may include a zoom lens. If the position of the zoom lens changes, the magnification of the two-dimensional image of the user 1 changes, and the resolution of the two-dimensional image on the image sensor 20 changes. Therefore, even if the distance from the user 1 is far, the area to be measured can be magnified and observed in detail.

[0092] In addition, the recognition device 100 may also be provided with a bandpass filter between the user 1 and the image sensor 20 that allows only light in the wavelength band or its vicinity emitted from the light source 10 to pass. In this way, the influence of interference components such as ambient light can be reduced. The bandpass filter is composed of a multilayer film filter or an absorption filter. Considering the temperature of the light source 10 and the shift of the wavelength band accompanying the oblique incidence to the filter, the bandwidth of the bandpass filter can have a width of about 20 to 100 nm.

[0093] In addition, the recognition device 100 may also include polarizing plates between the light source 10 and the user 1 and between the image sensor 20 and the user 1. In this case, the polarization directions of the polarizing plate arranged on the light source 10 side and the polarizing plate arranged on the image sensor side are in an orthogonal Nicol relationship. As a result, the regular reflection component of the surface reflection component I1 of the user 1, that is, the component with the same incident angle and reflection angle, can be prevented from reaching the image sensor 20. That is, the amount of light of the surface reflection component I1 reaching the image sensor 20 can be reduced.

[0094] [2. Time-decomposition camera operation]

[0095] The recognition device 100 of the present embodiment can distinguish and detect the surface reflection component I1 and the internal scattering component I2 by time-resolved imaging.

[0096] Hereinafter, an example of the operation of the recognition device 100 according to the present embodiment will be described.

[0097] like Figure 1A As shown, if the light source 10 irradiates pulse light to the user 1, a surface reflection component I1 and an internal scattering component I2 are generated. Parts of the surface reflection component I1 and the internal scattering component I2 reach the image sensor 20. The internal scattering component I2 is emitted from the light source 10 and passes through the inside of the user 1 before reaching the image sensor 20. That is, the optical path length of the internal scattering component I2 is longer than the optical path length of the surface reflection component I1. Therefore, the time when the internal scattering component I2 reaches the image sensor 20 is delayed on average relative to the surface reflection component I1.

[0098] In the time-resolved imaging, the surface reflection component I1 is acquired by the following operation.

[0099] Figure 2A This is a diagram showing an example of an optical signal in which rectangular pulse light is emitted from the light source 10 and the light returned from the user 1 reaches the image sensor 20. The horizontal axis represents time (t) in all signals (a) to (d), and the vertical axis represents intensity in signals (a) to (c). In signal (d), the electronic shutter is opened (OPEN) or closed (CLOSE). Signal (a) represents the surface reflection component I1. Signal (b) represents the internal scattering component I2. Signal (c) represents the total component of the surface reflection component I1 (a) and the internal scattering component I2 (b). Signal (d) is a diagram showing the timing of the electronic shutter used to obtain the surface reflection component I1 of user 1.

[0100] like Figure 2AAs shown in the signal (d) in FIG. 1 , by cutting off the shutter, the components arriving earlier in the reflected light returning to the image sensor 20 can be efficiently collected. The components arriving earlier mean that there is less scattering in the user 1 and contain the surface information of the user 1. In fact, the time for light accumulation is the shorter time of the front end of the pulse wave, but the shutter does not necessarily have to be only during this period. Figure 2A As shown in the signal (d) in FIG. 1 , if the charge is accumulated before the shutter is about to close, the shutter can be opened at a stage earlier than the front end of the pulse wave reaches the image sensor 20. Thus, an expensive high-time resolution camera device of the picosecond level is not required. The recognition device 100 of this embodiment can be composed of an inexpensive image sensor 20.

[0101] To execute Figure 2A According to the action shown by the signal (d) in FIG. 1 , the control circuit 30 causes the image sensor 20 to detect at least a portion of the components during the rise period of the reflected pulse light, and causes it to output a signal representing a two-dimensional image of the user 1. In this embodiment, the signal output from the image sensor 20 may include a signal obtained from at least a portion of the components during the rise period of the reflected pulse light.

[0102] The light source 10 irradiates a rectangular pulse wave. At this time, the pulse width does not need to be in the ps level, but can be several ns. Therefore, an inexpensive light source can be used. If the Tr characteristic of the front end of the pulse wave is a rapid rise, when only the front end is shuttered, the unnecessary internal scattering component I2 with time delay can be minimized from mixing into the acquired image.

[0103] In time-resolved imaging, the internal scattered component I2 is acquired by the following operation.

[0104] Figure 2B 1 is a diagram showing another example of an optical signal in which rectangular pulse light is emitted from the light source 10 and light returned from the user 1 reaches the image sensor 20 . Figure 2B The signals (a) to (c) in FIG. Figure 2A The signals in (a) to (c) have the same time variations. Figure 2B The signal (d) in FIG. 1 is a diagram showing the timing of the electronic shutter used to obtain the internal scattered component I2.

[0105] like Figure 2B As shown in the signal (a) in FIG, the surface reflection component I1 maintains a rectangular shape. On the other hand, Figure 2B As shown in the signal (b) in FIG. 1 , the internal scattered component I2 is the sum of the light that has passed through various optical path lengths, so it has the characteristic of trailing at the rear end of the pulse light. That is, compared with the surface reflected component I1, the falling period becomes longer. Figure 2B The signal (c) is extracted by increasing the proportion of the internal scattering component I2 in the optical signal, such as Figure 2B As shown in the signal (d) in , after the rear end of the surface reflection component I1, the electronic shutter begins to accumulate charge. After the rear end of the surface reflection component I1 is when the surface reflection component I1 decreases or thereafter. The shutter timing is adjusted by the control circuit 30. As described above, the recognition device 100 of this embodiment distinguishes and detects the surface reflection component I1 from the internal scattering component I2 that reaches the deep part of the user 1, so the pulse light width and shutter width are arbitrary. Therefore, unlike the previous method of using an ultra-high-speed scanning camera, the acquisition of the internal scattering component I2 can be achieved through a simple structure, which can greatly reduce the cost.

[0106] To execute Figure 2B According to the action shown by the signal (d) in FIG. 1 , the control circuit 30 causes the image sensor 20 to detect at least a part of the component of the falling period in the reflected pulse light, so as to output a signal representing the two-dimensional image of the user 1. In the present embodiment, the signal output from the image sensor 20 may include a signal obtained from at least a part of the component of the falling period in the reflected pulse light.

[0107] exist Figure 2B In the signal (a) in , the rear end of the surface reflection component I1 drops vertically. In other words, the time from the start of the drop to the end of the drop of the surface reflection component I1 is zero. However, in reality, there are cases where the pulse light irradiated by the light source 10 itself is not completely vertical, or there are fine irregularities on the surface of the user 1, or the rear end of the surface reflection component I1 does not drop vertically due to scattering in the epidermis. In addition, since the user 1 is an opaque object, the amount of light of the surface reflection component I1 is much larger than that of the internal scattering component I2. Therefore, even if the rear end of the surface reflection component I1 slightly exceeds the vertical drop position, the internal scattering component I2 may be masked. In addition, due to the time delay associated with the movement of electrons during the readout period of the electronic shutter, there is a possibility that it cannot be achieved. Figure 2B The ideal binary readout is shown in the signal (d) in . Therefore, the control circuit 30 may also delay the shutter timing of the electronic shutter slightly after the surface reflection component I1 just drops. For example, it may be delayed by about 0.5ns to 5ns. In addition, instead of adjusting the shutter timing of the electronic shutter, the control circuit 30 may adjust the light emission timing of the light source 10. The control circuit 30 adjusts the time difference between the shutter timing of the electronic shutter and the light emission timing of the light source 10. If the shutter timing is delayed too much, the originally small internal scattering component I2 is further reduced. Therefore, the shutter timing may also be kept near the rear end of the surface reflection component I1. Since the time delay caused by the scattering of user 1 assumed based on the sensor sensitivity is 4ns, the maximum delay amount of the shutter timing is about 4ns.

[0108] Alternatively, the light source 10 may emit pulse light a plurality of times, and each pulse light may be exposed a plurality of times with shutter timing of the same phase, thereby increasing the amount of light to be detected of the internal scattered component I2.

[0109] Alternatively, instead of placing a bandpass filter between the user 1 and the image sensor 20, or in addition thereto, the control circuit 30 may estimate the offset component by performing photography with the same exposure time without causing the light source 10 to emit light. The estimated offset component is subtracted and removed from the signal detected by each pixel of the image sensor 20. In this way, the dark current component and / or disturbance light generated in the image sensor 20 can be removed.

[0110] Next, an example of a method of detecting the surface reflection component I1 and the internal scattering component I2 for each frame will be described.

[0111] Figure 3A FIG. 4 is an example of a time chart for detecting the surface reflection component I1. In order to detect the surface reflection component I1, for example, Figure 3A As shown, the shutter can be set to open before the pulse light reaches the image sensor 20, and the shutter can be set to close before the rear end of the pulse light arrives. By controlling the shutter in this way, the mixing of the internal scattered component I2 can be reduced. The proportion of light passing through the surface near the user 1 can be increased. In particular, the timing of shutter closing can also be set to just after the light reaches the image sensor 20. In this way, signal detection of the proportion of the surface reflection component I1 with a shorter optical path length can be performed. As another method of obtaining the surface reflection component I1, the image sensor 20 can also obtain the entire pulse light, or irradiate continuous light from the light source 10 for detection.

[0112] Figure 3B An example of a time chart for detecting the internal scattered component I2 is shown. By opening the shutter while the rear end portion of the pulse reaches the image sensor 20, the signal of the internal scattered component I2 can be acquired.

[0113] If time-division video is taken by the same camera or the same sensor as in this embodiment, time and space deviations are less likely to occur. When the same sensor is used to obtain signals of both the surface reflection component I1 and the internal scattering component I2, the following can be done: Figure 3A and Figure 3B As shown in FIG. 1 , the components to be obtained are switched every 1 frame. Alternatively, as shown in FIG. Figures 1D to 1F As described above, the components to be acquired are alternately switched at high speed within one frame. In this case, the detection time difference between the surface reflection component I1 and the internal scattering component I2 can be reduced.

[0114] Furthermore, the signals of the surface reflection component I1 and the internal scattering component I2 may be obtained using two wavelengths of light. When the surface reflection component I1 and the internal scattering component I2 are obtained using two wavelengths, for example, as shown in FIG. Figures 1D to 1F As described above, it is possible to use a method of switching four types of charge accumulation at high speed within one frame. This method can reduce the time variation of the detection signal.

[0115] [3. Spatial resolution camera operation]

[0116] In addition to the above-mentioned time-resolved imaging, the surface reflection component I1 and the internal scattering component I2 can also be obtained by space-resolved imaging.

[0117] FIG. 4A to FIG. 4C 1 is a diagram schematically showing an example of distribution of two-dimensional pattern light projected from the light source 10 toward the user 1. FIG. 4A to FIG. 4C In the example shown, the white portion indicates a bright portion where light exists, and the black portion indicates a dark portion where no light exists. Figure 4A In FIG. 1 , a checkerboard pattern is shown in which a plurality of bright areas and a plurality of dark areas are periodically distributed. Figure 4B In FIG. 1 , a dot pattern of multiple dark areas is randomly distributed in the bright area. Figure 4C , a dot pattern in which a plurality of bright areas are randomly distributed in a dark area is shown.

[0118] The control circuit 30 causes the light source 10 to emit a two-dimensional pattern light that projects a two-dimensional pattern toward the user 1. FIG. 4A to FIG. 4C As shown, the two-dimensional pattern light includes at least one bright part and at least one dark part. The two-dimensional pattern light is obtained, for example, by disposing a light shielding mask with a two-dimensional distribution in front of the light source 10. Alternatively, the two-dimensional pattern light can also be formed using a digital micromirror device (DMD) or a spatial light modulator (SLM). The two-dimensional pattern light can be either continuous light or pulsed light.

[0119] like FIG. 4A to FIG. 4C As shown, a two-dimensional pattern light with spatially modulated intensity such as a checkerboard pattern or a dot pattern is emitted from the light source 10. If the two-dimensional pattern light is projected toward the user 1, more light is returned from the bright part and less light is returned from the dark part. The reflected light returned from the dark part contains an internal scattered component I2 scattered inside the user 1, and almost no surface reflected component I1.

[0120] In order to detect the internal scattered component I2 by the two-dimensional pattern light, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected light returned from at least one dark part of the user 1 on which the two-dimensional pattern light is projected, and outputs a signal corresponding to the intensity distribution of the at least a portion of the detected reflected light. Thus, if the two-dimensional pattern light is pulsed light, it is possible to obtain Figure 2Aor Figure 2B The portion from the front end to the rear end of the internal scattered component I2 shown in the signal (b) in FIG. 1. In the present embodiment, the signal output from the image sensor 20 may include the above-mentioned signal obtained from at least one dark portion.

[0121] On the other hand, the reflected light returned from the bright part includes both the surface reflection component I1 and the internal scattering component I2. Therefore, the surface reflection component I1 can be calculated by subtracting the detection data of the nearby dark part from the detection data of the bright part. In this case, the spatial resolution decreases. As a countermeasure, it can be considered not to obtain the signal with a single shot, but to obtain the signal multiple times by, for example, offsetting the distribution of the two-dimensional pattern light, or changing the distribution of the two-dimensional pattern light itself. In this way, the surface reflection component I1 can be obtained without reducing the spatial resolution.

[0122] In order to detect the surface reflection component I1 by the two-dimensional pattern light, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected light returned from at least one dark portion and at least a portion of the reflected light returned from at least one bright portion of the user 1 on which the two-dimensional pattern light is projected. The control circuit 30 causes the image sensor 20 to output a signal corresponding to the intensity distribution of the above-mentioned at least a portion of the reflected light returned from at least one dark portion and a signal corresponding to the intensity distribution of the above-mentioned at least a portion of the reflected light returned from at least one bright portion. The control circuit 30 causes the signal processing circuit 40 to calculate the surface reflection component I1 by subtracting the above-mentioned signal obtained from at least one dark portion from the above-mentioned signal obtained from at least one bright portion. Thus, if the two-dimensional pattern light is pulse light, it is possible to obtain Figure 2A or Figure 2B In the present embodiment, the signal output from the image sensor 20 may include the signal obtained from at least one bright part in addition to the signal obtained from at least one dark part.

[0123] It is also possible to combine spatial resolution imaging and temporal resolution imaging to obtain each of the surface reflection component I1 and the internal scattering component I2.

[0124] When the two-dimensional pattern light is pulse light, at least a portion of the rising period of the reflected pulse light returned from at least one bright part may be detected to obtain the surface reflection component I1. Similarly, at least a portion of the falling period of the reflected pulse light returned from at least one bright part or at least one dark part may be detected to obtain the internal scattering component I2. The operation of the control circuit 30 and the signal processing circuit 40 when detecting the components during the rising period and the components during the falling period of the reflected pulse light is as described above.

[0125] [4. Biometric Authentication Based on Surface Reflection Component I1 and Biometric Authentication Based on Internal Scattering Component I2]

[0126] A specific example of biometric authentication performed by the recognition device 100 of this embodiment will be described in comparison with a method using a general camera.

[0127] Figure 5A : This is a diagram schematically showing a state where a user 1 is photographed by a conventional camera 90. The light irradiating the user 1 also enters a few millimeters inside the surface of the user 1. Therefore, the conventional camera 90 detects both the surface reflection component I1 and the internal scattering component I2 contained in the reflected light returning from the user 1. The image in which both the surface reflection component I1 and the internal scattering component I2 are mixed may be somewhat blurred. Therefore, in face recognition based on the conventional camera 90, the acceptance rate of others may increase or the rejection rate of the person may increase. For example, a twin brother may be mistaken for the person himself, or a person may be mistaken for not the person himself if his hairstyle has been changed. That is, in the method using the conventional camera 90, the accuracy of face recognition may deteriorate.

[0128] Figure 5B 1 is a diagram schematically showing an example of photographing user 1 based on surface reflection component I1 of the present embodiment. In the recognition device 100 of the present embodiment, as described above, the surface reflection component I1 can be detected by time-decomposition photography or space-decomposition photography. As a result, the texture of the skin surface of user 1, such as wrinkles or micro-concavities, can be detected more clearly. By using the result of comparing the information obtained from the surface reflection component I1 with the information representing the texture of the skin surface of user 1 contained in the biometric data of the memory 25, the authentication accuracy can be improved. As a result, the rejection rate of the user or the acceptance rate of others is reduced.

[0129] Figure 5C 1 is a diagram schematically showing an example of photographing the user 1 based on the internal scattered component I2 according to the present embodiment. In the recognition device 100 according to the present embodiment, the internal scattered component I2 can be detected by time-resolved imaging or space-resolved imaging as described above. As a result, the distribution of blood vessels such as veins of the user 1 can be detected more clearly. By using the result of comparing the information obtained from the internal scattered component I2 with the information indicating the blood vessel distribution of the user 1 included in the biometric data of the memory 25, the authentication accuracy can be improved. As a result, the rejection rate of the user or the acceptance rate of others is reduced.

[0130] It is also possible to combine authentication based on the surface reflection component I1 with authentication based on the internal scattering component I2. For example, depending on the purpose, authentication can be considered successful only when both are authenticated, or authentication can be considered successful only when only one of the two is authenticated. In the case where the internal scattering component I2 is used for vein authentication, near-infrared light that is easier to penetrate a living body can be used. On the other hand, in the case where the surface reflection component I1 is used for facial feature authentication, short-wavelength light such as the blue band that is difficult to penetrate a living body can be used. In this way, different types of wavelengths of light can be used in the detection of the surface reflection component I1 and the internal scattering component I2.

[0131] The internal scattered component I2 contains deeper biological information of the user 1 than the surface reflected component I1. Therefore, it is possible to obtain information different from the surface reflected component I1, such as vein information or internal cells. Therefore, it is possible to implement multiple different types of authentication, such as authentication based mainly on facial features through the surface reflected component I1 and authentication based mainly on veins through the internal scattered component I2. Even if the authentication of one party fails, the authentication failure can be compensated by the authentication of the other party. This brings about stable and high-precision authentication, which can ensure higher security. In addition, in the case where someone else pretends to be the user 1, the authentication of both parties needs to be forged, and tampering becomes extremely difficult. In this way, through the recognition device 100 of this embodiment, an authentication system that is more difficult to forge or tamper with can be realized.

[0132] In addition to performing multiple authentications independently, authentication can also be performed by machine learning using data synthesized from the surface reflection component I1 and the internal scattering component I2. By including different features in the data, the amount of information increases, and the accuracy of machine learning recognition can be improved.

[0133] In addition to the above, the user 1 may be authenticated by the TOF method that calculates the distance to an object based on the round trip time of pulsed light. The TOF method can obtain a two-dimensional distribution of the distance between the image sensor 20 and the surface of the user 1. The two-dimensional distribution of the distance between the image sensor 20 and the surface of the user 1 can also be said to be a three-dimensional distribution of the surface of the user 1.

[0134] The distance between the image sensor 20 and the surface of the user 1 is calculated as follows. The round trip time from the pulse light emitted from the light source 10 to the pulse light reflected by the surface of the user 1 and detected by the image sensor 20 is τ, and the speed of light in the air is c. At this time, the distance d between the image sensor 20 and the surface of the user 1 is d=cτ / 2.

[0135] As a method of obtaining the round trip time τ, there are a direct TOF method and an indirect TOF method.

[0136] In the direct TOF method, the distance d to the object is calculated by directly measuring the round-trip time τ of light. In the direct TOF method, the time resolution used to achieve a distance resolution of Δd=1mm is Δτ=2Δd / c≈6.6 picoseconds. Thus, in the direct TOF method, a high-speed camera with a time resolution of picoseconds is used to achieve a distance resolution of millimeters.

[0137] On the other hand, in the indirect TOF method, which is often used in practice, the distance d to the object is calculated by measuring the round-trip time τ of the pulse light based on the phase difference of the reflected pulse light. The phase difference of the reflected pulse light is equivalent to the time interval between the emitted pulse light and the reflected pulse light. In the indirect TOF method, the shorter the pulse width Δt, the higher the measurement accuracy. In the indirect TOF method, in order to detect the phase difference, the light intensity of the emitted light is usually modulated by a rectangular wave or a sine wave.

[0138] Figure 6 This is a diagram for explaining the principle of measuring the round trip time τ of pulse light by the indirect TOF method. Figure 6 The signal (a) represents the time variation of the intensity of the emitted pulse light. Figure 6 Signals (b) and (c) represent the temporal variation of the intensity of the reflected pulse light.

[0139] exist Figure 6 In the example shown in the signal (a), the light intensity of the emitted pulse light is modulated by a rectangular wave. In order to accurately measure the phase difference, the light intensity of the emitted pulse light is basically modulated at a speed of the same order of magnitude as the round trip time τ of the light.

[0140] By opening the electronic shutter in the image sensor 20 and measuring the intensity, the phase difference of the light pulse can be detected. Figure 6 In the example shown in the signal (b), the image sensor 20 emits the pulse light at time t 0 Then the electronic shutter is opened to start the detection of reflected pulse light. Start time t 0 is longer than the pulse width Δt of the pulsed light and shorter than the round trip time τ of the light. <t 0 <τ. After the image sensor 20 opens the electronic shutter, it accumulates signal charges corresponding to the amount of reflected pulse light for each time Δt. Suppose the amount of signal charges accumulated during the initial time Δt is S 1 , let the amount of signal charge accumulated during the next time Δt be S 2 The round trip time τ of light is related to the starting time t 0 The difference is equal to [S 2 / (S 1 +S 2 )]Δt. Therefore, the round trip time of light is τ=t 0 +[S2 / (S 1 +S 2 )]Δt. According to the amount of signal charge S accumulated 1 , S 2 The round trip time τ of light is obtained indirectly, and the distance d to the object is calculated.

[0141] exist Figure 6 In the example shown in the signal (c), the reflected pulse light returns to the image sensor 20 before the emitted pulse light is emitted. That is, τ<Δt. In this case, the image sensor 20 opens the electronic shutter simultaneously with the emission of the pulse light from the light source 10, and accumulates the signal charge corresponding to the amount of reflected pulse light per time Δt. The round trip time of light is τ=[S 2 / (S 1 +S 2 )]Δt.

[0142] In the TOF method, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected pulse light and output a signal indicating the three-dimensional distribution of the surface of the user 1. In this embodiment, the signal output from the image sensor 20 may include a signal indicating the three-dimensional distribution of the surface of the user 1.

[0143] If the surface reflection component I1 included in at least a part of the reflected pulse light is used in the TOF method, the three-dimensional distribution of the surface of the user 1 including wrinkles and minute irregularities can be detected more clearly than with the conventional TOF method.

[0144] In the direct TOF method, the distance to the object can be calculated by detecting the surface reflection component I1 contained in the rising period of the reflected pulse light. Figure 6 As shown in the signals (b) and (c), the portion from the front end to the rear end of the reflected pulse light is detected to calculate the distance to the object. If spatial resolution imaging is used, the portion from the front end to the rear end of the surface reflection component I1 can be separated from the reflected pulse light.

[0145] Furthermore, if spatial resolution imaging is used, the portion from the front end to the rear end of the internal scattered component I2 can also be obtained from the reflected pulse light. If the portion from the front end to the rear end of the internal scattered component I2 is used in the indirect TOF method, the two-dimensional distribution of the distance between the image sensor 20 and the blood vessels of the user 1, that is, the three-dimensional distribution of the blood vessels of the user 1 can also be obtained.

[0146] Figure 7 FIG. 1 is a diagram schematically showing an example of photographing a user 1 using the TOF method according to the present embodiment. Figure 7In the example shown, the three-dimensional distribution of the surface of the user 1 is calculated based on the surface reflection component I1. By using the result of comparing the information obtained from the three-dimensional distribution of the surface of the user 1 with the information indicating the concavity and convexity of the characteristic part of the user 1 contained in the biometric data of the memory 25, the authentication accuracy can be improved. As a result, the rejection rate of the user or the acceptance rate of others is reduced. In addition, the three-dimensional distribution of the surface of the user 1 can also be calculated based on the reflected pulse light containing both the surface reflection component I1 and the internal scattering component I2.

[0147] Next, a series of processes for biometric authentication of the user 1 using the identification device 100 according to the present embodiment will be described.

[0148] Figure 8 This is a flowchart showing an example of processing executed by the control circuit 30 .

[0149] In step S201, the control circuit 30 causes the light source 10 to illuminate the user 1 with light. If time-resolved imaging is used, the light is pulsed light, and if space-resolved imaging is used, the light is two-dimensional patterned light. If time-resolved imaging and space-resolved imaging are combined, the light is two-dimensional patterned light and pulsed light.

[0150] In step S202, the control circuit 30 causes the image sensor 20 to detect at least a portion of the reflected light returned from the user 1, and outputs a signal representing a two-dimensional image of the user 1 corresponding to the intensity distribution of the detected light. If the reflected light is pulse light, at least a portion of the reflected light is, for example, a component during the rising period or a component during the falling period of the pulse light. The surface reflection component I1 is obtained from the component during the rising period, and the internal scattering component I2 is obtained from the component during the falling period. In the case of obtaining the three-dimensional distribution of the surface of the user 1 by the indirect TOF method, the portion from the front end to the rear end of the surface reflection component I1 or the portion from the front end to the rear end of the composite component of the surface reflection component I1 and the internal scattering component I2 in the reflected pulse light is detected.

[0151] In step S203, the control circuit 30 causes the signal processing circuit 40 to generate an image of the user 1 based on the output signal, and extracts data that characterizes the image. The data that characterizes the image generated based on the surface reflection component I1 includes information indicating the texture of the skin surface of the user 1. The data that characterizes the image generated based on the internal scattering component I2 includes information indicating the distribution of blood vessels of the user 1. In the case of the TOF method, the data that characterizes the image generated based on the surface reflection component I1 or the image generated based on the composite component of the surface reflection component I1 and the internal scattering component I2 includes information indicating the three-dimensional distribution of the surface of the user 1, that is, the concavity and convexity of the characteristic part of the user 1.

[0152] In step S204 , the control circuit 30 compares the data characterizing the image extracted from the generated image of the user 1 with the data characterizing the image of the user 1 stored in the memory.

[0153] In step S205, the control circuit 30 determines whether the difference between the two compared data is less than or equal to a reference value. For example, when the information of the three-dimensional distribution of the surface of the user 1 obtained by the TOF method is compared with the information indicating the concavity and convexity of the characteristic part of the user 1 included in the biometric data, an example of the determination reference is as follows. The control circuit 30 evaluates whether the difference between the two compared data is less than or equal to a reference value for all the measured points on the surface of the user 1 or for a certain proportion of measured points such as 80%.

[0154] If the difference is less than the reference value, the control circuit 30 determines in step S206 that the user 1 is the user himself. If the difference is not less than the reference value, the control circuit 30 determines in step S207 that the user 1 is someone else.

[0155] The identification device 100 of this embodiment can realize contactless biometric authentication of the user 1. Therefore, the burden on the user 1 during authentication is small.

[0156] In addition to the face, other parts of the body of the user 1, such as the hand, may also be used. For example, the surface reflection component I1 may be obtained from the fingerprint, and the internal scattering component I2 may be obtained from the vein of the finger. On the hand, it is easy to clearly present the fingerprint or vein unique to the individual, and the recognition accuracy is improved. In the case of using the hand as the inspected part of the user 1, the body movement can be suppressed by placing the hand on the glass. At this time, if there is space between the glass and the recognition device 100, there is an advantage in being able to focus the camera. In addition, in the case of hand authentication, by emitting a light amount exceeding level 1 from the light source 10, the SN ratio of the detection signal on the image sensor 20 can be improved.

[0157] Next, an application example of the recognition device 100 according to the present embodiment will be described.

[0158] Fig. 9 Schematically shows an example in which the recognition device 100 of this embodiment is installed in the interior of a car. Fig. 9As shown, the driver, i.e., user 1, can be identified by an identification device 100 installed in the vehicle. Instead of using the car key, the identification device 100 can be used to identify user 1 to start the vehicle. The identification device 100 can also be used to identify passengers during automatic driving, and the identification result can be fed back to the automatic driving customized according to the passenger. The feedback can be, for example, to make braking easier if the passenger is an elderly person, or to set a brisk driving feeling if the passenger likes driving. In this way, automatic driving suitable for individuals can be realized.

[0159] Fig.10 1 is a diagram schematically showing an example of applying the recognition device 100 of this embodiment to a mobile terminal. The mobile terminal is, for example, a smartphone, a tablet, or a personal computer. As the image sensor 20 in the recognition device 100, an image sensor built into the mobile terminal may also be used.

[0160] The recognition device 100 of this embodiment can be applied to an ATM installed in a bank or convenience store or at the entrance of a house, in addition to a car or a mobile terminal.

[0161] The identification device 100 of this embodiment can perform high-precision authentication based on a combination of more than two authentications. In addition to the authentication of logging into the terminal, the identification device 100 of this embodiment can also be used for security authentication of using the terminal. Security authentication using the terminal includes, for example, authentication after logging into the terminal, logging into a bank account, or logging into accounts of various services. It can also be designed to use the previous password authentication and the authentication of the present invention as an additional authentication system. As a result, compared with the previous system that only performs password authentication, the risk of hackers is reduced and higher security can be ensured.

[0162] The present invention also includes a method including actions performed by the control circuit 30 and the signal processing circuit 40 .

[0163] As described above, the present invention includes the recognition device and recognition method described in the following items.

[0164] [Project 1]

[0165] The recognition device related to the first item comprises: a light source; an image sensor; a memory storing biometric data representing the characteristics of a user's body; and a processor. The processor causes the light source to emit pulse light with a pulse width of not less than 0.2ns and not more than 1μs to irradiate the user, causes the image sensor to detect at least a portion of the reflected pulse light returning from the user, and outputs a signal corresponding to a two-dimensional distribution of the intensity of the at least a portion of the reflected pulse light, and identifies the user by comparing the signal with the biometric data.

[0166] [Project 2]

[0167] In the identification device according to the first item, the pulse width of the pulse light may be greater than or equal to 3 ns and less than or equal to 50 ns.

[0168] [Item 3]

[0169] In the recognition device related to the first or second item, it may also be that the image sensor includes a plurality of pixels, and the processor causes the image sensor to repeatedly accumulate the charge corresponding to the intensity of the at least a portion of the reflected pulse light for each of the plurality of pixels more than one thousand times and less than one hundred thousand times, and the signal represents a two-dimensional distribution of the total amount of charge accumulated in the plurality of pixels.

[0170] [Item 4]

[0171] In the recognition device according to any one of items 1 to 3, the biometric data may include first information indicating a texture of a skin surface of the user.

[0172] [Item 5]

[0173] In the identification device related to item 4, the processor may cause the image sensor to output a first signal, and identify the user by comparing information obtained from the first signal with the first information, wherein the first signal is obtained by detecting at least a portion of a component in the reflected pulse light, i.e., a rising period from the start of increase in the intensity of the reflected pulse light to the end of the increase.

[0174] [Item 6]

[0175] In the identification device according to any one of items 1 to 5, the biometric data may include second information indicating blood vessel distribution of the user.

[0176] [Item 7]

[0177] In the identification device related to item 6, the processor may cause the image sensor to output a second signal, and identify the user by comparing information obtained from the second signal with the second information, wherein the second signal is obtained by detecting at least a portion of the component in the reflected pulse light, i.e., a falling period, from the beginning of the decrease in intensity of the reflected pulse light to the end of the decrease.

[0178] [Item 8]

[0179] In the recognition device according to any one of items 1 to 7, the biometric data may include third information indicating concavity and convexity of a characteristic portion of the user.

[0180] In the recognition device according to any one of items 1 to 7, the biometric data may include fourth information indicating a shape of a characteristic portion of the user.

[0181] In the recognition device according to any one of items 1 to 7, the biometric data may include fifth information indicating a position of a characteristic portion of the user.

[0182] [Item 9]

[0183] In the identification device related to Item 8, the processor may cause the image sensor to output a third signal representing a two-dimensional distribution of the distance between the image sensor and the surface of the user, and identify the user by comparing information obtained from the third signal with the third information.

[0184] [Item 10]

[0185] In the identification device related to Item 9, the processor may cause the image sensor to detect at least a portion of the reflected pulse light, i.e., a rising period from the start of the increase in the intensity of the reflected pulse light to the end of the increase, and calculate the two-dimensional distribution of the distance based on the component.

[0186] [Item 11]

[0187] The recognition device related to item 11 comprises: a light source; an image sensor; a memory storing biometric data representing the characteristics of the user's body; and a processor. The biometric data includes first information representing the texture of the skin surface of the user. The processor causes the light source to emit irradiation light that projects a two-dimensional pattern including at least one bright part and at least one dark part onto the user, causes the image sensor to detect at least a portion of the reflected light returned from the user, and outputs a signal corresponding to the two-dimensional distribution of the intensity of the at least a portion of the reflected light, and identifies the user by comparing the signal with the biometric data stored in the memory.

[0188] [Item 12]

[0189] In the identification device according to item 11, the biometric data may further include second information indicating blood vessel distribution of the user.

[0190] [Item 13]

[0191] In the identification device related to Item 12, the processor may cause the image sensor to detect at least a portion of the first reflected light returned from the at least one dark portion of the user on which the two-dimensional pattern is projected, and output a first signal corresponding to the two-dimensional distribution of the intensity of the at least a portion of the first reflected light, and identify the user by comparing the information obtained from the first signal with the second information.

[0192] [Item 14]

[0193] In the identification device related to Item 13, the processor may cause the image sensor to detect at least a portion of the second reflected light returned from the at least one bright portion of the user on which the two-dimensional pattern is projected, and output a second signal corresponding to the two-dimensional distribution of the intensity of the at least a portion of the second reflected light, and identify the user by comparing information obtained by subtracting the first signal from the second signal with the first information.

[0194] [Item 15]

[0195] In the identification device related to Item 15, the processor may cause the image sensor to detect at least a portion of the first reflected light returned from the at least one dark portion of the user on which the two-dimensional pattern is projected, and to detect at least a portion of the second reflected light returned from the at least one bright portion, output a first signal corresponding to the two-dimensional distribution of the intensity of the at least one portion of the first reflected light, and output a second signal corresponding to the two-dimensional distribution of the intensity of the at least one portion of the second reflected light, and identify the user by comparing information obtained by subtracting the first signal from the second signal with the first information.

[0196] [Item 16]

[0197] In the identification device according to item 11, the irradiation light may be pulse light.

[0198] [Item 17]

[0199] In the identification device according to item 16, the pulse width of the pulse light may be greater than or equal to 3 ns and less than or equal to 50 ns.

[0200] [Item 18]

[0201] In the recognition device related to item 16 or 17, it may be that the image sensor includes a plurality of pixels, and the processor causes the image sensor to repeatedly accumulate the charge corresponding to the intensity of at least a portion of the reflected light for each of the plurality of pixels more than one thousand times and less than one hundred thousand times, and the signal represents a two-dimensional distribution of the total amount of charge accumulated in the plurality of pixels.

[0202] [Item 19]

[0203] In the recognition device according to any one of items 16 to 18, the biometric data may further include third information indicating concavity and convexity of a characteristic portion of the user.

[0204] [Item 20]

[0205] In the identification device related to Item 19, the processor may cause the image sensor to detect at least a portion of the first reflected pulse light returned from the at least one dark portion of the user on which the two-dimensional pattern is projected, detect at least a portion of the second reflected pulse light returned from the at least one bright portion, output a first signal corresponding to the two-dimensional distribution of the intensity of the at least one portion of the first reflected pulse light, and output a second signal corresponding to the two-dimensional distribution of the intensity of the at least one portion of the second reflected pulse light, and identify the user by comparing information obtained by subtracting the first signal from the second signal with the third information.

[0206] [Item 21]

[0207] In the identification device according to any one of items 16 to 18, the biometric data may further include second information indicating blood vessel distribution of the user.

[0208] [Item 22]

[0209] In the identification device related to Item 21, the processor may cause the image sensor to detect at least a part of a period from when the intensity of the first reflected pulse light returned from the at least one dark portion of the user on whom the two-dimensional pattern is projected, i.e., a falling period, and output a first signal corresponding to the two-dimensional distribution of the intensity of the at least a part of the falling period of the first reflected pulse light, and identify the user by comparing the information obtained from the first signal with the second information.

[0210] [Item 23]

[0211] In the identification device related to any one of items 16 to 18, the processor may cause the image sensor to detect at least a part of a rising period from the beginning to the end of the increase in the intensity of the second reflected pulse light returned from the at least one bright portion of the user on which the two-dimensional pattern is projected, and output a second signal corresponding to the two-dimensional distribution of the intensity of the at least a part of the rising period of the second reflected pulse light, and identify the user by comparing the information obtained from the second signal with the first information.

[0212] [Item 24]

[0213] The identification method related to Item 24 is an identification method using an identification device having a light source and an image sensor, comprising: causing the light source to emit pulse light with a pulse width of not less than 0.2ns and not more than 1μs to irradiate the user; causing the image sensor to detect at least a portion of the reflected pulse light returned from the user, and output a signal corresponding to a two-dimensional distribution of the intensity of the at least a portion of the reflected pulse light; and identifying the user by comparing the signal with biometric data representing the physical characteristics of the user.

[0214] [Item 25]

[0215] The identification method related to Item 25 is an identification method using an identification device having a light source and an image sensor, comprising: causing the light source to emit irradiation light that projects a two-dimensional pattern including at least one bright portion and at least one dark portion onto the user; causing the image sensor to detect at least a portion of the reflected light returned from the user, and output a signal corresponding to the two-dimensional distribution of the intensity of the at least a portion of the reflected light; and identifying the user by comparing the signal with biological data representing the texture of the skin surface of the user.

[0216] Industrial Applicability

[0217] The identification device of the present invention is useful for a camera or an authentication device that performs personal identification of a user without contact. The identification device can be applied to security services.

[0218] Description of symbols

[0219] 1 user

[0220] 10 Light Source

[0221] 20 Image Sensor

[0222] 22 Photoelectric conversion unit

[0223] 24 Charge storage unit

[0224] 25 Memory

[0225] 30 control circuit

[0226] 32 Light source control unit

[0227] 34Sensor control unit

[0228] 40 signal processing circuit

[0229] 100 identification devices

[0230] I1 Surface reflection component

[0231] I2 internal scattering component

Claims

1. A device, characterized in that: have: light source; Image sensor; as well as processor; The processor causes the light source to emit illumination light that projects a dot pattern onto the user's face; The processor causes the image sensor to detect light from the face to generate an image signal; The processor performs a first recognition process based on a surface reflection component of light from the face according to the image signal, The processor performs a second recognition process based on an internal scattered component of light from the face according to the image signal, The processor performs authentication of the user based on a result of the first identification process and a result of the second identification process.

2. The device according to claim 1, characterized in that The processor determines that the authentication of the user has succeeded, when it is determined that the first recognition process has succeeded and the second recognition process has succeeded.

3. The device according to claim 1 or 2, characterized in that The light source and the image sensor are arranged on a vehicle. The processor starts the vehicle when it is determined that the authentication of the user has succeeded.

4. The device according to claim 1 or 2, characterized in that The processor generates information representing the concavity and convexity of the face based on the surface reflection component, and executes the first recognition process based on the information representing the concavity and convexity.

5. The device according to claim 1 or 2, characterized in that The wavelength of the irradiation light is a wavelength within the wavelength range from red to near infrared rays.

6. A method, a processing method executed by at least one processor, characterized in that: include: causing the light source to emit illumination light that projects a dot pattern toward a user's face; causing an image sensor to detect light from the face to generate an image signal; performing a first recognition process based on a surface reflection component of light from the face according to the image signal; executing, based on the image signal, a second recognition process based on an internal scattered component of light from the face; and The user is authenticated based on the result of the first recognition process and the result of the second recognition process.

7. A program product, characterized in that The program product executes the method described in claim 6.

Images