Biological image acquisition device and biological image acquisition method

Abstract

This biological image acquisition device is provided with: an imaging unit that images at least part of a hand of the person to be authenticated; a control unit that controls the imaging unit on the basis of an exposure control parameter for controlling the imaging unit; a detection unit that detects a fingertip of the hand from the captured image having been captured by the imaging unit; an evaluation unit that evaluates the fingertip luminance value of the detected fingertip; an exposure control unit that determines the next exposure control parameter of the imaging unit on the basis of the fingertip luminance value, and causes the imaging unit to perform re-imaging of the fingertip using the next exposure parameter; and an acquisition unit that selects and acquires at least one captured image from among a plurality of captured images.

Description

Biometric image acquisition device and biometric image acquisition method

[0001] The present disclosure relates to a biometric image acquisition device and a biometric image acquisition method.

[0002] Patent Document 1 discloses a biometric authentication device that acquires multiple sets of palm images using light emitted from a light source under multiple different lighting conditions, adjusts the time interval between each of the multiple image sets when acquiring the multiple image sets depending on the shooting conditions, extracts biometric features from each of the multiple image sets, and compares each extracted biometric feature with pre-registered biometric features.

[0003] Japanese Patent Application Publication No. 2017-162302

[0004] It is desirable to capture images used for biometric authentication under more appropriate lighting conditions to prevent a decrease in the accuracy of biometric information extraction and the accuracy of biometric authentication using this biometric information. However, the configuration of Patent Document 1 performs biometric authentication using a set of images captured at adjusted time intervals and under multiple different lighting conditions. Therefore, depending on the adjusted time interval, the speed at which the user moves their palm, etc., the biometric authentication device may not be able to acquire a set of images captured under all lighting conditions. Furthermore, in the past, when synchronizing control of lighting and a camera during high-speed shooting, it was difficult to control the on / off of lighting to match the camera's shooting timing.

[0005] The present disclosure has been devised in view of the above-described conventional circumstances, and aims to provide a biometric image acquisition device and a biometric image acquisition method for acquiring a captured image more suitable for biometric authentication.

[0006] The present disclosure provides a biometric image acquisition device comprising: an imaging unit that images at least a portion of the hand of a person to be authenticated; a control unit that controls the imaging unit based on exposure control parameters that control the imaging unit; a detection unit that detects the fingertips of the hand from an image captured by the imaging unit; an evaluation unit that evaluates fingertip luminance values ​​of the detected fingertips; an exposure control unit that determines next exposure control parameters for the imaging unit based on the fingertip luminance values ​​and causes the imaging unit to re-image the fingertips using the next exposure parameters; and an acquisition unit that selects and acquires at least one captured image from a plurality of the captured images.

[0007] The present disclosure also provides a biometric image acquisition method performed by a device that captures an image of at least a portion of the hand of a person to be authenticated, the method comprising: controlling the camera that captures an image of the hand of the person to be authenticated based on exposure control parameters that control the camera; detecting the fingertips of the hand from the captured image; evaluating fingertip luminance values ​​of the detected fingertips; determining next exposure control parameters for the camera based on the fingertip luminance values; causing the camera to re-image the fingertips using the next exposure parameters; and selecting and acquiring at least one captured image from the plurality of captured images.

[0008] According to the present disclosure, a captured image suitable for biometric authentication can be acquired.

[0009] FIG. 1 is a block diagram showing an example of the internal configuration of a biometric authentication system according to a first embodiment; FIG. 2 is a flowchart illustrating an example of the overall operation procedure of a terminal device according to the first embodiment; FIG. 3 is a flowchart illustrating an example of the exposure control determination procedure of a terminal device according to the first embodiment; FIG. 4 is a diagram illustrating an example of the calculation of a fingertip luminance value;

[0010] (Background to the present disclosure) Here, as a comparative example of the embodiment described later, an example of capturing an image when conventional exposure control is performed will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating an example of exposure control in a conventional terminal device P1. The terminal device P1 shown in Fig. 9 has the same configuration as the terminal device P1 described later.

[0011] The terminal device P1 illuminates a user's hand UH, which is a biometric authentication target, with at least one light 13 (see FIG. 1), and captures an image of the user's hand UH with a camera 14 (see FIG. 1). The terminal device P1 performs biometric authentication (user authentication) using the captured image.

[0012] Here, the terminal device P1 captures an image of the user's moving hand UH by controlling the lighting 13 and the camera 14 under predetermined exposure control conditions. Each of the captured images IMG01, IMG02, and IMG03 is an image of the hand UH captured by the terminal device P1.

[0013] Captured image IMG01 is an image captured when the distance between the user's hand UH and the camera 14 of the terminal device P1 (hereinafter referred to as the "imaging distance") is short. The user's hand UH captured in captured image IMG01 is overexposed due to the short imaging distance, resulting in degraded image quality. Brightness graph LG01 indicates the brightness value (hereinafter referred to as the "fingertip brightness value") of the fingertip area of ​​the user's hand UH captured in captured image IMG01, which is used for biometric authentication and from which the user's biometric information is extracted. The fingertip brightness value indicated by brightness graph LG01 is greater than the fingertip brightness value Lm11, which is suitable for acquiring (extracting) biometric information used for biometric authentication. Therefore, captured image IMG01 is an image that is not suitable for acquiring biometric information.

[0014] Captured image IMG02 is an image captured at an appropriate imaging distance. Brightness graph LG02 shows the fingertip brightness value of captured image IMG02. The fingertip brightness value shown by brightness graph LG02 is approximately equal to fingertip brightness value Lm11, which is suitable for acquiring (extracting) biometric information used for biometric authentication. Therefore, captured image IMG02 is a captured image suitable for acquiring biometric information.

[0015] Captured image IMG03 is an image captured at a long imaging distance. Brightness graph LG03 shows the fingertip brightness value of captured image IMG03. The user's hand UH shown in captured image IMG03 is dark due to the long imaging distance, resulting in degraded image quality. The fingertip brightness value shown by brightness graph LG03 is smaller than the fingertip brightness value Lm11 that is suitable for acquiring (extracting) biometric information used for biometric authentication. Therefore, captured image IMG03 is not suitable for acquiring biometric information.

[0016] As described above, the terminal device P1 that executes conventional exposure control may have difficulty acquiring biometric information suitable for biometric authentication or achieving more accurate biometric authentication depending on the height at which the user holds their hand UH (i.e., the imaging distance), the angle or posture at which the user holds their hand UH, the shape of each user's hand UH, etc. Furthermore, the user must retake the image multiple times in order to capture an image from which biometric information suitable for biometric authentication can be acquired, which is very time-consuming.

[0017] Therefore, in the following embodiments, examples of a biometric image acquisition device, a biometric image acquisition method, and a biometric authentication system that acquire captured images more suitable for biometric authentication will be described.

[0018] Hereinafter, with reference to the drawings as appropriate, embodiments that specifically disclose the configuration and operation of a biometric image acquisition device and a biometric image acquisition method for acquiring captured images suitable for biometric authentication according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following explanation are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter recited in the claims.

[0019] A use case of the biometric authentication system 100 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an example of a use case of the biometric authentication system 100 according to the first embodiment. Note that the configuration of the biometric authentication system 100 is not limited to the example shown in Fig. 1.

[0020] The biometric authentication system 100 includes a terminal device P1 and an authentication device S1. The biometric authentication system 100 uses the terminal device P1 to capture an image of at least a portion of a user's hand (specifically, an area from which biometric information used for biometric authentication can be extracted) and acquires the user's biometric information shown in the captured image. The biometric authentication system 100 also uses the authentication device S1 to perform biometric authentication by comparing the user's biometric information with each of multiple pieces of biometric information registered in advance, and outputs and displays the biometric authentication result on a monitor 15 of the terminal device P1.

[0021] In the following description, biometric authentication will be mainly described using fingerprints as biometric information, but the biometric information is not limited to this. The biometric information may be a user's fingerprint, palm print, finger or palm veins, etc.

[0022] The biometric information referred to here may be the fingertip image itself, or may be feature quantities extracted from the fingertip image and indicating the individuality of the user. The feature quantities are extracted using known techniques (e.g., minutiae method or frequency feature analysis method, etc.). The minutiae method is a method for extracting fingerprint feature quantities by detecting the end points or bifurcations (branching points) of fingerprint ridges that indicate breaks in the fingerprint lines. The frequency feature analysis method is a method for extracting fingerprint feature quantities from a frequency waveform converted into a frequency based on the edges, bending points, etc. of the fingerprint ridges. Vein feature quantities are extracted by extracting the vein pattern of a user's finger from an image of the user's finger captured using near-infrared light, for example.

[0023] In the following description, the image of the user's hand may be captured while the user's hand is moving, or while the user's hand is stationary.

[0024] The terminal device P1 captures an image of the user's hand and acquires a fingertip image suitable for biometric authentication. The terminal device P1 is connected to the authentication device S1 via a network NW so as to be capable of wired or wireless communication, and transmits and receives data. Note that the wireless communication referred to here refers to communication provided in accordance with wireless communication standards such as a wireless local area network (LAN), a wireless wide area network (WAN), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system), or Wi-Fi (registered trademark).

[0025] The terminal device P1 includes a communication unit 10, a processor 11, a memory 12, a light 13, a camera 14, and a monitor 15. The light 13 and the camera 14 may be configured separately and connected to the terminal device P1 so as to be able to communicate with each other. The terminal device P1 may also include a sensor (not shown) capable of detecting the entry / exit of a user's hand into / from the angle of view of the camera 14. The monitor 15 is not essential and may be omitted.

[0026] The communication unit 10 transmits and receives data to and from the authentication device S1 via the network NW. The communication unit 10 outputs data transmitted from the authentication device S1 to the processor 11. The communication unit 10 also transmits data output from the processor 11 to the authentication device S1.

[0027] The processor 11 is configured using, for example, a central processing unit (hereinafter referred to as "CPU"), a system-on-chip (SOC), or a field programmable gate array (hereinafter referred to as "FPGA"), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 references programs and data stored in the memory 12 and executes the programs to realize the functions of each part of the terminal device P1. The processor 11 controls each of the lighting 13 and the camera 14 based on exposure control conditions.

[0028] The memory 12 includes, for example, a random access memory (hereinafter referred to as "RAM") that serves as a work memory used when the processor 11 executes each process, and a flash memory that stores programs and data that define the operation of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The flash memory stores programs that define the operation of the processor 11.

[0029] The memory 12 stores setting values ​​(hereinafter referred to as "exposure control conditions") of lighting parameters for controlling the lighting 13 executed by the processor 11 and imaging parameters for controlling the camera 14. The lighting parameters include, for example, the ON / OFF time, lighting pattern, and illuminance of the lighting 13. The imaging parameters include, for example, the aperture value, exposure time, International Organization for Standardization (ISO) sensitivity value, shutter opening / closing pattern, shutter cycle, and the like. It goes without saying that the lighting parameters and imaging parameters described above are merely examples and are not limited to these.

[0030] The memory 12 records the fingertip image, the number of frames in which the fingertip image is captured, an ID for identifying the finger appearing in the fingertip image, the fingertip luminance value of the fingertip image, and the exposure amount of the camera 14 when the fingertip image is captured in association with each other.

[0031] The lighting 13 is controlled by the processor 11 under predetermined exposure control conditions (lighting parameters) to illuminate the user's hand, which is the subject of imaging by the camera 14. The lighting 13 is configured with one or more light-emitting elements, such as a light-emitting diode (LED), a laser diode (LD), or an infrared (IR) light source. The lighting 13 may have multiple lighting areas (light-emitting areas) that can be turned on and off. For example, the lighting 13 and each lighting area may be configured with a surface light source, a point light source, or a linear light source, or may be configured with a combination of multiple different light sources selected from surface light sources, point light sources, and linear light sources. Furthermore, the lighting 13 may be configured with multiple lighting elements arranged in a polygonal, substantially rectangular, substantially annular, substantially U-shaped, or diamond shape, or may be arranged in a substantially parallel configuration.

[0032] The camera 14 is controlled by the processor 11 under predetermined exposure control conditions (imaging parameters) and captures images of the user's moving or stationary hand in a non-contact manner at a frame rate of, for example, 30 frames per second (fps). The camera 14 includes a lens (not shown) and an imaging sensor (not shown). The lens (not shown) forms an image of incident light on the imaging sensor (not shown). The imaging sensor (not shown) is a so-called image sensor, such as a charged-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) solid-state imaging element. The imaging sensor (not shown) controls the start and end of an electronic shutter (not shown), i.e., the start and end of exposure, based on a control signal output from the processor 11. Furthermore, the imaging sensor (not shown) converts an optical image formed on the imaging surface by a lens (not shown) into an electrical signal at a frame rate preset by an administrator, and outputs the electrical signal to the processor 11 .

[0033] The monitor 15 is configured using, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL). The monitor 15 outputs various screens (not shown) output from the processor 11. For example, the monitor 15 displays the results of biometric information registration performed by the authentication device S1, or the results of biometric authentication performed by the authentication device S1.

[0034] The operation unit 16 can accept user operations and outputs the contents of input operations based on the user operations to the processor 11. The operation unit 16 may be realized as a touch panel of the monitor 15 described above. The operation unit 16 may also include a microphone (not shown) and accept voice input operations based on the user's voice. The operation unit 16 is not an essential component and may be omitted.

[0035] The authentication device S1 performs biometric authentication based on a fingertip image or biometric information of a user acquired by the terminal device P1. The authentication device S1 is connected to the terminal device P1 via a network NW so as to be able to communicate with the terminal device P1 via wired or wireless communication, and performs data transmission and reception.

[0036] The authentication device S1 includes a communication unit 20, a processor 21, a memory 22, and a biometric information database DB. The biometric information database DB may be realized as an external storage device or an external storage medium connected to the authentication device S1 so as to be able to communicate data with the authentication device S1.

[0037] The communication unit 20 transmits and receives data to and from the terminal device P1 via the network NW. The communication unit 20 outputs data transmitted from the terminal device P1 to the processor 21. The communication unit 20 also transmits data output from the processor 21 to the terminal device P1.

[0038] The processor 21 is configured using, for example, a CPU, an SOC, or an FPGA, and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 21 references programs and data stored in the memory 12 and executes the programs to realize the functions of the authentication device S1. The processor 21 registers the user's fingertip image transmitted from the terminal device P1 or the user's biometric information extracted from the fingertip image in the biometric information database DB, and performs biometric authentication by matching the user's fingertip image or the user's biometric information transmitted from the terminal device P1 with the biometric information registered in the biometric information database DB.

[0039] The memory 22 includes, for example, a RAM as a work memory used when the processor 21 executes each process, and a flash memory that stores programs and data that define the operation of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The flash memory stores programs that define the operation of the processor 21.

[0040] The biometric information database DB is a storage medium device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores biometric information of each of multiple users registered in advance by an administrator. The biometric information stored in the biometric information database DB is stored (registered) in association with user information (e.g., name, date of birth, identification number that can identify the user, employee number, or facial photograph) and finger information (e.g., middle finger or index finger) corresponding to the biometric information.

[0041] Although FIG. 1 shows an example in which the terminal device P1 and the authentication device S1 are configured as separate devices, these devices may be configured as an integrated device.

[0042] Next, an example of the operation procedure of the terminal device P1 will be described with reference to Fig. 2. Fig. 2 is a flowchart illustrating an example of the overall operation procedure of the terminal device P1 in the first embodiment.

[0043] The terminal device P1 controls the lighting 13 and the camera 14 based on the currently set exposure control conditions, and captures an image of the user's hand (fingers) (St10).

[0044] The terminal device P1 detects at least one fingertip from a captured image of the user's hand (fingers) (St20). The terminal device P1 recognizes whether the detected finger is a thumb, index finger, middle finger, ring finger, or little finger, and assigns an ID (tag) corresponding to each detected finger. For example, when the terminal device P1 detects an index finger and a middle finger from the captured image, it assigns an ID to the index finger in the captured image indicating that the detected finger is an index finger, and an ID to the middle finger in the captured image indicating that the detected finger is a middle finger. Note that known technology may be used to recognize the fingers.

[0045] The terminal device P1 executes an exposure control determination process using the captured image (St30). The exposure control determination process will be described later with reference to FIGS.

[0046] The terminal device P1 performs image processing on the captured image (St40). The terminal device P1 generates fingertip images by cutting out an area including at least the first joint for each finger from the captured image.

[0047] The terminal device P1 calculates an evaluation value indicating the degree to which the fingerprints captured in the fingertip images are suitable for biometric authentication. The terminal device P1 determines whether the calculated evaluation value for each fingertip image is equal to or greater than a threshold value (St50). The threshold value here is an evaluation value for selecting fingertip images suitable for biometric authentication.

[0048] If the terminal device P1 determines that the evaluation value is equal to or greater than the threshold value (St50, YES), it determines that the fingertip image having the evaluation value equal to or greater than the threshold value is suitable for biometric authentication and acquires it (St60).

[0049] On the other hand, if the terminal device P1 determines that the evaluation value is not equal to or greater than the threshold (St50, NO), the terminal device P1 determines that the fingertip images having evaluation values ​​not equal to or greater than the threshold are not suitable for biometric authentication, and omits the acquisition process. The terminal device P1 repeatedly executes the processes of steps St10 to St50 until it has completed acquisition of fingertip images corresponding to all fingers that have been detected or will be used for biometric authentication.

[0050] Next, the exposure control determination process (step St30) shown in Fig. 2 will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating an example of an exposure control determination procedure of the terminal device P1 in the first embodiment.

[0051] The terminal device P1 determines whether or not at least one fingertip of the user is detected from the captured image captured by the camera 14 (St301).

[0052] When the terminal device P1 determines in the processing of step St301 that a fingertip has been detected (St301, YES), it determines whether the detected fingertip is the detection target, that is, the finger to be used for biometric authentication (St302). Note that, if the fingers to be used for biometric authentication, such as the index finger or the index finger and middle finger, have been designated in advance, the terminal device P1 may determine whether at least one finger to be used for biometric authentication has been detected. Furthermore, if multiple fingers to be used for biometric authentication have been designated in advance, the terminal device P1 may determine whether a fingertip whose fingertip image has not yet been acquired has been detected.

[0053] On the other hand, if the terminal device P1 determines in the processing of step St301 that a fingertip has not been detected (St301, NO), it determines the next exposure control mode of the camera 14 to be standard exposure control (auto exposure control mode). Note that the standard exposure control here is a type of auto exposure control mode, and is a mode that performs exposure control that continues to automatically adjust the imaging parameters of the camera 14 based on changes in the brightness of the imaging environment of the camera 14. The terminal device P1 sets the exposure control mode to standard exposure control (auto exposure control mode) at a predetermined timing and causes the camera 14 to capture an image of the user's hand (fingers) (St303). Note that the predetermined timing here refers to the timing at which the setting of the exposure control conditions is changed, which is the timing of times t21 and t51 shown in FIG. 8, and this also applies to the following description.

[0054] If the terminal device P1 determines in the processing of step St302 that the detected finger is the detection target (St302, YES), it determines whether the currently set exposure control mode is the auto exposure control mode (St304).

[0055] On the other hand, if the terminal device P1 determines in the processing of step St302 that the detected finger is not a detection target (NO in St302), it determines the standard exposure control (auto exposure control mode) as the exposure control mode of the next camera 14. The terminal device P1 sets the exposure control mode to the standard exposure control (auto exposure control mode) at a predetermined timing and causes the camera 14 to capture an image of the user's hand (fingers) (St303).

[0056] If the terminal device P1 determines in the processing of step St304 that the currently set exposure control mode is the auto exposure control mode (St304, YES), it determines whether the current auto exposure control mode is the fingertip exposure control mode (St305). Note that the fingertip exposure control mode here is a type of auto exposure control mode, and is a mode that performs exposure control that continues to automatically adjust the imaging parameters of the camera 14 based on the fingertip luminance value of the fingertip image.

[0057] On the other hand, if the terminal device P1 determines in the processing of step St304 that the currently set exposure control mode is not the auto exposure control mode (NO in St304), it determines the next exposure control mode of the camera 14 to be manual exposure control (non-auto exposure control mode). Note that the manual exposure control here refers to a non-auto exposure control mode in which the user sets the imaging parameters (exposure control conditions) of the camera 14 and performs exposure control of the camera 14 using the imaging parameters set by the user. The terminal device P1 accepts the setting operation related to the imaging parameters of the camera 14 via the operation unit 16, sets the exposure control mode to manual exposure control at a predetermined timing, and performs exposure control of the camera 14 based on the set imaging parameters (St306). The terminal device P1 then causes the camera 14 to capture an image of the user's hand (fingers) (St306).

[0058] If the terminal device P1 determines in step St305 that the current auto exposure control mode is the fingertip exposure control mode (St305, YES), it executes fingertip photometry processing to calculate the fingertip luminance value in the captured image (St307). Based on the result of the fingertip photometry processing, the terminal device P1 executes exposure control determination processing (St308), and executes exposure control based on the determined next exposure control condition (St309).

[0059] On the other hand, if the terminal device P1 determines in the processing of step St305 that the current auto exposure control mode is not the fingertip exposure control mode (NO in St305), it determines the next exposure control mode of the camera 14 to be conventional exposure control (auto exposure control mode). Note that the conventional exposure control here is a type of auto exposure control mode that accepts a user's operation to specify the position of the object to be imaged, i.e., the focus position, and performs exposure control by focusing the camera 14 on the specified position to capture the image. The terminal device P1 accepts an operation to specify the focus position of the camera 14 via the operation unit 16, sets the exposure control mode to conventional exposure control at a predetermined timing, and performs exposure control by focusing the camera 14 on the set focus position (St310). The terminal device P1 then causes the camera 14 to capture an image of the user's hand (fingers) (St310).

[0060] As described above, the terminal device P1 in the first embodiment can arbitrarily change and execute the exposure control mode and exposure control of the camera 14.

[0061] Next, the fingertip photometry process (step St307) shown in Fig. 3 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart illustrating an example of a photometry procedure of the terminal device P1 in embodiment 1. Fig. 5 is a diagram illustrating an example of calculation of a fingertip luminance value. Note that the entire fingertip area AR0 and partial area AR1 shown in Fig. 5 are merely examples and are not limiting.

[0062] The terminal device P1 repeatedly executes the processing of steps St307B to St307D until it calculates the brightness value of the entire fingertip area AR0 of each of the N (N: integer greater than or equal to 1) fingertips to be detected (hereinafter referred to as the "overall brightness value") and the brightness value of a partial area AR1 that is a part of the entire fingertip area AR0 and includes the center position Pt0 of the fingerprint (fingertip) (hereinafter referred to as the "partial brightness value") (St307A).

[0063] Specifically, the terminal device P1 extracts from the captured image an area that includes the first joint of the fingertip, which is the target of detection, and generates a fingertip image IMG1 (Step 307B). The generated fingertip image IMG1 has a width Fw that is approximately equal to the width of the fingertip, and a height Fh that includes the area from the tip of the finger to the first joint of the finger. The terminal device P1 also acquires an ID assigned to the finger depicted in the generated fingertip image IMG1 and identifies which finger, from the thumb to the little finger, the fingertip depicted in the fingertip image IMG1 is.

[0064] The terminal device P1 performs grayscale processing on each of the generated fingertip images IMG1 to convert the fingertip images IMG1 from color images to monochrome images (Step 307C). ​​The fingertip image IMG2 shown in FIG. 5 is an image obtained by converting the fingertip image IMG1 into a monochrome image. This allows the terminal device P1 to more accurately calculate the luminance value of the fingertip itself regardless of skin color, even if the skin color varies from user to user.

[0065] The terminal device P1 detects the entire fingertip region AR0 based on the outline of the fingertip shown in the fingertip image IMG2. The terminal device P1 calculates the brightness value of each pixel included in the entire fingertip region AR0 (St307D). The terminal device P1 also calculates the center position of the entire fingertip region AR0, i.e., the center position Pt0 of the fingerprint (fingertip), based on the outline of the fingertip shown in the fingertip image IMG2. The terminal device P1 calculates a partial region AR1 for calculating a partial brightness value, which is an area centered at the center position Pt0 and has a horizontal dimension Ew and a vertical dimension Eh, based on the ratios of the width Fw and height Fh of the entire fingertip region AR0 set in advance. The terminal device P1 calculates the brightness value of each pixel included in the partial region AR1 (St307D).

[0066] 5 shows an example of an ellipse-shaped partial area AR1, but it may also be a perfect circle. The partial area AR1 may be any area smaller than the entire fingertip area AR0, and may have a width Fw>horizontal Ew and a height Fh>vertical Eh.

[0067] After the terminal device P1 has completed the calculation of the overall luminance value and the partial luminance value of the N fingertips, it performs a weighting process on the overall luminance value and the partial luminance value using the weighting coefficient set for each finger based on the ID. The terminal device P1 calculates the average value of the overall luminance value and the partial luminance value of the N fingertips after the weighting process and acquires the calculated average value as the fingertip luminance value of the captured image (St307E). The terminal device P1 associates the frame number of the captured image with the calculated fingertip luminance value and records it in the memory 12.

[0068] The weighting coefficient may be different for the overall luminance value and the partial luminance value of the same finger. When the terminal device P1 calculates the fingertip luminance value of the captured image using only one of the overall luminance value and the partial luminance value, the terminal device P1 may set the weighting coefficient of the overall luminance value or the partial luminance value that is not used in calculating the fingertip luminance value to 0 (zero).

[0069] For example, if a weighting factor of 0.2 is set for the index finger and a weighting factor of 0.4 is set for the middle finger, the terminal device P1 executes a weighting process in which the overall luminance value and partial luminance value of the index finger are multiplied by a weighting factor of 0.2 and the overall luminance value and partial luminance value of the middle finger are multiplied by a weighting factor of 0.4, and calculates the fingertip luminance value of the captured image by calculating the average luminance value obtained by adding together the overall luminance value and partial luminance value of each finger after weighting.

[0070] Note that the method for calculating the fingertip luminance value shown in step St307E is not limited to this. The terminal device P1 may calculate the fingertip luminance value based on the sum or median of the overall luminance values ​​and partial luminance values ​​of the N fingertips, or the average, sum, or median of the overall luminance values ​​and partial luminance values ​​for each finger.

[0071] As a result of the above, the terminal device P1 in embodiment 1 can calculate a fingertip luminance value, which is an indicator of whether the captured image (fingertip image) captured under the currently set exposure control conditions is captured at a brightness suitable for biometric authentication.

[0072] Next, the exposure control condition determination process (step St308) shown in Fig. 3 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart illustrating an example of a procedure for determining the exposure control conditions of the terminal device P1 in embodiment 1. Fig. 7 is a diagram illustrating an example of determining the exposure control conditions (imaging parameters).

[0073] The terminal device P1 calculates the error between the fingertip luminance value and the target luminance value. The target luminance value is a fingertip luminance value suitable for biometric authentication and may be set arbitrarily based on the environment in which the camera 14 is installed, the time of day, etc. For example, if the captured image is an 8-bit image, the target luminance value is set to 160, which is a value between 0 (zero) and 255.

[0074] Based on the calculated error, the terminal device P1 acquires a control signal for performing exposure control using a Proportional Integral Derivative Control (hereinafter referred to as "PID control") algorithm to bring the current fingertip luminance value closer to the target luminance value, that is, to bring the error closer to 0 (zero) (St308A).

[0075] Specifically, the terminal device P1 calculates a correction amount (control amount) of the exposure control condition proportional to the error Ep through proportional control. The terminal device P1 calculates a correction amount (control amount) of the exposure control condition proportional to an integral value Ei of the error between the target luminance value and the fingertip luminance value of each frame captured after the current exposure control condition was set. The terminal device P1 calculates a correction amount (control amount) of the exposure control condition proportional to a rate of change Ed of the error between the target luminance value and the fingertip luminance value of each frame captured after the current exposure control condition was set.

[0076] The terminal device P1 determines a control signal based on the preset PID coefficients kp, ki, and kd, and the error Ep, integral value Ei, and rate of change Ed calculated by the PID control algorithm: Control signal = kp * (error Ep) + ki * (integral value Ei) + kd * (rate of change Ed).

[0077] The terminal device P1 acquires the exposure time TV and ISO sensitivity SV, which are the currently set exposure control conditions of the camera 14. Based on the acquired current exposure control conditions, the terminal device P1 calculates the current exposure amount (= exposure time TV - ISO sensitivity SV) (Step 308B).

[0078] The terminal device P1 refers to the memory 12 to obtain the exposure amount of the camera 14 from the current frame number to three frames before, and determines the exposure amount of the camera 14 under the current exposure control conditions (hereinafter referred to as the "reference exposure amount") (St308C).

[0079] The terminal device P1 determines a target exposure amount EVx for bringing the fingertip luminance value closer to the target luminance value based on the reference exposure amount and the control signal (Step 308D). For example, the terminal device P1 performs weighting processing by multiplying the control signal by a preset weighting coefficient kg, and determines the target exposure amount EVx by adding the control signal after the weighting processing to the reference exposure amount (target exposure amount EVx = reference exposure amount + control signal * kg).

[0080] The terminal device P1 refers to the program diagram (see FIG. 7 ) and determines the exposure time TV and ISO sensitivity SV for adjusting to the determined target exposure amount EVx. The terminal device P1 determines exposure time setting values ​​and ISO sensitivity setting values ​​for setting the determined exposure time TV and ISO sensitivity SV. The terminal device P1 determines the next exposure control condition based on the determined exposure time setting values ​​and ISO sensitivity setting values ​​(St308E).

[0081] In the program diagram shown in Figure 7, the vertical axis indicates the set value of ISO sensitivity SV, and the horizontal axis indicates the set value of exposure time TV. Furthermore, of the four corners of the program diagram, the upper right corner indicates a brighter environment in which the camera 14 is installed (i.e., a larger amount of exposure), and the lower left corner indicates a darker environment in which the camera 14 is installed (i.e., a smaller amount of exposure). The ISO sensitivity SV is set to decrease as the environment in which the camera 14 is installed becomes brighter, and to increase as the environment in which the camera 14 is installed becomes darker. The exposure time TV is set to decrease (shorten) as the environment in which the camera 14 is installed becomes brighter, and to increase (lengthen) as the environment in which the camera 14 is installed becomes darker. The diagonal lines connecting each set value of ISO sensitivity and each set value of exposure time shown in the program diagram indicate the combination of set values ​​of ISO sensitivity and exposure time for achieving the target exposure amount EVx.

[0082] The terminal device P1 selects one of the combinations of the ISO sensitivity setting value and the exposure time setting value that intersect with the diagonal line corresponding to the determined target exposure amount EVx. For example, when the target exposure amount EVx=5, the terminal device P1 determines the ISO sensitivity setting value "6" and the exposure time setting value "8" that intersect with the diagonal line corresponding to the target exposure amount EVx=5 as the next exposure control conditions.

[0083] The setting values ​​of ISO sensitivity and exposure time may be determined by setting a priority parameter to determine whether the setting value of exposure time should be determined based on (prioritized as) the setting value of ISO sensitivity for each target exposure amount EVx, or whether the setting value of ISO sensitivity should be determined based on (prioritized as) the setting value of exposure time, and setting a setting value for the prioritized parameter (ISO sensitivity or exposure time).

[0084] For example, the terminal device P1 may set the exposure time (priority parameter) setting value to "8" for a target exposure amount EVx = -2 to 7, "9" for a target exposure amount EVx = 10, "11" for a target exposure amount EVx = 11, "13" for a target exposure amount EVx = 12, "14" for a target exposure amount EVx = 13, "15" for a target exposure amount EVx = 14, "16" for a target exposure amount EVx = 15, etc. When the target exposure amount EVx = 5, the terminal device P1 determines the exposure time setting value to "8" and determines the ISO sensitivity setting value corresponding to these target exposure amount EVx and exposure time setting values ​​to "6."

[0085] The same applies when ISO sensitivity is set as the priority parameter. Furthermore, the priority parameter may be arbitrarily set for each target exposure level EVx. For example, when the target exposure level EVx is between -2 and 7, the terminal device P1 may set the exposure time as the priority parameter and then set the exposure time setting value to "8." When the target exposure level EVx is between 7 and 16, the terminal device P1 may set the ISO sensitivity as the priority parameter and then set the ISO sensitivity setting value to "4." This allows the terminal device P1 to determine priority parameters that are more suited to the specifications of the camera 14 and to determine exposure control conditions that enable acquisition of a fingertip image more suited to biometric authentication.

[0086] As described above, the terminal device P1 in the first embodiment can determine the exposure control conditions for acquiring a fingertip image suitable for biometric authentication.

[0087] In addition, when there are multiple detection targets, the terminal device P1 may determine exposure control conditions for simultaneously acquiring fingertip images of multiple fingers, or may determine exposure control conditions for acquiring fingertip images of each finger individually.

[0088] In the former case, the terminal device P1 selects any two or more fingers from the multiple fingers set as detection targets, and repeatedly determines exposure control conditions suitable for capturing fingertip images of the selected multiple fingers and captures the images until it acquires fingertip images suitable for biometric authentication corresponding to the selected multiple fingers. Specifically, the terminal device P1 calculates fingertip luminance values ​​from the fingertip images of the selected multiple fingers, and sets the next exposure control conditions more suitable for capturing the images of the selected multiple fingers based on the error between the fingertip luminance value and the target luminance value and the exposure amount of the camera 14. This allows the terminal device P1 to acquire fingertip images suitable for biometric authentication in a shorter time than if it were to acquire fingertip images of each finger one by one.

[0089] In the latter case, the terminal device P1 selects one of the multiple fingers set as detection targets, and repeatedly determines exposure control conditions suitable for capturing the fingertip image of the selected finger and captures the image until a fingertip image suitable for biometric authentication corresponding to the selected finger is acquired. Specifically, the terminal device P1 calculates a fingertip luminance value from the fingertip image of the selected finger, and sets the next exposure control condition more suitable for capturing the selected finger based on the error between the fingertip luminance value and the target luminance value and the exposure amount of the camera 14. This allows the terminal device P1 to capture a fingertip image more suitable for biometric authentication than if multiple fingers were captured simultaneously.

[0090] Next, a control example of the terminal device P1 will be described with reference to FIG. 8. FIG. 8 is a timing chart showing a control example of the terminal device P1 in embodiment 1. Note that, although an example will be described in which the terminal device P1 shown in FIG. 8 executes various controls in accordance with the frame rate (=30 fps) of the camera 14, this is merely an example and is not limiting. Furthermore, the terminal device P1 executes a setting change of the exposure control condition once every three frames, but the period and frequency of the setting change of the exposure control condition are not limited to this.

[0091] The numbers shown in each block in FIG. 8 indicate the number of frames captured by the camera 14, that is, the number of the captured image.

[0092] Camera 14 receives incident light with an imaging sensor (not shown) every 33.3 ms at times t0, t1, t2, t3, t4, t5, t6, ..., t(M-1), converts the received light into an electrical signal, and outputs it to processor 11. Note that in the example shown in Fig. 8, M is an integer of 8 or greater, but it goes without saying that it may be any integer of 1 or greater.

[0093] The camera 14 receives incident light with an image sensor (not shown) at times t0, t1, and t2, converts the received light into an electrical signal, and outputs it to the processor 11.

[0094] After the fingertip exposure control condition “fingertip exposure control mode pattern: 1” is set, the camera 14 receives incident light with an imaging sensor (not shown) at times t3, t4, and t5, converts the received light into an electrical signal, and outputs the signal to the processor 11.

[0095] After the fingertip exposure control condition is set to "fingertip exposure control mode pattern: 2," the camera 14 receives incident light with an imaging sensor (not shown) at each of times t6, ..., converts the received light into an electrical signal, and outputs the signal to the processor 11.

[0096] At time t1, the processor 11 generates a captured image with frame number "1" based on the electrical signal output from the camera 14. At time t2, the processor 11 executes fingertip photometry processing (step St307) and exposure control condition determination processing (step St308) based on the captured image with frame number "1".

[0097] The terminal device P1 determines the next exposure control condition, "fingertip exposure control mode pattern: 1." The processor 11 changes the setting of the exposure control condition currently set at time t21, "standard exposure control mode pattern: 1," to the next fingertip exposure control condition, "fingertip exposure control mode pattern: 1," determined at time t2.

[0098] In addition, the first setting change of the exposure control conditions executed at time t21 may be determined based on the fingertip luminance values ​​of N fingertip images generated based on the captured image with frame number "1" and the exposure amount of camera 14 when capturing the captured image with frame number "1".

[0099] The processor 11 also generates a captured image with frame number "2" at time t2, and executes fingertip photometry processing (step St307) based on the fingertip image generated at time t3. Similarly, the processor 11 generates captured images with frame numbers "3" and "4" and executes fingertip photometry processing (step St307) at times t4 and t5, respectively.

[0100] At time t5, the processor 11 determines the next exposure control condition "fingertip exposure control mode pattern: 2" based on the fingertip luminance values ​​of N fingertip images generated based on the captured images three frames before the current frame "4," i.e., frame numbers "2" to "4," and the exposure amounts of the camera 14 when the captured images of frame numbers "2" to "4" were captured. At time 51, the processor 11 changes the setting of the currently set exposure control condition "fingertip exposure control mode pattern: 1" to the next fingertip exposure control condition "fingertip exposure control mode pattern: 2" determined at time t5.

[0101] The processor 11 repeatedly executes the exposure control condition setting change process described above every three frames (i.e., every 1 second) until acquisition of a fingertip image suitable for biometric authentication is completed. In the example shown in Fig. 8, the processor 11 acquires a fingertip image based on the frame number "M-2" captured under the exposure control condition "fingertip exposure control mode pattern: K (K: any integer equal to or greater than 1)" at time t(M-1), and then terminates various controls.

[0102] 8 shows an example in which, after the exposure control mode is changed from the "standard exposure control mode" to the "fingertip exposure control mode," only the exposure control conditions in the "fingertip exposure control mode" (i.e., the ISO sensitivity setting value and the exposure time setting value) are changed, but this is not limiting. The exposure control mode may be changeable to any exposure control mode at any timing by the user.

[0103] (Additional Notes) The above description of each embodiment discloses the following techniques.

[0104] (Technology 1) A biometric image acquisition device (terminal device P1) comprising: an imaging unit (camera 14) that captures an image of at least a portion of a hand of a person to be authenticated (user); a control unit (processor 11) that controls the imaging unit (camera 14) based on exposure control parameters (exposure control conditions) that control the imaging unit (camera 14); a detection unit (processor 11) that detects the fingertips of the hand from an image captured by the imaging unit (camera 14); an evaluation unit (processor 11) that evaluates fingertip luminance values ​​of the detected fingertips; an exposure control unit (processor 11) that determines next exposure control parameters (exposure control conditions) for the imaging unit (camera 14) based on the fingertip luminance values, and causes the imaging unit to re-image the fingertips using the next exposure parameters; and an acquisition unit (processor 11) that selects and acquires at least one captured image from the plurality of captured images. With this configuration, the biometric image acquisition device (terminal device P1) can acquire an image (fingertip image) that is more suitable for biometric authentication by changing the exposure parameters of the camera 14 based on the fingertip luminance value of the fingertip shown in the captured image.

[0105] (Technology 2) The biometric image acquisition device (terminal device P1) according to (Technology 1) is configured such that the captured image is a color image, and the evaluation unit (processor 11) performs grayscale processing on the captured image to convert it into a monochrome image and evaluates the fingertip luminance value based on the monochrome image. This configuration allows the biometric image acquisition device (terminal device P1) to more effectively suppress a decrease in the accuracy of evaluation of fingertip luminance values ​​due to differences in user skin color. This allows the biometric image acquisition device (terminal device P1) to acquire a captured image (fingertip image) more suitable for biometric authentication by suppressing a decrease in the accuracy of selecting a captured image more suitable for biometric authentication based on the evaluated fingertip luminance value.

[0106] (Technology 3) The biometric image acquisition device (terminal device P1) according to (Technology 1) or (Technology 2), wherein the evaluation unit (processor 11) generates a fingertip image by cutting out a detection area (entire fingertip area AR0) where the fingertip is detected from the captured image, and performs the grayscale processing on the fingertip image. With this configuration, the biometric image acquisition device (terminal device P1) can evaluate the luminance value of only the fingertip, which is an area from which biometric information used for biometric authentication is extracted.

[0107] (Technology 4) The biometric image acquisition device (terminal device P1) described in (Technology 3) is configured such that the evaluation unit (processor 11) evaluates the overall luminance values ​​of pixels included in the detection area (entire fingertip area AR0) and the partial luminance values ​​of pixels included in a partial area (partial area AR1) that includes a center position Pt0 of the detection area (entire fingertip area AR0) and is part of the detection area (entire fingertip area AR0), and evaluates the fingertip luminance value based on the overall luminance values ​​and the partial luminance values. This configuration enables the biometric image acquisition device (terminal device P1) to evaluate the current exposure control conditions using the luminance values ​​of the entire fingertip, including the outer shape (edge) of the fingertip, and to evaluate the current exposure control conditions using the luminance values ​​of the central part of the fingerprint, which is more suitable for extracting biometric information. This allows the biometric image acquisition device (terminal device P1) to determine exposure control conditions for capturing an image (fingertip image) more suitable for biometric authentication.

[0108] (Technology 5) The biometric image acquisition device (terminal device P1) according to any one of (Technology 1) to (Technology 4), wherein the exposure control unit (processor 11) determines the next exposure control parameter (exposure control condition) based on an error between a fingertip luminance value of the fingertip and a target luminance value of the captured image acquired by the acquisition unit (processor 11). With this configuration, the biometric image acquisition device (terminal device P1) can determine the exposure control condition for acquiring a captured image (fingertip image) captured with a fingertip luminance value more suitable for biometric authentication.

[0109] (Technology 6) The biometric image acquisition device (terminal device P1) according to (Technology 5), wherein the exposure control unit (processor 11) acquires an exposure amount of the imaging unit (camera 14), determines a next exposure amount of the imaging unit (camera 14) based on the error and the exposure amount, and determines the next exposure control parameter (exposure control condition) based on the determined next exposure amount. With this configuration, the biometric image acquisition device (terminal device P1) can determine exposure control conditions for acquiring an image (fingertip image) captured with a fingertip luminance value suitable for biometric authentication, based on a fingertip luminance value of the captured image (fingertip image) and the exposure amount of the camera 14 that captured the image (fingertip image).

[0110] (Technology 7) The biometric image acquisition device (terminal device P1) according to any one of (Technology 1) to (Technology 6), wherein, when there are multiple detected fingertips, the evaluation unit (processor 11) evaluates the fingertip luminance values ​​of each of the fingertips, and the exposure control unit (processor 11) determines the next exposure control parameter (exposure control condition) based on an average value or a median value of the fingertip luminance values ​​of each of the fingertips. With this configuration, the biometric image acquisition device (terminal device P1) can determine the exposure control condition for acquiring captured images (fingertip images) of multiple fingertips each captured at a fingertip luminance value suitable for biometric authentication.

[0111] (Technology 8) The biometric image acquisition device (terminal device P1) according to any one of (Technology 1) to (Technology 7), wherein the exposure control parameters (exposure control conditions) include an ISO sensitivity of the imaging unit (camera 14) and an exposure time of the imaging unit (camera 14). With this configuration, the biometric image acquisition device (terminal device P1) can acquire an image (fingertip image) captured with a fingertip luminance value suitable for biometric authentication by changing the ISO sensitivity and exposure time, which are imaging parameters of the camera 14.

[0112] (Technology 9) A biometric image acquisition method performed by a device (terminal device P1) that captures an image of at least a portion of a hand of a person to be authenticated (user), comprising: controlling the camera 14 based on exposure control parameters (exposure control conditions) that control the camera 14 that captures an image of the hand of the person to be authenticated (user), detecting fingertips of the hand from the captured image, evaluating fingertip luminance values ​​of the detected fingertips, determining next exposure control parameters (exposure control conditions) for the camera based on the fingertip luminance values, causing the camera 14 to re-image the fingertips using the next exposure parameters, and selecting and acquiring at least one captured image from the multiple captured images. With this configuration, the device (terminal device P1) can acquire an image (fingertip image) that is more suitable for biometric authentication by changing the exposure parameters of the camera 14 based on the fingertip luminance values ​​of the fingertips appearing in the captured image.

[0113] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention.

[0114] This application is based on a Japanese patent application (Patent Application No. 2024-047277) filed on March 22, 2024, the contents of which are incorporated herein by reference.

[0115] The present disclosure is useful as a biometric image acquisition device and a biometric image acquisition method for acquiring a captured image more suitable for biometric authentication.

[0116] 10, 20 Communication unit 11, 21 Processor 12, 22 Memory 13 Lighting 14 Camera 15 Monitor 16 Operation unit 100 Biometric authentication system AR0 Whole fingertip area AR1 Partial area DB Biometric information database IMG1, IMG2 Fingertip image NW Network P1 Terminal device Pt0 Center position S1 Authentication device

Claims

1. A biometric image acquisition device comprising: an imaging unit that images at least a portion of the hand of a person to be authenticated; a control unit that controls the imaging unit based on exposure control parameters that control the imaging unit; a detection unit that detects the fingertips of the hand from the image captured by the imaging unit; an evaluation unit that evaluates the fingertip luminance values ​​of the detected fingertips; an exposure control unit that determines next exposure control parameters for the imaging unit based on the fingertip luminance values ​​and causes the imaging unit to re-image the fingertips using the next exposure parameters; and an acquisition unit that selects and acquires at least one of the multiple captured images.

2. The biometric image acquisition device according to claim 1, wherein the captured image is a color image, and the evaluation unit performs grayscale processing on the captured image to convert it into a black and white image, and evaluates the fingertip luminance value based on the black and white image.

3. The biometric image acquisition device according to claim 2, wherein the evaluation unit generates a fingertip image by cutting out a detection area in which the fingertip is detected from the captured image, and performs the grayscale processing on the fingertip image.

4. The biometric image acquisition device of claim 3, wherein the evaluation unit evaluates the overall luminance value of the pixels included in the detection area and the partial luminance value of the pixels included in a partial area that includes the center position of the detection area and is part of the detection area, and evaluates the fingertip luminance value based on the overall luminance value and the partial luminance value.

5. The biometric image acquisition device according to claim 1, wherein the exposure control unit determines the next exposure control parameter based on an error between a fingertip luminance value of the fingertip and a target luminance value of the captured image acquired by the acquisition unit.

6. The biometric image acquisition device according to claim 5, wherein the exposure control unit acquires the exposure amount of the imaging unit, determines the next exposure amount of the imaging unit based on the error and the exposure amount, and determines the next exposure control parameter based on the determined next exposure amount.

7. The biometric image acquisition device of claim 1, wherein, when there are multiple detected fingertips, the evaluation unit evaluates the fingertip luminance value of each of the fingertips, and the exposure control unit determines the next exposure control parameter based on the average or median of the fingertip luminance values ​​of each of the fingertips.

8. The biometric image acquisition device according to claim 1, wherein the exposure control parameters include an ISO sensitivity of the image capture unit and an exposure time of the image capture unit.

9. A biometric image acquisition method performed by a device that captures an image of at least a portion of the hand of a person to be authenticated, comprising: controlling the camera based on exposure control parameters that control the camera that captures the image of the hand of the person to be authenticated; detecting the fingertips of the hand from the captured image; evaluating the fingertip luminance values ​​of the detected fingertips; determining the next exposure control parameters for the camera based on the fingertip luminance values; causing the camera to re-image the fingertips using the next exposure parameters; and selecting and acquiring at least one captured image from the plurality of captured images.

Images