Biometric authentication system and authentication method

Abstract

To provide a biometric authentication system and a biometric authentication method which can maximize authentication accuracy in various capture environments when implementing biometric authentication based on biometric features captured by a universal camera.SOLUTION: A biometric authentication system 1000 includes a storage device in which a feature quantity of a living body is stored, an imaging apparatus (camera 9) which captures an image of the living body, and an authentication processing apparatus 10 comprising an image correction unit for converting an image quality of the image captured by the imaging apparatus and an authentication processing unit for performing biometric authentication based on an image outputted from the image correction unit. The image correction unit generates images before color correction of one or more images obtained by capturing the living body, converts the images before color correction with a plurality of values within a prescribed search range about color information to generate a plurality of color information conversion images, selects, for each biometric feature, a color information conversion image best representing the biometric feature from the plurality of color information conversion images and searches for optimum color information for each biometric feature on the basis of color information for obtaining the selected color information conversion image.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present invention relates to an authentication system and an authentication method for authenticating an individual using biometric information. [Background technology]

[0002] In recent years, biometric authentication technology has begun to be used not only for access management such as entrance / exit control and PC login, but also as a means of identity authentication when making payments for online shopping, etc. There are also expectations that biometric authentication will become a reality in retail stores. With the expansion of cashless payments using smartphones, the need for accurate identity authentication is also increasing, and there is a demand for highly accurate biometric authentication technology that can be easily used on smartphones.

[0003] Currently widely used biometric authentication methods include facial, iris, fingerprint, and finger or palm veins. Finger authentication technology, particularly based on finger veins, fingerprints, skin prints, and knuckle prints, is gaining widespread adoption in the real world due to its high authentication accuracy and ease of operation, thanks to the wide range of biometric features it can utilize. In practical applications, authentication devices with various sensing methods are used depending on the application, including devices that capture high-resolution images of fingers inserted into the device's holes, compact, open-type authentication devices that allow authentication simply by placing a finger on the device, contactless authentication devices that capture multiple fingers simultaneously by irradiating them with reflected infrared and visible light when they are held over the device, and authentication devices that capture color images of fingers under visible light using a general-purpose camera, such as the rear camera of a typical smartphone or the front camera of a laptop.

[0004] In particular, biometric authentication using general-purpose cameras is convenient because it does not require special equipment and registration and authentication can be performed anywhere. Therefore, it is believed that the spread of biometric authentication will accelerate if highly accurate authentication technology can be realized with simple operations. Applications include not only ensuring security for general-purpose devices such as smartphones and laptops, but also establishing compatibility of biometric authentication technology between general-purpose devices and dedicated devices will enable dedicated device registration to be performed on the general-purpose device. For example, when making a payment using biometric authentication using dedicated devices installed in a retail store, users previously had to go to the store to register. However, now it will be possible to register biometrics using a smartphone at any time and location, such as at home, and then make biometric payments at the retail store without any special procedures.

[0005] However, biometric authentication using general-purpose cameras poses a challenge, as the diversity of usage environments can degrade authentication accuracy. Because general-purpose camera authentication uses a visible-light color camera to capture biometric images, it is particularly susceptible to the effects of ambient light. Examples of ambient light include fluorescent lights, sunlight during the day, the setting sun, and LED lights (torches) on smartphones. Each type of light has different wavelengths and intensities, and the direction of illumination varies. The wavelength and direction of such ambient light can cause variations in the color and appearance of biometric features used for authentication. To minimize color variations, automatic white balance adjustments are typically used to maintain a consistent color. However, for biometric features such as vein patterns, which are detected based on color differences rather than just brightness, adjusting the overall color of the image can make the color differences essential for detecting vein patterns less noticeable. This can result in blurred patterns, preventing accurate feature extraction and degrading authentication accuracy. Similarly, if a light source does not uniformly illuminate a biometric subject, uneven brightness can appear in the biometric image, which can be mistakenly extracted as a biometric feature, resulting in reduced accuracy.

[0006] To solve these problems, it is necessary to develop an authentication technology that can extract stable biometric features regardless of changes in ambient light. However, with existing technologies, it has been difficult to extract biometric features stably even when the environment changes, such as adjusting the color to match the biometric features in response to changes in the wavelength of the ambient light, or extracting features without being affected by uneven brightness in images of living bodies captured due to changes in the illumination position of the ambient light.

[0007] Thus, the challenge is to provide an authentication device or authentication system that can achieve high authentication accuracy while being less susceptible to changes in the photographing environment.

[0008] Conventionally, there is a technology disclosed in Patent Document 1 that realizes biometric authentication that is robust against changes in environmental light. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-180660 Summary of the Invention [Problem to be solved by the invention]

[0010] In particular, finger-based biometric authentication using a general-purpose camera involves capturing color images of various biometric features, such as fingerprints, veins, joint patterns, fat patterns, and melanin patterns, and then extracting the features from the color images for authentication. To capture multiple biometric features properly, the image must be captured under conditions appropriate for each feature. Color balance is particularly important for color-based biometric features, such as veins and joint patterns, and the apparent color can change depending on the camera parameter settings and the lighting conditions of the shooting environment. While white balance is often automatically adjusted when capturing biometric images using a general-purpose camera, capturing images with an inoptimal white balance can result in biometric features not being clearly visible, resulting in degraded authentication accuracy.

[0011] Typical white balance adjustments involve digitally processing the color information of raw images—unprocessed image information acquired directly from sensors such as CMOS devices in color cameras—to ensure that white objects appear white under any lighting environment. However, typical automatic white balance adjustments are not necessarily optimal for capturing biometric features. Suboptimal white balance can obscure essential biometric information, making it difficult to extract. Therefore, manually setting an optimal white balance is crucial, but some cameras may not allow for such fine adjustments. Similarly, while it is possible to generate images with any desired white balance by acquiring raw image information, not all cameras can extract raw images. Even if raw images can be extracted, various adjustments, such as demosaicing, noise reduction, lens distortion, and chromatic aberration, are required in addition to white balance. In particular, biometric authentication using general-purpose cameras requires optimal customization for each camera, as CMOS sensors and lenses have different characteristics. This increases the complexity of the process and makes it difficult to optimize for each camera. Therefore, in order to make it work universally with various cameras, it is useful to use images obtained by utilizing the image conversion function that is already built into the camera, rather than directly processing RAW images.

[0012] Furthermore, biometric authentication using multiple biometric features requires optimal white balance settings for each biometric feature, such as finger veins, fingerprints, skin patterns, and joint patterns. However, if raw images cannot be directly processed, multiple captures must be performed with optimal white balance settings for each biometric feature. Furthermore, even when capturing the same biometric feature, it is conceivable that the image will appear clearer if different white balance settings are used depending on the ambient lighting conditions. For example, optimal color correction is required for various ambient lighting conditions, such as indoors, by a window, midday sunlight, and sunset. Furthermore, in situations where the ambient light is partially intense, the dynamic range of the image may be insufficient, resulting in missing biometric features. In other words, there are issues with the optimal white balance and color correction methods differing for each biometric feature, and accuracy degradation due to variations in the lighting conditions of the capture environment.

[0013] Patent Document 1 aims to realize a biometric authentication device that can perform authentication with high accuracy even if the lighting environment or imaging device is different between enrollment and authentication. To this end, it discloses a technology that uses enrollment information stored in a storage unit to generate a color transformation matrix that minimizes the color difference between the representative colors of a reference biometric image and the biometric image at the time of authentication, generates a color-converted biometric image using the generated color transformation matrix, and extracts features. While this document mentions the viewpoint of making both features similar by suppressing the color difference between enrollment and authentication caused by changes in ambient light, it does not mention a technology related to color transformation suitable for capturing the biometric features to be used.

[0014] The above-mentioned problems can be said to be true not only for finger veins, but also for various other biometrics such as face, iris, auricle, facial veins, subconjunctival veins, palm veins, back of the hand veins, palm prints, inner and outer finger joint prints, dorsal finger veins, subcutaneous fat patterns, and melanin patterns, as well as for multimodal biometric authentication that combines these.As such, conventional technologies have had various issues with biometric authentication based on general-purpose cameras, such as image quality degradation caused by shooting with an inappropriate white balance, which leads to various reductions in accuracy.

[0015] The present invention aims to provide a biometric authentication system and a biometric authentication method that can maximize authentication accuracy in various imaging environments when implementing biometric authentication based on biometric features captured by a general-purpose camera. [Means for solving the problem]

[0016] A preferred example of the biometric authentication device of the present invention includes a storage device that stores biometric feature amounts, an image capture device that captures a biometric image, an image correction unit that converts the image quality of the image captured by the image capture device, and an authentication processing unit that performs biometric authentication using the image output by the image correction unit. The image correction unit generates pre-color-corrected images of one or more images of the biometric image, converts the pre-color-corrected images using multiple values ​​within a predetermined search range for color information to generate multiple color-information-converted images, selects from the multiple color-information-converted images the color-information-converted image that best represents each biometric feature, and searches for optimal color information for each biometric feature based on the color information used to obtain the selected color-information-converted image. [Effects of the Invention]

[0017] According to the present invention, it is possible to achieve highly accurate authentication even if the image capturing environment at the time of registration is different from the image capturing environment at the time of authentication. [Brief explanation of the drawings]

[0018] [Figure 1A] 1 is a diagram illustrating an overall configuration of a biometric authentication system according to a first embodiment. [Figure 1B] FIG. 2 is a diagram illustrating an example of a functional configuration of a program stored in a memory according to the first embodiment. [Figure 2] 10 is a diagram illustrating an example of a processing flow for generating an image optimal for authentication from an image captured with an arbitrarily set white balance and emphasizing biometric features according to the first embodiment. [Figure 3A] 1 is a diagram illustrating an example of the configuration of a neural network that estimates a pre-color-correction image from images with a plurality of white balances according to the first embodiment. [Figure 3B]10 is a diagram illustrating an example of a configuration of a learning phase of a neural network that estimates a pre-color-correction image from a plurality of images with different white balances according to the first embodiment. [Figure 4] 10 is a diagram illustrating an example of a processing flow for searching for an optimum color temperature for extracting a biometric feature according to the first embodiment. [Figure 5] FIG. 4 is a diagram illustrating an example of a processing flow for searching for an optimal color temperature according to the first embodiment. [Figure 6A] 10 is a diagram illustrating an example of a configuration at the time of inference of a neural network for processing to correct a luminance gradient caused by ambient light according to the second embodiment. [Figure 6B] 10 is a diagram illustrating an example of a configuration during learning of a neural network for processing to correct a luminance gradient caused by ambient light according to the second embodiment. [Figure 6C] 10 is a diagram illustrating an example of generating learning data for a neural network for processing to correct a luminance gradient caused by ambient light according to a second embodiment. [Figure 6D] 10 is yet another example of generating learning data for a neural network for processing to correct a luminance gradient caused by ambient light according to the second embodiment. [Figure 7] 10 is a diagram illustrating an example of a processing flow of authentication processing according to a third embodiment, in which the influence of wavelength changes in ambient light is reduced by normalizing a ratio of wavelength components of received light through spectral processing. [Figure 8] FIG. 11 is a diagram illustrating an example of the spectral sensitivity characteristics of various wavelengths of ambient light and an image sensor of a camera according to the third embodiment. [Figure 9] 13 is a diagram illustrating an example of a processing flow of authentication processing including guidance for reducing the influence of unnecessary ambient light according to a fourth embodiment. [Figure 10] 10 is a schematic diagram showing an example of a method for detecting a luminance gradient caused by external light irradiated onto a finger according to a fourth embodiment. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of a process for estimating the irradiation direction of an external light component according to the fourth embodiment. [Figure 12A] FIG. 11 is a diagram showing an example of a guidance display at the start of authentication when external light is detected during authentication according to the fourth embodiment. [Figure 12B]FIG. 13 is a diagram showing an example of guidance display during biometric image capture when external light is detected during authentication according to the fourth embodiment. [Figure 12C] FIG. 13 is a diagram showing an example of guidance displayed when external light is detected during authentication according to the fourth embodiment. [Figure 12D] FIG. 11 is a diagram showing an example of a user's posture to avoid external light based on a guidance display when external light is detected during authentication according to the fourth embodiment. [Figure 13] 10 is a diagram illustrating an example of a processing flow for searching for an optimum hue for extracting a biometric feature according to the first embodiment. [Figure 14A] 1A to 1C are diagrams illustrating an example of a method for measuring biological contrast according to Example 1. [Figure 14B] FIG. 4 is a diagram showing an example of biological contrast measurement in vein enhancement processing according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0020] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0021] Furthermore, in the following description, processing performed by executing a program may be described, but the program is executed by a processor (e.g., a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit)) to perform the specified processing while appropriately using storage resources (e.g., memory) and / or interface devices (e.g., communication ports), and therefore the processor may be the subject of the processing. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing a program may be any computing unit, and may include a dedicated circuit (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs specific processing.

[0022] In this specification, biometric features refer to finger veins, fingerprints, joint patterns, skin patterns, finger contour shapes, fat lobule patterns, finger length ratios, finger widths, finger areas, melanin patterns, palm veins, palm prints, back veins, facial veins, ear veins, or face, ears, irises, etc. Furthermore, biometric information may be information that means anatomically different biometric features, or biometric information that means multiple feature amounts extracted from information containing a mixture of multiple anatomically classified biometric parts and divided in any manner.

[0023] According to the present invention, highly accurate authentication can be achieved even when the image capturing environment at the time of registration and the time of authentication are different. For example, not only can biometric authentication be easily used in entrance / exit management, PC login, and automatic payment at stores, but biometric information can also be registered using a general-purpose camera such as a smartphone, eliminating the need to go to a dedicated terminal for registration, thereby increasing convenience. This can facilitate the widespread use of biometric authentication technology. [Example]

[0024] FIG. 1A is a diagram showing an example of the overall configuration of a biometric authentication system 1000 using biometric features in this embodiment. It goes without saying that the configuration of this embodiment may be configured as an authentication device in which all or part of the configuration is mounted in a housing, rather than as an authentication system. The authentication device may be a personal authentication device that includes authentication processing, or may perform authentication processing externally and be a finger image acquisition device or finger feature image extraction device specialized in acquiring finger images. It may also be embodied as a terminal.

[0025] A configuration that includes at least an imaging unit that captures biometric images and an authentication processing unit that processes the captured images and authenticates the biometrics is called a biometric authentication device, and a configuration in which authentication processing is performed by a device connected to the imaging unit that captures biometric images via a network is called a biometric authentication system. Authentication systems include biometric authentication devices and biometric authentication systems.

[0026] 1A includes an input device 2 serving as an imaging unit, an authentication processing device 10, a storage device 14, a display unit 15, an input unit 16, a speaker 17, and an image input unit 18. The input device 2 includes an imaging device 9 installed inside a housing, and may also include a light source 3 installed in the housing. The authentication processing device 10 has an image processing function.

[0027] The light source 3 is a light-emitting element such as an LED (Light Emitting Diode), and irradiates light onto a certain area of ​​the user's body presented to the input device 2, for example, onto a finger 1. Depending on the embodiment, the light source 3 may be capable of irradiating light of various wavelengths, or may be capable of irradiating light transmitted through the body, or may be capable of irradiating light reflected from the body. The light source 3 may also be composed of multiple light-emitting elements, and may have a function that allows the light emission intensity of each element to be adjusted by the authentication processing device 10.

[0028] The imaging device 9 captures an image of the fingers 1 presented to the input device 2. It may also capture images of other living organisms, such as the face, iris, back of the hand, and palm. The imaging device 9 is an optical sensor capable of capturing light of a single or multiple wavelengths, such as a color camera. The optical sensor may be a monochrome camera or a multispectral camera capable of simultaneously capturing light of various wavelengths, such as ultraviolet or infrared light in addition to visible light. It may also be a distance camera capable of measuring the distance to a subject, or a stereo camera configuration combining multiple identical cameras. The imaging device 9 may have either or both of an automatic exposure adjustment function and a manual exposure adjustment function, as in a camera used in a smartphone, and either or both of an automatic white balance adjustment function and a manual white balance adjustment function. If it has a manual white balance adjustment function, it may have either or both of a function to set an arbitrary color temperature or a function to set several predetermined color temperatures.

[0029] A plurality of such imaging devices may be included in the input device 2. Furthermore, the finger 1 may be one or more fingers, and may include multiple fingers on both hands at the same time.

[0030] The image input unit 18 acquires an image captured by the imaging device 9 in the input device 2, and outputs the acquired image to the authentication processing device 10. As the image input unit 18, for example, various reader devices for reading images (for example, a video capture board) can be used.

[0031] The authentication processing device 10 is configured by a computer including, for example, a central processing unit (CPU) 11, a memory 12, and various interfaces (IF) 13. The CPU 11 executes programs stored in the memory 12 to realize various functional units such as authentication processing.

[0032] FIG. 1B is a diagram showing an example of the functional configuration of a program stored in the memory 12 for realizing each function of the authentication processing device 10. As shown in FIG.

[0033] As shown in FIG. 1B, the authentication processing device 10 includes a registration processing unit 20 that associates an individual's biometric features with a personal ID (registration ID) and registers them in advance; an authentication processing unit 21 that authenticates the biometric features extracted from a current image based on the registered biometric features and outputs the authentication result; and a biometric information processing unit 27 that performs image processing and signal processing necessary for biometric authentication. The biometric information processing unit 27 includes various processing blocks, such as an image capture control unit 22 that captures a presented biometric image under appropriate conditions; an image correction unit 23 that corrects the biometric image to an ideal state so that biometric features can be more reliably extracted; a feature extraction unit 24 that extracts biometric features while appropriately correcting the biometric posture during the registration and authentication processes; a matching processing unit 25 that calculates the similarity between biometric features; and an authentication determination unit 26 that determines the authentication result based on one or more matching process results. These various processes will be described in detail later. The memory 12 stores programs executed by the CPU 11. The memory 12 also temporarily stores images input from the image input unit 18.

[0034] In FIG. 1B, the feature extraction unit 24 may be configured as a part of the image correction unit 23, and the authentication determination unit 26 may be configured as a part of the authentication processing unit 21.

[0035] The interface 13 connects the authentication processing device 10 to an external device. Specifically, the interface 13 is a device having ports for connecting to the input device 2, the storage device 14, the display unit 15, the input unit 16, the speaker 17, the image input unit 18, etc.

[0036] The interface 13 also functions as a communication unit that enables the authentication processing device 10 to communicate with external devices via a communication network (not shown). The communication unit is a device that performs communication in accordance with the IEEE802.3 standard if the communication network 30 is a wired LAN, and is a device that performs communication in accordance with the IEEE802.11 standard if the communication network 30 is a wireless LAN.

[0037] The storage device 14 is configured, for example, with a hard disk drive (HDD) or a solid state drive (SSD), and stores user registration data, etc. The registration data is information obtained during registration processing for verifying users, and is stored in association with multiple biometric features for each user. For example, the registration data is image and biometric feature data such as facial features, finger features, and finger vein patterns linked to a registration ID as user identification information.

[0038] A finger vein pattern image is an image of the finger veins, which are blood vessels distributed under the skin of the finger, captured as a dark shadow pattern or a slightly bluish pattern.Finger vein pattern feature data is data obtained by converting the image of the vein portion into a binary or 8-bit image, or data consisting of feature values ​​generated from the coordinates of feature points such as the bends, branches, and endpoints of the vein or brightness information around the feature points, or data obtained by encrypting and converting such data into an undecipherable form.

[0039] The display unit 15 is, for example, a liquid crystal display, and is an output device that displays information received from the authentication processing device 10, biometric guidance information, and various determination results.

[0040] The input unit 16 is, for example, a keyboard or a touch panel, and transmits information input by the user to the authentication processing device 10. The display unit 15 may also have an input function such as a touch panel. The speaker 17 is an output device that transmits information received from the authentication processing device 10 as an acoustic signal such as voice.

[0041] Figure 2 shows an example of a processing flow for generating an optimal image for authentication from an image captured with an arbitrarily set white balance and emphasizing biometric features. In this example, one or more images with different white balances are captured, and multiple biometric features to be used are appropriately captured regardless of the ambient light conditions. The optimal white balance is determined, and color conversion is performed to emphasize the biometric features, thereby improving the accuracy of biometric authentication for each biometric feature and making authentication robust to changes in ambient light. The following processing will be described assuming that the feature extraction unit 24 in Figure 1B is configured as part of the image correction unit 23 and the authentication determination unit 26 is configured as part of the authentication processing unit 21.

[0042] Hereinafter, color conversion may be referred to as conversion of color information, where "color information" includes "color temperature" and "hue." Therefore, conversion of color information includes conversion of color temperature and conversion of hue.

[0043] First, the imaging device 9 captures multiple images of the user's biometric data using different white balance values ​​(S201). Assume that the biometric features used for authentication are, for example, the finger vein pattern and knuckle prints present on the fingers, and capture an image of the entire hand. The input device 2 is a smartphone, and the imaging device 9 is the smartphone's rear camera. The light source 3 is an LED or torch attached to the smartphone, which can be turned on to enable authentication even in dark environments. However, it is also possible to capture images with the light turned off. The camera has an autofocus function that automatically focuses on the biometric data held over the camera, and also an autoexposure function that appropriately corrects the brightness of the subject. Regarding white balance correction, which corrects the color tone of the image, several pre-defined white balance values ​​(preset values) can be manually set. However, it is also possible to capture images by specifying an arbitrary color temperature. However, to increase the versatility of the authentication system, we assume that the ability to specify an arbitrary color temperature is not available. The parameter that determines the white balance is color temperature, generally expressed in units of K (Kelvin). For example, an image with a color temperature of 2000K will be corrected to have a blue tint, and an image with a color temperature of 6000K will be corrected to have a red tint.

[0044] Furthermore, the camera's frame rate is sufficiently high that the position of the subject's fingers does not change significantly even when multiple images are continuously captured at different white balance values ​​(color temperatures). However, taking multiple images continuously will result in significant deviation in the subject's position, so this explanation assumes that the camera will be capturing images at three different white balance settings. As will be described later, the camera can also operate by capturing a single image at any white balance setting, or by capturing images at more than three different white balance values.

[0045] Next, the image correction unit 23 estimates and generates a "pre-color-corrected image" before the white balance was changed (S202). In this embodiment, the captured biometric image is converted into an image before the white balance was changed, i.e., into an image with the colors immediately after the CMOS sensor received light, based on a generative adversarial network (GAN). Here, the image converted into an image before the white balance was changed, i.e., into an image with the colors immediately after the CMOS sensor received light, is referred to as the "pre-color-corrected image." The pre-color-corrected image is a RAW image in which demosaicing processing and various aberration corrections have been optimized according to the functions of each camera, but only the colors remain in the state of the RAW image. The conversion method will be described in detail later.

[0046] Next, the image correction unit 23 searches for the optimal color temperature for extracting each biometric feature (S203). This is a process of searching for a white balance that allows each biometric feature used for authentication to be observed more clearly and stably. Here, the color temperature of the pre-color-correction image is arbitrarily changed while actually extracting and matching one or more biometric features to be used for authentication, thereby estimating the optimal white balance for stable extraction of the biometric features. This allows for the acquisition of the optimal white balance value that maximizes authentication accuracy. Details will be described later.

[0047] Thereafter, the image correction unit 23 extracts each biometric feature from the image with the optimal color temperature for each biometric feature (S204). If the feature extraction unit 24 and the image correction unit 23 are configured separately, the image correction unit 23 can simply read out the biometric features from the feature extraction unit 24. Note that since the biometric features are extracted in the search for the optimal color temperature in the previous stage, the results can be reused here.

[0048] Then, the matching processing unit 25 performs matching with pre-registered biometric characteristic data to obtain a final matching score (S205).

[0049] In this embodiment, it is assumed that authentication is successful if two of the four captured fingers match. As one example of matching, the matching processor 25 first calculates a matching score for each biometric feature by performing a brute force comparison between all registered fingers and all authenticated fingers. Next, taking into account that there may be fingers that could not be detected or that were incorrectly detected, all combinations of registered and authenticated fingers are found for the E registered fingers and the V detected fingers during authentication under conditions that satisfy constraints such as no geometric contradictions occurring, for example, the positions of the index finger and ring finger not being reversed.

[0050] For example, if there are four E fingers and three V fingers, there are three combinations of selecting two from the three V fingers, and six combinations of matching pairs of the four E fingers and the two V fingers, for a total of 18 possible combinations. Then, for each combination, a "finger-level matching score" is calculated as the matching score for each finger. The finger-level matching score can be calculated, for example, by linearly combining the finger vein matching score and the epidermal matching score, and the matching score for the two fingers can be calculated as the average of these. This matching score, which uses the linear combination of the finger vein matching score and the epidermal matching score, is specifically called the "fusion matching score." The lowest fusion matching score calculated for all of the above combinations is specifically called the "final matching score."

[0051] Then, the authentication processing unit 21 determines whether the final matching score is lower than a predetermined authentication threshold (S206), and if so, performs the authentication process as a success (S209) and ends the process.

[0052] Furthermore, if the final matching score is higher than the predetermined authentication threshold, the authentication processing unit 21 checks whether the authentication timeout period has elapsed (S207).

[0053] If the timeout time has not elapsed, the authentication processing unit 21 returns to step S201, and if the authentication is not successful within the timeout time, the authentication fails (S208) and ends without performing the authentication process.

[0054] 3A and 3B show an example configuration of the image correction unit 23 shown in (S202) of FIG. 2, which is an example configuration of a neural network that estimates a pre-color-corrected image from images captured with multiple white balance settings. Because white balance adjustment methods are designed individually for each camera, the algorithms are generally not publicly available. Furthermore, once the white balance is changed, the color information of essential biometric features can be lost, making it difficult to restore the color temperature. Therefore, the image correction unit 23 of this embodiment uses a generative adversarial network (GAN) to generate a "pre-color-corrected image" (before the white balance is changed) from multiple images captured with different white balance settings. Here, we describe an example using pix2pix, a GAN technique known to be capable of style conversion.

[0055] As shown in Figure 3A, the image correction unit 23 performs position correction using a registration layer on three color-converted images Fi (F1 to F3) captured by the image capture device 9 at multiple different white balance settings to minimize misalignment between the multiple images. The pix2pix algorithm used in this method generates images based on a pair of two training images, but it is known that accuracy degrades when the training images contain geometric misalignment. Therefore, it is important to reduce the misalignment of the subject when capturing and inputting multiple images. One example of position correction using a registration layer involves first removing the background using a finger region detection process, then dividing the two images to be corrected into multiple patches (small regions). The misalignment between two patches in the two images that have the same positional relationship is estimated using CNN. The x- and y-direction misalignments are then obtained for all patches. A grid of control points is then generated to match the patch divisions and perform B-spline transformation on the image. By applying the misalignment information to these control points and performing image transformation, a smoothly deformed image can be generated. However, one of the two images is used as the reference image, and the other image is deformed. Similarly, by performing similar processing when aligning multiple images, it is possible to reduce pixel-by-pixel misalignment across all images.

[0056] The reference image X is composed of multiple planes so that images with multiple white balances can be processed, and in this embodiment, the number of planes is large enough to include images with three different white balances. Specifically, there are nine planes because three color images with three colors are held. Furthermore, the generator G receives the reference image X containing images with multiple white balances as input, and generates one pre-color-corrected image before correcting the white balance as the generated image G(X). With this configuration, a pre-color-corrected image can be obtained by passing three images with any white balance.

[0057] FIG. 3B shows an example of the configuration of the learning phase of the image correction unit 23 shown in FIG. 3A. In the learning phase, a RAW image F of a living body is captured, and multiple white balance (color temperature) images F(Ti) (here, i = 1 to 3) of the living body are generated from the RAW image F. Ti is a color temperature determined by a random number, and the color temperature is randomly changed when generating the three color-converted images. However, the camera used to construct the learning data is assumed to be capable of capturing RAW images, and there is no geometric transformation between the RAW image F and the multiple white balance images F(Ti). A specific method for converting a RAW image to an arbitrary color temperature will be described later, but commonly known methods can be used. This allows for the acquisition of three images with randomly changed white balance. These images are then set as reference images X. Note that the three randomly selected color temperatures may overlap.

[0058] In general, GAN training involves alternating between training the generator and training the discriminator. First, we will describe the training method for generator G. When given a reference image X, generator G outputs a generated image G(X). However, when the pair of reference image X and generated image G(X) is given to discriminator D (described below), generator G is trained so that discriminator D will make an incorrect judgment (i.e., judge it to be genuine). After training generator G for a certain number of times, the training of generator G is stopped and training of discriminator D is started. Discriminator D is trained to judge that when given a pair of reference image X and generated image G(X), it is a fake, and when given reference image X and correct image Y (which is the same as RAW image F), it is genuine. Similarly, after a certain number of repetitions, the training of discriminator D is stopped and training of generator G is started again. By repeating this training process alternately, the generated image G(X) output by generator G gradually becomes so similar that it becomes indistinguishable from the correct image Y.

[0059] Then, the same RAW image F is multiplied by a different random number Ti and re-learned, and then RAW images F taken in different postures or in different shooting environments are input and re-learned, and finally similar learning is performed using biometric images of various subjects. As a result, when any three white balance corrected biometric images are given, it becomes possible to obtain a pre-color correction image in the RAW image state, that is, a pre-color correction color image having the spectral sensitivity characteristics of the camera used in learning.

[0060] While this embodiment describes a network configuration that provides three color-converted images, if the number of captured images is known in advance, the number of planes in the reference image X can be adjusted to accommodate that number. Alternatively, if a network is configured to accept a larger number of images than expected, such as a maximum of five images, the network can be trained by providing all images with the same color temperature, or by providing three of the five images with the same color temperature and the remaining two with different color temperatures. This allows the network to acquire a pre-color-corrected image even when any number of images, from one to five, is input without changing the network configuration. In particular, if a slow frame rate prevents the capture of images with multiple white balances and only one image is captured, the pre-color-corrected image can be acquired by copying five copies of the captured image. While this has the advantage of obtaining the desired pre-color-corrected image even with only one captured image, image quality may be degraded, for example, when converting a reddish image to a bluish image. Therefore, if multiple images with different white balances can be captured and input in a balanced manner, providing multiple images as input can result in higher-quality images.

[0061] In the above example, an uncorrected white balance image was first obtained from one or more images with different white balances, and then an image with a desired color temperature was generated by a separate method. However, a neural network that directly estimates an image with a desired color temperature can also be constructed. In this case, in the training phase, biometric images with various color temperatures are prepared in advance, and the color temperature is digitally converted from the RAW images in 500K increments from 2000K to 7000K, for example, with no geometric misalignment. In this case, 11 correct images are provided as training data. From these 11 images, multiple reference images X are created using all combinations of three images, allowing overlaps, and each reference image X is paired with 11 more correct images y. Training pairs are then similarly created for various biometric postures and subjects, building a large number of datasets. Pix2pix is ​​then trained using this training data, enabling it to generate images with a specific color temperature by inputting any three white balance images. In this case, we are assuming that the conversion will result in 11 correct images in increments of 500K from 2000K to 7000K, so we will prepare 11 different pix2pix networks for each color temperature of the correct image and proceed with the learning.

[0062] Furthermore, during inference, by selecting the pix2pix network, which can output images at the desired color temperature in 500K increments from 2000K to 7000K, and then passing three images with any white balance, it is possible to generate an image at the specified color temperature.

[0063] The advantage of this method is that it directly obtains an image with the specified color temperature, eliminating the need to implement a color temperature conversion algorithm during authentication. However, in this example, the color temperature that can be obtained is limited to 500K units. To overcome this limitation, it is necessary to prepare training data in finer units, or to process the latent vector in style conversion so that the white balance value changes smoothly. In either case, it can be applied as desired depending on the specifications of the system being designed.

[0064] 4 shows an example of the processing flow for searching for an optimum color temperature for extracting biometric features by the image correction unit 23, as shown in (S203) of Fig. 2 above. In this example, biometric features of finger veins and knuckle patterns (skin patterns on the palm side of the finger) are used, but here we will explain the search for a white balance suitable for vein features. Searching for a white balance suitable for knuckle patterns can be similarly achieved by replacing veins with knuckle patterns and repeating the process, so we will not explain it here.

[0065] First, the image correction unit 23 performs finger region detection on the pre-color-corrected image acquired in the previous process (S401). Finger region detection can be performed using a deep learning-based segmentation technology, such as Mobile Net v3, which provides training data paired with hand images and mask images of the finger regions. During inference, the finger region can be acquired as a mask image of the finger region for any hand image. The results of finger region detection are also used to acquire rectangle information for each finger. The rectangle information for each finger is obtained by determining the positions of the fingertip and base and the center line of the finger from the finger's outline, and then determining a rectangle that includes the fingertip and base positions and whose center line coincides with the center line of the frame.

[0066] Next, the image correction unit 23 sets a lower limit, an upper limit, and a median as multiple values ​​of a predetermined search range related to color information as initial values ​​of the range of color temperatures to be searched (S402). In this embodiment, an image with a color temperature between 2000K and 7000K, which generally covers the range when general indoor lighting and sunlight are used as ambient light, is to be generated, and the lower limit is set to 2000K and the upper limit to 7000K. In addition, 4500K is set as the initial median value for these upper and lower limit values.

[0067] Next, the image correction unit 23 converts the pre-color-correction image into an image with the color temperature indicated by the lower, median, and upper values, respectively, to obtain Image 1, Image 2, and Image 3 as color-information-converted images (S403). In this embodiment, to convert color information, the image is converted to an arbitrary color temperature using a color temperature conversion algorithm designed in-house. A commonly known calculation method for converting an image to a specific color temperature can be used, such as utilizing the relationship between color temperature and RGB images based on the International Commission on Illumination color system "CIE 1931." For example, if the pixel values ​​representing blue, green, and red in the pre-color-converted image are {b, g, r}, the color temperature of the ambient light in which the pre-converted image was captured is Torg, the color temperature to be converted is Tdst, and the coefficients are αb(Torg, Tdst) and αr(Torg, Tdst), the pixel values ​​{b', g', r'} of the converted image can be expressed as follows: b' = b * αb(Torg, Tdst) g' = g r' = r * αr(Torg, Tdst)

[0068] For example, if the color temperature Torg of the shooting environment is 6500K and the desired color temperature Tdst is 5000K, it is known that αb(Torg, Tdst) and αr(Torg, Tdst) are approximately 0.74 and 1.29, respectively. In this way, the color temperature of an image can be converted by preparing a table of the coefficients αb and αr in advance. Also, since the color temperature of the shooting environment is often unknown, it may be possible to fix the value of Torg to, for example, 5000K.

[0069] Thereafter, the image correction unit 23 superimposes random noise on image 1, image 2, and image 3 to generate image 4, image 5, and image 6 (S404). The random noise is, for example, additive Gaussian noise, and adding noise to an extent that could actually occur can change the image quality. However, the random noise added to image 1, image 2, and image 3 is assumed to be exactly the same. Note that the same effect can be achieved by using two images taken consecutively in time instead of adding random noise.

[0070] Next, the image correction unit 23 extracts vein features for each finger from the acquired images 1 to 6 (S405). As a specific example of feature extraction, the image is first extracted for each finger based on the rectangular information for each finger described above, and then the veins are emphasized by subtracting the green and blue components from the red component of the image. For example, the veins are emphasized by darkening them by subtracting the green and blue components from the red component at a ratio of 2.0*r' - 0.8*g' - 0.2*b'. The vein line pattern is then sharpened using a spatial feature enhancement filter such as unsharp masking. The center line of the dark line pattern is extracted from this image using a commonly known method, and the result is binarized or ternarized to obtain the vein feature. The articular crest features can be obtained by obtaining an image of the surface skin pattern using an average image of the green and blue components of the image, similarly emphasizing the epidermal features using unsharp masking or the like, extracting the epidermal line pattern, and then binarizing or ternarizing the result.

[0071] Next, the image correction unit 23 performs a brute-force matching using the vein features obtained from all finger images (S406). Alternatively, the image correction unit 23 may instruct the matching processing unit 25 to perform matching and obtain the results. In the brute-force matching, first, matching between the same fingers (identity matching) and different fingers (non-identity matching) is performed for the finger vein patterns of each finger extracted from image 1 and image 4, which have color temperatures indicated by the lower limit value, and the results are defined as matching score group 1. In this embodiment, since four fingers are basically used for authentication, identity matching is performed between the same fingers in images with different random noise, resulting in a total of four identity matchings. Non-identity matching involves matching with three other fingers, regardless of whether the random noise matches, for four fingers, resulting in 24 matchings. Here, the score distribution for identity matching is referred to as the identity distribution, and the score distribution for non-identity matching is referred to as the non-identity distribution. Similarly, the same process is performed on images with color temperatures indicated by the median and upper limit values ​​to obtain matching score group 2 and matching score group 3. These brute-force comparisons reveal the stability and clarity of pattern extraction due to the influence of random noise. In other words, when comparing the comparison score distribution for an image of one color temperature with that of another, the greater the difference between the score distribution for true identity verification and the score distribution for false identity verification, the higher the authentication accuracy, i.e., the more stable the color temperature at which biometric features can be acquired.

[0072] In addition, when comparing different people, it is possible to geometrically invert each finger horizontally or vertically to generate different pseudo patterns and increase the number of combinations, thereby obtaining a more reliable distribution of different people. It goes without saying that the number of fingers to be compared is not limited to the above number and can be changed as appropriate depending on the application.

[0073] Thereafter, the image correction unit 23 normalizes the score distribution for each matching score group so that the other distribution becomes a standard normal distribution (S407). As a result, the mean value of the other distribution for each matching score group becomes 0 and the variance becomes 1, so that the degree of separation between the true distribution and the other distribution can be grasped by comparing the true distributions with each other.

[0074] Here, the image correction unit 23 selects the image that best represents each biometric feature. Specifically, the degree of separation is measured using the average value of the individual's distribution, and a larger absolute value of this average value indicates a higher degree of separation, i.e., higher authentication accuracy. The color temperature of the image that yields the highest degree of separation, including all of the loop processing up to this point, is then obtained, and the color temperature that generated the image that best represents each biometric feature is designated as the optimal value (S408) as the value that best represents each biometric feature (S408). The search range is also set to half the difference between the current upper and lower limits. The search range defines the range of color temperatures to be searched in the next loop of the processing flow, and is halved with each iteration of the loop. This allows the search range to be gradually narrowed, enabling a detailed color temperature search to be performed with as few iterations as possible.

[0075] Furthermore, if the search range becomes smaller than a predetermined range, the image correction unit 23 determines that the color temperature has been sufficiently narrowed down and ends the search for the optimal value. Therefore, it determines whether the search range is smaller than a threshold (S409), and if so, it sets the current optimal value as the final result (S412) and ends the search. If not, it sets the current optimal color temperature as the median, and sets the upper and lower limits around it (S410).

[0076] However, if the lower limit value falls below the originally set initial lower limit value, the image correction unit 23 adds the difference to the lower limit value, median value, and upper limit value. Similarly, if the upper limit value exceeds the initial upper limit value, the image correction unit 23 subtracts the difference from the lower limit value, median value, and upper limit value (S411). Then, the color temperature conversion process (S403) is performed again. However, since color temperatures for which the degree of separation was previously calculated may be included in the second and subsequent iterations, in such cases the amount of calculation is reduced by reusing the processing results. Here, an example has been described in which the search range is halved each time the loop is repeated, but the search range can also be narrowed for each loop, for example, by 1 / 3 or 1 / 4.

[0077] The above process allows the image correction unit 23 to gradually narrow the range in which it searches for the optimal color temperature while searching for the optimal value, which contributes to faster processing speed. However, if calculation speed is not an issue, it is also possible to search for the optimal color temperature based on the above criteria while covering color temperatures at regular intervals.

[0078] Furthermore, when multiple biometric features are used for authentication, for example when using joint prints, it is possible to extract and compare the joint prints while searching for the color temperature that is optimal for the joint prints, and to improve the matching accuracy for each by using biometric images with multiple different color temperatures.Similarly, when three or more biometric features are used for authentication, it goes without saying that the same process can be repeated the same number of times as the number of biometric features.

[0079] FIG. 5 shows an example of an optimum color temperature search by the image correction unit 23 shown in FIG. 4 above. FIG. 5(a) shows the initial state when searching for the optimum color temperature. In this embodiment, the optimum value is searched for from 2000K to 7000K, with the horizontal axis representing color temperature, and the lower limit of the range to be searched corresponding to 2000K and the upper limit corresponding to 7000K. First, the range median, which is the midpoint of this range, is found. In this example, the median is 4500K.

[0080] Next, as shown in Figure 5(b), images with color temperatures indicated by the lower limit, median, and upper limit are obtained using the procedure described above, and the true person's score distribution and the other person's score distribution are calculated. The vertical axis in the figure indicates the matching score, with crosses representing other people's scores and circles representing true people's scores. The other person's distribution is standardized, so it is distributed in the same position for all color temperatures. In this example, it is assumed that the mean true person's score after standardization calculated from the 2000K image is distributed at a lower value than the other color temperatures. In other words, it can be seen that of these three color temperatures, the 2000K color temperature is the color temperature most suitable for authentication.

[0081] Next, as shown in Figure 5(c), the new lower limit, median, and upper limit values ​​calculated by halving the search range are 2000K, 3250K, and 4500K, respectively. This is the result of performing step (S411) in addition to step (S410) in Figure 4. A normalized score distribution group for 3250K, which had not been calculated previously, is obtained, and it is assumed that 3250K is the closest to the mean of the subject's distribution. In this case, using the above and step (S410) in Figure 4, the lower and upper limits are obtained with 3250K, which has the best separation, as the center of the range, resulting in 2625K, 3250K, and 3875K, as shown in Figure 5(d). Similarly, when score distribution groups are calculated for uncalculated color temperatures, it is found that the lower limit of the range, 2625K, provides the best separation. If the threshold for the search range is set to 1000, the current search range is 1250, but when the next loop starts, it will be half that, 625, so the search will end at this point. As a result, it was found that 2625K was the best color temperature.

[0082] If the optimal color temperature could be determined in advance, it would be possible to always take photographs at that color temperature; however, the optimal color temperature may differ depending on the shooting environment, so this method is effective as it can dynamically detect the optimal value during the authentication process even when the ambient light fluctuates.

[0083] Furthermore, while the above embodiment describes the search for the optimal color temperature of an image, it is also possible to appropriately correct the "hue" of an image in a similar manner. That is, by replacing the color temperature conversion in FIG. 4 with hue conversion, the processing related to color temperature conversion (S403) may be changed to, for example, the following processing.

[0084] First, level correction is performed on each of the B / G / R color planes of the resulting uncorrected image. Level correction converts the minimum and maximum values ​​of the image's brightness histogram distribution to zero and maximum values ​​of image brightness (255 for 8-bit gradation), maximizing the dynamic range of each color. Next, this image is converted to an HSV image and its saturation is maximized, thereby emphasizing the color features in the image. Hue is generally expressed on a hue wheel, with an angle ranging from 0° to 360°. If the hue of the previously given uncorrected image is considered to be 0°, the optimal hue will be somewhere between 0° and 360°. Therefore, by setting the initial lower limit of the search process in Figure 4 to 0°, the upper limit to 360°, and the median to 180°, three different hue conversions can be used to generate Images 1, 2, and 3. Note that x° hue conversion means rotating the hue of each pixel by +x° from its current state, so for example, if red (color code #FF0000) is converted to green (color code #00FF00) by a 120° hue conversion, it will be converted to green (color code #00FF00). By executing the processing flow in Figure 4 in this way, the optimal hue can be obtained.

[0085] Hue conversion can change color more dramatically than color temperature conversion, which has the potential to significantly improve the contrast of biometric features. In other words, by stably obtaining biometric features, we can expect to see improvements in authentication accuracy and resistance to environmental fluctuations.

[0086] Fig. 13 shows another example of the processing flow of the image correction unit 23 for searching for the optimum hue or color temperature for extracting biometric features. The method shown in Fig. 4 is an example in which the optimum hue is searched for by matching multiple different fingers, but the example described below can be implemented without matching processing on a finger-by-finger basis, so it can be applied even when multiple fingers are not photographed, and is effective in that it also leads to faster processing speed.

[0087] First, finger area detection is performed on the image before color correction acquired in the previous process (S1301), initial values ​​of the median, lower limit, and upper limit hues are set (S1302), and hue conversion to each hue is performed (S1303). These are the same as the method shown in Figure 4, so a description will be omitted.

[0088] Next, the biometric contrast is measured for each image, and an evaluation value is obtained for each (S1304). A specific processing example will be described later. After that, the hue with the highest contrast evaluation value is held as the optimal value, and a search range is determined (S1305). Thereafter, the threshold processing of the search range (S1306), updating of the lower limit, upper limit, and median (S1307), correction processing of the lower limit, median, and upper limit (S1308), and determination processing of the optimal hue (S1309) are the same as those in FIG. 4, and therefore their explanations will be omitted. Here, the optimal hue is the hue value at which each biometric feature is most clearly represented.

[0089] FIG. 14A is a diagram illustrating an example of a method for measuring biometric contrast, as shown in the process (S1304) of FIG. 13 . For each of images 1, 2, and 3, which are obtained by converting the hue or color temperature in three ways, the ROI image 221 of the finger shows a different color for finger 1. Furthermore, the appearance of veins 281 and joint wrinkles 286 also differs, and it is assumed that veins may appear clearer in some cases and joint wrinkles may appear clearer in others. Here, assuming that the biometric feature to be enhanced is vein 281, an example of a method for measuring and maximizing the contrast of the vein will be described. First, the image immediately after hue conversion (left side of the figure) is subjected to the vein enhancement process (described below) to obtain a biometric feature-enhanced image 285 shown on the right side of the figure. While the image after hue conversion shown on the left side is a color image, the biometric feature-enhanced image 285 in this embodiment is converted to a grayscale image to simplify processing.

[0090] Here, we will explain one example of the vein enhancement process. First, brightness correction is performed on each of the B / G / R color planes of a color image so that their average brightness values ​​match. For example, one method for adjusting each color plane to an average value of 64 is to first calculate the current average value for each color plane, then divide the brightness value of each pixel by the current average brightness value and multiply the result by 64. Next, the brightness values ​​of each of the B / G / R color planes of the image are used to blend the color planes so that the veins are most emphasized. Here, if the pixel values ​​of the B / G / R color planes are Fb, Fg, and Fr, and α and β are parameters with a range of 0≦α, β≦1.0, a vein-enhanced image expressed as Fr - α(βFr+(1-β)Fg) is obtained. For example, by varying α and β in 0.1 increments, 11 different values ​​are possible, resulting in the generation of 121 different images. This formula sharpens only the vein pattern by subtracting blue and green images, which highlight the epidermis and joint wrinkles, from the red image, where the vein pattern is most visible. This enables vein enhancement. Here, one method for obtaining the optimum values ​​of α and β will be described.

[0091] Figure 14B shows an example of biometric contrast measurement using vein enhancement processing. When the above-mentioned α and β are varied, the appearance of veins and joint wrinkles in vein-enhanced images varies depending on the values ​​of α and β. In this case, by setting an image cross section 282 as shown in the figure, a cross-sectional intensity profile 283 of the image, which is the intensity curve present on this cross section, can be obtained. Because vein patterns generally run horizontally, setting the image cross section 282 vertically results in many veins crossing this cross section. In this example, two veins 281 cross this cross section. The cross-sectional intensity profile becomes darker where the veins 281 are located, revealing a downward depression at the location of the veins. In this embodiment, a biometric contrast detection probe 284 is set to detect these depressions. This is a virtual probe that calculates the curvature of the intensity profile curve and calculates the difference in intensity values ​​at three locations. This probe has three circles and functions to double the pixel value at the center circle and subtract the pixel values ​​at the left and right circles from that value. The distance between the three circles is always equal and is set to match the typical width of a vein. If the result of this calculation is a negative value, the intensity profile curve will be convex downward, indicating the presence of a vein at that location. The smaller the value, the deeper the valley formed in the profile curve, indicating the presence of a vein with high contrast. The above calculation is then performed while shifting the probe position by one or several pixels. When the result is negative, the calculation results are summed up and the absolute value is taken, which determines the number of veins present in the cross-sectional intensity profile and how high their contrast is. By calculating and summing this for vertical image cross sections at every location within the finger 1 area, the contrast strength of the veins running horizontally can be quantitatively obtained.

[0092] Similarly, if the image cross section is set horizontally and the same processing is performed on the cross-sectional brightness profile of the entire finger region, the contrast intensity of the joint wrinkles, which tend to run vertically, can be obtained. However, the distance between the three circles of the biological contrast detection probe 284 is adjusted according to the width of the joint wrinkles.

[0093] Next, the contrast intensities of the veins and joint wrinkles obtained above are combined to obtain a contrast evaluation value. One method for obtaining a contrast evaluation value is to subtract the contrast intensity of the joint wrinkles from the contrast intensity of the veins. A similar effect can also be achieved by focusing only on the contrast of the joint wrinkles to search for the conditions under which the joint wrinkles are least visible. In this case, for example, the contrast intensity of the joint wrinkles is subtracted from a fixed value and this maximum value is searched for. By obtaining the α and β values ​​that maximize this evaluation value, the contrast intensity of the veins can be maximized and the contrast intensity of the joint wrinkles can be minimized, resulting in an image in which only the veins are emphasized.

[0094] The contrast evaluation value for the optimal α and β obtained by the above method becomes the contrast evaluation value for that hue image. The contrast evaluation values ​​for images 1, 2, and 3 shown in Figure 14A are then extracted as evaluation value 1, evaluation value 2, and evaluation value 3, respectively (S1304), and the hue that maximizes these values ​​is selected, thereby enabling the search for the optimal hue.

[0095] While the above describes the process of enhancing veins, when similarly enhancing joint wrinkles, the positions of veins and joint wrinkles can be reversed to maximize the contrast evaluation value. In addition, in the above-described embodiment, Fr - α(βFr + (1-β)Fg) was calculated as the biometric enhancement process, but simple subtraction may not eliminate the desired information. In response to this, subtraction processing may be performed on the log image, e.g., log(Fr) - α(βlog(Fr) + (1-β)log(Fg)). This has the advantage of homogenizing the brightness unevenness within the finger region, improving robustness against brightness gradients, and enabling enhancement processing that is suited to the physical light absorption characteristics of veins and joint wrinkles.

[0096] As described above, in this embodiment, even if an undesired white balance has been set due to camera settings, the captured biometric image can be converted into an image before color correction, and the optimal color temperature or hue can be automatically determined for each biometric feature, thereby improving authentication accuracy and resistance to environmental fluctuations.

[0097] Note that some general-purpose cameras allow you to change the white balance by setting an arbitrary color temperature. In such cases, it goes without saying that you can acquire an image with the desired color temperature directly from the camera and then search for the optimal color temperature as described above, without having to generate an image before color correction as described above. However, since it is important to match the camera to a low-spec model in order to support any general-purpose camera, this method is effective from the perspective of improving versatility.

[0098] Alternatively, instead of searching for the optimal white balance, the image correction unit 23 may acquire and combine images with multiple white balances to improve image quality on average. In one specific example, several white balances capable of emphasizing biometric features are first determined through preliminary evaluation, and all images captured at these color temperatures are integrated to increase the dynamic range. Then, as shown in the hue conversion described above, the integrated image is subjected to range correction for each color, maximizing saturation, and converting to the hue determined through preliminary evaluation to emphasize color features. While this method does not necessarily achieve optimal enhancement for each biometric feature, it does improve biometric contrast on average, which is expected to improve authentication accuracy and environmental resistance, and is therefore effective because it also achieves faster processing speeds. [Example]

[0099] 6A and 6B show an example of the configuration of the image correction unit 23 for correcting a luminance gradient caused by factors other than biometric features, such as ambient light. The above-described embodiment employs a technique for obtaining a color temperature or hue that best captures biometric features while tracking changes in ambient light. However, the influence of ambient light not only changes color characteristics but can also cause partial luminance saturation and luminance gradients in biometric images, resulting in a deterioration in accuracy. While overall luminance saturation can be easily eliminated by exposure control, luminance gradients cannot be eliminated by simple exposure control.

[0100] "Brightness gradient" refers to a state in which there is a large variation in the illumination intensity of environmental light, such as when standing near a window where strong sunlight shines in, causing one side of the subject to be particularly brightly illuminated, resulting in an irregular gradation in the brightness of the subject. If the subject has irregular variations or gradients, the uneven brightness areas may be mistakenly extracted as biometric features, so it is desirable to remove brightness gradients as much as possible before extracting biometric features.

[0101] Although uneven brightness or gradation can cause some biometric images to have both saturated and non-saturated areas, this can be resolved by applying HDR (High Dynamic Range), a dynamic range improvement technology. However, applying HDR does not necessarily eliminate the brightness gradient itself.

[0102] Figure 6A shows an example of the GAN configuration of the image correction unit 23 for removing brightness gradients. Here, pix2pix is ​​used, and the configuration is similar to that shown in Figure 3. The reference image X is a hand image taken under arbitrary ambient light, and the generated image G(X) is obtained through the generator G. At this time, the generated image G(X) is trained to generate a hand image taken under ideal conditions.

[0103] FIG. 6B shows an example of the configuration of the learning phase of the image correction unit 23 that trains FIG. 6A. Ideal environment Y is a hand image captured in an ideal environment, such as indoors. Reference image X is the same hand image captured under any environmental luminance, such as indoors, indoors under a different lighting environment, by a window, outdoors during the day, or outdoors in the evening. First, reference image X and ideal image Y are input as a pair of images into the registration layer. Position correction is performed on reference image X so that it matches ideal image Y pixel-by-pixel in the same manner as in the above-described embodiment, resulting in a position-corrected reference image X' with the same shape as ideal image Y. Note that the position correction process may be performed in advance on all training data using batch processing. Next, generator G and classifier D are trained. These procedures are the same as those in FIG. 3, so a detailed description is omitted. Through this training process, hand images captured in any environment are converted into images captured in an ideal environment, thereby suppressing the occurrence of luminance gradients.

[0104] This process can be performed in combination with the authentication process flow described in FIG. 2 in the above-described embodiment, for example, immediately after the process of photographing a living body with multiple white balance values ​​(S201), thereby realizing authentication process that further reduces the influence of ambient light.

[0105] FIG. 6C shows an example of generating training data. First, an indoor hand image 121 taken in an ideal environment with almost no tilt, a hand image 122 of the same hand taken near a window, and a hand image 123 including shadows taken in a situation where light and dark shadows are cast on the fingers due to the influence of window blinds or the like are taken. In addition to this example, it is desirable to take images in a wide variety of environments by many subjects. In this case, registration becomes difficult if various objects are reflected in the background of the hand and fingers, so it is desirable to remove the background using finger region detection processing. However, as will be described later, there are methods that allow learning even when the background is included, so either method may be adopted.

[0106] Next, an ideal image of a hand 121 taken indoors and an ideal image of a hand 122 taken by a window are paired and used as training data. Similarly, an ideal image of a hand 121 taken indoors and an ideal image of a hand 123 including shading are paired and used as training data. The former of this pair is then input as ideal image Y and the latter as reference image X to the network configuration for training.

[0107] If it is possible to continuously generate paired images using a dedicated image capture device 9 that can instantly switch between an indoor environment and a windowside environment without pixel-level misalignment, the registration layer in Figure 6B can be eliminated. When using a registration layer, one method for accurately matching the geometric deformation between the two is to use the texture of the skin. Since the skin has a fine spatial pattern, it is a useful feature for correcting minute misalignments. Therefore, if the skin features are enhanced using an image enhancement filter before being input into the registration layer, more accurate alignment can be achieved.

[0108] FIG. 6D shows another example of generating training data. While the above example shows how to convert a hand image into an ideal image, it is also possible to generate a finger ROI image by extracting an image for each finger with the finger angle and finger width normalized to a certain value and filling in the background, and then generate pairs of finger ROI images captured in different environments to generate training pair images for each finger image. Here, an example is shown in which a finger ROI image 124 taken indoors and a finger ROI image 125 taken by a window are paired. When extracting images by finger, registration is relatively easy compared to when extracting images for the entire hand and fingers, and because unnecessary image areas are not included, improved image quality and processing speed can be expected after conversion.

[0109] Similarly, an ideal biometric feature pattern extracted from a biometric image captured under an ideal environment may be determined in advance, and an original finger image captured under any environment and the ideal biometric feature pattern may be used as a training pair image so that this ideal biometric feature pattern can be obtained regardless of the capture environment. Here, an example is shown in which a finger vein pattern 126 extracted from a finger image captured indoors, which is an ideal environment, is paired with a finger ROI image 125 captured by a window. By setting such a pair as training data, an ideal biometric feature pattern can be directly obtained regardless of the capture environment, thereby enabling conversion to an ideal environment image and feature extraction processing to be performed at the same time, which also has the effect of reducing processing costs.

[0110] The method of estimating the ideal image from the entire hand image shown in Figure 6C paints out the background of the hand, but if the image of the window or blinds reflected in the background is left as is, information about the shooting environment will be included in the image, and it is possible to generate an ideal image with higher accuracy. One method for converting to an ideal image while retaining background information is to use CycleGAN. CycleGAN is a method that does not assume image matching on a pixel-by-pixel basis, and has the advantage of not requiring registration and allowing for relatively free collection of training data, so it may be used. [Example]

[0111] FIG. 7 shows an example of a processing flow of authentication processing in which the influence of wavelength changes in ambient light is reduced by normalizing the ratio of wavelength components of received light through spectral processing.

[0112] This processing flow is roughly the same as that shown in FIG. 2 in the above embodiment, so a description of the same processing will be omitted.

[0113] First, the image capturing device 9 captures an image of a living body at one or more white balance settings (S701). As in the above-described embodiment, the image capturing may be performed at, for example, three different white balances, or the image may be captured at one white balance using an automatic white balance setting, or at three or more different white balances.

[0114] Next, similarly to FIG. 2, the image correction unit 23 estimates and generates an image before color correction (S702).

[0115] Next, the image correction unit 23 performs spectral processing (S703), thereby converting the signal (pixel value for each color of B / G / R) received by an imaging element such as a CMOS sensor of a camera into light intensity for each wavelength.

[0116] Thereafter, the image corrector 23 performs hue conversion suitable for capturing each biometric feature (S704), and acquires the biometric feature from those images (S705).

[0117] As explained in the above embodiment, the hue conversion (S704) can be performed using the process (S203) in FIG. 2 and the embodiment in which color temperature is replaced with hue in the color temperature search shown in FIG. 4. Also, the acquisition of biometric features (S705) can be performed in the same manner as the process (S204) in FIG. 2. Next, the image correction unit 23 performs matching of each biometric feature (S706). Alternatively, the image correction unit 23 may instruct the matching processing unit 25 to perform matching and obtain the results.

[0118] The authentication processing unit 21 determines whether the matching score is below a preset authentication threshold (S707), and if it is below the authentication threshold, performs authentication processing (S710) and ends authentication. If not, a timeout determination (S708) is performed, and if a timeout occurs, authentication fails (S709) and the authentication processing ends, but if not, the process returns to capturing a biological image (S701). The following describes in detail the spectroscopic processing (S703), which is new to this embodiment.

[0119] 8 is a diagram showing an example of the wavelengths of various ambient light and the spectral sensitivity characteristics of the image sensor of a camera. The spectral processing (S703) in FIG. 7 will be described with reference to FIG.

[0120] First, we explain the background of spectral processing. The top and middle panels of Figure 8 show an example of a typical spectrum of ambient light seen in everyday life. For example, white LEDs used in indoor lighting and smartphone flashlights (torches) have a sharp blue peak around 450 nm and another peak around 580 nm, but the red component around 700 nm is somewhat weak. Similarly, fluorescent lights have peaks at multiple wavelengths. The sunlight spectrum varies depending on the time of day. Daytime sunlight has roughly uniform intensity across blue, green, and red, while evening sunlight has less blue but is more intense at longer wavelengths. As such, ambient light has various spectral characteristics. Changes in ambient light change the color of captured biometric images, which in turn changes the appearance of biometric features. For example, the epidermis of the palm of the hand, one of the biological tissues, is a mixture of melanin and blood, and appears to be a mixture of yellow and red due to the absorption characteristics of melanin and hemoglobin.

[0121] On the other hand, veins located in the subcutaneous tissue below the epidermis cannot be reached by blue or green wavelength components, so they reflect red, blue, and green in roughly equal amounts, appearing a slightly bluish gray. When red light is irradiated as ambient light, the colors of the epidermis and veins both turn red, eliminating contrast, making the vein pattern difficult to observe, especially in sunlight in the evening. As such, changes in ambient light are expected to cause changes in the contrast of biometric features, and fluctuations in ambient light can be a factor in degrading authentication accuracy.

[0122] In contrast, normalizing the color of the ambient light allows the color of the subject to remain constant regardless of differences in the ambient light spectrum. For example, assuming that the spectral intensity is uniform, including the color of the subject's biological part and the color of the ambient light, each color can be normalized by multiplying each color by a coefficient so that the average brightness of each R / G / B biological part in the captured color image is consistent. If the ambient light has a low blue component, the average brightness of the R component of the image will be low, so a coefficient will be applied to emphasize the R component, resulting in a uniform overall color component. In this case, although the spectrum of the ambient light is unknown, the color of the subject's biological features will be constant if it is the same person. Therefore, uniforming the R / G / B brightness for the image area where the biological part is observed will eliminate fluctuations in appearance due to changes in the ambient light spectrum.

[0123] However, when capturing images using a typical color camera, the R / G / B colors in the image do not necessarily correspond to physical wavelengths. A typical color camera has three types of optical elements: R / G / B, each with different sensitivity characteristics depending on the wavelength, as illustrated in Figure 8 as B light-receiving sensitivity characteristics 161, G light-receiving sensitivity characteristics 162, and R light-receiving sensitivity characteristics 163. When light of a certain wavelength is received, each of the R / G / B elements responds, and the color of the received light can be identified based on the ratio of these responses. However, even when capturing a single wavelength of blue light, the R light-receiving element, which primarily responds to red wavelengths, responds slightly, so the R brightness does not necessarily correspond to capturing a red wavelength. Similarly, capturing only red light increases the B brightness. Therefore, when capturing blue wavelengths, spectral processing is performed to more accurately handle wavelength components, so that only the B brightness has a value and the G and R brightness values ​​do not have a value. Spectroscopic processing can be performed by using a multispectral camera to separate a continuous spectrum, but a typical color camera only has three types of color receiving elements, so in principle it can only separate three wavelengths.

[0124] Therefore, in this example, we estimate the average intensity of multiple wavelength bands in an image containing color information before color correction. To enable spectral separation using three types of light-receiving elements in a color camera, three wavelength bands with a fixed bandwidth are defined, and the average spectral intensity of the three wavelength bands is obtained. In other words, the wavelength band of 400 nm to 700 nm, which can be captured by a typical visible light camera, is roughly divided equally, with the 400 to 490 nm wavelength band designated as blue wavelength band 164, the 490 to 580 nm wavelength band designated as green wavelength band 165, and the 580 to 700 nm wavelength band designated as red wavelength band 166. Considering that smartphone torches are always lit, the wavelength of the light source must be set to avoid incorrect spectral separation. Generally, white LEDs have a wavelength peak at 450 nm and a valley around 490 nm. In this case, if the boundary between the blue wavelength band 164 and the green wavelength band 165 is set to coincide with the valley of the peak of the blue wavelength component of the LED, even if there is variation between models and the wavelength peak shifts, the blue wavelength component of the LED will not be mistakenly dispersed as green, so here the boundary between the blue and green wavelength bands is set to 490 nm. Similarly, the boundary between the green and red wavelength bands is set to 580 nm so that the intensity of the white LED will be roughly the same.

[0125] Next, we will explain one example of spectral processing. Here, the signal intensities of three types of light receiving elements in a color camera are converted into average spectral intensities in the three wavelength bands mentioned above. First, let us assume that the B / G / R pixel values ​​of an image, which are the signal intensities of the light receiving elements, are FB, FG, and FR, the average intensities in the blue / green / red wavelength bands are Ib, Ig, and Ir, and the wavelength band average spectral sensitivity characteristic of light receiving element k when it receives light in wavelength band j is Wjk. Then, we can write it as follows: [FB FG FR] T = [[Wbb Wbg Wbr] T [Wgb Wgg Wgr] T [Wrb Wrg Wrr] T ] * [Ib Ig Ir] T

[0126] This is expressed as a matrix as follows: F = W * I

[0127] Here, to calculate the wavelength band average intensity I, the inverse matrix W of the spectral response characteristic matrix is -1 In other words, it can be written as follows: I = W -1 * F

[0128] This conversion makes it possible to obtain the blue, green, and red wavelength components, which are the average intensity of each wavelength band. This allows us to estimate the average intensity of multiple wavelength bands in the color information image before color correction. However, Wbb and nine other parameters are calculated using the average value of each wavelength band on the color camera's spectral sensitivity curve in Figure 8. For example, the spectral sensitivity of element B in the blue wavelength band varies smoothly between 30% and 45%, but the average value 167 of the light receiving sensitivity characteristic of element B in the blue wavelength band is assumed to be approximately 40%, so we set Wbb = 0.4 here.

[0129] In this way, the light intensities of the three wavelength bands can be calculated from the R / G / B luminance values ​​of the received image. Then, by converting these light intensities so that the ratios between them remain constant, the influence of the wavelengths of ambient light can be normalized. This normalization is performed by the level correction process for each color in the hue conversion process (S704) in Figure 7 described in the above embodiment.

[0130] It is also possible to roughly estimate the type of ambient light based on the spectral intensities of the above-mentioned wavelength bands. For example, daytime sunlight has a large difference in the intensity of the blue wavelength band, and if the intensity of the blue wavelength band is clearly stronger than the intensity of other wavelength bands, it can be determined that daytime sunlight is irradiating the subject. Similarly, if the intensity of the red wavelength band is clearly stronger than other wavelength bands, it can be determined that evening sunlight is irradiating the subject. In this way, it is possible to roughly estimate the type of ambient light based on the intensity ratio of each wavelength band. By utilizing this result, it is possible to change the selection of multiple white balance color temperatures when photographing living organisms, for example, thereby generating optimal image quality for each environment.

[0131] A specific example of the optimal hue conversion (S704) in FIG. 7 in this embodiment has been described above. As another example, instead of searching for the optimal hue while performing feature extraction and matching, a method can be used in which the hue of the image in the blue and green wavelength bands is changed so that it is most different from the image in the red wavelength band. Generally, light in the blue and green wavelength bands is diffusely reflected without reaching the veins located below the dermis, so the vein features are hardly included in the brightness information of these images. As described above, veins appear slightly bluish gray because the red component is relatively darker than the blue. However, this is only true for images with specific hues. By freely changing the hue, the vein parts can be converted to appear bright blue or green, or the red of the vein parts can be converted to appear darker. However, because the location of the veins is unknown, it is not possible to determine in advance which parts should be hue converted to what color. Therefore, by taking advantage of the fact that neither blue nor green wavelength band images contain vein information in terms of physical wavelengths, we consider consolidating vein features into the red wavelength band image by adjusting the hue so that the red and blue, and red and green images are most different.

[0132] Hue conversion is a correction that rotates three colors, so by adjusting the hue in various ways, it is possible to include vein information in the blue and green images, or to eliminate veins from the red image. For example, if the hue is rotated so that veins are slightly visible not only in the red image but also in the blue image, some of the vein information in the red image will be missing, and the veins in the red image will not appear as clearly as possible.

[0133] Similarly, if the hue is adjusted so that veins are present in the red and green images but not in the blue image, calculating the dissimilarity between the red and green images and the red and blue images reveals a moderate similarity between the red and green images and a large dissimilarity between the red and blue images. On the other hand, if the hue is set so that veins are present only in the red image, it can be assumed that vein features are almost absent in the blue and green images. In other words, comparing the red image with the blue and green images should clearly differ significantly in appearance. In this case, both the dissimilarity between the red and blue images and the dissimilarity between the red and green images are considered maximized. Therefore, by calculating the dissimilarity between the red and blue images and the dissimilarity between the red and green images and adding them together, and adjusting the hue to maximize this sum, veins can be observed only in the red image, thereby concentrating vein information in the red image. In this state, the red image exhibits the highest contrast in the vein pattern, stabilizing vein pattern extraction. Furthermore, since the veins are removed from the remaining blue and green images but the epidermal features remain, features such as articular marks can be extracted from these images.

[0134] This method allows two different biometric features to be separated and extracted, improving authentication accuracy. In the above-described embodiment, a method was shown in which a vein pattern was emphasized by subtracting a blue or green image from the red image. However, in this case, the veins contained in the red image are already emphasized, eliminating the need to optimize the subtraction parameter settings, making this a method robust to environmental fluctuations. Note that normalized cross correlation (NCC) and zero-means normalized cross correlation (ZNCC) can be used to calculate the dissimilarity. In particular, because images in each wavelength band may have different average brightness levels, a dissimilarity calculation method that is robust to brightness fluctuations, such as the latter, is effective.

[0135] If it is assumed that biometric features have different distributions for each color, then, for example, independent component analysis (ICA) can be performed on the three color information within the finger area to convert it into new image components distinct from the R / G / B planes. This converts the image into three independent color image planes—for example, hemoglobin, melanin, and veins—and allows the biometric features of each plane to be used for authentication. However, biometric features obtained through ICA do not necessarily correspond to commonly known biological tissues. The advantage of using independent biometric features is that they complement each other, so even if authentication fails with the first biometric component, the second or third biometric component may be successful, potentially improving overall authentication accuracy. [Example]

[0136] FIG. 9 shows an example of a processing flow of authentication processing including guidance for reducing the influence of unnecessary ambient light.

[0137] First, the user follows the guidance displayed on the screen and holds their fingers over the guide (S901).

[0138] Thereafter, the image correction unit 23 performs a biological image capturing process and a finger region detection process (S902). Thereafter, the image correction unit 23 performs a spectroscopic process (S903), and then performs an external light component detection process and an external light component irradiation direction estimation process (S904). Then, the image correction unit 23 determines whether external light is present (S905).

[0139] If there is external light, the image correction unit 23 controls the display unit 15 to display guidance to the user that external light is being irradiated (S906), and the process returns to the living body image capturing process (S902).

[0140] At this time, if the user checks the guide indicating that external light is being irradiated, he or she can photograph the biometric image while avoiding the external light. If the external light component is smaller than the threshold, the authentication process continues, and the image correction unit 23 performs a biometric feature extraction process (S907).

[0141] The authentication matching unit 21 performs matching processing (S908), and determines whether the matching score is lower than a certain value (S909). If it is lower, the authentication is deemed successful and the authentication processing (S912) is performed and terminated. If not, the system appropriately checks whether a timeout has occurred (S910) and returns to the finger photography processing (S902). If authentication is not completed by the timeout, the authentication fails (S911) and terminates.

[0142] Note that the specific processing methods of each process in this embodiment can be appropriately used as described in the above-described embodiments, and therefore a description thereof will be omitted. The following describes new processes, namely, the process of detecting external light components and the process of estimating the irradiation direction by the image correction unit 23 (S904), the process of determining whether or not there is an external light component (S905), and the display of guidance for external light (S906).

[0143] FIG. 10 is a schematic diagram showing an example of a method for detecting a luminance gradient due to external light irradiated onto a finger, as an example of the external light component detection process and external light component irradiation direction estimation process (S904) shown in FIG. 9 by the image correction unit 23.

[0144] A user holds their fingers over a camera to perform biometric authentication. First, biometric photography and finger detection are performed as described above, and spectroscopic processing is performed using the method described above to obtain a spectroscopic image. Next, each detected finger is converted into a finger ROI image. As described above, the finger ROI image is an image in which the finger is cut out into a rectangle according to the finger orientation, with the finger angle and finger width normalized and background information filled in. Here, assume that indoor finger ROI image 124 is detected from indoor hand image 121, and similarly, windowside finger ROI image 125 is cut out from windowside hand image 122.

[0145] Next, three image cross sections of the finger ROI image are set for each finger in a direction perpendicular to the center of the finger. By selecting these three locations - the center of the finger, the center between the fingertip and the center of the finger, and the center between the base of the finger and the center of the finger - it is possible to confirm the overall brightness profile trend of the finger while minimizing the number of areas to be processed. Here, image cross sections 201a, 201b, and 201c of the finger ROI image indoors and image cross sections 202a, 202b, and 202c of the finger ROI image by a window are set as shown in the figure.

[0146] Next, for the three image cross sections set above, luminance profiles for each of the blue, green, and red wavelength bands are extracted, and the average luminance profiles are calculated for each wavelength band and aggregated into a single average luminance profile for each wavelength band. The average slope angle of the average luminance profile for each wavelength band and the average luminance around the finger contour are then measured, and the maximum average slope angle is determined for each wavelength band. The maximum average slope angle is the angle of greatest slope obtained by linearly approximating the luminance profile and calculating the slope from horizontal for each wavelength band. Figure 10 shows the maximum average cross-sectional luminance profile 205 on the indoor average image cross-sectional axis, which yielded the maximum average slope angle among the average cross-sectional luminance profiles calculated for the blue, green, and red wavelength bands for indoor finger ROI image 124, plotted on average image cross-sectional axis 203, and also shows maximum average slope angle 207. Similarly, for finger ROI image 125 near a window, maximum average cross-sectional luminance profile 206 is plotted on average image cross-sectional axis 204, and the maximum average slope angle 208 is also shown. In this case, because the wavelengths of external light vary, such as being strongly blue or red, it is effective to process each wavelength band separately and obtain the maximum value. In the example of Figure 10, assume that a luminance gradient occurs due to light near a window during the day. In this case, the intensity of the blue wavelength band increases, so the maximum average cross-sectional luminance profile 206 shown in the figure is in the blue wavelength band. Since the components of other wavelengths are weak, no gradient of this magnitude occurs. In this way, by using the maximum result among the wavelength bands, if a luminance gradient is detected because the maximum intensity at each wavelength exceeds a predetermined threshold, it can be determined that the image is affected by external light.

[0147] When a luminance gradient occurs due to external light, the maximum average gradient angle increases. Therefore, if the absolute value of the luminance gradient angle is greater than a certain value, it is determined that external light is being irradiated, and guidance is provided to the user. The direction of external light irradiation can be roughly estimated based on the positive or negative gradient angle. The image correction unit 23 displays the intensity and direction of external light determined to be affected by external light on the display unit 15. For example, if the luminance gradient direction from the horizontal direction in FIG. 10 is determined to be a positive angle downward, it can be determined that strong light is being irradiated from the upper side of the drawing. Therefore, by displaying on the display unit 15 that external light is being irradiated from that direction, the user can take action to mitigate the effects of external light. Generally, since the luminance gradient is significantly affected by the direction of external light, it is possible to display only the direction of external light, or the intensity of external light together with the direction.

[0148] Fig. 11 shows another example of estimating the irradiation direction of the external light component by the image correction unit 23. Fig. 10 above shows a method for estimating the irradiation direction of external light, but it was only possible to determine a rough direction, such as whether it was upward or downward. Therefore, this section shows a method for detecting the direction of external light more accurately by measuring the brightness of the fingertip, taking advantage of the fact that the fingertip has a three-dimensional spherical surface.

[0149] First, as shown in Figure 11(a), a finger 1 including the fingertip is observed in the finger ROI image 221. The contour near the fingertip generally forms a gentle semicircle. Therefore, the center and radius of the circle that best fits the contour near the tip of the fingertip are determined, for example, using a Hough transform, to obtain a circular fingertip region 222 as shown in Figure 11(b). This circular fingertip region is then divided into, for example, eight equal parts as shown in Figure 11(c), resulting in fingertip divided regions 223. The average brightness values ​​within each divided region are then calculated for each of the blue, green, and red wavelength bands. It can be seen that the wavelength band with the highest total average brightness value is the wavelength most affected by external light. Therefore, for that wavelength band, the difference in average brightness values ​​obtained between a certain divided region 223 and the opposing divided region 224 is calculated. For example, as shown in FIG. 11(d), when external light 225 is irradiated from the position shown in the figure, the average brightness of divided area 223 tends to be high, and the average brightness of the opposing divided area 224 tends to be low due to being in shadow.

[0150] On the other hand, the difference in luminance between the other divided areas and the divided areas facing each other is not so large. Therefore, when the difference in luminance between all divided areas and the divided areas facing each other is calculated, it can be determined that the external light 225 is on the extension line of the divided area where the difference in luminance is greatest.

[0151] By using the above method, it is possible to determine whether a luminance gradient is occurring due to external light and the direction in which the external light is being irradiated.

[0152] In this embodiment, the determination of the intensity of external light and the determination of the direction of external light are performed separately. However, it is desirable to perform the determination of the intensity of external light in the finger area from which the biometric features are actually extracted, while it is effective to use the three-dimensionally rounded area of ​​the fingertip to detect the direction of external light. Therefore, in this embodiment, the two are separated.

[0153] In addition, methods for estimating external light intensity and direction can be implemented using methods other than those described above, such as deep learning. As an example, a large number of biometric images, both those without and those with external light gradients, are collected, and training data is constructed by assigning correct labels indicating whether or not an external light gradient exists and the external light direction, ranging from 0° to 360° clockwise. However, if there is no brightness gradient, any external light direction can be set. Then, a typical convolutional neural network (CNN) is input with three pieces of information: image information, the presence or absence of an external light gradient, and the external light direction. The output is designed to obtain two pieces of information: the presence or absence of an external light gradient and the external light direction. Training is then performed to ensure that the given image information and labels match the training data. However, when training with images without external light gradients, inconsistencies in the external light direction are not an issue, and are not included in the loss function during training.

[0154] As a result, when any biometric image is input, the presence or absence of external light and the direction of the external light can be obtained. Brightness gradients caused by external light often occur, for example, near windows, but by using CNN to learn the direction of external light based on the image features, it is possible to determine the direction of external light using the image conditions of the surrounding environment as clues, enabling more accurate determination.

[0155] Another method for estimating the direction of external light involves determining the shadow boundary line separating the bright and dark areas of the fingertip. As described above, external light creates a shadow on the fingertip. However, the direction of the shadow boundary line changes smoothly depending on the direction of the external light. Therefore, by determining the line direction, it is possible to determine that a light source is present in a direction perpendicular to the line. One specific example of how to determine the shadow boundary line involves first determining a circle that fits the fingertip, as described above, and then bisecting the circle with a line passing through the center of the circle at an arbitrary angle. The average luminance of one semicircular region and the average luminance of the other semicircular region are then calculated to determine the difference between the two. The angle of the line at which the absolute value of the difference is maximized when the angle is varied is obtained. Since this angle coincides with the angle of the shadow boundary line, it can be determined that external light is present in a direction perpendicular to this direction and in the direction of the bright region of the semicircular region. This method allows for more robust determination even with finer angle adjustments than the above-described method, thereby enabling more accurate detection of the direction of external light.

[0156] If you are only interested in estimating the position of sunlight near a window, it is possible to determine the direction of the sun without using image information by using, for example, the GPS, magnetic sensor, acceleration sensor, gyro sensor, and time information installed in a smartphone. This makes it possible to detect brightness gradients caused by sunlight and provide guidance to the user, and it can also determine the possibility of backlighting, where the subject appears crushed due to the background being too bright, thereby preventing image quality degradation caused by backlighting.

[0157] Figures 12A, 12B, 12C, and 12D show an example of guidance provided when external light is detected during authentication. As shown in Figure 12A, when biometric authentication begins, a camera image 243 and a finger guide 244 are superimposed on the screen of a smartphone 242, and a message "Please hold your hand over the screen" is displayed on the guidance display 245 to prompt the user to hold their hand over the screen. When the user captures their finger 1 in the camera, biometric image capture and authentication processing begins as described in the above embodiment. As shown in Figure 12B, the guidance display 245 displays a message indicating that authentication is being performed, such as "Capture in progress." If external light is irradiated from the upper left of the drawing, a luminance gradient 247 due to the external light will appear on the finger 1, and authentication accuracy will deteriorate unless special measures are taken. Therefore, using the methods shown in Figures 10 and 11, the presence or absence of a luminance gradient due to external light irradiation is determined. If a luminance gradient is detected, the direction of the luminance gradient is detected, and a message indicating that external light has been detected and the direction of the luminance gradient is displayed to the user. For example, as shown in FIG. 12C, an external light mark 246 is displayed so as to overlap the display portion of a camera image 243.

[0158] In this embodiment, a symbol representing the sun and an arrow are displayed to indicate the presence and direction of external light. Furthermore, the guidance display unit 245 displays a message such as "External light detected. Avoid external light" or "External light is shining from the direction of the arrow." In response to this, as shown in FIG. 12D, the user 241 recognizes the presence of external light 225, which can adversely affect authentication, and can perform authentication in an appropriate environment by turning their back to the external light 225 when photographing their body or by moving to a location without external light. As a result, highly accurate authentication can be achieved without being affected by ambient light.

[0159] As a variation of brightness gradients and brightness unevenness on a living body, a shadow cast by ambient light on the living body being photographed can also be considered. Shadows can occur, for example, due to the light and dark patterns caused by blinds hanging near a window, or when the user or their smartphone partially blocks external light. To detect such shadows, the brightness distribution within the finger region of the finger ROI image described above can be acquired, its variance calculated, and determined to be greater than a preset value. If this variance is greater than a certain value, it indicates that bright and dark regions are distributed within the finger region, indicating the occurrence of a brightness gradient or brightness unevenness. If this condition is detected, a guidance message such as "Shadow detected. Please move away from external light" can be displayed, prompting the user to change the shooting environment.

[0160] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to facilitate a better understanding of the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0161] 1 finger 2 Input devices 3 light source 9 Camera 10 Authentication processing device 11 Central Processing Unit 12 Memory 13 Interface 14 Storage device 15 Display section 16 Input section 17 Speaker 18 Image input unit 20 Registration processing section 21 Authentication processing section 22 Imaging control unit 23 Image correction section 24 Feature Extraction Unit 25 Matching processing unit 26 Authentication Judgment Unit 27 Biometric Information Processing Unit 121 indoor hand images 122 Image of hands and fingers at the window 123 Hand Images with Shading 124 indoor finger ROI images 125 Finger ROI image at the window 126 Finger Vein Pattern Extracted from Indoor Finger Images 161 Light receiving sensitivity characteristics of element B 162 Light receiving sensitivity characteristics of element G 163 Light receiving sensitivity characteristics of element R 164 Blue wavelength band 165 green wavelength band 166 Red Wavelength Band 167 Average value of the light receiving sensitivity characteristics of element B in the blue wavelength band 201a Image cross section of finger ROI image indoors 201b Image cross section of finger ROI image indoors 201c Image cross section of finger ROI image indoors 202a Image cross section of finger ROI image at window 202b Image cross section of finger ROI image at window 202c Image cross section of finger ROI image at window 203 Image cross-sectional axis averaged over indoor luminance profiles 204 Image cross-sectional axis averaged over the brightness profile near the window 205 Maximum average cross-sectional luminance profile on the average image cross-sectional axis indoors 206 Maximum average cross-sectional brightness profile on the average image cross-sectional axis at the window 207 Maximum average tilt angle of finger ROI images indoors 208 Maximum average tilt angle of finger ROI image at window 221 Finger ROI Images 222 Fingertip area 223 Fingertip Segmentation Area 224 Opposite division areas 225 Pleinair 241 User 242 smartphones 243 camera images 244 Finger Guide 245 Guidance display 246 Plein Air Mark 247 Brightness gradient due to external light 1000 Biometric Authentication Systems 282 Image cross section 283 Cross-sectional brightness profile of the image 284 Biological Contrast Detection Probe 285 Biometric Feature Enhanced Images 286 Joint wrinkles

Claims

1. A biometric authentication device having a storage device that stores biometric feature amounts, an image capturing device that captures a biometric image, an image correction unit that converts the image quality of an image captured by the image capturing device, and an authentication processing unit that performs biometric authentication using the image output by the image correction unit, The image correction unit generating a color-corrected image of one or more images of a living body; converting the pre-color-corrected image with a plurality of values ​​within a predetermined search range for color information to generate a plurality of color information-converted images; selecting, from the plurality of color information converted images, a color information converted image that best represents each biometric feature; Searching for optimal color information for each biometric feature based on the color information for obtaining the selected color information converted image. Biometric authentication device.

2. The biometric authentication device according to claim 1, The image correction unit A color-uncorrected image is generated from a plurality of images captured by the image capture device using a generative adversarial network. Biometric authentication device.

3. The biometric authentication device according to claim 2, The image correction unit The search range is narrowed based on the color information for obtaining the selected color information converted image, thereby searching for optimal color information for each biometric feature. Biometric authentication device.

4. The biometric authentication device according to claim 3, The image correction unit calculating a biometric feature matching score by matching the color information converted image with the same finger and a different finger; searching for an optimal value of color information of the color information converted image in which the matching score of the same finger and the matching score of the different finger are most separable; Biometric authentication device.

5. The biometric authentication device according to claim 3, The image correction unit The color information that maximizes the evaluation value of the contrast of the biological information of the color information converted image is set as the optimal value. Biometric authentication device.

6. The biometric authentication device according to claim 3, The image correction unit The conversion of the color information is a conversion of the color temperature. Biometric authentication device.

7. The biometric authentication device according to claim 3, The image correction unit The conversion of the color information is a conversion of hue. Biometric authentication device.

8. The biometric authentication device according to claim 3, The image correction unit Correcting brightness gradients caused by non-biometric features in the image. Biometric authentication device.

9. The biometric authentication device according to claim 3, The image correction unit Estimating the average intensity of a plurality of wavelength bands in the uncorrected image. Biometric authentication device.

10. The biometric authentication device according to claim 3, The image correction unit changing the hue of an image of average intensities of a plurality of wavelength bands in the image before color correction; determining a degree of difference between the image in the first wavelength band and the image in the second wavelength band, and a degree of difference between the image in the first wavelength band and the image in the third wavelength band; Calculating the sum of the two obtained dissimilarities; selecting the hue that maximizes the sum of the dissimilarity values ​​as the optimum value; Biometric authentication device.

11. The biometric authentication device according to claim 3, Further comprising a display unit, The image correction unit Calculating the direction of external light irradiating the living body to be photographed by the photographing device; The direction of external light is displayed on the display unit. Biometric authentication device.

12. The biometric authentication device according to claim 11, The image correction unit Calculating the intensity of external light irradiated onto the living body to be photographed by the photographing device; The presence or absence of external light and the direction of the external light are displayed on the display unit. Biometric authentication device.

13. A biometric authentication method using a biometric authentication device having a storage device that stores biometric feature amounts, an image capture device that captures a biometric image, an image correction unit that converts image quality of an image captured by the image capture device, and an authentication processing unit that performs biometric authentication using the image output by the image correction unit, The image correction unit generating a color-corrected image of one or more images of a living body; converting the pre-color-corrected image with a plurality of values ​​within a predetermined search range for color information to generate a plurality of color information-converted images; selecting, from the plurality of color information converted images, a color information converted image that best represents each biometric feature; Searching for optimal color information for each biometric feature based on the color information for obtaining the selected color information converted image. Biometric authentication methods.

Images