Biometric authentication system and method

Abstract

To provide an authentication system and method for achieving high-accuracy and high-speed multimodal biometric authentication while suppressing postural fluctuation when multiple biological bodies are photographed and suppressing computational complexity for processing a plurality of biometric features.SOLUTION: In a biometric authentication system 1000 comprising a memory 12 storing, for each user, feature quantities of multiple biological bodies in association with each other, and a camera 9 for imaging the biological bodies, and configured to perform biometric authentication using input images by the camera, the camera images a first biological body of a first user in a first period and images a second biological body and a third biological body of the first user in a second period which is different from the first period. An authentication processing apparatus 10 calculates a first feature quantity from the first biological body, calculates a second feature quantity and a third feature quantity from the second biological body and the third biological body, respectively, and compares the feature quantities for the biological bodies of each user stored in the memory with the first feature quantity, the second feature quantity, and the third feature quantity, to authenticate the user.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] The present invention relates to an authentication system and an authentication method for authenticating an individual using biometric information. [Background technology]

[0002] In recent years, biometric authentication technology has been used as a means to reduce the risk of information leakage and unauthorized use of mobile terminals such as smartphones and laptops. In particular, mobile terminals used in remote environments are at high risk of being used unauthorizedly by others. For this reason, it is necessary to authenticate the user each time they access a terminal or information system. However, entering a password every time is troublesome, and there is also the risk of passwords being forgotten or leaked, so there are an increasing number of cases where simple and reliable biometric authentication is being introduced.

[0003] Cashless payments are also becoming more common in convenience stores and other retailers, as well as restaurants. Cashless payments are highly convenient as they do not require the hassle of paying cash on the spot, and by linking them with various point services, they can encourage customers to make purchases, so there are also many benefits to introducing them for stores. If biometric authentication is used for such cashless payments, there is no need to carry cards, and the identity of the person can be confirmed with certainty, providing a convenient and effective service.

[0004] Thus, although there are many benefits to introducing biometric authentication in preventing unauthorized access and enabling cashless payments, the need for additional dedicated biometric authentication devices increases implementation costs, making it difficult for the system to become widespread.

[0005] Therefore, if biometric authentication could be performed using images of the body captured by general-purpose cameras installed in smartphones, laptops, etc., the barrier to introducing biometric authentication could be lowered. Furthermore, if the authentication operation could be performed without contact, the risk of the spread of infectious diseases, which is a recent social problem, could be reduced, and it is believed that biometric authentication could be introduced and used with peace of mind.

[0006] Biometric identity authentication involves holding a user's own biometrics, such as a finger, hand, or face, over an authentication terminal and verifying their identity through matching with previously registered biometric information. Login and payment are only permitted if the user is authenticated as a registered user. Among the various biometric authentication technologies, biometric authentication based on internal characteristics of the body, such as finger veins, is known to be able to achieve high accuracy. Finger vein authentication achieves high authentication accuracy by using the complex vein patterns inside the finger, and is more difficult to forge and tamper with than fingerprint authentication, achieving high security.

[0007] Compared to biometric authentication using a dedicated sensor specialized for capturing biometric images, biometric authentication using a general-purpose camera is more difficult to capture images under favorable conditions, and the degradation of image quality tends to result in lower authentication accuracy. To compensate for this deterioration in accuracy, multimodal biometric authentication technology is effective, which improves authentication accuracy by using multiple pieces of biometric information. Basically, it is relatively easy to capture images simultaneously, and by combining multiple biometric features that have low correlation with each other or low correlation in the capture conditions, it is possible to effectively improve authentication accuracy by having each biometric feature play a complementary role.

[0008] As one of the multimodal biometric authentication methods using a general-purpose camera, a method has been proposed in which the biometric information of the face and fingers is held over the front camera. Conventional methods include a method in which the face is photographed first and then the fingers are photographed, and a method in which the face and fingers are photographed simultaneously by separately setting positions for holding the face and the fingers over the front camera.

[0009] The former tends to take a long time to capture images, while the latter has the difficulty of holding the body of the body over the screen. In particular, when using the latter, the face may be tilted as a result of looking into the screen to place the finger in the correct position, or the finger may block the face.

[0010] In order to realize multimodal biometric authentication in which multiple biometric devices are held up at the same time, it is necessary to provide authentication technology that is less susceptible to the effects of changes in the posture of the biometric device or occlusion.

[0011] Furthermore, since the use of multiple biometric data tends to require long calculation times, it is desirable to perform authentication processing as quickly as possible.

[0012] A multimodal authentication technology that simultaneously performs face and fingerprint matching is disclosed in Patent Document 1. Also, a technology that uses face and fingerprint for authentication while measuring the flatness of a face image is disclosed in Patent Document 2. [Prior art documents] [Patent documents]

[0013] [Patent Document 1] JP 2009-20735 A [Patent Document 2] JP 2004-62846 A Summary of the Invention [Problem to be solved by the invention]

[0014] In multimodal biometric authentication, where multiple biometric features are presented simultaneously, there are cases where it is difficult to capture good quality biometric features due to posture changes or occlusions, resulting in degradation of authentication accuracy. In addition, there is also the issue of long authentication times due to the need to process multiple biometric features.

[0015] Patent Document 1 discloses a technology that has a first phase in which face matching is performed using face data, and a second phase in which finger matching and face photographing are performed, and in the second phase, detects the face orientation (up, down, left, right) of the face image while photographing the face image in parallel with the finger matching process. However, Patent Document 1 uses the photographing of the face image in the second phase for gesture judgment rather than face authentication, and does not mention a technology that solves the problems of improving the accuracy and speed of multimodal authentication.

[0016] Patent Document 2 discloses a technology in which a photographing light is turned on toward the face when a finger is placed on a fingerprint sensor, the face is photographed by a camera, each feature amount of face authentication and each feature amount of fingerprint authentication are categorized in the same category, and personal authentication is performed using a minimum distance identification method, etc. Patent Document 2 discloses a viewpoint of implementing highly accurate multimodal biometric authentication using face and fingerprint, but in addition to the necessity of a fingerprint sensor, there is no mention of a technology for improving accuracy by suppressing posture variation.

[0017] The above-mentioned problems are not limited to multimodal biometric authentication of face and fingers, but also apply to various other biometric elements such as iris, auricle, facial veins, subconjunctival blood vessels, palm veins, back of the hand veins, palm prints, knuckle prints on the inside and outside of fingers, veins on the back of the fingers, etc. Thus, in the conventional technology, in multimodal biometric authentication using various biometric elements, the biometric elements cannot be observed correctly due to posture changes or occlusions, which leads to a decrease in authentication accuracy.

[0018] An object of the present invention is to provide a biometric authentication system and a biometric authentication method that can achieve high-precision and high-speed authentication even when posture changes or occlusion occur during multimodal biometric imaging. [Means for solving the problem]

[0019] In a preferred example of the biometric authentication device of the present invention, in an authentication system including a storage device that stores a plurality of biometric features associated with each user, an image capturing device that captures a biometric image, and an authentication processing device that inputs an image captured by the image capturing device and performs biometric authentication using the input image, the image capturing device captures a first biometric image of a first user in a first period, and captures a second biometric image and a third biometric image of the first user in a second period different from the first period. The authentication processing device calculates a first feature from the first biometric image captured in the first period, A second feature and a third feature are calculated from the second biometric and the third biometric captured during the second period, respectively, and the user is authenticated by comparing the biometric feature of each user stored in the storage device with the first feature, the second feature, and the third feature. Effect of the Invention

[0020] According to the present invention, it is possible to realize highly accurate authentication even when posture changes or occlusions occur during capturing images of biometric data in multimodal biometric authentication in which multiple biometric data are simultaneously presented. [Brief description of the drawings]

[0021] [Figure 1A] FIG. 1 is a diagram showing an overall configuration of a biometric authentication system according to a first embodiment. [Figure 1B] FIG. 2 is a diagram illustrating an example of a functional configuration of a program stored in a memory according to the first embodiment. [Diagram 2] FIG. 1 is a schematic diagram showing a configuration of a multimodal biometric authentication device using a general-purpose front camera according to a first embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of a processing flow of a registration processing unit of the biometric authentication system according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a processing flow of an authentication processing unit of the biometric authentication system according to the first embodiment. [Figure 5A] FIG. 13 is an example of a screen transition diagram during authentication using multimodal biometric authentication technology that performs authentication by simultaneously guiding a face and fingers. [Figure 5B] FIG. 13 is an example of a screen transition diagram during authentication using multimodal biometric authentication technology that performs authentication by simultaneously guiding a face and fingers. [Figure 5C] FIG. 13 is an example of a screen transition diagram during authentication using multimodal biometric authentication technology that performs authentication by simultaneously guiding a face and fingers. [Figure 5D] FIG. 13 is an example of a screen transition diagram during authentication using multimodal biometric authentication technology that performs authentication by simultaneously guiding a face and fingers. [Figure 5E]FIG. 13 is an example of a screen transition diagram during authentication using multimodal biometric authentication technology that performs authentication by simultaneously guiding a face and fingers. [Figure 6A] FIG. 11 is an example of a screen transition diagram during authentication using a multimodal biometric authentication technique in which authentication is performed by simultaneously holding a face and a finger over the device, according to the first embodiment. [Figure 6B] FIG. 11 is an example of a screen transition diagram during authentication using a multimodal biometric authentication technique in which authentication is performed by simultaneously holding a face and a finger over the device, according to the first embodiment. [Figure 6C] FIG. 11 is an example of a screen transition diagram during authentication using a multimodal biometric authentication technique in which authentication is performed by simultaneously holding a face and a finger over the device, according to the first embodiment. [Figure 6D] FIG. 11 is an example of a screen transition diagram during authentication using a multimodal biometric authentication technique in which authentication is performed by simultaneously holding a face and a finger over the device, according to the first embodiment. [Figure 7A] FIG. 4 is an explanatory diagram of a method of buffering and selecting face feature amounts according to the first embodiment. [Figure 7B] FIG. 4 is an explanatory diagram of a method of buffering and selecting face and finger feature amounts according to the first embodiment. [Figure 7C] FIG. 4 is an explanatory diagram of a method for generating feature pairs based on face and finger feature amounts according to the first embodiment. [Figure 8A] FIG. 1 is an explanatory diagram showing an example of a multi-modal biometric authentication technique using a face and fingers according to a first embodiment. [Figure 8B] FIG. 1 is an explanatory diagram showing an example of a multi-modal biometric authentication technique using a face and fingers according to a first embodiment. [Figure 9] FIG. 11 is a diagram illustrating an example of a processing flow of an authentication processing unit of a biometric authentication system capable of matching a face and a finger separately according to a second embodiment. [Figure 10] FIG. 11 is an explanatory diagram showing an example of a multimodal biometric authentication technique in which face or finger processing is omitted according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0023] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0024] In addition, in the following description, processing performed by executing a program may be described, but the program is executed by a processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) to perform a defined process using storage resources (e.g., a memory) and / or an interface device (e.g., a communication port) as appropriate, so the subject of the processing may be the processor. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing a program may be a calculation unit, and may include a dedicated circuit that performs specific processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0025] In this specification, biometric features refer to biometric information that refers to anatomically different biological features such as finger veins, fingerprints, joint patterns, skin patterns, finger contour shapes, fat lobule patterns, finger length ratios, finger widths, finger areas, melanin patterns, palm veins, palm prints, dorsal veins, facial veins, ear veins, or the face, ears, and irises. EXAMPLES

[0026] 1A is a diagram showing an example of the overall configuration of a biometric authentication system 1000 using biometric features in this embodiment. It goes without saying that the configuration of this embodiment may be configured as an authentication device in which all or part of the configuration is mounted in a housing, rather than as an authentication system. The authentication device may be a personal authentication device including authentication processing, or may be a finger image acquisition device or finger feature image extraction device specialized in acquiring finger images, with authentication processing performed outside the device. Also, it may be an embodiment as a terminal.

[0027] A configuration that includes at least an imaging unit that captures biometric images and an authentication processing unit that processes the captured images and performs biometric authentication is called a biometric authentication device, and a configuration in which authentication processing is performed by a device connected to the imaging unit that captures biometric images via a network is called a biometric authentication system. Authentication systems include biometric authentication devices and biometric authentication systems.

[0028] 1A includes an input device 2 serving as an imaging unit, an authentication processing device 10, a storage device 14, a display unit 15, an input unit 16, a speaker 17, and an image input unit 18. The input device 2 includes an imaging device 9 installed inside a housing, and may also include a light source 3 installed in the housing. The authentication processing device 10 has an image processing function.

[0029] The light source 3 is, for example, a light emitting element such as an LED (Light Emitting Diode), and irradiates light onto the face 4 and finger 1 as a certain area of ​​the user's living body presented to the input device 2. The light source 3 may be capable of irradiating light of various wavelengths depending on the embodiment, may be capable of irradiating light transmitted through the living body, or may be capable of irradiating light reflected from the living body.

[0030] The imaging device 9 captures images of the finger 1 and face 4 presented to the input device 2. It is also possible to simultaneously capture images of living body parts such as the iris, the back of the hand, and the palm. The imaging device 9 is an optical sensor capable of capturing images of light of a single or multiple wavelengths, and may be a monochrome camera or a color camera, or may be a multispectral camera capable of capturing visible light as well as ultraviolet or infrared light at the same time. It may also be a distance camera capable of measuring the distance to a subject, or may be configured as a stereo camera in which multiple identical cameras are combined.

[0031] A plurality of such imaging devices may be included in the input device 2. Furthermore, the number of fingers 1 may be multiple, and may include multiple fingers on both hands at the same time.

[0032] The image input unit 18 acquires an image captured by the imaging device 9 in the input device 2, and outputs the acquired image to the authentication processing device 10. As the image input unit 18, for example, various reader devices for reading images (for example, a video capture board) can be used.

[0033] The authentication processing device 10 is composed of a computer including, for example, a central processing unit (CPU) 11, a memory 12, and various interfaces (IF) 13. The CPU 11 executes programs stored in the memory 12 to realize various functional units such as authentication processing.

[0034] FIG. 1B is a diagram showing an example of the functional configuration of a program stored in the memory 12 for implementing each function of the authentication processing device 10. As shown in FIG.

[0035] As shown in FIG. 1B, the authentication processing device 10 includes various processing blocks, such as a registration processing unit 20 that links an individual's biometric features to a personal ID and registers them in advance, an authentication processing unit 21 that authenticates the biometric features currently captured and extracted based on the registered biometric features and outputs an authentication result, a biometric detection unit 22 that performs, for example, detection of the position of a biometric feature and removal of unnecessary background from an input image, an image capture control unit 23 that captures the presented biometric feature under appropriate conditions, a quality determination unit 24 that determines the quality of the image of the biometric feature and the posture of the biometric feature, a feature extraction unit 25 that appropriately corrects the posture of the biometric feature during registration processing and authentication processing to extract biometric features, a matching unit 26 that compares the similarity of biometric features, and an authentication determination unit 27 that determines the result of authentication from the matching result of a plurality of biometric features. These various processes will be described in detail later. The memory 12 stores a program executed by the CPU 11. The memory 12 also temporarily stores an image input from the image input unit 18, etc.

[0036] The interface 13 connects the authentication processing device 10 to an external device. Specifically, the interface 13 is a device having ports for connecting to the input device 2, the storage device 14, the display unit 15, the input unit 16, the speaker 17, the image input unit 18, and the like.

[0037] The interface 13 functions as a communication unit for allowing the authentication processing device 10 to communicate with external devices via a communication network (not shown). The communication unit is a device that performs communication in accordance with the IEEE802.3 standard if the communication network 30 is a wired LAN, and is a device that performs communication in accordance with the IEEE802.11 standard if the communication network 30 is a wireless LAN.

[0038] The storage device 14 is composed of, for example, a hard disk drive (HDD) or a solid state drive (SSD), and stores user registration data, etc. The registration data is information obtained during registration processing for verifying users, and is stored in association with a plurality of biometric features for each user. For example, the registration data is image or biometric feature data such as facial features, finger features, finger vein patterns, etc., linked to a registrant ID as user identification information.

[0039] A finger vein pattern image is an image of the finger veins, which are blood vessels distributed under the skin of the finger, captured as a dark shadow pattern or a slightly bluish pattern.Finger vein pattern feature data is data obtained by converting an image of the vein portion into a binary or 8-bit image, or data consisting of feature values ​​generated from the coordinates of characteristic points such as the bends, branches, and ends of the vein or brightness information around the characteristic points, or data obtained by encrypting and converting the above data into an undecipherable state.

[0040] The display unit 15 is, for example, a liquid crystal display, and is an output device that displays information received from the authentication processing device 10, biometric posture guidance information, and posture determination results.

[0041] The input unit 16 is, for example, a keyboard or a touch panel, and transmits information input by the user to the authentication processing device 10. The display unit 15 may also have an input function such as a touch panel. The speaker 17 is an output device that transmits information received from the authentication processing device 10 as an acoustic signal such as voice.

[0042] 2 is a schematic diagram showing the configuration of a multimodal biometric authentication device using a general-purpose front camera described in this embodiment. Here, an example is described in which multimodal biometric authentication is performed using the biometric features of the face and the fingers of the left hand when a user logs in to a notebook PC.

[0043] When working on a notebook PC, a user activates an authentication function to log in to the PC. At this time, the user often stands in front of the notebook PC 41, and in a typical notebook PC 41, camera 9 is installed above display 42 to make it easier to capture the user's face 4. Camera 9 is installed so that it can capture the area near the front of display 42, that is, so that the user's face 4 and left hand 45 are captured in their entirety, and as a result, the user's face 4 is captured near the center of the angle of view of the image captured by camera 9.

[0044] The authentication system activates the camera 9 to capture the biometric features of the user, and in order to make it easier for the user to understand how to hold the biometric device, the image captured by the camera 9 is displayed on the display 42, and a preview image 47, which is an image in which a face guide 43 and a finger guide 44 are overlaid on the display 42, can also be displayed. However, in cases where there is no particular need to display the guide, such as when authentication can be performed by holding the biometric device in any position, the guide display may be omitted. By displaying the guide as necessary, the authentication operation for the user becomes easier, and the effect of improving convenience is obtained.

[0045] The user aligns the face 4, left hand 45, and fingers 1 with the displayed guide positions while looking at the preview image 47. At this time, a guide message 46 may be displayed on the preview screen to clearly inform the user that each biometric information should be presented. When the authentication system detects that the biometric information has been presented, it performs authentication based on a plurality of biometric features, and if it determines that the user is a pre-registered user, it transitions the notebook PC 41 to a login state. Specific methods of registration and authentication will be described in detail below.

[0046] 3 and 4 are diagrams showing an example of the schematic flow of an enrollment process and an authentication process of a multimodal biometric authentication technique using a plurality of biometric features described in this embodiment.

[0047] The registration process and authentication process are realized, for example, by a program executed by the CPU 11 of the authentication processing device 10. Note that, in this embodiment, the description is based on the premise that the face and four fingers of the left hand of the user are photographed, but the face may be a partial feature rather than the entire face, and the fingers may be one finger or any number of fingers, or multiple fingers on a hand, and any number of types of biometric features other than the face and fingers that are generally widely known, such as irises, veins, fingerprints, palm patterns, etc., may be used.

[0048] First, the flow of the registration process in FIG. 3 will be described.

[0049] The registration processing unit 20 is started by the user's instruction for registration processing, and first, the authentication system displays a preview image on the display unit 15 to explain to the user that biometric information will be registered (S301). The displayed preview image allows the user to understand the flow of the registration process. The preview image explains the flow of the registration process to the user by showing the ideal way to hold the face and four fingers over the screen, the procedure for holding the fingers over the screen, and a sentence such as "First, take a picture of your face, then hold the four fingers of your left hand over the screen." This can reduce errors in the registration operation.

[0050] Also, to allow the user to visually confirm that he or she is being photographed by the camera, the current camera image is displayed as a preview image on the display unit 15. The display may be performed on the entire screen, or may be a small image on a part of the screen. At this time, the camera image is displayed in a left-right inverted manner so that the left and right of the user and the left and right of the image match, making it easier for the user to hold his or her own body over the image.

[0051] Next, face detection processing is performed (S302), which is a pre-processing for registering facial features. As an example of this, a deep neural network that has previously learned the relationship between a face image and the position of the face and the positions of facial organs (facial parts or landmarks) is prepared, and a facial image is input to this network to acquire a rectangular face region (bounding box) including the face or a face ROI image (ROI: Region of Interest). Facial landmarks are composed of, for example, the center of the eyes, the tips of the inner and outer corners of the eyes, the edges of the eyelids (eyeliner), the tip of the nose, the left and right corners of the mouth, the center position between the eyebrows, and the like, and a rectangle that includes these can be defined as a facial region. In this example, facial biometric features are registered without the user being aware that their face is being photographed as a biometric feature. When logging in to a PC as shown in FIG. 2, the user is usually positioned in front of the PC, and facial biometric features can be registered without the user being aware of it. However, to let the user know that face detection is being performed, a bounding box of the face may be displayed superimposed on the preview image, or a guide such as "Please face forward" may be displayed. Furthermore, a face guide imitating the outline of the face may be displayed so that the user can visually understand that the current phase is to capture a face and where to place the face.

[0052] Next, the photographing control unit 23 is started to perform face photographing control, in which the face is photographed while appropriately adjusting camera parameters such as the exposure time, gain, white balance, and focus of the camera (S303). Here, the exposure time of the camera is adjusted so that whiteout or blackout does not occur inside the detected face ROI image, and the focus of the camera is adjusted so that the focus of the camera is on the face.

[0053] In addition, the white balance of the entire image is automatically adjusted based on a method such as the gray hypothesis, which assumes that the average color of the entire image is the color of the ambient lighting. The exposure time may be adjusted by adjusting the exposure time of the camera, but it may also be adjusted by software, such as by weighting and integrating the pixel values ​​of multiple consecutive image frames. Software exposure adjustment methods have the advantage that the exposure of the image can be partially corrected, so that, for example, the face and multiple fingers of the hand can be optimally corrected independently.

[0054] Next, the acquired face ROI image is normalized (S304). Examples of normalization include enlarging or reducing the face so that the size (area of ​​the face ROI, etc.) is constant, generating a pseudo frontal face by perspective projection transformation to correct the face orientation to face forward, and multiplying by a constant so that the brightness of the face becomes a constant value. This normalization is a preprocessing performed to stabilize the results of facial feature extraction performed in a later stage.

[0055] Next, facial feature extraction is performed (S305). As an example of facial feature extraction, there is a method of acquiring a feature vector from a facial ROI image by using a deep neural network that inputs a facial ROI image and outputs an arbitrary fixed-length feature vector, and learning the deep neural network so that the L1 distances between feature vectors obtained from a plurality of facial ROI images of the same face are minimized and the L1 distances between feature vectors obtained from facial ROI images of different faces are maximized.

[0056] According to this, features obtained from the same face image have a small L1 distance from each other, while features obtained from different face images have a large L1 distance from each other, so it becomes possible to evaluate whether face images are the same or not by the distance (dissimilarity) between patterns. Although L1 distance is described here, any distance space can be used, and distance learning between features is generally performed to classify the person himself / herself from others. Specific widely known methods include a method that uses Triplet Loss and a method such as ArcFace, which can achieve distance learning simply by learning a general class classification problem.

[0057] Next, the quality value of the face image is calculated (S306). As an example of a specific method of calculating the quality value of the face image, each item of the face size, brightness of the face area, face direction, facial expression, face movement speed, and temporal fluctuation of the face feature amount extracted from the face image is quantified, and the quality value is determined based on the weighted sum of these. The face size is the size of the face ROI image described above, and if this is small, it is determined that the face was not photographed at a sufficient size, so the quality value is reduced.

[0058] In addition, the brightness of the face is calculated from the average brightness of the face ROI image before normalization, and if it is darker than expected, or too bright, or if there are many blown-out pixels, the quality value can be determined to be low.

[0059] In addition, the face orientation is estimated by estimating the three-dimensional rotation angle based on the landmark positions that occur in an average frontal face with respect to the above-mentioned facial landmarks. If the weighted sum of the pitching rotation angle, rolling rotation angle, and yawing rotation angle of the face is close to 0, the quality is determined to be high, and if the value is large, it is determined that the face is not a frontal face and the quality value is low.

[0060] In addition, it is considered ideal for facial expressions to be expressionless, and common machine learning methods are used to calculate factors such as the degree of smile, with the quality value increasing the more expressionless the face.

[0061] Regarding the speed of facial movement, if the center point of the face ROI in the preceding and succeeding frame images moves significantly, the face is not stationary and the quality value is determined to be low. Regarding temporal fluctuations in facial features, as described above, facial features are extracted from the face ROI image, and the similarity of the facial features in the preceding and succeeding frame images is determined in a brute force manner, and if there is a large variation, the extracted features are deemed unstable and the quality value of the facial image is determined to be low.

[0062] Each of these evaluation items is quantified, and the values ​​are summed up with weighting to obtain the final quality score for the face image.

[0063] As another method, a deep neural network that inputs multiple face ROI images arranged in chronological order and outputs an arbitrary scalar value can learn to output a low scalar value (e.g., 10) when a time-series image of face ROI in which the distance between the facial features of the same face tends to be large is input, and output a high value (e.g., 1000) when the distance tends to be small, thereby acquiring a high value when a time-series image of face ROI that is likely to be successful in authentication is input. The value acquired in this way becomes higher as it is more suitable for authentication, so it can be used as a quality value. This method has the advantage that it is not necessary to manually list each evaluation item as described above, which increases development efficiency, and it is easy to increase the correlation between the quality value and authentication success.

[0064] In addition, the evaluation items of the face quality value may include an item for detecting blinking. Before capturing a face image, a guidance such as "Please blink several times" is displayed and a certain period of time is set for blinking. If blinking is detected during that period, the quality value is increased as the face is genuine, and if not, the quality value is decreased as the face is a fake such as a printed matter. This makes it possible to reject fake face images that at least do not blink as having a low quality value.

[0065] Next, a decision is made as to whether or not to register the face (S307). The quality value of the face image is a criterion for determining whether the image quality and face posture are suitable for registration and authentication, so this quality value can be used to determine whether the currently acquired face features are suitable for registration. For this reason, a predetermined threshold value is set for the quality value of the face image, and if this threshold is exceeded, it can be determined that registration is possible.

[0066] However, if the quality value accidentally increases and the data is registered, authentication may become unstable. Therefore, it may be determined that the facial features can be registered if they continuously exceed a predetermined threshold value, or if the quality value of the current facial image is integrated over time and exceeds a certain value.

[0067] In this case, multiple features will have face quality values ​​that exceed a predetermined threshold, and as an example of a method for determining the final enrollment data, there is a method of selecting the one with the highest quality value, or a method of enrolling the one with the lowest total degree of dissimilarity when comparing each feature in a brute force manner. The latter method has the advantage of improving authentication accuracy, since it selects the feature that can be most stably authenticated on average even if there are fluctuations in the image quality of the face image or the posture of the face.

[0068] Next, it is determined whether or not face registration data has been acquired by the registration determination (S308). If the registration data could not be uniquely determined in the previous registration determination, the process starts again from face detection. If the face registration data has been determined, it is temporarily stored in a memory or the like and the process moves to the next step of finger registration. Although not shown, a timeout period may be set for the registration process, and if face registration data still cannot be acquired after the time period has elapsed, the process may transition to registration failure (S321) and exit the process.

[0069] Next, finger registration is performed. First, as shown in Fig. 6C, a finger guide is displayed on the screen as an overlay on the preview image, which is the image captured by the user's camera (S309). While checking the finger guide and the preview image of the user displayed on the screen, the user places their left hand over the finger guide. At this time, a guide message such as "Place your left hand over" may be displayed.

[0070] Next, finger detection processing is performed (S310). In the finger detection processing, first the finger is separated from the background, and then a finger ROI image is obtained by cutting out the finger on a finger-by-finger basis. One example of background separation processing is a method in which a deep neural network that receives a camera image of a finger held up as input and outputs a finger mask image in which only the finger area is 1 and the rest is 0 is trained to output a finger mask image correctly for every input image, and the finger mask image is obtained by the network to mask (remove) the background.

[0071] In addition, as a method for acquiring ROI images of fingers by cutting them out on a finger-by-finger basis, fingers can be cut out into rectangles based on deep learning that can extract finger landmarks such as the fingertips, bases, and joints, similar to the facial landmark detection method described above.

[0072] Then, finger photography control is performed (S311). Here, the processing is the same as the face photography processing described above, except that the area of ​​the finger ROI image is controlled to have appropriate brightness, and the white balance and focus are controlled to be appropriate, so a description thereof will be omitted.

[0073] Next, the finger images are normalized (S312). As an example of normalizing the finger images, the finger thickness and three-dimensional inclination may be corrected to a constant value based on perspective projection transformation, or may be corrected based on the landmarks implemented in the face normalization as described above. For example, as a posture correction process for normalizing the thickness and orientation of all detected fingers, an image is generated that includes the fingertip point of each finger and two points of the finger crotch on both sides, rotates so that the central axis of the finger is parallel to the horizontal axis of the image, and scales so that the finger width of each finger is a constant value. This unifies the orientation and thickness of the fingers on the image reflected in the ROI image of all fingers.

[0074] Next, the feature extraction unit 26 is started to extract the features of the finger (S313). The feature extraction of the finger can be performed in the same manner as the feature extraction of the face described above. Note that, as the feature amount of the finger, the finger vein, fingerprint, joint pattern, epidermal pattern, melanin pattern, fat pattern, etc. may be extracted independently, or they may be mixed.

[0075] As another feature extraction method, biometric features such as line pattern features of epidermis and blood vessels and spot features of fat lobules can be emphasized by filtering using a general edge enhancement filter, Gabor filter, matched filter, etc., and the result can be binarized or ternarized to acquire biometric features. Alternatively, the biometric features may be acquired by a method of extracting brightness gradient features from key points such as SIFT (Scale-Invariant Feature Transform) features. In any case, any feature that can extract biometric features from an image and calculate the similarity between them may be used.

[0076] Next, a quality value of the finger image is calculated (S314). One example of calculating the quality value of the finger image is a method of performing finger posture detection to extract information on the fingertips, bases, and widths of multiple fingers from a finger image, and judging whether the finger posture at that time is appropriate. In finger posture judgment, evaluation items include whether the finger is in an appropriate position and whether the finger is stationary for a certain period of time, by confirming that the finger is not significantly deviated from the displayed finger guide based on the result of finger posture detection.

[0077] As an example of finger rest detection, it is sufficient to confirm that finger posture information such as the position of the fingertip does not change over time. Since it is difficult to completely stop a finger, it may be determined that the finger is at rest when it is within a certain range of movement. However, if the finger is still not at rest or the finger is too far away (the finger is far from the camera and the hand looks small), or the posture is not appropriate, a guide to that effect is displayed, and although not shown in the figure, the process may return to the process (S309) of prompting the user to present the finger again.

[0078] In addition, a data suitability determination may be performed to determine whether the pattern extracted during this process is appropriate and whether the photographed finger is not a foreign object or a counterfeit. If the result of this determination is inappropriate, the quality value is significantly reduced so that the finger is not selected as a candidate. As an example of the data suitability determination process, if a highly continuous pattern, such as a blood vessel pattern, cannot be extracted despite being a line feature, or if a strong edge that is not observed in a real finger is observed in the original image, the pattern extraction may be failed or a counterfeit may be input and the data may be rejected. Alternatively, a method may be used in which pulsation of image brightness accompanying changes in blood flow in the finger is detected from a moving image, and if no pulsation is detected, the data may be rejected.

[0079] Next, a finger registration determination is made (S315). This determination is made based on the quality value of the finger image as described above, and the method for this determination can be the same as the method based on the quality value of the face image described above.

[0080] Then, by repeating these steps (S309 to S315), it is determined whether or not the registration candidates have been accumulated three times (S316), and if they have been accumulated three times, a registration selection process is performed (S317). Note that the number of registration candidates is not limited to three.

[0081] One example of the registration selection process is a method in which the feature data of three registration candidates are compared in a brute-force manner to calculate the similarity between each candidate, and the one registration candidate with the highest total similarity with the other two candidates is selected as the registration data. This method improves authentication accuracy because the most stable feature data that is most likely to be reproduced among the three images is registered.

[0082] However, if the similarity between the selected registration data and the other candidates is not recognized as the same pattern, all three registration candidates are regarded as unstable biometric features, and the registration data is not determined. Then, it is determined whether one feature data suitable for registration has been determined (S318), and if so, the finger feature data at that time and the facial feature data acquired in the previous stage are linked to the enrollee ID entered by the enrollee at the start of the enrollment process and stored in the storage device 14 as the enrollment data of the biometric feature (S319). On the other hand, if the feature data has not been determined, the process is repeated until a timeout occurs (S320), and if a timeout occurs, a report is issued regarding the failure of the registration and the process is terminated (S321).

[0083] Next, a description will be given of the flow of the authentication process in Fig. 4. The authentication process is a process in which a user who has already registered his / her personal biometric characteristics through a registration process (basically, it is assumed that the user is the same throughout the authentication process) is authenticated by the biometric authentication system 1000 as the registered user.

[0084] The authentication process involves capturing an image of the biometric data presented by the user, extracting the biometric features, and comparing them with each feature data in the registered data. If there is registered data that can identify the person, the authentication success result and the registered user ID are output; if there is no registered data that can identify the person, an authentication failure notification is output.

[0085] The authentication processing unit 21 is started by an instruction for authentication processing by a user, and a preview image indicating that authentication has started is displayed (S401). For example, "Starting to photograph your left hand." The displayed preview image allows the user to understand the sequence of steps in the authentication processing.

[0086] For authentication, the face and hands are photographed, but as described above in Figure 2, since the user is usually positioned in front of the device, such as a laptop computer, the face can be photographed without the user being particularly aware of the photograph being taken, and so the user can be guided to only photograph the left hand. This allows the user to prepare in advance to hold their left hand over the screen, allowing for smooth biometric authentication. Also, a preview image from the camera is displayed, just like in the registration process.

[0087] Next, for a predetermined period of time, face detection processing (S402) to face image quality value calculation (S406) are performed. These processing steps are similar to the face detection processing (S302) to face image quality value calculation (S306) in the registration processing in FIG. 3, so a description thereof will be omitted.

[0088] Then, the facial feature amount having a quality value of the facial image exceeding a certain predetermined standard is recorded in the memory 12 (buffer) of the authentication processing device 10 (S407). Then, the process up to this point is repeated for a certain period of time (S408). Note that the display of the above-mentioned guide text may be continued during this process.

[0089] This series of processes (S402 to S408) is a process related to photographing a face alone, and is referred to here as a "face alone photographing phase" as the first period.

[0090] Although the face-only photographing phase is set to a fixed time here, this phase may be ended when a fixed number of high-quality facial feature values ​​are collected in the buffer, or when a face is successfully compared with registered data. However, the advantage of setting a fixed time is that if the loop is repeated until a high-quality facial feature value is obtained, it may take too long to get through this phase if photographing is performed in an environment where the quality value is not easily increased, resulting in a delay in authentication. In contrast, limiting this phase to a fixed time has the advantage of making it less likely to be delayed. Furthermore, if the time to get through the phase changes depending on the quality value, the quality value may be guessed, which may be exploited to create counterfeits, so it is also advantageous to be able to prevent this.

[0091] Next, for example, as shown in Fig. 6C, a finger guide is displayed (S409) to encourage the user to place their finger over the image, and then face and finger detection processing (S410) is performed to calculate the quality values ​​of the face image and finger image (S414). These processing steps are almost the same as the registration processing in Fig. 3 or the face-only photographing phase in Fig. 4, and the only difference is that processing is performed on the biometric features of both the face and fingers, so a description thereof will be omitted here.

[0092] Next, the facial feature amounts are buffered (stored) in the memory 12 (buffer) of the authentication processing device 10, and a selection process of the stored facial feature amounts is performed (S415). Here, similar to the registration process of Fig. 3, facial feature amounts that have reached a certain quality value or higher are buffered, and facial feature amounts to be used in the matching process performed later are selected from the buffer.

[0093] An embodiment of the selection method will be described in detail later with reference to Fig. 7. In any case, if it is possible to preferentially select facial features in the buffer that are more likely to lead to successful authentication, it is possible to achieve successful authentication at an early stage.

[0094] In addition, among the buffered facial features, those captured in the face-only capture phase tend to have a relatively forward-facing posture, as described below, and because this is the stage before the hand is held up, there is less chance of missing facial images due to occlusion, so there is an advantage that a higher quality facial image is more likely to be selected than if a facial image taken when the hand and face are held up at the same time is used.

[0095] Thereafter, it is confirmed that the quality values ​​of both the face and the fingers are sufficiently high (S416), but if either quality value is lower than a certain value, the process is repeated from the face and finger detection process (S410) again, and is repeated until face and finger feature values ​​suitable for use in the authentication process are obtained. This repetitive period is called the "multi-photographing phase" as the second period. The face feature values ​​in the "single face photographing phase" and the "multi-photographing phase" are calculated at different times for the same area of ​​the same user. The finger feature values ​​in the "multi-photographing phase" are calculated for a different area from the face of the same user.

[0096] The facial feature values ​​in the "single face shooting phase" and "multiple shooting phase" are values ​​obtained from the images of the face. For example, if the captured images are a video of 30 frames per second, 30 facial feature values ​​can be calculated per second. The same applies to the finger feature values ​​in the "multiple shooting phase".

[0097] When it is confirmed that the quality values ​​of both the face and finger images are high, the facial features and finger features are fused, and the matching unit 26 is started to sequentially match the authentication data of the facial features and finger features acquired by the process shown in FIG. 4 with one or more registered data items (usually, it is assumed that multiple registered persons are registered) previously registered in the storage device 14, and obtain a matching score (S417).

[0098] The matching process may be performed by internally separating the finger feature amount and the face feature amount, calculating a matching score as the degree of difference between each registered feature amount, and finally obtaining the result of weighting and summing up the matching scores, or by matching the finger feature amount and the face feature amount with the registered data without separating them. Also, a match with the registered data may be confirmed based on information obtained by converting the biometric feature amount into an encrypted feature amount using, for example, PBI (Public Biometric Infrastructure) technology. The matching score may be a scalar value or a vector value, and may be a binary or multi-valued value.

[0099] Finally, authentication is determined based on the calculated matching score (S418). As an example of the determination, a determination level fusion determination is performed based on AND determination, in which a matching score for the face feature amount alone and a matching score for the finger feature amount are obtained, and if both of the matching results (dissimilarity) are below a threshold value that indicates a similarity with the registered person, it is finally determined that the person is the registered person.

[0100] This determination method requires that both biometric characteristics are similar to those of the registered person, and has the effect of reducing the false acceptance rate, in which unregistered people are mistakenly determined to be registered people.

[0101] Similarly, a decision level fusion judgment based on OR judgment can be used, in which if either the face or finger matching score falls below a threshold where similarity with the registered person is recognized, the person is determined to be the registered person. In this case, it is sufficient for authentication to be successful using either biometric data, so it is possible to reduce the false rejection rate, in which the person is mistakenly rejected.

[0102] These can be set arbitrarily according to the security policy of the authentication system. Alternatively, a score level fusion judgment in which the matching scores of each biometric are linearly combined may be used, or the matching scores of each biometric may be treated as a two-dimensional matching score vector, a threshold boundary hyperplane may be defined in a multidimensional space, and if the matching score vector is included in an area in which it can be determined that the person is the person in question, it may be determined that the data is similar to the registered data. In particular, the method of treating the data as a vector allows the authentication threshold to be set flexibly, and if there is a correlation between the matching scores of each biometric, a boundary according to the correlation can be defined, making it possible to realize a highly accurate authentication judgment.

[0103] In the finger matching process, the similarity between the extracted skin feature and vein feature and registered data (skin feature and vein feature of one piece of registered data) may be calculated.

[0104] Finally, if it is determined that the data is similar to the registered data, the authentication success result and the registered user ID linked to the corresponding registered data are output (S419); if not, the authentication attempt continues by repeating the multi-photography phase until the authentication process times out (S420); if the timeout occurs, a notification that authentication with all registered data has failed (a notification that the registered user could not be authenticated) is output and the authentication process ends (S421).

[0105] Note that, although the above description is of 1:N authentication in which a unique registrant is determined from multiple registrants, it goes without saying that the authentication may also be configured as 1:1 authentication in which a registrant ID is specified in advance before authentication and then identity of the registrant is verified.

[0106] The authentication judgment (S418) may be performed based on a requirement that the data match the registered data consecutively. In this case, the authentication is not successful even if the data matches the registered data in the first authentication judgment, but is successful when a predetermined number of matches or a consecutive number of matches are confirmed. This makes it possible to prevent false acceptance errors, in which a false person is accidentally authenticated as a false person, and to achieve stable and highly accurate authentication.

[0107] In this embodiment, the face and finger features are fused as in S417, but a method of matching the face alone and the finger features alone may also be used, and this method will be described later with reference to FIG. 9. The advantage of fusing both features is that it is possible to increase the strength against attacks using a large number of forged face images and finger images. If an authentication system is designed to be able to match the face features and the finger features individually, the matching results of each can be confirmed individually, so that if forgeries that have been successfully used to attack each of them are used simultaneously, the multimodal authentication attack will also be successful. On the other hand, if the features are fused so that they cannot be matched individually, the multimodal authentication attack can only be successful if both are successfully attacked at the same time, so it becomes necessary to try countless combinations of the two, which results in a combination explosion and increases the difficulty of the attack. In order to suppress such fraud, a feature level fusion method in which both features are fused and then matched is effective.

[0108] Figures 5A-E are examples of screen transition diagrams during authentication in multimodal biometric authentication technology that guides the face and fingers simultaneously for authentication. Here, we will explain a typical example of authentication failure that occurs when, unlike the example shown in Figure 4 above, the face and fingers are guided simultaneously without a face-only photographing phase and face image buffering is not performed.

[0109] 5A shows a preview image 47 immediately after a user launches an authentication screen to log in to a laptop PC. As described above, the user is often positioned in front of the terminal, and camera 9 mainly captures images near the front of display 42, so that user's face 4 is captured near the center of the captured image.

[0110] 5B, face guide 43 and finger guide 44 for holding the face and hand over the screen at the same time are displayed. To guide the user to hold the face and left hand over the screen at the same time, finger guide 44 is displayed on the left side of the screen and face guide 43 is displayed on the right side of the screen, and guide message 46 or the like is displayed to guide the user to hold the living body parts over the screen within the frames.

[0111] First, the user places his / her face in the correct position while checking his / her own face 4 and face guide 43 in preview image 47 as shown in FIG. 5C.

[0112] Next, the user holds his / her left hand 45 over finger guide 44 as shown in Fig. 5D. At this time, the user holds his / her left hand in front of the face, but the left hand may block the view and make it difficult to see the screen. Therefore, to ensure a clear view, the user may tilt his / her head and turn face 4 to the side as shown in Fig. 5D. In this case, the face may not be detected correctly, or the facial features may change because the face is photographed in a different position from when it was registered, making it difficult to recognize the face as registered.

[0113] Or, as shown in Fig. 5E, to secure visibility, the user may look at the screen through the fingers of the left hand, but in this case the face is hidden by the hand, making it impossible to capture a correct facial image. Also, although the color of the fingers is often similar to the color of the face, if the face is directly behind the fingers, the boundary between the fingers and the face becomes unclear, which makes it difficult to detect the fingers, and in some cases makes it impossible to perform finger detection accurately. In either case, it becomes impossible to capture a facial image of at least the same quality as the registered facial image.

[0114] As described above, multimodal biometric authentication technology for faces and hands, which only involves the phase of holding the face and hand over the sensor at the same time, has had the issue of degradation in authentication accuracy due to holding multiple biometric sensors over the sensor at the same time.

[0115] Figures 6A-E are examples of screen transition diagrams for face and finger multimodal biometric authentication technology including a face-only photographing phase, as shown in Figure 4. Here, we explain the example of logging in to the same laptop computer as Figure 5 above.

[0116] First, as shown in Fig. 6A, when a user opens an authentication screen to log in to a notebook PC, the authentication system starts up the camera 9 to capture a biometric image. The image captured at that time is presented to the user as a preview image 47, similar to Fig. 5 described above.

[0117] Next, as shown in FIG. 6B, a guide message 46 is displayed for a certain period of time as a preview image 47 to indicate that authentication will be performed. The certain period of time is, for example, one second, and can be set to any number of seconds. At this time, the user does not need to perform any operation to align the face or hand with the guide, but this is the face-only photographing phase shown in FIG. 4, and the authentication system photographs the face, extracts features, and calculates the quality value in the background. As described above, in the face-only photographing phase, the user is positioned in front of the laptop computer and is not holding up his or her hand. Therefore, the face is not tilted intentionally, and is not blocked by the hand. Therefore, a high-quality face image can be captured without the user being aware of any operation.

[0118] After a certain period of time has passed, the system transitions to the multi-photographing phase. As shown in Fig. 6C, a finger guide 44 for holding the left hand is displayed. Since the purpose of the multi-photographing phase is to capture a high-quality image of the left hand, the face guide may be omitted in this embodiment.

[0119] The user holds their left hand in the correct position while checking their own image and the guide for the left hand, as shown in Fig. 6D. As mentioned above, since the hand is held in front of the face, it may be difficult to see the guide screen, and the user may tilt their head sideways to secure the field of view blocked by the hand. However, since a relatively high-quality face image is captured and buffered in the face-only capture phase, variations in face posture during this phase do not have a significant impact.

[0120] Finally, multimodal biometric authentication is performed by combining the face image in the buffer with the finger image captured in the multiple capture phase, and authentication can be performed with high-quality features from both. Of course, if the face image captured in the multiple capture phase exceeds a certain quality value, authentication processing can also be performed using the face image captured in the multiple capture phase. In other words, for face images, the face images captured in the "single face capture phase" and the "multiple capture phase" can be used as information for authentication, and by increasing the amount of data that can be compared with the registered authentication data (face features), high-precision and high-speed authentication processing can be achieved.

[0121] Therefore, in this embodiment illustrated in Fig. 4 and Fig. 6, even if the inclination of the face changes when the hand is held over the camera, the authentication is more likely to succeed without being affected by the change, and even if multiple biometric features are held over the camera at the same time, the user can concentrate on holding the hand over the camera, so that a highly accurate and convenient multimodal biometric authentication can be provided. Another advantage of dividing the phase into a face-only photographing phase and a multi-photographing phase is that the hand is generally held over the camera in front of the face. In this case, the distances of the hand and the face from the camera are different, so that it may be difficult to focus on both the hand and the face at the same time due to the focus control of the camera. Therefore, as in this embodiment, the face is focused on in the face-only photographing phase, and the hand is focused on in the multi-photographing phase, so that both biometric features can be optimally photographed. In addition, in order to photograph multiple biometric features simultaneously with optimal focus, not only the face and the hand, multiple images with different focal points may be photographed in a short period of time, and the blur of the entire image may be corrected by estimating a PSF (Point Spread Function) from the difference in the degree of blur between the images, and an all-focus image in which all subjects are in focus may be generated. This allows multiple biometric features to be captured clearly.

[0122] In this embodiment, the face-only phase transitions to the next multi-photographing phase after a certain period of time, but the face may be individually compared with registered face data in the face-only phase, and transition to the next phase may occur when it is confirmed that the face matches the registered face data. Also, transition to the next phase may occur when a predetermined number of face images exceeding a predetermined quality value are collected.

[0123] In this case, the next phase will occur when it is determined that facial recognition will definitely be successful or when the possibility of success has increased, which has the advantage of shortening the shooting time while at least preventing any deterioration in the accuracy of facial recognition.

[0124] However, as mentioned above, a drawback of the method of matching a single face is the possibility of information used to create a counterfeit being leaked. Therefore, by always setting up a face-only phase of a certain period of time so that the user cannot guess whether face authentication has been successful or not, for example, when multiple forged face images are presented, it becomes impossible to infer from the behavior of the authentication system whether authentication will be successful with a single face, making counterfeiting more difficult. As for which method should be adopted, any method can be adopted depending on the security policy of the authentication system.

[0125] As shown in FIG. 6C, only the finger guide 44 is displayed in the multi-photographing phase. In this case, the face guide may be displayed on the right side in another embodiment. In this case, the effect of moving the face to the right side can be obtained. In yet another embodiment, only the finger guide 44 may be displayed for a certain period of time, and the multi-modal authentication of the face and fingers may be repeated. If the authentication is not successful after a certain period of time, the face guide may be additionally displayed. In this method, since only the hand guide is displayed at first, the user can focus on aligning the position of the hand. On the other hand, if the face has not been photographed successfully and the multi-modal authentication is not successful, it is expected that the user will move the position of the face away from the position of the hand by displaying the face guide midway. As a result, the hand authentication may be more likely to be successful, or the possibility of successful authentication may be increased by increasing the variety of the face.

[0126] 7A to 7C are diagrams for explaining a method of buffering and selecting the facial feature amounts in FIG. 4 proposed in this embodiment.

[0127] FIG. 7A is a graph plotting the facial feature amounts and their quality values ​​in a time series in the face alone photographing phase, and diagrammatically illustrates the part corresponding to the processes in S405 to S408 in FIG.

[0128] The horizontal axis is time, and the vertical axis is quality value, showing the transition of the quality value when the facial feature amount at time t is Ft. A threshold value (high quality threshold value) that can be determined as high quality is also set, and features that exceed this threshold value are buffered. In this figure, facial feature amounts F3, F4, F5, F8, and F9 exceed the high quality threshold value, and it can be seen that the facial feature amounts F3, F4, F5, F8, and F9 are selected and stored in chronological order in the facial feature buffer 141 for the face single photographing phase.

[0129] Next, when the multi-photographing phase begins, two feature amounts, face and finger, are extracted as shown in the processing of S410 to S416 in FIG. 4. As shown in FIG. 7B, quality values ​​can be plotted in time series for both the face feature amount and the finger feature amount. In this embodiment, the face feature amounts F10, F14, F17, and F18 are assumed to be of high quality, and it can be seen how they are stored in the buffer 142 for the face feature amount in the multi-photographing phase. Although not shown in FIG. 7B, the face feature amounts F10, F14, F17, and F18 are also stored in the buffer 141 for the face feature amount in time series order.

[0130] Also, Ht indicates the finger feature amount, and here, H12, H15, H17, H18, and H19 are assumed to exceed the high quality value. In this embodiment, the buffers are explicitly divided according to the face alone and the multi-photographing phase, but it goes without saying that they may be managed in the same buffer. In this embodiment, no buffer is provided for the finger feature amount, but the finger may be buffered in the same way as the face, and the finger feature amount to be used may be selected according to the selection method described later.

[0131] Then, the selection process of the face and finger features is performed in S415 and S416 in Fig. 4. As shown in Fig. 7A and Fig. 7B, the face and finger features are not always of high quality, and only one of them may be of high quality, or both may be of high quality, or both may be of high quality. In this case, if the design is such that authentication is performed only when both are of high quality, there is no need to buffer the features, but on the other hand, in this embodiment, there are only two opportunities for authentication, when the time t is 17 and 18, as shown in Fig. 7B. This makes authentication more likely to fail.

[0132] Therefore, in this embodiment, as shown in FIG. 7C, face features and finger features are selected from the buffered features and combined, that is, a feature pair (fused feature) is generated, thereby increasing the number of authentication opportunities and performing authentication processing as early and as many times as possible, thereby increasing the success rate of authentication.

[0133] Here, a method for selecting feature pairs in this embodiment shown in FIG. 7C will be described.

[0134] First, when the quality value of a finger feature is higher than the threshold, it is always a candidate for selection. In addition, the method for selecting the paired face feature is to first select the face feature from the buffer for the face single shooting phase, and then select it from the buffer for the multi-shooting phase on the next occasion, and so on, by alternately selecting the buffers for the two phases. In addition, in each buffer, selection is made from the oldest to the newest. However, if the most recently selected feature is consecutive in chronological order, the feature of both features is highly likely to be similar, so the feature stored next is selected.

[0135] As shown in Fig. 7C, first, H12 is selected as the finger feature, and as the facial feature to be paired with H12, F3, which was saved the oldest in the buffer of the face-only photographing phase, is selected. First, at time t=12, the finger feature of H12 and the facial feature of F3 are paired and a matching process is performed. In other words, the matching process is performed based on the facial feature and finger feature of each user stored in the storage device using the feature pair.

[0136] If authentication is not successful, H15 will be selected as the next finger feature, but since the facial feature paired with this was previously selected from the buffer for the face-only shooting phase, it will be selected here from the buffer for the multi-shooting phase, and F10, which was saved the earliest, will be selected.

[0137] If authentication is still unsuccessful, H17 is selected at time t=17, but the facial feature value paired with this is selected from the buffer for the face-only photography phase, as the buffers are alternately switched. Since F3 was selected earlier, F4 could be a candidate if selection were made in chronological order, but as mentioned above, F3 and F4 are consecutive in time, so F5 is selected by skipping one. In other words, H17 and F5 are paired and matching processing is performed.

[0138] Similarly, the facial feature pair for H18 is selected from the multi-photography phase buffer, and since F10 was selected first, F14 is selected this time. Finally, H19 and F8 are selected as a pair.

[0139] An advantage of this method of alternately selecting facial features from the face-single shooting phase and facial features from the multi-shooting phase is that it is possible to increase the variety of facial features by incorporating not only facial images from the face-single shooting phase that have been buffered in the past, but also facial images from the current multi-shooting phase, in which the facial pose is expected to be different, thereby increasing the chances of successful authentication.

[0140] Similarly, by skipping features extracted from temporally consecutive frames among the buffered facial features, it is possible to increase the variation in features compared to using facial features with small changes, which is expected to increase the success rate of authentication and reduce the time required for successful authentication.

[0141] In this embodiment, the facial feature amount and the finger feature amount having different timings are combined and matched, but from the viewpoint of more robustly preventing attacks by counterfeits, a method of determining that authentication is successful only when multiple biometric features similar to the registered data are observed at the same time (existing simultaneously in the same image) can be considered. Therefore, for example, at the time when the finger feature H17 is obtained in FIG. 7C, a process may be performed to confirm that the facial feature F5 to be combined with the finger feature H17 is the same person as F17 (high similarity) by utilizing the facial feature F17 obtained at the same time. This allows a process to be performed such that if the facial image at the current time is replaced with the facial image captured in the face-only photographing phase, the authentication is not successful as fraudulent behavior, making it possible to provide a more secure authentication system.

[0142] In the above embodiment, the buffered facial features are arranged in chronological order, but they may be arranged in descending order of the quality value of the facial images and used in that order. In the registration, the facial images are selected so that the quality value is as high as possible, so that if the quality value of the facial image is high, it is highly likely that the image is similar to the registration data. Therefore, by fusing the facial features in descending order of quality value, the probability of successful authentication can be increased as early as possible. At this time, the acquisition time may be buffered as well as the feature and quality value, and if the feature to be currently selected and the feature selected last time are close in time as described above, the feature may be selected by skipping one. In addition, the facial features in the buffer may be mutually compared with each other, and features with high similarity may not be selected (thinned out). In addition, the quality value includes a parameter related to the face direction, but the face direction may be selected to have some variation. For example, after selecting an image in which the pitching angle of the face is slightly upward from the front, an image facing downward may be preferentially selected, or an image in which the yawing angle of the face is slightly leftward from the front and an image facing rightward may be alternately extracted. This makes it possible to comprehensively use features of different facial orientations for matching, thereby increasing the success rate of authentication at an early stage.

[0143] In either selection method, consecutive selection of facial features that are similar is avoided, and as early as possible a large variety of facial features can be used for authentication, which has the effect of increasing the success rate of authentication at an early stage.

[0144] When all the facial feature quantities in the buffer have been used, the facial feature quantities in the buffer may be reused again, starting from the top. At this time, if there are any feature quantities that have not been selected as described above, they may be used preferentially, or the number of times each feature quantity has been selected may be recorded, and the feature quantity that has been selected the least may be reused preferentially.

[0145] As described above, in the authentication process, the feature pair of the authentication data (facial features, finger features) to be matched with the registered facial features (facial features) and finger features (finger features) is obtained by combining the facial features acquired in the face-only photographing phase and the facial features acquired in the multi-photographing phase with the finger features acquired in the multi-photographing phase. This provides the above-mentioned effects. In particular, there is an effect of increasing the variation of the features, increasing the success rate of authentication, and shortening the time until successful authentication.

[0146] 8A-B are diagrams illustrating an example of a multimodal biometric authentication technique using a face and fingers in a mutual authentication manner.

[0147] As described above, in the multi-photography phase, the user holds up both their face and hand, and multimodal biometric authentication is performed through face detection and facial feature extraction, and finger detection and finger feature extraction. In this case, performance may be degraded by processing both the face and the fingers at the same time.

[0148] Therefore, in this embodiment, as shown in FIG. 8A, finger feature extraction is performed by skipping one frame. The horizontal axis represents the passage of time t, Ft and Ht represent face feature and finger feature, and it can be seen that H1 is used for the finger feature at time t=2 as in the case of time t=1. That is, at time t=2, finger detection and feature extraction are not performed, and the finger feature at time t=1 is used as is. In this embodiment, finger feature extraction is performed for a maximum of four fingers, and matching is performed by matching each finger one by one in a brute force manner, so that the processing time is longer than that of face matching. Therefore, the extraction processing of finger feature is simplified every other frame by utilizing the temporal locality that features close in time are unlikely to change significantly. This makes it possible to simplify the processing without reducing the authentication accuracy as much as possible, that is, to speed up the authentication processing.

[0149] As shown in Fig. 8B, not only the finger processing but also the face processing can be simplified to every other frame. As with Fig. 8A, the previous finger feature value is reused when the time is an even number, but the previous face feature value is reused when the time is an odd number. Since each is processed alternately, new feature pairs are always generated. This makes it possible to reduce the average processing time by half while always generating new feature pairs, and improve the perceived speed with almost no degradation in authentication accuracy. EXAMPLES

[0150] In the above-mentioned embodiment 1, a processing example is described in which a matching method based on feature level fusion that simultaneously uses both face feature amounts and finger feature amounts is adopted, but when face matching alone and finger matching alone can be performed independently, it is not necessary to generate feature pairs, and another method can be adopted. In embodiment 2, an embodiment in which each biometric element can be matched independently will be described.

[0151] Figure 9 shows an example of the processing flow of multimodal biometric authentication that independently matches a face and a finger. The above-mentioned Figure 4 shows an example of fusing face and finger patterns, but Figure 9 shows a processing flow when matching is performed on the face alone or on the finger alone, and the results are combined into a score level. Note that the registration process can be performed in the same way as in Figure 3, so a description of it will be omitted.

[0152] First, the steps from displaying the guide message encouraging the user to place their finger to calculating the quality value of the facial image (S901 to S906) are the same as those in Fig. 4, and therefore will not be described here. If the quality value is higher than the standard (S907), the facial feature amount alone is matched with the registered data, and the facial matching result at that time is stored (S908). The facial matching result may be a matching score indicating the similarity with the registered data, or may be a match or mismatch with the registered data determined by threshold processing of the matching score.

[0153] Next, a process of resetting the face matching result according to the expiration date of the face matching result is performed (S909). In this embodiment, the face matching result obtained at a certain time is held for a predetermined period of time, and the result is invalidated thereafter. Here, the certain period of time during which the result is held is called the expiration date, and invalidating the result is called resetting.

[0154] In multimodal biometric authentication, which is a method of matching multiple biometric features independently, the success or failure of matching can be obtained for each of the multiple modalities, but if matching is repeated until it is successful for all modalities at exactly the same time, authentication may take a long time. In contrast, by setting an expiration date for each matching result and always considering a successful matching within the expiration date, the possibility of authentication being successful for all modalities increases, which has the effect of increasing the authentication success rate and shortening the processing time until authentication. In this case, if a successful matching is made permanently valid, for example, if a different person accidentally succeeds in authentication in one modality, it becomes easy for false acceptance of a different person to occur if the result is permanently valid.

[0155] Therefore, results that have exceeded the expiration date are reset to invalidate them, preventing acceptance of other people. The expiration date can be a value of, for example, 0.5 seconds or 1 second. This process is repeated until a certain amount of time has passed (S910). This repetition will yield multiple matching results, but as mentioned above, all matching results within the expiration date are recorded. The loop from S902 to S910 is the face-only photography phase, as in FIG. 4.

[0156] Next, a loop of processing (S911 to S917) is entered to confirm that the quality values ​​of the face and fingers are sufficiently high from the display of the finger guide, and this processing is basically almost the same as that in FIG. 4. However, in this embodiment, processing for modals for which matching has been successful within the validity period at present is omitted. In other words, if it is already determined that the matching of the facial features is sufficiently similar to the registered data at this time, that is, that the face alone has been successfully authenticated within the validity period, face matching is omitted. Similarly, if a result that can be determined to be sufficiently similar to the registered data by matching the finger features in the loop exists within the validity period, finger matching can be omitted. In this way, the matching process is performed at high speed only for those of the facial or finger features that have not been confirmed to be sufficiently similar to the registered data, and each matching score is recorded in chronological order (S918), and each result is reset according to the validity period of the face and finger matching results as described above (S919), and authentication is determined by score level fusion using the matching score group obtained so far (S920). If the result of the determination is that the authentication is successful, the authentication success process is performed (S919, S921) and the authentication process ends. If the authentication is not successful, it is determined whether or not a timeout has occurred (S922), and if the timeout has not occurred, the process is repeated from the display of the finger guide, but if the timeout has occurred, the authentication failure process is performed (S923) and the authentication process ends.

[0157] As an example of authentication judgment by score level fusion using a matching score group, which is performed in process S920, there is a method of first extracting the matching scores with the smallest difference among the matching scores obtained in the past within the validity period for the face and the finger, and judging whether the score is below a predetermined authentication threshold for each of the face and the finger. For example, an AND judgment may be used in which authentication is successful only when both the face and the finger are below the threshold, or an OR judgment may be used in which authentication is successful when either one is below the threshold. Similarly, a score level fusion may be adopted in which the smallest scores are extracted respectively, and then a fusion score is obtained by multiplying the scores by a predetermined weight and taking a sum, and authentication is successful when the fusion score is below a predetermined threshold. Generally, score level fusion is preferable because it can achieve authentication with higher accuracy than AND judgment or OR judgment. Also, authentication may be successful only when the fusion scores are consecutively below the authentication threshold when arranged in chronological order, thereby suppressing accidental authentication acceptance errors by other people.

[0158] FIG. 10 is an explanatory diagram of an example of a multimodal biometric authentication technique using a face and fingers in a multimodal biometric authentication in which a face and fingers are independently matched.

[0159] If the method does not implement feature level fusion, matching can be performed for each biometric body alone, so a method different from that shown in FIG. 8 above can be implemented. First, from time t=1 to t=6, it is a face-only photographing phase, in which only facial features are extracted and matched. Here, it is assumed that the facial features F1 to F3 have low similarity to the registered data, and the similarity increases at the time of F4 (face matching is successful), and the above-mentioned expiration date is set to 3. Since a high similarity result is obtained at the time of F4, the current time t=4 and the expiration date 3 are added together, and this result is held until the time t=7. Therefore, it is possible to simplify the extraction and matching of facial features from F5 to F7 here. However, since the face-only photographing phase is limited to only the face, the original amount of calculation is small, so the processing of F5 and F6 may be performed. In that case, for example, if the similarity is high even at F6, the processing of facial features up to F9 in the subsequent multi-photographing phase can be omitted, so the amount of calculation in the subsequent stages can be further reduced.

[0160] Next, from time t=7, the multi-photographing phase begins, and processing is performed on the face and the fingers. However, as described above, the face matching success validity period is within the period up to t=7, so the extraction and matching of the face feature F7 can be omitted. Therefore, only the finger feature H7 is extracted and matched here. Here, it is assumed that the finger feature H7 has low similarity to the registered data, and the finger feature H8 is processed next. At the same time, the face feature F8 is also processed at the same time, because the face feature matching success validity period expires here. Here, it is assumed that the finger feature H8 has high similarity to the registered data, while the face feature F8 has low similarity. At this time, the finger feature is already similar to the registered data, so processing is omitted for three periods from time t=9 to t=11. On the other hand, the face feature F8 has low similarity, so extraction and matching are performed again for the face features F9 to F11. At this time, since it is possible to focus on processing only the face features during this period, the processing speed is improved. Here, it is assumed that none of F8 to F11 is similar to the registered data. Then, at time t=12, the validity period for the result that finger feature H8 had a high similarity to the registered data expires, and finger feature H12 is extracted and compared again. Then, F12 is extracted for the face. If only facial feature F12 has a high similarity to the registered data, then at time t=13, facial feature extraction is simplified and only finger feature H13 is extracted and compared. If it is confirmed that H13 is similar to the registered data, then facial feature F12 at time t=12 is similar to the registered data, and finger feature H13 is also similar. Since the similarity of F12 is within the validity period, it can be determined that the authentication was successful at time t=13 as both biometrics are similar.

[0161] In this way, when matching can be performed for each biometric feature individually, the processing can be simplified more effectively than when feature level fusion is performed, and the processing speed can be significantly improved. Note that the advantage of setting the expiration date is that even if a different person is accidentally authenticated successfully, this situation can be prevented from continuing forever, and false acceptance of a different person can be prevented.

[0162] The present invention is not limited to the above-mentioned embodiment, but includes various modifications. For example, the above-mentioned embodiment has been described in detail for a better understanding of the present invention, and is not necessarily limited to those including all of the configurations described. Also, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0163] 1 finger 2 Input Devices 3 light source 4. Face 9. Camera 10 Authentication processing section 11 Central Processing Unit 12. Memory 13 Interface 14 Storage device 15 Display section 16 Input section 17 Speaker 18 Image input section 20 Registration processing section 21 Authentication processing section 22 Biometric detection unit 23 Shooting control section 24 Quality Judgment Department 25 Feature Extraction Unit 26 Matching section 27 Authentication Judgment Unit 41 Notebook PC 42 Display 43 Face Guide 44 Finger Guide 45 left hand 46 Guide Message 47 Preview Images 141 Buffer of face feature values ​​during face-only photography phase 142 Buffering of facial features from multiple capture phases 1000 Biometric Authentication System

Claims

1. An authentication system including an image capturing device for capturing images of a living body, a storage device connected to the image capturing device for storing a plurality of biometric features for each user in association with each other, and an authentication processing device for inputting an image captured by the image capturing device and performing biometric authentication using the input image, The imaging device is capturing an image of a first biometric data of a first user during a first period; capturing an image of a second biometric data set and a third biometric data set of a first user during a second period different from the first period; The authentication processing device includes: Calculating a first feature amount from a first living body imaged during the first period; calculating a second feature amount and a third feature amount from the second living body and the third living body photographed during the second period, respectively; selecting a third feature amount having a quality value exceeding a predetermined value calculated during the second period; alternately selecting the first feature calculated in the first period and the second feature calculated in the second period from among the first feature calculated in the first period and the feature having a quality value exceeding a predetermined value for the second feature calculated in the second period, and generating a feature pair with the selected third feature; Using the feature pair, a matching process is performed based on the first biometric feature amount or the second biometric feature amount and the third biometric feature amount for each user stored in the storage device.

1. An authentication system comprising:

2. 2. The authentication system according to claim 1, the storage device stores user identification information, a first biometric feature, and a second biometric feature for each user in association with each other; the first biometric data and the second biometric data captured by the image capturing device during the first period and the second period are of the same region of the same user; the second feature calculated by the authentication processing device is a feature of a region of the same user that is different from the first biometric feature; The authentication system according to claim 1, wherein the storage device stores a first feature amount calculated during the first period and a second feature amount calculated during the second period.

3. 3. The authentication system according to claim 2, The authentication processing device includes: calculating quality values ​​for the first feature amount and the second feature amount, and if the quality value exceeds a predetermined value, storing the first feature amount, the second feature amount and the corresponding quality value in the storage device; 1. An authentication system comprising:

4. 2. The authentication system according to claim 1, The authentication processing device includes: an authentication system using a fusion feature obtained by fusing the first feature calculated in the first period, the second feature calculated in the second period, and a third feature calculated in the second period.

5. 2. The authentication system according to claim 1, storing, in the storage device, in chronological order, a first feature calculated during the first period and a feature having a quality value exceeding a predetermined value for a second feature calculated during the second period; When generating the feature pair, the authentication processing device alternately selects a first feature amount calculated in the first period and a second feature amount calculated in the second period in order of oldest to newest.

1. An authentication system comprising:

6. 2. The authentication system according to claim 1, the storage device stores the first feature amount and the second feature amount in descending order of quality value; The authentication processing device selects the feature quantity in descending order of the quality value and generates a feature pair with the third feature quantity.

1. An authentication system comprising:

7. 2. The authentication system according to claim 1, The authentication processing device includes: extracting the second feature amount and the third feature amount alternately during the second period; 1. An authentication system comprising:

8. 2. The authentication system according to claim 1, The authentication processing device includes: The first feature amount, the second feature amount, and the third feature amount are independently compared, and when the comparison result has a high similarity to the previously registered feature amount, comparison of the feature amount is omitted for a certain period of time.

1. An authentication system comprising:

9. A biometric authentication method for an authentication system including an image capturing device for capturing images of a living body, a storage device connected to the image capturing device for storing a plurality of biometric features for each user in association with each other, and an authentication processing device for inputting an image captured by the image capturing device and performing biometric authentication using the input image, comprising: The imaging device is capturing an image of a first biometric data of a first user during a first period; capturing an image of a second biometric data set and a third biometric data set of a first user during a second period different from the first period; The authentication processing device includes: Calculating a first feature amount from a first living body imaged during the first period; calculating a second feature amount and a third feature amount from the second living body and the third living body photographed during the second period, respectively; selecting a third feature amount having a quality value exceeding a predetermined value calculated during the second period; alternately selecting the first feature calculated in the first period and the second feature calculated in the second period from among the first feature calculated in the first period and the feature having a quality value exceeding a predetermined value for the second feature calculated in the second period, and generating a feature pair with the selected third feature; Using the feature pair, a matching process is performed based on the first biometric feature amount or the second biometric feature amount and the third biometric feature amount for each user stored in the storage device. A biometric authentication method comprising:

10. The biometric authentication method according to claim 9, The authentication processing device includes: generating a feature pair using a third feature amount calculated during the second period and a first feature amount calculated during the first period or a second feature amount calculated during the second period; Using the feature pair, a matching process is performed based on the first biometric feature amount and the second biometric feature amount for each user stored in the storage device. A biometric authentication method comprising:

Images