Authentication system and authentication method

Abstract

This authentication system inputs, to an inference model, an eye circumference image including one eye circumference image that has been previously determined from a left eye circumference image and a right eye circumference image and that has been subjected to image conversion including one or both of flip image conversion including flip vertical or flip horizontal of the one eye circumference image, and rotation image conversion in which the one eye circumference image is rotated by a predetermined angle with respect to a predetermined center of the one eye circumference image to acquire the feature amount of the eye circumference image. The authentication system authenticates a user on the basis of the result of a comparison between the acquired feature amount and a feature amount in a registration template in which a class ID including a user ID for identifying the user and the feature amount of the left eye circumference image or the right eye circumference image of the user are previously associated and registered.

Description

Authentication system and authentication method

[0001] The present invention relates to an authentication system and an authentication method.

[0002] In recent years, with the spread of smartphones and tablets, the use of e-commerce and internet banking in mobile environments via the internet has expanded. Furthermore, cashless payments using smartphones and QR codes (registered trademark) are becoming increasingly common at retail stores. Thus, remote transactions via the internet have been expanding rapidly in recent years.

[0003] However, with the expansion of remote trading, there has been no end to the damage caused by fraudulent use such as the leakage of account information and impersonation, and there are concerns that the scale of damage will continue to expand in the future. Therefore, in order to prevent fraudulent use, there is a growing trend to adopt identity authentication methods that use biometric information such as face, fingerprint, iris, and vein pattern to authenticate the identity of the person in remote trading with high accuracy.

[0004] Here, facial recognition using facial images is a highly accurate and convenient means of personal authentication, but has not yet been fully adopted in stores and other locations, except for use on personal devices. This is because it is difficult to obtain an image of the mouth of the person being authenticated when wearing a mask, making it difficult to use in ID-less authentication, which is becoming more popular for improved convenience. Therefore, as a means to complement facial recognition, a method has been proposed that uses periocular features based on an image of the area around the eyes that is captured simultaneously with the face (see, for example, Non-Patent Document 1).

[0005] Karan Ahuja, etc., “A Preliminary Study of CNNs for Iris and Periocular Verification in the Visible Spectrum.”, [online], 2016 23rd International Conference on Pattern Recognition (ICPR).

[0006] In the conventional technology disclosed in the aforementioned Non-Patent Document 1, the periocular images of the left and right eyes are flipped horizontally to expand the learning data, and then a model for performing authentication is trained, thereby improving authentication accuracy. However, when the images are flipped left and right to align the orientation, it is difficult to distinguish between the left and right eyes of the person, and there is room for improvement in accuracy.

[0007] The present invention has been made in view of the above-mentioned problems, and aims to further improve authentication accuracy in face authentication.

[0008] In order to achieve the above-mentioned object, one aspect of the present invention is an authentication system for authenticating a user, the authentication system having a processor and a memory, wherein the processor acquires periocular images from a face image of the user, the periocular images including a left periocular image including the user's left eye and the area around the left eye, and a right periocular image including the user's right eye and the area around the right eye, and performs one of image inversion, including vertical or horizontal inversion of the one periocular image, and image rotation, rotating the one periocular image by a predetermined angle about a predetermined center of the one periocular image, on a predetermined one of the acquired periocular images. performs an image transformation including both, and inputs the periocular image including the one of the periocular images on which the image transformation has been performed into an inference model, wherein the inference model is an inference model that outputs features of the input periocular image, acquires the features of the periocular image output from the inference model in response to the input of the periocular image, and authenticates the user based on the comparison result between the acquired features and the features in a registered template in which a class ID including a user ID that identifies the user and features of the user's left periocular image and right periocular image are registered in advance in correspondence with each other.

[0009] According to the present invention, it is possible to further improve the authentication accuracy in face authentication.

[0010] 1 is a configuration diagram of an authentication system according to embodiment 1. FIG. 2 is a flowchart showing a user ID estimation process according to embodiment 1. FIG. 3 is a flowchart showing a parameter update process according to embodiment 1. FIG. 4 is an explanatory diagram of back error propagation in the parameter update process according to embodiment 1. FIG. 5 is an explanatory diagram of an overview of image transformation of a periocular image according to embodiment 1. FIG. 6 is an explanatory diagram of a modified example of image transformation of a periocular image according to embodiment 1. FIG. 7 is a flowchart showing user authentication process according to embodiment 2. FIG. 8 is a flowchart showing parameter update process according to embodiment 3.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Examples in the embodiments are illustrative for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0012] Examples of various types of information may be described using expressions such as "table," "list," "queue," and "database," but the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0013] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0014] In the embodiments, the program may be used as the subject to describe the processing performed by executing the program. Here, a computer executes the program using a processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) and performs the processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0015] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiments, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] [First Embodiment] (Configuration of Authentication System 10 According to First Embodiment) Fig. 1 is a configuration diagram of an authentication system 10 according to First Embodiment. The authentication system 10 is configured by connecting an information processing terminal 3 and a server 5 via a network 4 such as the Internet.

[0017] The information processing terminal 3 photographs the face of the user to be authenticated and acquires a facial image.The information processing terminal 3 then acquires a left eye peripheral image including the area around the user's left eye and a right eye peripheral image including the area around the user's right eye from the user's facial image.The information processing terminal 3 then acquires feature quantities of the eye peripheral images including the user's left eye peripheral image and right eye peripheral image from the inference model.The information processing terminal 3 also acquires feature quantities of the user's facial image.

[0018] The server 5 performs parameter optimization processing of an inference model that estimates the feature quantities of a user's periocular image using the user's periocular image as input, based on data of the user's periocular image, user ID, and side ID. The user ID is user identification information. The side ID is identification information indicating whether the periocular image is a left periocular image or a right periocular image. The combination of the user ID and side ID is called a class ID.

[0019] The server 5 also compares the side ID and feature amounts of the periocular image received from the information processing terminal 3 with feature amounts of registered templates stored in the server 5, which are registered by associating feature amounts of the periocular image with class IDs. The similarity score of the feature amounts can be calculated using well-known techniques such as cosine similarity. The server 5 then narrows down the user IDs to registered templates whose similarity scores of the periocular image feature amounts are equal to or greater than a threshold.

[0020] Furthermore, the server 5 compares the feature amounts of the user's facial image received from the information processing terminal 3 with feature amounts corresponding to the narrowed-down user IDs among registered templates that are stored in the server 5 and in which feature amounts of facial images are associated with class IDs. The server 5 then extracts user IDs whose similarity scores of feature amounts of facial images are equal to or greater than a threshold, and returns the extracted user IDs to the information processing terminal 3.

[0021] The information processing terminal 3 is configured by a computer including, for example, a processor 30, a memory 31, a storage device 32, a communication device 33, an input device , an image capturing device 35, and a display device .

[0022] The processor 30 executes programs stored in the memory 31. The memory 31 includes a non-volatile storage element, Read Only Memory (ROM), and a volatile storage element, Random Access Memory (RAM). The ROM stores unchanging programs (e.g., a Basic Input / Output System (BIOS)). The RAM is a high-speed, volatile storage element such as a Dynamic Random Access Memory (DRAM), and temporarily stores programs executed by the processor 30 and data used when the programs are executed. The memory 31 stores, for example, a periocular region extraction program 311, an image inversion program 312, and a class estimation program 313.

[0023] The storage device 32 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores the program executed by the processor 30 and data used when the program is executed. That is, the program is read from the storage device 32, loaded into the memory 31, and executed by the processor 30. Alternatively, the program is loaded into the memory 31 from the server 5 and executed by the processor 30. The storage device 32 holds a trained inference model 321, which is loaded into the memory 31 by a class estimation program 313 in the memory 31 and used in the user ID estimation process.

[0024] The input device 34 is a device such as a keyboard or mouse that accepts user input. The display device 36 is a device such as a display device or printer that outputs the results of program execution in a format that can be viewed by the user. The communication device 33 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0025] The server 5 is configured by a computer including, for example, a processor 50, a memory 51, a storage device 52, a communication device 53, an input device 54, and a display device 56. The processor 50, the memory 51, the storage device 52, the communication device 53, the input device 54, and the display device 56 are similar hardware to the processor 30, the memory 31, the storage device 32, the communication device 33, the input device 34, the image capturing device 35, and the display device 36, and therefore a description thereof will be omitted.

[0026] The memory 51 stores a periocular region extraction program 511 , an image inversion program 512 , a model parameter update program 513 , a class probability calculation program 514 , a model parameter update suppression program 515 , and a feature similarity score calculation program 516 .

[0027] The storage device 52 stores teacher data 521, a training inference model 522, and a registered template 523. The teacher data 521 includes, for example, a user ID, a side ID, and a periocular image. The registered template 523 is, for example, data that combines the user ID, the side ID, and feature amounts.

[0028] (User ID Estimation Process According to Embodiment 1) Fig. 2 is a flowchart showing the user ID estimation process according to embodiment 1. In the user ID estimation process according to embodiment 1, user IDs are narrowed down by periocular authentication processing in the authentication system 10. The server 5 is an authentication server that manages enrolled templates for biometric authentication.

[0029] First, in step S101, the periocular region extraction program 311 acquires a facial image of the user using the image capturing device 35. Next, in step S102, the periocular region extraction program 311 acquires a periocular image of the left eye and a periocular image of the right eye from the facial image acquired in step S101. To extract the periocular images from the facial image, specific landmark coordinate output of MediaPipe / FaceMesh, which is an existing technology, can be used.

[0030] Next, in step S103, the image inversion program 312 determines whether the periocular image acquired in step S102 is of a predetermined side (e.g., the right eye side). If the periocular image is of the predetermined side (step S103 YES), the image inversion program 312 proceeds to step S104. If the periocular image is of the opposite side from the predetermined side (step S103 NO), the image inversion program 312 proceeds to step S105.

[0031] In step S104, the image inversion program 312 inverts the periocular image determined to be on the predetermined side in step S103 upside down. After step S104 is completed, the image inversion program 312 proceeds to step S105.

[0032] In steps S103 and S104, the image around the eye on the predetermined side is inverted upside down, and the image around the eye on the opposite side different from the predetermined side is not converted and remains in its original state.

[0033] Next, in step S105, the class estimation program 313 inputs the periocular image of the left eye and the periocular image of the right eye into the trained inference model 321 and obtains features for estimating the user ID + side ID from the trained inference model 321. An example of a feature is a fixed-length vector in which each element is a real number.

[0034] Next, in step S106, the class estimation program 313 generates an authentication template for the left eye and an authentication template for the right eye, respectively, from the feature amounts of the periocular image for the left eye and the periocular image for the right eye acquired in step S105. Here, the authentication template has the same configuration as the registered template 523 stored in the storage device 52 of the server 5, and a user ID has not been set. The authentication template generated in step S106 is not limited to one for each side ID of the right eye and the left eye, and multiple templates may be generated for each of the right eye and the left eye from a series of images being captured.

[0035] Next, in step S107, the class estimation program 313 transmits the authentication templates for the left and right eyes generated in step S106 and the side ID to the server 5.

[0036] Next, in step S108, the feature similarity score calculation program 516 compares the feature of the authentication template for the left eye received from the information processing terminal 3 with the feature of the registered templates 523 of multiple people who have been registered in advance. Then, the feature similarity score calculation program 516 extracts authentication templates (referred to as authentication template group 1) whose similarity scores are equal to or greater than a threshold.

[0037] In step S109, similarly to step S108, the feature similarity score calculation program 516 compares the feature of the authentication template for the right eye received from the information processing terminal 3 with the feature of authentication templates of multiple people who have been registered in advance. Then, the feature similarity score calculation program 516 extracts authentication templates (referred to as authentication template group 2) whose similarity score is equal to or greater than a threshold. Steps S108 and S109 can be performed in any order.

[0038] Next, in step S110, the feature similarity score calculation program 516 integrates the authentication template groups 1 and 2, and extracts a final list of candidate user IDs. Some examples of the integration procedure are shown below.

[0039] (1) Integration procedure 1: An authentication template common to authentication template groups 1 and 2 is extracted, and the user IDs included in the extracted authentication templates are used as the integrated user ID list. (2) Integration procedure 2: An authentication template included in authentication template group 1 or 2 is extracted, and the user IDs included in the extracted authentication templates are used as the integrated user ID list. (3) Integration procedure 3: Of the authentication template group 1' generated for each of a plurality of right eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are used as the authentication template group 1. Also, of the authentication template group 2' generated for each of a plurality of left eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are used as the authentication template group 2. Integration procedure 1 is applied to authentication template groups 1 and 2. (4) Integration procedure 4: Of the authentication template group 1' generated for each of a plurality of right eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are used as the authentication template group 1. Among the authentication template group 2' generated for each of the multiple left eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are defined as authentication template group 2. Integration procedure 2 is applied to authentication template groups 1 and 2.

[0040] The candidate user ID list, which is obtained by narrowing down candidates presumed to be users to be authenticated from the pre-registered registration templates 523 of multiple people through the above steps, can be used as input for a subsequent separate process. In other words, by using the candidate user ID list as a list of initial candidate user IDs, ID-less authentication, which performs authentication without inputting a user ID in steps S111, S112, and S113, can be applied to face authentication.

[0041] In step S111, the class estimation program 313 extracts features from the facial image acquired in step S101 and generates an authentication template. This authentication template, like the authentication template in step S106, has the same configuration as the enrollment template 523. However, unlike the enrollment template 523, this authentication template does not have side ID information and has facial features (second features) instead of periocular features. Existing technology (e.g., FaceNet) can be used to extract the features of the facial image.

[0042] Next, in step S112, the class estimation program 313 transmits the authentication template generated in step S111 to the server 5.

[0043] Next, in step S113, the feature similarity score calculation program 516 calculates a similarity score between the feature amounts of the face image received from the information processing terminal 3 and the facial feature amounts of the registered template 523′ corresponding to the candidate user ID list acquired in step S110. The registered template 523′ (second registered template) stores facial feature amounts instead of the periocular feature amounts in the registered template 523. The feature similarity score calculation program 516 then acquires user IDs whose similarity scores are equal to or greater than a threshold as a further narrowed down final candidate user ID list, and transmits the list to the information processing terminal 3.

[0044] Next, in step S114, the feature extraction / authentication program 314 receives the final user ID list transmitted from the server 5 in step S113. The feature extraction / authentication program 314 then determines whether the user IDs are uniquely defined in the received final user ID list. If the user IDs are uniquely defined (YES in step S114), the feature extraction / authentication program 314 determines that the authentication is successful and proceeds to step S115. On the other hand, if the user IDs are not uniquely defined (NO in step S114), the feature extraction / authentication program 314 returns the process to step S101, or proceeds to step S115 after a certain period of time has elapsed and displays an authentication error on the display device 36.

[0045] In step S115, the feature extraction / authentication program 314 displays the authentication status (authentication success or authentication error) on the display device 36. In step S115, the feature extraction / authentication program 314 may display the uniquely narrowed down user ID on the display device 36 in addition to or instead of the authentication status.

[0046] Note that steps S108, S109, and S110 do not necessarily have to be performed by the server 5. That is, the information processing terminal 3 may store the registered template 523 in the storage device 32, and steps S108, S109, and S110 may be executed by the feature similarity score calculation program 316.

[0047] In addition, steps S111 to S114 can be omitted, and authentication result information indicating that authentication is successful if one or more candidate user IDs are obtained as a result of the processing of step S110, or that authentication is an error if no candidate user IDs are obtained, may be output in step S115.

[0048] In this embodiment, the processes of steps S102 to S112 are performed by the information processing terminal 3. However, the information processing terminal 3 may only acquire a face image in step S101, and the processes of steps S102 to S112 may be performed by the server 5.

[0049] According to this embodiment, the accuracy of estimating the user ID (+side ID) of each of the left and right periocular images can be improved using the trained inference model 321.

[0050] 3 is a flowchart showing the parameter update process according to embodiment 1. In the parameter update process according to embodiment 1, the server 5 updates the parameters of the training inference model 522 for estimating the class ID of each periocular image.

[0051] First, in step S201, the periocular region extraction program 511 acquires a periocular image from the training data 521 in the storage device 52. Next, in step S202, the model parameter update program 513 determines whether the periocular image acquired in step S201 is on a predetermined side (e.g., the right side). If the periocular image is on the predetermined side (YES in step S202), the model parameter update program 513 proceeds to step S203. If the periocular image is not on the predetermined side (NO in step S202), the model parameter update program 513 proceeds to step S203.

[0052] In step S203, the image inversion program 512 inverts the periocular image determined to be the predetermined side in step S202 upside down. After step S203, the model parameter update program 513 moves the process to step S204.

[0053] In step S204, the model parameter update program 513 inputs the periocular images of the left and right eyes into the inference model under training 522 to acquire features for estimating the user ID and side ID. The features are, for example, fixed-length vectors in which each element is a real number. SE-ResNet, a deep learning model, can be used as a feature output means.

[0054] Next, in step S205, the class probability calculation program 514 converts the feature quantities of the left and right eye periocular images into a probability distribution of class IDs with dimensions equal to the number of types of class IDs (user IDs + side IDs). The conversion means can use a fully connected layer (the number of input dimensions is the number of feature dimensions, and the number of output dimensions is the number of user IDs + side IDs) following the feature output layer of SE-ResNet, and a SoftMax function that converts the output value into a probability value between 0 and 1. The SoftMax function is the SoftMax function 608 in FIG. 4, which will be described later.

[0055] Next, in step S206, the model parameter update program 513 has a vector with the same number of output dimensions as the number of class IDs (user IDs + side IDs), and calculates the error with a probability distribution in which the class ID in the correct answer data corresponding to the class ID is 1 and the rest are 0. One method of calculating the error is cross-entropy, that is, cross-entropy loss calculation with the teacher distribution.

[0056] Next, in step S207, the model parameter update suppression program 515 suppresses the update of parameters of some dependent classes of the inference model under training 522 during error backpropagation. Parameter update suppression is performed by setting the gradient information of all edges connected to nodes in the fully connected layer of the class ID corresponding to the user ID + opposite side ID among the class IDs (user ID + side ID) to 0. The opposite side ID is the ID of a side (left eye) different from one side (e.g., the right eye) determined in advance.

[0057] Next, in step S208, the model parameter update program 513 updates the parameters of the inference model under training 522 using backpropagation.

[0058] Next, in step S209, the model parameter update program 513 determines whether a parameter update termination condition is satisfied. If the parameter update termination condition is satisfied (step S209 YES), the model parameter update program 513 terminates the parameter update. If the parameter update termination condition is not satisfied (step S209 NO), the process returns to step S201. The parameter update termination condition may be when parameter updates have been performed a predetermined number of times, when the error calculated in step S206 has not improved from the previous step, or when the error is sufficiently small.

[0059] The training inference model 522 is adopted as the post-training inference model 321 at any time after the parameter update process is completed.

[0060] In this embodiment, the upside-down flip of the image around the eyes is performed only by the parameter update suppression process, and even if the user ID estimation process (Figure 2) is not performed, it is expected that the estimation accuracy of the class ID of the image around the left and right eyes will be improved.

[0061] According to this embodiment, the accuracy of estimating the class IDs of the images around the left and right eyes can be improved using the trained inference model 321.

[0062] (Backward Error Propagation in Parameter Update Processing According to Embodiment 1) Fig. 4 is an explanatory diagram of backward error propagation in parameter update processing according to embodiment 1. In this embodiment, during model parameter update processing, updating of some parameters of the inference model 522 during training is suppressed.

[0063] Image 603 represents an example of a periocular image. User ID 601 "002" represents the user ID of image 603. Side ID 602 ​​"L" represents a side ID indicating whether image 603 is of the right eye ("R") or the left eye ("L"). Inference model 604 represents inference model 522 in training, which takes an image as input and outputs feature 605.

[0064] The feature 605 represents a feature for estimating the user ID+side ID of the input image 603. The class ID classifier 606 is an example of the implementation of the class probability calculation program 514, and constitutes a fully connected layer (the number of input dimensions is the number of feature dimensions, and the number of output dimensions is the number of types of user IDs+side IDs).

[0065] Class-specific output 607 represents an output vector with the number of dimensions (user ID + side ID) output by class ID classifier 606. SoftMax function 608 represents a SoftMax function that normalizes class-specific output 607 to a probability value between 0 and 1. Class probability distribution 609 represents a probability distribution made up of vectors with the number of dimensions equal to the number of types of user IDs + side IDs.

[0066] The teacher probability distribution 610 has a vector with the same number of dimensions as the number of types of user IDs + side IDs, and represents a probability distribution in which the class ID in the correct answer data corresponding to the user ID + side ID is 1 and the rest are 0. The error calculation 611 represents a calculation process for calculating the error between the class probability distribution 609 and the teacher probability distribution 610. Cross-entropy can be used as one method of error calculation.

[0067] ID 612 represents the ID of the input image (user ID + side ID). ID 613 represents the user ID + opposite side ID. Error backpropagation 621 represents the state of backpropagation of the error of a node corresponding to the user ID + side ID in the class-specific output 607. Error backpropagation 622 represents the state of backpropagation of the error of a node corresponding to the user ID + opposite side ID in the class-specific output 607. In the class-specific output 607, node 623 represents a node (or output) corresponding to the user ID + side ID.

[0068] 3 in this embodiment, the error between the class probability distribution 609 and the teacher probability distribution 610 is calculated (error calculation 611). Then, in step S207, the gradient of all error backpropagation 622 connected to the node 623 corresponding to the user ID + opposite side ID in the class-specific output 607 is set to 0, and error backpropagation connected to other nodes is performed as usual.

[0069] According to this embodiment, when learning images of user ID + side ID, parameter updates related to user ID + opposite side ID can be suppressed, and the post-training inference model 321 can improve the estimation accuracy of the class IDs of images around the left and right eyes.

[0070] (Outline of Image Conversion of Periocular Images According to Embodiment 1) Next, an outline of image conversion of periocular images according to embodiment 1 will be described with reference to Fig. 5 . Fig. 5 is a schematic diagram showing a specific example of an image after image inversion processing performed in step S104. Images 701 and 702 represent periocular images acquired in step S102 from a face image acquired by the image capturing device 35. Image 701 represents a periocular image of the right eye, and image 702 represents a periocular image of the left eye. Image 703 represents an image obtained by flipping image 701 horizontally. Image 705 represents an image obtained by flipping image 701 vertically.

[0071] Here, horizontal flipping refers to the process of swapping each pixel of an image with respect to a certain axis from left to right as shown in Figures 5(a) and 5(b), and vertical flipping refers to the process of swapping each pixel of an image with respect to a certain axis from top to bottom as shown in Figures 5(a) and 5(c).

[0072] In the combination of images 705 and 706, which are upside-down images, there is a clear difference in the position of the eyebrows and the shape of the eyes, and even if the user ID is the same, the probability of incorrect authentication of the images around the left and right eyes is extremely low, resulting in an improvement in overall authentication accuracy.

[0073] In contrast, the combination of image 701 and image 702 without image inversion processing may be able to capture the characteristics of the images around the left and right eyes, but the difference between the images around the left and right eyes is not as clear as when using image 705 that has been inverted upside down.

[0074] Furthermore, in the combination of images 703 and 704, which are mirror-inverted images, the similarity score of a pair (different class pair) with the same user ID but different side IDs may be close to the similarity score of a pair (same class pair) with the same user ID and side ID. This occurs when one mirror-inverted periocular image is similar to the other periocular image. This may hinder the improvement of the estimation accuracy of the class IDs of the left and right periocular images.

[0075] From the above, by performing a process of flipping the image around the right eye, for example, with a predetermined side ID, upside down, the accuracy of estimating the class ID of the image around the left and right eyes can be improved in the post-training inference model 321 after parameter update.

[0076] (Modification of Periocular Image Conversion Processing According to First Embodiment) Next, a modification of the periocular image conversion processing according to the first embodiment will be described with reference to FIG.

[0077] In the first embodiment, the image around the eye of a predetermined side (left eye or right eye) is flipped upside down. However, the image transformation of the image around the eye is not limited to flipping upside down. Alternatively, the image 707 of the image around the eye of the predetermined side shown in FIG. 6A may be rotated counterclockwise (or clockwise) by a predetermined angle θ (θ is a fixed value, for example, 90°) with respect to the X-axis in an XY coordinate system with its center O as the origin, as shown in FIG. 6B.

[0078] Images after image rotation, in which the periocular image is rotated counterclockwise by a predetermined angle θ, are expected to improve the estimation accuracy of the class IDs of the left and right periocular images in the post-training inference model 321 after parameter update, just like images after image inversion, which is upside down.

[0079] Furthermore, the image transformation of the predetermined side periocular image may be an appropriate combination of image inversion (left-right or up-down inversion) and the above-mentioned image rotation. In the image transformation that combines image inversion and image rotation, the order of execution of the image inversion and image rotation does not matter.

[0080] (Effects of Embodiment 1) In this embodiment, image transformation is performed on a predetermined one of the user's left and right periocular images, including image inversion (including up-down or left-right inversion) and image rotation (rotation by a predetermined angle about a predetermined center). The periocular image including the one periocular image that has undergone image transformation is input to an inference model, and user authentication is performed based on the feature quantities of the periocular image output from the inference model. Therefore, this embodiment can improve the estimation accuracy of the class IDs of the left and right periocular images.

[0081] In this embodiment, candidates who are presumed to be the user are narrowed down based on the comparison result between the feature amount of the periocular image and the feature amount of the registered template, and the user is authenticated based on the comparison result between the feature amount of the face image of the candidate and the feature amount of the face image of the registered template. Thus, according to this embodiment, ID-less authentication can be applied to face authentication.

[0082] Furthermore, in this embodiment, when updating the parameters of the inference model for the feature quantities of the periocular image, the parameters are updated to reduce errors in the class IDs of the periocular image with the opposite side ID in the correct answer data for the user corresponding to the periocular image. Alternatively, updating of parameters related to the class IDs of the periocular image with the opposite side ID is stopped. Therefore, according to this embodiment, the estimation accuracy of the class IDs of the left and right periocular images using the inference model can be improved. Furthermore, even if the up-down flip of the periocular image is performed only in the parameter update suppression process and not in the user ID estimation process (Figure 2), the estimation accuracy of the class IDs of the left and right periocular images can be improved.

[0083] [Embodiment 2] In embodiment 1, ID-less face authentication is performed, in which face authentication is performed based on an input face image of a user without inputting a user ID. However, this is not limited to this, and user authentication can also be performed based on an input user ID and a face image of the user. Hereinafter, as embodiment 2, a form in which user authentication is performed based on an input user ID and a face image of the user will be described.

[0084] In the description of the second embodiment, the differences from the first embodiment will be mainly described, and the description of the parts that overlap with the first embodiment will be omitted.

[0085] (User authentication processing according to the second embodiment) Fig. 7 is a flowchart showing user authentication processing according to the second embodiment. The user authentication processing according to the second embodiment differs from the user ID estimation processing according to the first embodiment in the following respects. That is, the difference is that step S101a is executed before step S101. The difference is also that step S107a is executed instead of step S107. The difference is also that steps S108a to S112a are executed instead of steps S108 to S114. Otherwise, the same steps as those in the user ID estimation processing according to the first embodiment are assigned the same step numbers, and their explanations will be omitted.

[0086] First, in step S101a, the information processing terminal 3 acquires the input user ID.

[0087] In step S107a, the class estimation program 313 transmits to the server 5 the authentication templates for the left and right eyes generated in step S106 and the user ID and side ID received in step S101a.

[0088] Next, in step S108 a, the feature similarity score calculation program 516 acquires, from the registered templates 523, an authentication template that corresponds to the combination of the user ID and side ID received from the information processing terminal 3.

[0089] Next, in step S109a, the feature similarity score calculation program 516 compares the authentication template for the left eye received from the information processing terminal 3 with the feature of the authentication template acquired in step S108a to calculate the similarity score for the left eye. In step S110a, the feature similarity score calculation program 516 compares the authentication template for the right eye received from the information processing terminal 3 with the feature of the authentication template acquired in step S108a to calculate the similarity score for the right eye. Steps S109a and S110a may be performed in any order.

[0090] Next, in step S111a, the feature similarity score calculation program 516 integrates the similarity scores calculated in steps S109a and S110a to perform authentication. The feature similarity score calculation program 516 transmits the authentication result to the information processing terminal 3.

[0091] The authentication determination in step S111a is performed as follows: Authentication determination 1 or Authentication determination 2. Authentication determination 1: If the similarity score for the left eye or the similarity score for the right eye is equal to or greater than a preset threshold Th, authentication is deemed successful; if it is less than the threshold Th, authentication is deemed unsuccessful. Authentication determination 2: If the similarity score for the left eye or the similarity score for the right eye is equal to or greater than a preset threshold Th, authentication is deemed successful; if it is less than the threshold Th, authentication is deemed unsuccessful.

[0092] The similarity scores of the left and right periocular images may be selected by selecting the highest similarity score among a plurality of consecutive periocular images.

[0093] In step S112a, the class estimation program 313 receives authentication result information from the server 5. If the class estimation program 313 receives information indicating successful authentication, it displays the result on the display device 36 in step S115. On the other hand, if the authentication fails, the class estimation program 313 returns the process to step S101, or displays an authentication error on the display device 36 after a certain period of time has elapsed.

[0094] (Effects of embodiment 2) According to this embodiment, the post-training inference model 321 can be used to improve the estimation accuracy of the user ID + side ID of the periocular images of the left and right eyes, even in ID authentication in which the user ID is entered during authentication.

[0095] [Embodiment 3] In embodiment 1, when updating parameters, an error calculation of back error propagation is performed, and then the update of parameters of a specific class of the inference model during training 522 is stopped. However, this is not limited to this. The update of parameters of the inference model during training 522 can also be stopped by converting the left and right periocular features into class probability distributions and setting the class probability distribution of the specific class to 0. Hereinafter, as embodiment 3, an embodiment will be described in which the update of parameters of a specific class of the inference model during training 522 is stopped by setting the class probability distribution of the specific class to 0.

[0096] In the description of the third embodiment, differences from the first embodiment will be mainly described, and descriptions of parts that overlap with the first embodiment will be omitted.

[0097] (Parameter Update Processing According to Embodiment 3) Fig. 8 is a flowchart showing the parameter update processing according to embodiment 3. The parameter update processing according to embodiment 3 differs from the parameter update processing according to embodiment 1 in the following respects. That is, step S206a is executed instead of step S206, and step S207a is executed instead of step S207. Other steps that are the same as those in the parameter update processing according to embodiment 3 are assigned the same step numbers, and descriptions thereof will be omitted.

[0098] In step S206a, the model parameter update suppression program 515 sets the probability value of the class ID corresponding to the user ID + opposite side ID in the probability distribution of class IDs with dimensions equal to the number of types of class IDs (user ID + side ID) in step S205 to 0. The opposite side ID is the side ID on the opposite side of a given side ID. Next, in step S207a, the model parameter update program 513 performs error calculation.

[0099] (Effects of embodiment 3) According to this embodiment, similar to embodiments 1 and 2, in ID-less authentication, the accuracy of estimating the class ID of the images around the left and right eyes can be improved using the trained inference model 321.

[0100] The present invention is not limited to the above-described embodiments, and includes various modifications. Furthermore, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the described configurations. Furthermore, not only are such configurations removed, but replacement or addition of configurations is also possible. Furthermore, embodiments of the present invention also include those in which some or all of the above-described embodiments are appropriately combined to be consistent. Furthermore, the order of execution of each step of the various processes in the above-described embodiments can be changed as long as the processing results are the same.

[0101] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that implements the functions of the embodiments. In this case, a recording medium on which the program code is recorded is provided to a computer, and a processor within the computer reads the program code stored in the recording medium.

[0102] In this case, the program code itself read from the recording medium will realize the functions of the above-described embodiment, and the program code itself and the recording medium on which it is stored constitute the present invention. Examples of recording media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0103] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, and Java (registered trademark).

[0104] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0105] 1: Input biometric information 1, 3: Information processing terminal, 5: Server, 30: Processor, 31: Memory, 50: Processor, 51: Memory, 321: Post-trained inference model, 522: Inference model under training, 523: Enrollment template.

Claims

1. An authentication system for authenticating a user, the authentication system comprising a processor and a memory, wherein the processor: acquires periocular images from a facial image of the user, including a left periocular image including the user's left eye and the area surrounding the left eye, and a right periocular image including the user's right eye and the area surrounding the right eye; performs image transformation on a predetermined one of the acquired periocular images, including either or both of image inversion, including up-down or left-right inversion of the one periocular image, and image rotation, which rotates the one periocular image by a predetermined angle about a predetermined center of the one periocular image; inputs the periocular image including the one periocular image after the image transformation into an inference model, wherein the inference model outputs feature values ​​of the input periocular image; acquires the feature values ​​of the periocular image output from the inference model in response to the input of the periocular image; An authentication system characterized by authenticating the user based on a comparison result between the acquired features and the features in a registered template in which a class ID including a user ID that identifies the user and features of the image around the left eye and the image around the right eye of the user are registered in advance in correspondence with each other.

2. An authentication system according to claim 1, wherein the image transformation is vertical inversion.

3. An authentication system according to claim 1, wherein the image transformation is an image rotation that rotates the periocular image by 90° about the predetermined center.

4. An authentication system as described in claim 1, wherein, when authenticating the user, the processor narrows down candidates who are presumed to be the user based on a comparison result between the acquired features and the features in the registered template, extracts second features from the facial image, and authenticates the user based on a comparison result between the second features and the features in a second registered template in which the class ID of the candidate and the features of the facial image of the user are registered in advance in correspondence with each other.

5. An authentication system as described in claim 4, wherein the processor: extracts the features from the periocular image including the one periocular image on which the image transformation has been performed; calculates a probability distribution of the class ID of the periocular image by inputting the periocular image into the inference model; calculates the error between the calculated probability distribution and the probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; updates the parameters of the inference model to reduce the calculated error; and updates the parameters to reduce the error between the class ID of the one periocular image in the correct answer data and the class ID of the other periocular image, or stops updating the parameters related to the class ID of the other periocular image.

6. An authentication system for authenticating a user, the authentication system comprising a processor and a memory, wherein the processor: acquires periocular images, including a left periocular image including the user's left eye and the area surrounding the left eye, and a right periocular image including the user's right eye and the area surrounding the right eye, from teacher data of the user's face image; performs image transformation on a predetermined one of the periocular images, including either or both of image inversion, including up-down or left-right inversion of the one periocular image, and image rotation, which rotates the one periocular image by a predetermined angle about a predetermined center of the one periocular image; inputs the periocular image including the one periocular image after the image transformation into an inference model, wherein the inference model outputs feature values ​​of the input periocular image; acquires the feature values ​​of the periocular image output from the inference model in response to the input of the periocular image; and calculates a probability distribution of a class ID including a user ID that identifies the user of the periocular image based on the feature values ​​of the acquired periocular image; An authentication system characterized by calculating the error between the calculated probability distribution and the probability distribution of the class ID in the correct data corresponding to the user corresponding to the periocular image, updating the parameters of the inference model to reduce the calculated error, and updating the parameters to reduce the error between the class ID of one periocular image in the correct data and the class ID of the other periocular image, or stopping the update of the parameters related to the class ID of the other periocular image.

7. An authentication method performed by an authentication system that authenticates a user, the authentication system having a processor and a memory, wherein the processor: acquires periocular images from the user's face image, including a left periocular image including the user's left eye and the area surrounding the left eye, and a right periocular image including the user's right eye and the area surrounding the right eye; performs image transformation on a predetermined one of the acquired periocular images, including either or both of image inversion, including vertical or horizontal inversion of the one periocular image, and image rotation, which rotates the one periocular image by a predetermined angle about a predetermined center of the one periocular image; inputs the periocular image including the one periocular image that has undergone the image transformation into an inference model, wherein the inference model outputs feature values ​​of the input periocular image; acquires the feature values ​​of the periocular image output from the inference model in response to the input of the periocular image; An authentication method characterized by comprising the steps of: authenticating the user based on a comparison result between the acquired feature amount and a class ID including a user ID that identifies the user, and the feature amount in a registered template in which the feature amount of the image around the left eye and the image around the right eye of the user is registered in advance in correspondence with the class ID.

8. An authentication method according to claim 7, wherein the image transformation is vertical inversion.

9. An authentication method according to claim 7, wherein the image transformation is an image rotation that rotates the periocular image by 90° about the predetermined center.

10. An authentication method according to claim 7, wherein, when authenticating the user, the processor narrows down candidates who are presumed to be the user based on a comparison result between the acquired feature amount and the feature amount in the registered template, extracts a second feature amount from the facial image, and authenticates the user based on a comparison result between the second feature amount and the feature amount in a second registered template in which the class ID of the candidate and the feature amount of the facial image of the user are registered in advance in correspondence with each other.

11. An authentication method as described in claim 10, characterized in that the processor comprises the following processes: extracting the features from the periocular image including the one periocular image on which the image transformation has been performed; calculating a probability distribution of the class ID of the periocular image by inputting the periocular image into the inference model; calculating the error between the calculated probability distribution and the probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; updating the parameters of the inference model to reduce the calculated error, and updating the parameters to reduce the error between the class ID of the one periocular image in the correct answer data and the class ID of the other periocular image, or stopping the update of the parameters related to the class ID of the other periocular image.

12. An authentication method performed by an authentication system for authenticating a user, the authentication system having a processor and a memory, wherein the processor: acquires periocular images including a left periocular image including the user's left eye and the area surrounding the left eye, and a right periocular image including the user's right eye and the area surrounding the right eye, from teacher data of the user's face image; performs image transformation on a predetermined one of the periocular images, including either or both of image inversion, including up-down or left-right inversion of the one periocular image, and image rotation, which rotates the one periocular image by a predetermined angle about a predetermined center of the one periocular image; inputs the periocular image including the one periocular image after the image transformation into an inference model, wherein the inference model outputs feature values ​​of the input periocular image; acquires the feature values ​​of the periocular image output from the inference model in response to the input of the periocular image; and calculates a probability distribution of a class ID including a user ID that identifies the user of the periocular image based on the feature values ​​of the acquired periocular image; An authentication method characterized by comprising the steps of: calculating the error between the calculated probability distribution and the probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; updating the parameters of the inference model to reduce the calculated error; and updating the parameters to reduce the error between the class ID of one periocular image in the correct answer data and the class ID of the other periocular image, or stopping the update of the parameters related to the class ID of the other periocular image.

Images