Authentication system and authentication method

Abstract

To improve authentication accuracy in face authentication.SOLUTION: The authentication system inputs, to the inference model, an eye surrounding image including one eye surrounding image obtained by performing image conversion including one or both of flip image conversion including vertical inversion or horizontal inversion of a predetermined one eye surrounding image of the left eye surrounding image and the right eye surrounding image and rotation image conversion of rotating the one eye surrounding image by a predetermined angle about a predetermined center of the one eye surrounding image, and acquires a feature amount of the eye surrounding image. Then, the authentication system authenticates the user on the basis of a comparison result between the acquired feature amount and a feature amount in a registered template in which a class ID including a user ID for identifying the user and a feature amount of a left eye surrounding image or a right eye surrounding image of the user are registered in advance in association with each other.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an authentication system and an authentication method. [Background technology]

[0002] In recent years, with the spread of smartphones and tablets, the use of e-commerce and internet banking in mobile environments via the internet has expanded. In addition, cashless payments using smartphones and QR codes (registered trademark) are becoming more common at retail stores. In this way, remote transactions via the internet have been expanding rapidly in recent years.

[0003] However, with the expansion of remote trading, there has been no end to the damage caused by fraudulent use such as the leakage of account information and impersonation, and there are concerns that the scale of damage will continue to expand in the future. Therefore, in order to prevent fraudulent use, there is a growing trend to adopt identity authentication methods that use biometric information such as face, fingerprint, iris, and vein pattern to authenticate the identity of the person in remote trading with high accuracy.

[0004] Here, facial recognition using facial images is a highly accurate and convenient means of personal authentication, but it has not yet been fully adopted in stores and other locations, except for use on personal devices. This is because it is difficult to capture an image of the person's mouth when wearing a mask, making it difficult to use in ID-less authentication, which is becoming more popular for improving convenience. Therefore, as a means to complement facial recognition, a method has been proposed that uses periocular features based on an image of the area around the eyes that is captured at the same time as the face (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Karan Ahuja, etc., “A Preliminary Study of CNNs for Iris and Periocular Verification in the Visible Spectrum.”, [online], 2016 23rd International Conference on Pattern Recognition (ICPR). Summary of the Invention [Problem to be solved by the invention]

[0006] In the conventional technology disclosed in the above-mentioned Non-Patent Document 1, the periocular images of the left and right eyes are flipped horizontally to expand the learning data, and then a model for performing authentication is trained, thereby improving authentication accuracy. However, when the images are flipped left and right to align the orientation, it is difficult to distinguish between the left and right eyes of the person, and there is room for improvement in accuracy.

[0007] The present invention has been made in view of the above-mentioned problems, and aims to further improve authentication accuracy in face authentication. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, one aspect of the present invention is an authentication system for authenticating a user, the authentication system having a processor and a memory, wherein the processor acquires periocular images from a face image of the user, the periocular images including a left periocular image including the user's left eye and the area around the left eye, and a right periocular image including the user's right eye and the area around the right eye, and performs one of image inversion, including vertical or horizontal inversion of the one periocular image, and image rotation, rotating the one periocular image by a predetermined angle about a predetermined center of the one periocular image, on a predetermined one of the acquired periocular images. performs an image transformation including both, and inputs the periocular image including the one of the periocular images on which the image transformation has been performed into an inference model, wherein the inference model is an inference model that outputs features of the input periocular image, acquires the features of the periocular image output from the inference model in response to the input of the periocular image, and authenticates the user based on the comparison result between the acquired features and the features in a registered template in which a class ID including a user ID that identifies the user and features of the user's left periocular image and right periocular image are registered in advance in correspondence with each other. [Effects of the Invention]

[0009] According to the present invention, it is possible to further improve the authentication accuracy in face authentication. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram of an authentication system according to a first embodiment. [Figure 2] 10 is a flowchart showing a user ID estimation process according to the first embodiment. [Figure 3] 4 is a flowchart showing a parameter update process according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of back error propagation in the parameter update processing according to the first embodiment. [Figure 5] FIG. 2 is an explanatory diagram illustrating an outline of image conversion of a periocular image according to the first embodiment. [Figure 6] FIG. 10 is an explanatory diagram of a modified example of image conversion of a periocular image according to the first embodiment. [Figure 7] 10 is a flowchart showing a user authentication process according to the second embodiment. [Figure 8] 10 is a flowchart showing a parameter update process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Examples in the embodiments are illustrative for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0012] Examples of various types of information may be described using expressions such as "table," "list," "queue," and "database," but the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0013] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0014] In the embodiments, the program may be used as the subject to describe the processing performed by executing the program. Here, a computer executes the program using a processor (e.g., a central processing unit (CPU) or a graphics processing unit (GPU)), and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a complex programmable logic device (CPLD).

[0015] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] [Embodiment 1] (Configuration of authentication system 10 according to embodiment 1) 1 is a configuration diagram of an authentication system 10 according to embodiment 1. The authentication system 10 is configured by connecting an information processing terminal 3 and a server 5 via a network 4 such as the Internet.

[0017] The information processing terminal 3 photographs the face of the user to be authenticated and acquires a facial image. Then, the information processing terminal 3 acquires a left eye peripheral image including the area around the user's left eye and a right eye peripheral image including the area around the user's right eye from the user's facial image. Then, the information processing terminal 3 acquires feature amounts of the periocular image including the left eye peripheral image and the right eye peripheral image from the inference model. The information processing terminal 3 also acquires feature amounts of the user's facial image.

[0018] The server 5 performs parameter optimization processing of an inference model that estimates the feature quantities of a periocular image by inputting the user's periocular image based on data of the user's periocular image, user ID, and side ID. The user ID is user identification information. The side ID is identification information that identifies whether the periocular image is a left periocular image or a right periocular image. The combination of the user ID and side ID is called a class ID.

[0019] The server 5 also compares the side ID and feature amounts of the periocular image received from the information processing terminal 3 with feature amounts of registered templates stored in the server 5, which are registered by associating feature amounts of periocular images with class IDs. The similarity score of the feature amounts can be calculated using well-known techniques such as cosine similarity. The server 5 then narrows down the user IDs to registered templates whose similarity scores of the periocular image feature amounts are equal to or greater than a threshold.

[0020] Furthermore, the server 5 compares the feature amounts of the user's facial image received from the information processing terminal 3 with the feature amounts corresponding to the narrowed-down user IDs among registered templates in which the feature amounts of facial images are associated with class IDs and stored in the server 5. The server 5 then extracts user IDs whose similarity scores of the feature amounts of the facial images are equal to or greater than a threshold, and returns the extracted user IDs to the information processing terminal 3.

[0021] The information processing terminal 3 is configured by a computer including, for example, a processor 30, a memory 31, a storage device 32, a communication device 33, an input device , an image capturing device 35, and a display device .

[0022] The processor 30 executes programs stored in the memory 31. The memory 31 includes a ROM (Read Only Memory), which is a non-volatile storage element, and a RAM (Random Access Memory), which is a volatile storage element. The ROM stores unchanging programs (e.g., a Basic Input / Output System (BIOS)) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 30 and data used when the programs are executed. The memory 31 stores, for example, a periocular region extraction program 311, an image inversion program 312, and a class estimation program 313.

[0023] The storage device 32 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores the program executed by the processor 30 and data used when the program is executed. That is, the program is read from the storage device 32, loaded into the memory 31, and executed by the processor 30. Alternatively, the program is loaded into the memory 31 from the server 5 and executed by the processor 30. The storage device 32 holds a trained inference model 321, which is loaded into the memory 31 by a class estimation program 313 in the memory 31 and used in the user ID estimation process.

[0024] The input device 34 is a device such as a keyboard or mouse that accepts user input. The display device 36 is a device such as a display device or printer that outputs the results of program execution in a format that can be viewed by the user. The communication device 33 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0025] The server 5 is configured by a computer including, for example, a processor 50, a memory 51, a storage device 52, a communication device 53, an input device 54, and a display device 56. The processor 50, the memory 51, the storage device 52, the communication device 53, the input device 54, and the display device 56 are similar hardware to the processor 30, the memory 31, the storage device 32, the communication device 33, the input device 34, the image capturing device 35, and the display device 36, and therefore a description thereof will be omitted.

[0026] The memory 51 stores a periocular region extraction program 511, an image inversion program 512, a model parameter update program 513, a class probability calculation program 514, a model parameter update suppression program 515, and a feature similarity score calculation program 516.

[0027] The storage device 52 stores teacher data 521, a training inference model 522, and a registered template 523. The teacher data 521 includes, for example, a user ID, a side ID, and a periocular image. The registered template 523 is, for example, data that combines the user ID, the side ID, and features.

[0028] (User ID Estimation Process According to the First Embodiment) 2 is a flowchart showing a user ID estimation process according to embodiment 1. In the user ID estimation process according to embodiment 1, user IDs are narrowed down by a periocular authentication process in the authentication system 10. The server 5 is an authentication server that manages enrolled templates for biometric authentication.

[0029] First, in step S101, the periocular region extraction program 311 acquires a facial image of the user using the image capturing device 35. Next, in step S102, the periocular region extraction program 311 acquires a periocular image of the left eye and a periocular image of the right eye from the facial image acquired in step S101. To extract the periocular images from the facial image, specific landmark coordinate output of MediaPipe / FaceMesh, which is an existing technology, can be used.

[0030] Next, in step S103, the image inversion program 312 determines whether the periocular image acquired in step S102 is of a predetermined side (for example, the right eye side). If the periocular image is of a predetermined side (step S103 YES), the image inversion program 312 proceeds to step S104, and if the periocular image is of the opposite side from the predetermined side (step S103 NO), the image inversion program 312 proceeds to step S105.

[0031] In step S104, the image inversion program 312 inverts the periocular image determined to be on the predetermined side in step S103 upside down. After step S104 is completed, the image inversion program 312 moves the process to step S105.

[0032] In steps S103 and S104, the periocular image on the predetermined side is flipped upside down, and the periocular image on the opposite side different from the predetermined side is not transformed and remains in its original state.

[0033] Next, in step S105, the class estimation program 313 inputs the periocular image of the left eye and the periocular image of the right eye into the trained inference model 321 and obtains features for estimating the user ID + side ID from the trained inference model 321. An example of a feature is a fixed-length vector in which each element is a real number.

[0034] Next, in step S106, the class estimation program 313 generates an authentication template for the left eye and an authentication template for the right eye, respectively, from the feature amounts of the periocular image for the left eye and the periocular image for the right eye acquired in step S105. Here, the authentication template has the same configuration as the registered template 523 stored in the storage device 52 of the server 5, and a user ID has not been set. The authentication template generated in step S106 is not limited to one for each side ID of the right eye and the left eye, and multiple templates may be generated for each of the right eye and the left eye from a series of images being captured.

[0035] Next, in step S107, the class estimation program 313 transmits to the server 5 the authentication templates for the left and right eyes generated in step S106 and the side ID.

[0036] Next, in step S108, the feature similarity score calculation program 516 compares the feature of the authentication template for the left eye received from the information processing terminal 3 with the feature of registered templates 523 of multiple people who have been registered in advance. Then, the feature similarity score calculation program 516 extracts authentication templates (referred to as authentication template group 1) whose similarity scores are equal to or greater than a threshold.

[0037] In step S109, similarly to step S108, the feature similarity score calculation program 516 compares the feature of the authentication template for the right eye received from the information processing terminal 3 with the feature of authentication templates of multiple people who have been registered in advance. Then, the feature similarity score calculation program 516 extracts authentication templates (referred to as authentication template group 2) whose similarity scores are equal to or greater than a threshold. Steps S108 and S109 can be performed in any order.

[0038] Next, in step S110, the feature similarity score calculation program 516 integrates the authentication template groups 1 and 2, and extracts a final list of candidate user IDs. Some examples of the integration procedure are shown below.

[0039] (1) Integration procedure 1: Extract authentication templates common to authentication template groups 1 and 2, and use the user IDs included in the extracted authentication templates as a list of integrated user IDs. (2) Integration procedure 2: An authentication template included in authentication template group 1 or 2 is extracted, and the user IDs included in the extracted authentication template are used as an integrated user ID list. (3) Integration procedure 3: Among the authentication template group 1' generated for each of a plurality of right eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are set as authentication template group 1. Also, among the authentication template group 2' generated for each of a plurality of left eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are set as authentication template group 2. Integration procedure 1 is applied to authentication template groups 11 and 2. (4) Integration procedure 4: Among the authentication template group 1' generated for each of a plurality of right eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are set as authentication template group 1. Also, among the authentication template group 2' generated for each of a plurality of left eye images, those that appear n times or more (or those whose similarity score is equal to or greater than a predetermined threshold th) are set as authentication template group 2. Integration procedure 2 is applied to authentication template groups 1 and 2.

[0040] The candidate user ID list, which is created by narrowing down candidates presumed to be users to be authenticated from the pre-registered multiple people's registered templates 523 through the above steps, can be used as input for a subsequent separate process. In other words, by using the candidate user ID list as a list of initial candidate user IDs, ID-less authentication, which performs authentication without inputting a user ID in steps S111, S112, and S113, can be applied to face authentication.

[0041] In step S111, the class estimation program 313 extracts features from the face image acquired in step S101 and generates an authentication template. This authentication template, like the authentication template in step S106, has the same configuration as the enrollment template 523. However, unlike the enrollment template 523, this authentication template does not have side ID information and has facial features (second features) instead of periocular features. Existing technology (for example, FaceNet) can be used to extract the features of the face image.

[0042] Next, in step S112, the class estimation program 313 transmits the authentication template generated in step S111 to the server 5.

[0043] Next, in step S113, the feature similarity score calculation program 516 calculates a similarity score between the feature of the face image received from the information processing terminal 3 and the facial feature of the registered template 523′ corresponding to the candidate user ID list acquired in step S110. The registered template 523′ (second registered template) stores facial feature instead of the periocular feature in the registered template 523. The feature similarity score calculation program 516 then acquires user IDs whose similarity scores are equal to or greater than a threshold as a further narrowed down final candidate user ID list, and transmits the list to the information processing terminal 3.

[0044] Next, in step S114, the feature extraction / authentication program 314 receives the final user ID list transmitted from the server 5 in step S113. The feature extraction / authentication program 314 then determines whether the user IDs are uniquely determined in the received final user ID list. If the user IDs are uniquely determined (YES in step S114), the feature extraction / authentication program 314 determines that the authentication is successful and proceeds to step S115. On the other hand, if the user IDs are not uniquely determined (NO in step S114), the feature extraction / authentication program 314 returns the process to step S101, or proceeds to step S115 after a certain period of time has elapsed and displays an authentication error on the display device 36.

[0045] In step S115, the feature extraction / authentication program 314 displays the authentication status (authentication success or authentication error) on the display device 36. In step S115, the feature extraction / authentication program 314 may display the uniquely narrowed down user ID on the display device 36 in addition to or instead of the authentication status.

[0046] Note that steps S108, S109, and S110 do not necessarily have to be performed by the server 5. That is, the information processing terminal 3 may store the registered template 523 in the storage device 32, and steps S108, S109, and S110 may be executed by the feature similarity score calculation program 316.

[0047] Furthermore, steps S111 to S114 can be omitted, and authentication result information indicating successful authentication if one or more candidate user IDs are obtained as a processing result of step S110, or an authentication error if no candidate user IDs are obtained, may be output in step S115.

[0048] In this embodiment, the processes of steps S102 to S112 are performed by the information processing terminal 3. However, the information processing terminal 3 may only acquire a face image in step S101, and the processes of steps S102 to S112 may be performed by the server 5.

[0049] According to this embodiment, the accuracy of estimating the user ID (+side ID) of each of the left and right periocular images can be improved using the trained inference model 321.

[0050] (Parameter update process according to the first embodiment) 3 is a flowchart showing the parameter update process according to embodiment 1. In the parameter update process according to embodiment 1, the server 5 updates the parameters of the inference model in training 522 for estimating the class ID of each periocular image.

[0051] First, in step S201, the periocular region extraction program 511 acquires a periocular image from the training data 521 in the storage device 52. Next, in step S202, the model parameter update program 513 determines whether the periocular image acquired in step S201 is a predetermined side (e.g., the right side). If the periocular image is a predetermined side (step S202 YES), the model parameter update program 513 proceeds to step S203, and if it is not a predetermined side (step S202 NO), the model parameter update program 513 proceeds to step S203.

[0052] In step S203, the image inversion program 512 inverts upside down the periocular image determined to be the pre-determined side in step S202. When step S203 is completed, the model parameter update program 513 moves the process to step S204.

[0053] In step S204, the model parameter update program 513 inputs the periocular images of the left and right eyes into the inference model under training 522 to acquire features for estimating the user ID and side ID. The features are, for example, fixed-length vectors in which each element is a real number. SE-ResNet, a deep learning model, can be used as a means for outputting the features.

[0054] Next, in step S205, the class probability calculation program 514 converts the features of the left and right periocular images into a probability distribution of class IDs with dimensions equal to the number of types of class IDs (user IDs + side IDs). The conversion means can use a fully connected layer (the number of input dimensions is the number of feature dimensions, and the number of output dimensions is the number of user IDs + side IDs) following the feature output layer of SE-ResNet and a SoftMax function that converts the output value into a probability value between 0 and 1. The SoftMax function is SoftMax function 608 in Figure 4, which will be described later.

[0055] Next, in step S206, the model parameter update program 513 has a vector with the same number of output dimensions as the number of class IDs (user IDs + side IDs), and calculates the error with a probability distribution in which the class ID in the correct answer data corresponding to the class ID is 1 and the others are 0. One method of calculating the error is cross-entropy, that is, cross-entropy loss calculation with the teacher distribution.

[0056] Next, in step S207, the model parameter update suppression program 515 suppresses the update of parameters of some dependent classes of the inference model under training 522 during error backpropagation. Parameter update suppression is performed by setting the gradient information of all edges connected to nodes in the fully connected layer of the class ID corresponding to the user ID + opposite side ID among the class IDs (user ID + side ID) to 0. The opposite side ID is the ID of a side (left eye) different from one side (e.g., the right eye) determined in advance.

[0057] Next, in step S208, the model parameter update program 513 updates the parameters of the inference model under training 522 using backpropagation.

[0058] Next, in step S209, the model parameter update program 513 determines whether a parameter update termination condition is satisfied. If the parameter update termination condition is satisfied (step S209 YES), the model parameter update program 513 terminates the parameter update. If the parameter update termination condition is not satisfied (step S209 NO), the process returns to step S201. The parameter update termination condition may be when parameter updates have been performed a predetermined number of times, when the error calculated in step S206 has not improved from the previous step, or when the error is sufficiently small.

[0059] The inference model in training 522 is adopted as the post-training inference model 321 at any time after the parameter update process is completed.

[0060] In this embodiment, the upside-down flip of the image around the eyes is performed only by the parameter update suppression process, and even if the user ID estimation process (Figure 2) is not performed, it is expected that the estimation accuracy of the class IDs of the images around the left and right eyes will be improved.

[0061] According to this embodiment, the accuracy of estimating the class IDs of the left and right periocular images can be improved using the trained inference model 321.

[0062] (Backward Error Propagation in Parameter Update Processing According to Embodiment 1) 4 is an explanatory diagram of back error propagation in the parameter update process according to embodiment 1. In this embodiment, during the model parameter update process, updating of some parameters of the inference model 522 during training is suppressed.

[0063] Image 603 represents an example of a periocular image. User ID 601 "002" represents the user ID of image 603. Side ID 602 ​​"L" represents the side ID indicating whether image 603 is of the right eye ("R") or the left eye ("L"). Inference model 604 represents inference model in training 522, which takes an image as input and outputs feature 605.

[0064] The feature 605 represents the feature for estimating the user ID+side ID of the input image 603. The class ID classifier 606 is an example of implementing the class probability calculation program 514, and configures a fully connected layer (the number of input dimensions is the number of feature dimensions, and the number of output dimensions is the number of types of user ID+side ID).

[0065] Class output 607 represents an output vector with the number of dimensions (user ID + side ID) output by class ID classifier 606. SoftMax function 608 represents a SoftMax function that normalizes class output 607 to a probability value between 0 and 1. Class probability distribution 609 represents a probability distribution consisting of a vector with the number of dimensions equal to the number of types of user ID + side ID.

[0066] The teacher probability distribution 610 has a vector with the same number of dimensions as the number of types of user IDs + side IDs, and represents a probability distribution in which the class ID in the correct answer data corresponding to the user ID + side ID is 1 and the rest are 0. The error calculation 611 represents a calculation process for calculating the error between the class probability distribution 609 and the teacher probability distribution 610. Cross-entropy can be used as one method for calculating the error.

[0067] ID 612 represents the ID of the input image (user ID + side ID). ID 613 represents the user ID + opposite side ID. Error backpropagation 621 represents the state of backpropagation of the error of the node corresponding to the user ID + side ID in the class-specific output 607. Error backpropagation 622 represents the state of backpropagation of the error of the node corresponding to the user ID + opposite side ID in the class-specific output 607. In the class-specific output 607, node 623 represents the node (or output) corresponding to the user ID + side ID.

[0068] 3 in this embodiment, the error between the class probability distribution 609 and the teacher probability distribution 610 is calculated (error calculation 611). Then, in step S207, the gradient of all error backpropagation 622 connected to the node 623 corresponding to the user ID + opposite side ID in the class-specific output 607 is set to 0, and error backpropagation connected to other nodes is performed as usual.

[0069] According to this embodiment, when learning images of user ID + side ID, parameter updates related to user ID + opposite side ID can be suppressed, and the post-training inference model 321 can improve the estimation accuracy of the class IDs of images around the left and right eyes.

[0070] (Outline of Image Conversion of Periocular Image According to Embodiment 1) Next, an overview of image conversion of periocular images according to the first embodiment will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing a specific example of an image after image inversion processing performed in step S104. Images 701 and 702 represent periocular images acquired in step S102 from a face image acquired by the image capturing device 35. Image 701 represents a periocular image of the right eye, and image 702 represents a periocular image of the left eye. Image 703 represents an image obtained by flipping image 701 horizontally. Image 705 represents an image obtained by flipping image 701 vertically.

[0071] Here, horizontal flipping is a process of swapping each pixel of an image with respect to a certain axis from left to right as shown in Figures 5(a) and 5(b), and vertical flipping is a process of swapping each pixel of an image with respect to a certain axis from top to bottom as shown in Figures 5(a) and 5(c).

[0072] In the combination of images 705 and 706, which are upside-down images, there are clear differences in the position of the eyebrows and the shape of the eyes, and even if the user ID is the same, the probability of incorrect authentication of the images around the left and right eyes is extremely low, resulting in an improvement in overall authentication accuracy.

[0073] In contrast, the combination of image 701 and image 702 without image inversion processing may be able to capture the characteristics of the images around the left and right eyes, but the difference between the images around the left and right eyes is not as clear as when using image 705 that has been inverted upside down.

[0074] Furthermore, in the combination of images 703 and 704, which are left-right reversed images, the similarity score of a pair (different class pair) with the same user ID but different side IDs may be close to the similarity score of a pair (same class pair) with the same user ID and side ID. This occurs when one periocular image, which is a left-right reverse of the other, is similar to the other periocular image. This may hinder the improvement of the estimation accuracy of the class IDs of the left and right periocular images.

[0075] From the above, by performing a process of flipping the image around the eye with a predetermined side ID, for example, the right side, upside down, the accuracy of estimating the class IDs of the images around the left and right eyes can be improved in the trained inference model 321 after parameter update.

[0076] (Modification of Periocular Image Conversion Process According to Embodiment 1) Next, a modified example of the periocular image conversion process according to the first embodiment will be described with reference to FIG.

[0077] In the first embodiment, the periocular image of a predetermined side (left eye or right eye) is flipped upside down. However, the image transformation of the periocular image is not limited to flipping upside down. Alternatively, the image 707 of the periocular image of the predetermined side shown in FIG. 6(a) may be rotated counterclockwise (or clockwise) by a predetermined angle θ (θ is a fixed value, for example, 90°) with respect to the X-axis in an XY coordinate system with its center O as the origin, as shown in FIG. 6(b).

[0078] Images after image rotation, in which the periocular image is rotated counterclockwise by a predetermined angle θ, are expected to improve the estimation accuracy of the class IDs of the left and right periocular images in the post-training inference model 321 after parameter update, just like images after upside-down inversion.

[0079] Furthermore, the image transformation of the predetermined side periocular image may be an appropriate combination of image inversion (left-right or up-down inversion) and the above-mentioned image rotation. In the image transformation that combines image inversion and image rotation, the order of execution of the image inversion and image rotation does not matter.

[0080] (Effects of the First Embodiment) In this embodiment, image transformation is performed on a predetermined one of the user's left and right periocular images, including image inversion (including up-down or left-right inversion) and image rotation (rotation by a predetermined angle about a predetermined center).The periocular image including the one periocular image that has undergone image transformation is input to an inference model, and user authentication is performed based on the feature values ​​of the periocular image output from the inference model.Therefore, this embodiment can improve the estimation accuracy of the class IDs of the left and right periocular images.

[0081] In this embodiment, candidates who are presumed to be the user are narrowed down based on the comparison result between the feature amount of the periocular image and the feature amount of the registered template, and the user is authenticated based on the comparison result between the feature amount of the face image of the candidate and the feature amount of the face image of the registered template. Therefore, according to this embodiment, ID-less authentication can be applied to face authentication.

[0082] Furthermore, in this embodiment, when updating the parameters of the inference model for the feature quantities of the periocular image, the parameters are updated to reduce errors in the class IDs of the periocular image with the opposite side ID in the correct answer data for the user corresponding to the periocular image. Alternatively, updating of parameters related to the class IDs of the periocular image with the opposite side ID is stopped. Therefore, according to this embodiment, the estimation accuracy of the class IDs of the left and right periocular images using the inference model can be improved. Furthermore, even if the upside-down flipping of the periocular image is performed only in the parameter update suppression process and not in the user ID estimation process (Figure 2), the estimation accuracy of the class IDs of the left and right periocular images can be improved.

[0083] [Embodiment 2] In the first embodiment, ID-less face authentication is performed, in which face authentication is performed based on an inputted user's face image without inputting a user ID. However, this is not limiting, and user authentication can also be performed based on an inputted user ID and a user's face image. Hereinafter, as a second embodiment, a form in which user authentication is performed based on an inputted user ID and a user's face image will be described.

[0084] In the explanation of the second embodiment, the differences from the first embodiment will be mainly explained, and explanations of parts that overlap with the first embodiment will be omitted.

[0085] (User authentication process according to the second embodiment) 7 is a flowchart showing user authentication processing according to the second embodiment. The user authentication processing according to the second embodiment differs from the user ID estimation processing according to the first embodiment in the following respects. That is, step S101a is executed before step S101. Furthermore, step S107a is executed instead of step S107. Furthermore, steps S108a to S112a are executed instead of steps S108 to S114. Other steps that are the same as those in the user ID estimation processing according to the first embodiment are assigned the same step numbers, and descriptions thereof will be omitted.

[0086] First, in step S101a, the information processing terminal 3 acquires the input user ID.

[0087] In step S107a, the class estimation program 313 transmits to the server 5 the authentication templates for the left and right eyes generated in step S106, and the user ID and side ID received in step S101a.

[0088] Next, in step S108 a, the feature similarity score calculation program 516 acquires, from the registered templates 523, an authentication template that corresponds to the combination of the user ID and side ID received from the information processing terminal 3.

[0089] Next, in step S109a, the feature similarity score calculation program 516 compares the authentication template for the left eye received from the information processing terminal 3 with the feature of the authentication template acquired in step S108a, and calculates the similarity score for the left eye. In step S110a, the feature similarity score calculation program 516 compares the authentication template for the right eye received from the information processing terminal 3 with the feature of the authentication template acquired in step S108a, and calculates the similarity score for the right eye. Steps S109a and S110a can be performed in any order.

[0090] Next, in step S111a, the feature similarity score calculation program 516 integrates the similarity scores calculated in steps S109a and S110a and performs authentication. The feature similarity score calculation program 516 transmits the authentication result to the information processing terminal 3.

[0091] The authentication decision in step S111a is performed as follows: authentication decision 1 or authentication decision 2. Authentication decision 1: If the similarity score for the left eye or the right eye is equal to or greater than a preset threshold Th, the authentication is deemed successful; if it is less than the threshold Th, the authentication is deemed unsuccessful. Authentication decision 2: If the similarity score for the left eye and the similarity score for the right eye are equal to or greater than a preset threshold Th, the authentication is deemed successful; if they are less than the threshold Th, the authentication is deemed unsuccessful.

[0092] Note that the similarity scores of the left and right periocular images may be selected by selecting the highest similarity score among a plurality of consecutive periocular images.

[0093] In step S112a, the class estimation program 313 receives authentication result information from the server 5. If the class estimation program 313 receives information indicating successful authentication, it displays the result on the display device 36 in step S115. On the other hand, if the authentication fails, the class estimation program 313 returns the process to step S101, or displays an authentication error on the display device 36 after a certain period of time has elapsed.

[0094] (Effects of the second embodiment) According to this embodiment, the trained inference model 321 can be used to improve the estimation accuracy of the user ID + side ID of the periocular images of the left and right eyes, even in ID authentication where the user ID is entered during authentication.

[0095] [Embodiment 3] In the first embodiment, when updating the parameters, an error calculation of back error propagation is performed, and then the update of the parameters of the specific class of the inference model during training 522 is stopped. However, this is not limited to this, and the update of the parameters of the inference model during training 522 can also be stopped by converting the left and right periocular features into a class probability distribution and setting the class probability distribution of the specific class to 0. Hereinafter, as the third embodiment, an embodiment will be described in which the update of the parameters of the specific class of the inference model during training 522 is stopped by setting the class probability distribution of the specific class to 0.

[0096] In the explanation of the third embodiment, the differences from the first embodiment will be mainly explained, and explanations of parts that overlap with the first embodiment will be omitted.

[0097] (Parameter update process according to the third embodiment) 8 is a flowchart showing the parameter update processing according to the third embodiment. The parameter update processing according to the third embodiment differs from the parameter update processing according to the first embodiment in the following respects. That is, step S206a is executed instead of step S206, and step S207a is executed instead of step S207. Other steps that are the same as those in the parameter update processing according to the third embodiment are given the same step numbers, and descriptions thereof will be omitted.

[0098] In step S206a, the model parameter update suppression program 515 sets 0 to the probability value of the class ID corresponding to the user ID + opposite side ID in the probability distribution of class IDs with dimensions equal to the number of types of class IDs (user ID + side ID) in step S205. The opposite side ID is the side ID on the opposite side to a certain side ID. Next, in step S207a, the model parameter update program 513 performs error calculation.

[0099] (Effects of the third embodiment) According to this embodiment, similar to the first and second embodiments, in ID-less authentication, the accuracy of estimating the class IDs of the images around the left and right eyes can be improved using the trained inference model 321.

[0100] The present invention is not limited to the above-described embodiments, and includes various modifications. Furthermore, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the described configurations. Furthermore, not only are such configurations removed, but replacement or addition of configurations is also possible. Furthermore, embodiments of the present invention also include those in which some or all of the above-described embodiments are appropriately combined to be consistent. Furthermore, the order of execution of each step of the various processes in the above-described embodiments can be changed as long as the processing results are the same.

[0101] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that implements the functions of the embodiments. In this case, a recording medium on which the program code is recorded is provided to a computer, and a processor within the computer reads the program code stored in the recording medium.

[0102] In this case, the program code itself read from the recording medium will realize the functions of the above-described embodiment, and the program code itself and the recording medium on which it is stored constitute the present invention. Examples of recording media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0103] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, and Java (registered trademark).

[0104] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected. [Explanation of symbols]

[0105] 1: Input biometric information 1, 3: Information processing terminal, 5: Server, 30: Processor, 31: Memory, 50: Processor, 51: Memory, 321: Post-trained inference model, 522: Inference model under training, 523: Enrollment template.

Claims

1. An authentication system for authenticating a user, comprising: The authentication system includes: a processor and a memory, The processor: Obtaining periocular images from the user's face image, the periocular images including a left periocular image including the user's left eye and the area around the left eye, and a right periocular image including the user's right eye and the area around the right eye; performing image transformation on a predetermined one of the acquired periocular images, the image transformation including either or both of image inversion, including vertical inversion or horizontal inversion of the one periocular image, and image rotation, rotating the one periocular image by a predetermined angle about a predetermined center of the one periocular image; The periocular image including the one periocular image on which the image transformation has been performed is input to an inference model, wherein the inference model outputs a feature quantity of the periocular image for the input periocular image; acquiring a feature of the periocular image output from the inference model in response to the input of the periocular image; The user is authenticated based on a comparison result between the acquired feature amount and a registered template in which a class ID including a user ID for identifying the user and feature amounts of the image around the left eye and the image around the right eye of the user are registered in advance in association with each other. An authentication system comprising:

2. 2. The authentication system according to claim 1, The image transformation is the upside-down An authentication system comprising:

3. 2. The authentication system according to claim 1, The image transformation is the image rotation rotating the periocular image by 90° about the predetermined center. An authentication system comprising:

4. 2. The authentication system according to claim 1, The processor: When authenticating the user, narrowing down candidates who are presumed to be the user based on a comparison result between the acquired feature amount and the feature amount in the registered template; extracting a second feature amount from the face image; The user is authenticated based on a comparison result between the second feature amount and the feature amount in a second registered template in which the class ID of the candidate and the feature amount of the facial image of the user are registered in advance in association with each other. An authentication system comprising:

5. 5. The authentication system according to claim 4, The processor: extracting the feature amount from the periocular image including the one periocular image on which the image transformation has been performed; The periocular image is input to the inference model to calculate a probability distribution of the class ID of the periocular image; Calculating an error between the calculated probability distribution and a probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; The parameters of the inference model are updated to reduce the calculated error, and the parameters are updated to reduce the error between the class ID of one of the periocular images in the correct answer data and the class ID of the other periocular image, or the update of the parameters relating to the class ID of the other periocular image is stopped. An authentication system comprising:

6. An authentication system for authenticating a user, comprising: The authentication system includes: a processor and a memory, The processor: Obtaining periocular images from teacher data of the user's face image, the periocular images including a left periocular image including the user's left eye and the area around the left eye, and a right periocular image including the user's right eye and the area around the right eye; performing image transformation on a predetermined one of the periocular images, the image transformation including either or both of image inversion including vertical inversion or horizontal inversion of the one periocular image, and image rotation for rotating the one periocular image by a predetermined angle about a predetermined center of the one periocular image; The periocular image including the one periocular image on which the image transformation has been performed is input to an inference model, wherein the inference model outputs a feature quantity of the periocular image for the input periocular image; acquiring a feature of the periocular image output from the inference model in response to the input of the periocular image; calculating a probability distribution of class IDs including a user ID for identifying the user of the periocular image based on the feature amount of the acquired periocular image; Calculating an error between the calculated probability distribution and a probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; The parameters of the inference model are updated to reduce the calculated error, and the parameters are updated to reduce the error between the class ID of one of the periocular images in the correct answer data and the class ID of the other periocular image, or the update of the parameters relating to the class ID of the other periocular image is stopped. An authentication system comprising:

7. An authentication method performed by an authentication system that authenticates a user, comprising: The authentication system includes: a processor and a memory, the processor: Obtaining periocular images from the user's face image, the periocular images including a left periocular image including the user's left eye and the area around the left eye, and a right periocular image including the user's right eye and the area around the right eye; performing image transformation on a predetermined one of the acquired periocular images, the image transformation including either or both of image inversion, including vertical inversion or horizontal inversion of the one periocular image, and image rotation, rotating the one periocular image by a predetermined angle about a predetermined center of the one periocular image; The periocular image including the one periocular image on which the image transformation has been performed is input to an inference model, wherein the inference model outputs a feature quantity of the periocular image for the input periocular image; acquiring a feature of the periocular image output from the inference model in response to the input of the periocular image; The user is authenticated based on a comparison result between the acquired feature amount and a registered template in which a class ID including a user ID for identifying the user and feature amounts of the image around the left eye and the image around the right eye of the user are registered in advance in association with each other. An authentication method comprising the steps of:

8. 8. The authentication method according to claim 7, The image transformation is the upside-down 1. An authentication method comprising:

9. 8. The authentication method according to claim 7, The image transformation is the image rotation rotating the periocular image by 90° about the predetermined center.

1. An authentication method comprising:

10. 8. The authentication method according to claim 7, The processor: When authenticating the user, narrowing down candidates who are presumed to be the user based on a comparison result between the acquired feature amount and the feature amount in the registered template; extracting a second feature amount from the face image; The user is authenticated based on a comparison result between the second feature amount and the feature amount in a second registered template in which the class ID of the candidate and the feature amount of the facial image of the user are registered in advance in association with each other.

1. An authentication method comprising:

11. 11. The authentication method of claim 10, the processor: extracting the feature amount from the periocular image including the one periocular image on which the image transformation has been performed; The periocular image is input to the inference model to calculate a probability distribution of the class ID of the periocular image; Calculating an error between the calculated probability distribution and a probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; The parameters of the inference model are updated to reduce the calculated error, and the parameters are updated to reduce the error between the class ID of one of the periocular images in the correct answer data and the class ID of the other periocular image, or the update of the parameters relating to the class ID of the other periocular image is stopped. An authentication method comprising the steps of:

12. An authentication method performed by an authentication system that authenticates a user, comprising: The authentication system includes: a processor and a memory, the processor: Obtaining periocular images from teacher data of the user's face image, the periocular images including a left periocular image including the user's left eye and the area around the left eye, and a right periocular image including the user's right eye and the area around the right eye; performing image transformation on a predetermined one of the periocular images, the image transformation including either or both of image inversion including vertical inversion or horizontal inversion of the one periocular image, and image rotation for rotating the one periocular image by a predetermined angle about a predetermined center of the one periocular image; The periocular image including the one periocular image on which the image transformation has been performed is input to an inference model, wherein the inference model outputs a feature quantity of the periocular image for the input periocular image; acquiring a feature of the periocular image output from the inference model in response to the input of the periocular image; calculating a probability distribution of class IDs including a user ID for identifying the user of the periocular image based on the feature amount of the acquired periocular image; Calculating an error between the calculated probability distribution and a probability distribution of the class ID in the correct answer data corresponding to the user corresponding to the periocular image; The parameters of the inference model are updated to reduce the calculated error, and the parameters are updated to reduce the error between the class ID of one of the periocular images in the correct answer data and the class ID of the other periocular image, or the update of the parameters relating to the class ID of the other periocular image is stopped. An authentication method comprising the steps of:

Images