Information processing apparatus

Abstract

To more efficiently perform specification of a person representing a person to be authenticated and selection of a selection item.SOLUTION: An information processing apparatus has: specification means that specifies a person representing a person to be authenticated on the basis of a face feature amount extracted from one or more face images including the face of the person to be authenticated; recognition means that recognizes face input performed by the person to be authenticated from the one or more face images including the face of the person to be authenticated; and selection means that selects one of a plurality of selection items as a selection item corresponding to the person specified by the specification means, on the basis of the face input recognized by the recognition means.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an authentication system, and a processing method and program for the information processing device. [Background technology]

[0002] In recent years, many techniques have been proposed for extracting useful information by advanced image processing. In particular, face recognition, which uses a multi-layered neural network called a deep net to compare an input face image with face images of multiple people registered in advance and determine who the input face image is, has been actively researched and developed. The deep net is also called a deep neural network or deep learning.

[0003] Furthermore, as the accuracy of facial recognition improves, empty-handed payment systems and empty-handed automatic ticket gate systems that use facial recognition are also being developed. These systems use what is known as 1:N authentication. In 1:N authentication, the unique facial feature information (facial feature amounts) of the person to be authenticated (person to be authenticated) obtained from the face image of that person is compared with facial feature amounts pre-registered in the system to identify who the person is. With 1:N authentication, the person to be authenticated can make payments and pass through automatic ticket gates using only their face, without the need for an ID card or other accessories.

[0004] When using a system that uses face recognition, convenience is improved if the authenticated person can make some choices in addition to identifying the person through face recognition. One example is the choice of payment method. It is convenient to be able to choose to pay sometimes with a payment method offered by credit company A and sometimes with a payment method offered by credit company B.

[0005] Another example is account selection. For example, when checking into a hotel, it would be convenient to be able to check in as an individual member at one time and as a corporate member at another time. Another example is service selection. It would be convenient to be able to select a desired service at the same time as identifying the user by face recognition.

[0006] Facial recognition uses the face, which is a unique piece of biometric information for each person. Therefore, in order to allow the person to be authenticated to make some kind of choice at the same time, the system must be specially designed.

[0007] The information processing device disclosed in Patent Document 1 uniquely identifies the person to be authenticated by face recognition, and then displays a list of accounts linked to the person to be authenticated on a display unit.Then, the person to be authenticated is prompted to select an account to be used by the person to be authenticated using an input device such as a mouse or a keyboard.

[0008] Furthermore, in the information processing device disclosed in Non-Patent Document 1, the person to be authenticated uses input devices such as a mouse and a keyboard to select an account. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2011-108148 A [Non-patent literature]

[0010] [Non-Patent Document 1] Deng, Jiankang, et al. “Arcface: Additive angular margin loss for deep face recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. Summary of the Invention [Problem to be solved by the invention]

[0011] In the information processing devices disclosed in Patent Document 1 and Non-Patent Document 1, the person to be authenticated uses input devices such as a mouse or keyboard to select an account, and there is a risk that the process of selecting an account requires a lot of work, which can be time-consuming.

[0012] It is an object of the present disclosure to enable more efficient identification of a person representing a subject and selection of a selection item. [Means for solving the problem]

[0013] The information processing device has an identification means for identifying a person representing the person to be authenticated based on facial features extracted from one or more facial images including the face of the person to be authenticated, a recognition means for recognizing a facial input made by the person to be authenticated from one or more facial images including the face of the person to be authenticated, and a selection means for selecting one of a plurality of selection items as a selection item corresponding to the person identified by the identification means based on the facial input recognized by the recognition means. Effect of the Invention

[0014] According to the present disclosure, it is possible to more efficiently identify a person representing a person to be authenticated and select an option. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an example of a configuration of an authentication system. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of an authentication device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an authentication system. [Figure 4] 13 is a flowchart showing the flow of a registration process by the authentication system. [Diagram 5] FIG. 13 is a diagram showing an example of a screen displayed on the registration device. [Figure 6] FIG. 11 is a diagram showing an example of member information stored in the registration process. [Figure 7] 13 is a flowchart showing a flow of authentication processing by the authentication system. [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of an authentication system. [Figure 9] 13 is a flowchart showing a flow of authentication processing by the authentication system. [Figure 10] FIG. 13 is a diagram illustrating an example of a screen displayed on the authentication device. [Figure 11] 13 is a flowchart showing the flow of a registration process by the authentication system. [Figure 12] 13 is a flowchart showing a flow of authentication processing by the authentication system. [Figure 13] FIG. 13 is a diagram illustrating an example of a selection table. [Figure 14] FIG. 13 is a diagram illustrating an example of a screen displayed on the authentication device. [Figure 15] 13 is a flowchart showing a flow of authentication processing by the authentication system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings.

[0017] (First embodiment) In the first embodiment, a method will be described in which, when a person to be authenticated passes through an automatic ticket gate using face recognition, the person to be authenticated is identified and at the same time, the person to be authenticated is allowed to select a fare payment method.

[0018] 1 is a diagram showing an example of a configuration of an authentication system 1 according to the first embodiment. The authentication system 1 includes a management device 10, a registration device 20, and an authentication device 30.

[0019] Management device 10 is an example of an information processing device, and is a device for storing and managing member information of the person to be authenticated who uses authentication system 1. The member information includes, for example, a facial image or facial feature amount of the person to be authenticated, a member ID, account information, etc. One management device 10 is placed in a cloud, a data center, etc.

[0020] Registration device 20 is a device for performing registration processing for the person to be authenticated to use authentication system 1. Registration device 20 is, for example, a smartphone carried by the person to be authenticated or a terminal installed in a station.

[0021] The authentication device 30 is a device for performing authentication processing for an authenticatee to use the authentication system 1 and receive a service. In this embodiment, the service means passing through a ticket gate. The authentication device 30 is, for example, a terminal provided at each entrance side or exit side of an automatic ticket gate.

[0022] The management device 10, the registration device 20, and the authentication device 30 are connected to each other via a network 40. Data transmitted and received by these devices via the network 40 is encrypted by a known encryption method to prevent tampering and eavesdropping. In the configuration example of FIG. 1, there is one registration device 20 and one authentication device 30, but there may be multiple registration devices 20 and multiple authentication devices 30. In addition, the management device 10 may also be divided into multiple devices and distributed.

[0023] In addition, the management device 10, the registration device 20, and the authentication device 30 do not need to be independent devices, and may be an integrated device. For example, a single device may function as both the registration device 20 and the authentication device 30. In addition, a single device may function as both the management device 10 and the registration device 20.

[0024] Fig. 2 is a diagram showing an example of a hardware configuration of the authentication device 30 in Fig. 1. The authentication device 30 has, as its hardware configuration, a control device 31, a storage device 32, a calculation device 33, an input device 34, an output device 35, and an I / F device 36. Note that the management device 10 and the registration device 20 can also be configured in the same manner as the authentication device 30, and therefore a description thereof will be omitted.

[0025] The control device 31 controls the entire authentication device 30. The storage device 32 holds programs and data necessary for the operation of the control device 31. The arithmetic device 33 executes necessary arithmetic processing based on the control of the control device 31.

[0026] The input device 34 is a human interface device or the like, and inputs operations of the person to be authenticated to the authentication device 30. The output device 35 is a display or the like, and presents the processing results of the authentication device 30 to the person to be authenticated. The input device 34 and the output device 35 may be configured as a single device that serves as both the input device 34 and the output device 35, such as a touch display.

[0027] The I / F device 36 is a wired interface such as a universal serial bus, Ethernet, or optical cable, or a wireless interface such as Wi-Fi or Bluetooth. The I / F device 36 has functions such as connecting a camera or the like to input a captured image to the authentication device 30, transmitting a processing result obtained by the authentication device 30 to the outside, and inputting programs, data, etc. required for the operation of the authentication device 30 to the authentication device 30.

[0028] Fig. 3 is a diagram showing an example of a functional configuration of the authentication system 1 of Fig. 1. The authentication system 1 includes a management device 10, a registration device 20, and an authentication device 30. Below, examples of the functional configurations of the management device 10, the registration device 20, and the authentication device 30 will be described in detail.

[0029] The functional configuration of Fig. 3 is realized by the control device 31 of Fig. 2 reading and executing a predetermined program from the storage device 32. The control device 31 is, for example, a CPU. However, all or part of the functional configuration of Fig. 3 may be executed by a hardware processor other than the control device 31. Examples of hardware processors other than the control device 31 include, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0030] [Management device] The management device 10 includes an image acquisition unit 101 , a facial feature extraction unit 102 , a facial authentication unit 103 , a recognition unit 104 , a determination unit 105 , a member information management unit 106 , and a storage unit 107 .

[0031] The image acquisition unit 101 acquires a facial image sequence required for facial authentication and facial input recognition from the registration device 20 or the authentication device 30. A facial image sequence refers to one or more images that are continuous in time and contain the face of the person to be authenticated. The facial image sequence may be configured to include the surrounding area of ​​the face. Facial input will be described later. The facial image sequence acquired by the image acquisition unit 101 is sent to a facial feature extraction unit 102 and a recognition unit 104.

[0032] The facial feature amount extraction unit 102 acquires facial features to be used for face authentication from the sequence of face images acquired by the image acquisition unit 101. The facial feature amount acquired by the facial feature amount extraction unit 102 is sent to the face authentication unit 103.

[0033] The face authentication unit 103 compares the face feature amount acquired by the face feature amount extraction unit 102 with the face feature amounts (registered face feature amounts) for N people stored in the storage unit 107, and specifies whether the person to be authenticated is someone among the N people stored in the storage unit 107, or a person different from any of the N people. The authentication result of the face authentication unit 103 is sent to the determination unit 105.

[0034] The recognition unit 104 recognizes a face input made by the person to be authenticated from the face image sequence of the person to be authenticated acquired by the image acquisition unit 101. The recognized face input is sent to the determination unit 105.

[0035] Here, face input is a coined term that means both putting the face, body parts other than the face, a specific object, or an attached object into a specific state or changing them according to a predetermined method in order for the person to be authenticated to select one item from a plurality of selection items. The specific form of face input is not important as long as it is mechanically recognizable by the recognition unit 104 described later. However, it is desirable that face input recognition be performed using the face of the person to be authenticated and its surrounding area so that face authentication and face input recognition can be performed based on a sequence of images captured by the same imaging device. This provides economic advantages such as limiting the number of imaging devices to one and allowing the imaging parameters (angle of view, shutter speed, etc.) of the imaging device to be set only once.

[0036] The face input recognition by the recognition unit 104 may be performed from one image in the sequence of face images acquired by the image acquisition unit 101, or from a plurality of images in time series.

[0037] For example, consider the case where a face input of "face facing right" is to be recognized. In this case, the recognition unit 104 uses only one image in the face image sequence to determine whether or not the face of the person to be authenticated is facing right.

[0038] Consider the case where a face input of "turning to the right" is to be recognized. In this case, the recognition unit 104 uses a plurality of images in a time series of the face image sequence to determine whether or not there is a gradual change in the face direction from facing forward to turning to the right.

[0039] When face input is recognized from multiple images, the cost of implementing face input recognition by the recognition unit 104 is higher than when face input is recognized from a single image. However, it provides improved defense against presentation attacks by malicious third parties (attacks that attempt to break through authentication by holding up a paper with the face of the target printed on it to the camera).

[0040] The following (1) to (7) are examples of face inputs that can be recognized by the recognition unit 104 using only one image. These face inputs are merely examples and do not limit the types of face inputs.

[0041] (1) Condition of the facial organs The recognition unit 104 can acquire position information of facial organs by previously learning an organ point position detector that detects the positions of the end points of facial organs such as the eyes, nose, mouth, eyebrows, etc. For example, the recognition unit 104 can recognize face inputs such as "raised right eyebrow" and "raised left eyebrow."

[0042] In addition, the recognition unit 104 can distinguish between the open and closed states of the eyes and mouth by learning in advance a classifier that distinguishes between the open and closed states of facial features such as the eyes and mouth. For example, the recognition unit 104 can recognize face inputs such as "open mouth," "closed right eye," and "closed left eye."

[0043] The recognition unit 104 can also train a classifier to distinguish states more complicated than simple opening and closing. For example, the recognition unit 104 can recognize face inputs such as "mouth for 'a'" and "mouth for 'i'" by training an estimator that estimates characters from the shape of the mouth in advance.

[0044] These face inputs can also be recognized by a method that utilizes the position information of the face organs obtained from the above-mentioned organ point position detector. For example, if the organ point position detector is configured to output position information of the right end, left end, upper end, and lower end of the right eye, the recognition unit 104 can recognize whether the right eye is open or closed based on the organ point positions.

[0045] (2) Face direction The recognition unit 104 can identify the face direction by learning in advance a face direction estimator that estimates the yaw angle, pitch angle, and roll angle of the face direction. For example, the recognition unit 104 can recognize face inputs such as "face facing right," "face facing left," "face facing up," "face facing down," "face tilted to the right," and "face tilted to the left."

[0046] (3) Facial expression state The recognition unit 104 can recognize facial expressions by learning a facial expression classifier that recognizes facial expressions in advance. For example, the recognition unit 104 can recognize facial inputs such as "smile," "anger," "surprise," and "sadness."

[0047] (4) Line of sight direction The recognition unit 104 can acquire the gaze direction by learning in advance a gaze direction detector that detects the gaze direction. For example, the recognition unit 104 can recognize a face input such as "gaze to the right."

[0048] (5) Condition of body parts other than the face For example, consider a case where a hand is used as a specific body part. The recognition unit 104 can detect the position and state of fingers by learning in advance a hand detector that detects the positions of the finger joints. For example, the recognition unit 104 can recognize face inputs such as "a right hand with n fingers standing up (n is an integer between 1 and 5) placed next to the right ear" and "a right hand placed over the right eye."

[0049] (6) State of a specific object The recognition unit 104 can acquire the position of a specific object by learning in advance a detector for the specific object (for example, an employee ID card, a badge, a hat, etc., that only employees of a certain company have). For example, the recognition unit 104 can recognize a face input such as "an employee ID card placed next to the right ear."

[0050] Similarly, the recognition unit 104 can acquire the state of a specific object by learning a classifier for the state of the specific object in advance. For example, the recognition unit 104 can recognize face inputs such as a "face-up badge" and a "face-down badge."

[0051] (7) Condition of the equipment The recognition unit 104 can acquire the presence or absence of an article being worn (such as sunglasses or a mask) by learning a detector for the article in advance. For example, the recognition unit 104 can recognize a face input such as "with sunglasses."

[0052] Above, we have listed examples of facial input based on the state of the face, body parts other than the face, specific objects, and attached items. Time-series fluctuations in which the face, body parts other than the face, specific objects, and attached items change in sequence from state s1, state s2, ..., state sK (K is an integer of 2 or more) can also be defined as facial input. In other words, the following (8) to (14) are all recognizable facial inputs.

[0053] (8) Changes in facial organs (9) Changes in face direction (10) Changes in facial expression (11) Changes in gaze direction (12) Changes in body parts other than the face (13) Changes in specific objects (14) Changes in clothing

[0054] A facial input including such time-series changes is assumed to be recognized using a plurality of facial images in a time series. The recognition unit 104 applies the above-mentioned detector and classifier to each of the plurality of facial images in a time series, and by using the results, it becomes possible to recognize the facial input including time-series changes.

[0055] Below are examples of face inputs that can be recognized by the recognition unit 104 using a plurality of face images in time series. These face inputs are merely examples and do not limit the types of face inputs.

[0056] Examples of "variations in facial organs" include facial inputs such as "raising the right eyebrow", "raising the left eyebrow", "opening the mouth", "closing the right eye", "closing the left eye", and "lip-syncing the string of characters". Lip-syncing refers to the movement of the mouth when pronouncing a string of characters. The recognition unit 104 becomes able to recognize this facial input by previously training an estimator that estimates a string of characters from lip-syncing.

[0057] Examples of "changes in facial direction" include facial inputs such as "turn to the right," "turn to the left," "turn up," "turn down," "tilt the face to the right," "tilt the face to the left," and "shake the face from side to side."

[0058] An example of a "change in facial expression" is a facial input such as "changing one's facial expression from a straight face to a smiling face."

[0059] An example of "change in gaze direction" is face input such as "moving the gaze from right to left."

[0060] Examples of "movements of body parts other than the face" include facial inputs such as "swinging the right hand from right to left," "drawing a circle in the air with one's fingers," and "brushing one's hair back with the right hand."

[0061] An example of a "change in a specific object" is a facial input such as "turning a badge from face up to face down."

[0062] An example of a "change in wearable item" is a facial input such as "putting on sunglasses."

[0063] The face input recognition method described above is just an example, and any existing method can be applied to face input recognition.

[0064] The determination unit 105 determines the item selected by the person to be authenticated, based on the person authentication result by the face authentication unit 103, the face input recognized by the recognition unit 104, and the member information stored in the storage unit 107.

[0065] The member information management unit 106 cooperates with the registration device 20 to acquire the member information of the person to be authenticated, and stores it in the storage unit 107. The member information corresponds to, for example, a general account table or an account table, which will be described later. Details will be described later.

[0066] The storage unit 107 stores the member information of the person to be authenticated. The member information is stored in a machine-readable format such as a database or text.

[0067] [Registration device] Next, a detailed description will be given of an example of the functional configuration of the registration device 20. The registration device 20 includes an image capturing unit 201 and an image transmitting unit 202.

[0068] The image capturing unit 201 controls the image capturing device to capture images necessary for face authentication and face input recognition. The images captured by the image capturing unit 201 are sent to the image transmitting unit 202. The image transmitting unit 202 transmits the images captured by the image capturing unit 201 to the management device 10.

[0069] [Authentication device] Finally, a detailed description will be given of an example of the functional configuration of the authentication device 30. The authentication device 30 includes an image capturing unit 301 and an image transmitting unit 302.

[0070] The image capturing unit 301 controls the image capturing device to capture images necessary for face authentication and face input recognition. The images captured by the image capturing unit 301 are sent to the image transmitting unit 302. The image transmitting unit 302 transmits the images captured by the image capturing unit 301 to the management device 10.

[0071] An example of the functional configuration of the authentication system 1 has been described above. The above is just an example, and the functional configuration method is not limited to this. For example, the authentication device 30 may have the function of the facial feature extraction unit 102 of the management device 10, and the authentication device 30 may transmit facial features instead of facial images to the management device 10.

[0072] The authentication system 1 according to this embodiment performs registration processing and authentication processing. The flow of the registration processing and the flow of the authentication processing will be described below in this order.

[0073] [Registration process] Fig. 4 is a flowchart showing the flow of registration processing by the authentication system 1. Fig. 5(a) to (e) are diagrams showing examples of screens displayed on a touch display (input device / output device) provided in the registration device 20. An example of a registration processing method by the management device 10 and the registration device 20 will be described with reference to Fig. 4 and Fig. 5(a) to (e).

[0074] Here, the registration device 20 is assumed to be a smartphone held by the person to be authenticated, but may be another terminal such as a terminal installed at a station. The person to be authenticated first creates his / her own general account in the traditional manner using an email address and password through the operation of an app or the like pre-installed on the registration device 20, and performs the following processes while logged in to the general account.

[0075] When a general account is created, a new unique ID for distinguishing the person to be authenticated is issued by the member information management unit 106. This ID will hereinafter be referred to as the person ID. FIGS. 5(a) to 5(e) show examples of screens displayed on the touch display of the registration device 20. When the person to be authenticated operates the touch display of the registration device 20 to perform the registration process, the registration device 20 and the management device 10 cooperate as appropriate, and the member information of the person to be authenticated is stored in the storage unit 107 of the management device 10.

[0076] In step S401, the image capturing unit 201 captures a sequence of face images necessary for face authentication. When capturing the sequence of face images, as shown in Fig. 5(a), the registration device 20 may display a guide 501 to instruct the person to be authenticated to capture the face in a size suitable for face authentication. Here, the sequence of face images may be a single image.

[0077] The image transmitting unit 202 transmits the sequence of face images captured by the image capturing unit 201 to the image acquiring unit 101 of the management device 10.

[0078] In step S402, the image acquisition unit 101 acquires the sequence of face images transmitted by the image transmission unit 202. The face feature amount extraction unit 102 extracts (acquires) face features to be used for face authentication based on the sequence of face images of the person to be authenticated acquired by the image acquisition unit 101. A feature extractor that has been trained in advance is used to extract the facial feature amount. In the facial feature amount extraction unit 102, when a facial image that has been subjected to a predetermined normalization process is input to the feature extractor, biometric information (facial feature amount) called a feature amount for identifying the identity of a face is output from the feature extractor. Generally, the facial feature amount is often a feature vector of fixed dimensions (for example, 256 dimensions or 512 dimensions). The feature extractor is trained in advance so that facial feature amounts extracted from facial images of the same person show a high similarity, and facial feature amounts extracted from facial images of other people show a low similarity. The feature extractor can be trained by a known method, for example, as shown in Non-Patent Document 1.

[0079] The facial feature registration process from step S401 to S402 usually needs to be performed only once for one general account. However, since the appearance of a face changes over time, for example, the date and time when the facial feature was registered may be recorded, and if a predetermined number of years (for example, 5 years) have passed since the date and time when the facial feature was registered, the person to be authenticated may be prompted to re-register the facial feature.

[0080] The member information management unit 106 links the extracted facial feature amount to a person ID and stores (registers) it as member information in the storage unit 107. As an example, this member information is stored as a general account table shown in FIG. 6(a). The general account table is a table that stores data related to a general account, and stores personal information such as a member's name and email address and facial feature amount linked to a person ID. The general account table may additionally store a facial image in addition to the facial feature amount. By additionally storing a facial image, it becomes possible to regenerate the facial feature amount from the facial image when the facial feature amount extractor is updated in the future.

[0081] Steps S403 to S411 form a loop for creating an account. For example, if the person to be authenticated wishes to register two types of fare payment methods, the authentication system 1 repeats this loop twice.

[0082] In step S404, member information management unit 106 prompts the person to be authenticated to create an account. An example of the display on the touch display of registration device 20 is shown in Fig. 5(b). For example, when the person to be authenticated presses account addition button 502, member information management unit 106 creates an initial account.

[0083] In step S405, the member information management unit 106 presents to the person to be authenticated a list of face input options that the person to be authenticated can set in the form of a list or the like. An example of the display on the touch display provided in the registration device 20 is shown in Fig. 5(c). For example, when the person to be authenticated selects the pull-down menu 503, a list of options for face input that the person to be authenticated can select is displayed.

[0084] The face inputs selectable by the person to be authenticated may be controlled by an administrator depending on the situation in which the authentication device 30 is used. For example, in a situation where a large number of authentications per unit time are required, such as at a station ticket gate, only face inputs that can be executed and recognized at high speed can be included. Also, in a situation where a large amount of payment is likely to be made, only face inputs that can also be used for spoofing judgment (described in detail in the second embodiment) can be included to reduce the risk of mistaking the person for another person.

[0085] In step S406, the person to be authenticated selects one face input from the presented face inputs. This can be achieved, for example, by the person to be authenticated selecting one face input from the pull-down menu 503 in FIG. 5(c).

[0086] In step S407, the image capturing unit 201 captures an image of the person to be authenticated performing the face input selected in step S406. The image capturing is performed, for example, in the following procedure. First, after the person to be authenticated selects face input in step S406, the member information management unit 106 presents the screen of the registration device 20 as shown in FIG. 5(d) to the person to be authenticated. This screen displays a message 504 encouraging the person to perform face input and a guide 505 for matching faces, and prompts the person to be authenticated to capture the face input. Next, the member information management unit 106 requests the image capturing unit 201 of the registration device 20, and the image capturing unit 201 captures a face image sequence of the face input performed by the person to be authenticated. The captured face image sequence is sent to the image transmission unit 202. The image transmission unit 202 sends the received face image sequence to the image acquisition unit 101 of the management device 10.

[0087] In step S408, the image acquisition unit 101 acquires the sequence of face images transmitted by the image transmission unit 202. The recognition unit 104 recognizes the face input made by the person to be authenticated from the sequence of face images acquired by the image acquisition unit 101. An example of the implementation of face input recognition is as described above.

[0088] In step S409, the member information management unit 106 determines whether the face input recognized in step S408 matches the face input selected by the person to be authenticated in step S406. If they match, the process proceeds to step S410. If they do not match, the process returns to step S405 and starts over from displaying the list of face inputs. This process prevents the person to be authenticated from erroneously adopting a face input that cannot be implemented.

[0089] In step S410, the member information management unit 106 prompts the person to be authenticated to register a payment method to be linked to the account. The person to be authenticated inputs, for example, information required for payment at a credit company (such as an ID and password) and instructs to link the method to this account.

[0090] Next, the member information management unit 106 links the face input selected in step S406 and the payment information registered in step S410 to the initial account created in step S404, and stores them as an account table in the storage unit 107. A specific example will be described later.

[0091] In step S412, the member information management unit 106 prompts the person to be authenticated to set one of the created accounts as a default account. The default account is an account that is automatically selected when face input recognition fails in the authentication process described below. It is not always necessary to have the person to be authenticated explicitly select the default account, and the first account created may be set as the default account.

[0092] Fig. 5(e) shows an example of the screen of the registration device 20 presented to the person to be authenticated after completing the account creation process loop from steps S403 to S411 twice. In Fig. 5(e), information on two accounts is displayed in boxes 506 and 507, and the person to be authenticated can check the payment information and face input associated with each account. Also, mark 508 is a symbol indicating that this is the default account.

[0093] The membership information after the account creation process loop from steps S403 to S411 has been completed twice is represented, for example, as an account table shown in FIG. 6(b).

[0094] The account table is a table that stores member information related to accounts created by the person to be authenticated. In this example, there are two records in the account table because the person to be authenticated created two accounts. Each account is associated with information such as face input, payment method, and whether it is a default account or not. There is only one default account for each person to be authenticated.

[0095] As described above, in step S410, the member information management unit 106 functions as a registration unit, and registers an account table of multiple associations between face inputs and payment methods for each person ID, as shown in Fig. 6(b). Specifically, the member information management unit 106 controls to display face input options in Fig. 5(c) in step S405, and registers the account table of Fig. 6(b) of associations between face input selected from the face input options and payment methods in step S410.

[0096] An example of the flow of the registration process by the registration device 20 according to this embodiment has been described above. The flow of the registration process is not limited to this example, and various modifications are possible.

[0097] For example, a modification is possible in which multiple types of facial inputs are linked to one account. For example, multiple types of facial inputs, such as "waving the right hand" and "closing the right eye," can be linked to one account. This modification allows, for example, a person to be authenticated to normally execute a facial input of waving the right hand, but when the person's hands are full with luggage, to execute a facial input of closing the right eye, thereby increasing the convenience of the person to be authenticated.

[0098] In addition, if the facial inputs associated with multiple accounts are similar to each other, there is a risk that an account not intended by the person to be authenticated will be selected during authentication processing due to a failure to recognize the facial input. Therefore, a modification can be considered that makes it possible to determine the type of facial input to be associated with an account so that the facial inputs associated with the accounts are not similar to each other, in other words, so that the facial inputs can be easily distinguished by a machine.

[0099] An example of how to achieve this is shown below. Let us assume that the n face inputs that the person to be authenticated can set are numbered from a1 to an. Let us assume that the person to be authenticated has already linked k-1 face inputs b1, ..., b(k-1) to one of his / her accounts.

[0100] First, a function f(ai, aj) that defines the similarity between ai (i-th face input) and aj (j-th face input) is determined in advance. The similarity may be determined in any manner, for example, manually based on rules. Alternatively, the similarity may be determined statistically so that a high similarity is obtained for a pair of face inputs that are likely to cause mistakes in face input recognition by the recognition unit 104. In the following, f(ai, aj) returns a real number between 0 and 1.

[0101] Next, in step S406, it is assumed that the person to be authenticated is about to select face input a as the k-th face input. In step S406, the member information management unit 106 can be configured to allow the selection of face input a only if face input a is not similar to any of face inputs b1, ..., b(k-1) already selected by the person to be authenticated. This determination can be made, for example, by calculating a score max(f(a,b1),f(a,b2), ...,f(a,b(k-1))) for face input a. If this score is below a first threshold, it can be determined that "face input a is not similar to any of face inputs b1, ..., b(k-1) already selected by the person to be authenticated."

[0102] Also, in step S405, the member information management unit 106 may present to the person to be authenticated only face inputs that are not similar to any of the face inputs b1, ..., b(k-1) that the person to be authenticated has already selected. The member information management unit 106 performs control to display options for some face inputs that are easy to distinguish from each other by a machine among the multiple face inputs. This makes it easy for the person to be authenticated to select a face input only from among the face inputs that the person to be authenticated can set.

[0103] Furthermore, the member information management unit 106 can also recommend to the person to be authenticated a face input selected by the authentication system 1. For example, the member information management unit 106 defines a function g(a) that calculates a recommendation score for a face input a that has not yet been selected by the person to be authenticated. The member information management unit 106 then calculates a recommendation score for all face inputs a that have not yet been selected by the person to be authenticated, and presents the face input list to the person to be authenticated in order of the recommendation score sorted in descending order.

[0104] The recommendation score can be calculated by any method. For example, a function h(a) that quantifies the simplicity of the action of face input a is determined in advance. The function h(a) can be determined based on rules or machine learning. Then, a recommendation score g(a) for face input a that has not yet been selected by the person to be authenticated is determined as follows:

[0105] g(a)=(1-max(f(a,b1),f(a,b2),…,f(a,b(k-1))))*h(a)

[0106] This method allows the person to be authenticated to perform simple operations and makes it easier for the authentication system 1 to set a face input that reduces the chance of face input recognition failure.

[0107] [Authentication process] 7 is a flowchart showing the flow of authentication processing by the authentication system 1. An example of an authentication processing method by the management device 10 and authentication device 30 according to this embodiment will be described with reference to FIG.

[0108] Here, the authentication device 30 is a terminal provided at each entrance or exit side of the gate of the automatic ticket gate. The imaging device connected to the authentication device 30 is installed, for example, at a height of 1.6 m from the floor, at an angle that makes it difficult for the face of a person passing through the automatic ticket gate to be blocked.

[0109] An imaging range in which the face of the person to be authenticated can be imaged under suitable conditions is set in advance in the authentication device 30. The imaging range is set, for example, to a range from 2 m to 1 m in front of the gate of the automatic ticket barrier.

[0110] When walking toward the automatic ticket gate, the person to be authenticated looks at the imaging device while in the imaging range, and inputs his / her face for account selection at any timing while in the imaging range.

[0111] The authentication system 1 performs face authentication and account selection based on a sequence of face images captured within the imaging range. It is assumed that a person to be authenticated who passes through an automated ticket gate has already completed the above-mentioned registration process.

[0112] In step S701, the image capturing unit 301 captures a sequence of face images necessary for face authentication and face input recognition. This sequence of face images is composed of one or more face images including the face of the person to be authenticated. The sequence of face images is captured, for example, when the person to be authenticated enters the imaging range as a trigger. When the person to be authenticated turns toward the imaging device, the accuracy of face authentication and face input recognition increases, so the person to be authenticated may be encouraged to cooperate by using a method such as changing the color of the floor in the imaging range, placing symbols such as footprints on the floor in the imaging range, or notifying the person to be authenticated by light or sound when he or she enters the imaging range.

[0113] Furthermore, a touch display (both input device 34 and output device 35) may be installed at the automatic ticket gate. By displaying the image captured by the imaging device on the touch display in live view and superimposing a guide frame for aligning the face on the touch display, an image of the person to be authenticated can be captured more appropriately. In face input recognition described later, when face input using an object other than the face (e.g., a hand or an employee ID card) is to be recognized, the guide frame may be adjusted so that the surrounding area other than the face is also captured.

[0114] The image transmitting unit 302 transmits the sequence of face images captured by the image capturing unit 301 to the image acquiring unit 101 of the management device 10 .

[0115] In step S702, the image acquisition unit 101 acquires the face image sequence transmitted by the image transmission unit 302. The management device 10 performs 1:N authentication. This is realized, for example, by the following procedure.

[0116] First, the facial feature extraction unit 102 extracts facial features of the person to be authenticated from the sequence of facial images acquired by the image acquisition unit 101. This can be achieved by the method described in step S402. The same algorithm is basically used for extracting facial features in the registration process and in the authentication process.

[0117] There are various methods for determining the image from which the facial feature is extracted. For example, there is a method using a face image quality assessment score. The face image quality score is a numerical representation of how suitable a face image is for face recognition, and can be calculated using existing methods such as FaceQNet v1 (see Non-Patent Document 2 below).

[0118] (Non-patent document 2) Hernandez-Ortega, Javier, et al. “Biometric quality: Review and applica tion to face recognition with faceqnet.” arXiv preprint arXiv:2006.03298. 2020.

[0119] First, the facial feature amount extraction unit 102 calculates a facial image quality score for each facial image constituting the facial image sequence acquired by the image acquisition unit 101. Next, the facial feature amount extraction unit 102 extracts facial features from the top n (n is an integer equal to or greater than 1) facial images with the highest facial image quality scores. Finally, the facial feature amount extraction unit 102 calculates a simple average or a weighted average based on the facial image quality scores for the extracted facial feature amounts, and extracts the final facial feature amounts.

[0120] For example, when the person to be authenticated inputs his / her face by turning to the side or closing one eye, the facial image quality score is expected to be low, and when the person to be authenticated faces the imaging device with a straight face, the facial image quality score is expected to be high. Therefore, even if the person to be authenticated does not make any facial input action for a moment, highly accurate facial authentication is possible using the captured image.

[0121] Next, the face authentication unit 103 calculates a matching score between the face feature amount acquired by the face feature amount extraction unit 102 and the face feature amount (registered face feature amount) of N people stored in the storage unit 107. The registered face feature amount is acquired by referring to the member information (general account table). The matching score is calculated, for example, by the cosine similarity or L2 distance between the features.

[0122] In step S703, the face authentication unit 103 functions as an identification unit, and identifies the person with the highest matching score among the matching scores of the N people calculated in step S702 as person A representing the person to be authenticated. Then, the face authentication unit 103 compares the matching score of person A with a second threshold value, and if the matching score of person A is less than the second threshold value, it determines that the person to be authenticated is different from any of the N people, and proceeds to step S704. In step S704, the face authentication unit 103 determines that the identification of the person to be authenticated has failed. In this case, for example, the management device 10 closes the gate of the automatic ticket barrier, etc., to prompt the person to be authenticated to pass through by a method other than face authentication. At this time, the management device 10 may display the fact that the person identification has failed on a touch display installed in the authentication device 30.

[0123] Furthermore, if the matching score of person A is equal to or greater than the second threshold, the face authentication unit 103 determines that the person to be authenticated is person A, and proceeds to step S705. At this time, the management device 10 may display the person ID, name, etc. of person A on a touch display installed in the authentication device 30, and have the person to be authenticated correct any errors in the authentication result.

[0124] In step S705, the determination unit 105 refers to the member information (account table in FIG. 6(b)) stored in the storage unit 107 and counts the number of accounts linked to person A. If the number of accounts linked to person A is zero, the determination unit 105 proceeds to step S706. In step S706, the determination unit 105 determines that the selection of an account has failed. In this case, for example, the management device 10 may close the gate of the automatic ticket barrier, etc., to prompt the person to be authenticated to pass through using a method other than facial authentication. At this time, the management device 10 may display on a touch display installed on the authentication device 30 that an account does not exist.

[0125] If the number of accounts linked to person A is one, the determination unit 105 selects the one account, and the process proceeds to step S710. If the number of accounts linked to person A is two or more, the process proceeds to step S707.

[0126] In step S707, the recognition unit 104 recognizes a face input made by the person to be authenticated, based on the face image sequence of the person to be authenticated acquired by the image acquisition unit 101. The face input recognition can be performed by the same method as that described in step S408.

[0127] In step S708, the determination unit 105 checks the face input recognized in step S707 against the membership information (account table in FIG. 6(b)) stored in the storage unit 107, and identifies the account selected by the person to be authenticated. This can be achieved by sequentially referring to the records linked to person A in the account table, and searching for a record in which the same face input as the face input recognized in step S707 is registered.

[0128] If the determination unit 105 succeeds in identifying the account, it selects the identified account and proceeds to step S710. At this time, the management device 10 may display information on the identified account on a touch display installed in the authentication device 30, and have the person to be authenticated correct any errors. The correction may be performed, for example, by the person to be authenticated touching the touch display, or by the person to be authenticated performing a specific face input, such as turning his / her head, toward the imaging device.

[0129] Furthermore, if the determination unit 105 fails to identify the account, the process proceeds to step S709. This can occur, for example, when the recognition of the face input made by the person to be authenticated fails (including the case where the person to be authenticated does not perform face input in the first place) or when the face input made by the person to be authenticated is not linked to any account in the account table.

[0130] In step S709, the determination unit 105 refers to the member information stored in the storage unit 107 (the account table in FIG. 6(b)), identifies one default account, and selects the identified account. Steps S708 and S709 enable the authenticated person to pass through the automated ticket barrier without inputting a face image if he / she wishes to use the default payment method, and to input a face image of some kind if he / she wishes to use a payment method other than the default. After that, the process proceeds to step S710.

[0131] In step S710, the management device 10 performs entry processing and exit processing. In the entry processing, the management device 10 records information about the station where the authenticated person entered in the storage unit 107. In the exit processing, the management device 10 calculates the fare from the entry station to the exit station, selects a payment method linked to the account selected in any of steps S705, S708, and S709, and performs payment using the selected payment method.

[0132] As described above, the management device 10 functions as a selection unit, selects one of the multiple payment methods as a selection item corresponding to the person identified in step S703 based on the face input recognized in step S707, and performs payment using the selected payment method. Specifically, the management device 10 selects the payment method corresponding to the face input recognized in step S707 based on the account table of FIG. 6(b) showing multiple associations between face inputs and payment methods for the person identified in step S703. Note that the payment method is an example of a selection item, and is not limited thereto. That is, the management device 10 selects one of the multiple selection items as a selection item corresponding to the person identified in step S703 based on the face input recognized in step S707.

[0133] In step S709, if there is no payment method corresponding to the face input recognized in step S707 in the account table of FIG. 6(b), the management device 10 selects a default payment method.

[0134] The above describes the flow of the authentication process by the authentication device 30 of the first embodiment. However, the method of the authentication process is not limited to the above, and various modifications are possible.

[0135] For example, in the flowchart of Fig. 7, a method of performing 1:N face authentication and face input recognition in sequence has been described. This processing order does not need to be fixed, and may be modified, for example, so that 1:N face authentication and face input recognition are performed in parallel. This modification can further reduce the time required for authentication processing.

[0136] In addition, in the flowchart of FIG. 7, a method of using the face image sequence captured in step S701 for both 1:N face authentication and face input recognition has been described. This process may be modified so that imaging for 1:N face authentication is performed first, and then imaging for face input recognition is performed immediately after that. In this case, for example, the management device 10 may notify the authenticatee by light or sound on the touch display of the authentication device 30 whether imaging for 1:N face authentication or face input recognition is being performed, thereby encouraging the authenticatee to cooperate. For example, the authenticatee looks at the imaging device while a red lamp is on, and performs face input when the lamp changes to blue. Although the time required for the entire authentication process is longer, the accuracy of 1:N face authentication is improved because the front face for 1:N face authentication can be more reliably captured.

[0137] 7, a method for selecting a default account when face input recognition fails has been described. Alternatively, when face input recognition fails, the management device 10 may display a list of accounts on the touch display of the authentication device 30 and have the person to be authenticated explicitly select one of the accounts.

[0138] As described above, the authentication system 1 of the first embodiment allows the authenticated person to select a fare payment method at the same time as identifying the authenticated person when the authenticated person passes through an automatic ticket gate using face authentication. Also, by setting one default account in advance, the authenticated person only needs to input some face information when he / she wants to use a payment method other than the default, which increases the convenience of the authenticated person.

[0139] In the above example, the payment method is stored in the account table in FIG. 6(b), but various applications are possible by storing other information in the account table.

[0140] The first application is the proper use of multiple accounts on a personal computer or smartphone. As an example, consider a case where a person to be authenticated has created two accounts: account A without administrator authority and account B with administrator authority. By storing an account table such as that shown in Fig. 6(c) in advance in the storage unit 107, the accounts and face inputs are linked. The person to be authenticated can log in using a certain account by performing face input linked to that account.

[0141] In the example of Fig. 6(c), if the person to be authenticated undergoes face authentication without making any face input, he or she can log in with an account without administrator privileges, but if the person to be authenticated undergoes face authentication while making a specific face input (here, "close your right eye"), he or she can log in with an account with administrator privileges. In other words, it can be interpreted that some authority is added to the person to be authenticated only if the person to be authenticated makes an additional specific face input.

[0142] This concept can also be applied to other cases, such as when the person to be authenticated uses a bank's smartphone app. The authentication system 1 can be configured so that when the person to be authenticated logs in using only face recognition, only operations with a low security level, such as checking the balance, can be performed, but when the person to be authenticated performs face recognition while entering a specific face, all operations, such as transfers, can be performed.

[0143] The second application is the use of different account types. For example, consider a case where a person to be authenticated has created two accounts, a personal account and a corporate account (an account created by the corporation to which the person to be authenticated belongs), and sometimes checks into a hotel as a personal member and sometimes as a corporate member. By storing an account table such as that shown in Fig. 6(d) in advance in the storage unit 107, the account and the face input are linked. The person to be authenticated can check in with the desired account by entering the face stored in the account table.

[0144] The third application is the use of different services. For example, a certain payment method has a mechanism whereby points are generated according to the payment amount at the time of payment. The person to be authenticated creates in advance an account for accumulating points at the time of payment and an account for using points at the time of payment, as shown in the account table in Fig. 6(e). By entering the face image saved in the account table, the person to be authenticated can sometimes choose to accumulate points and other times to use points. Similarly, various types of choices can be made by linking the type of service the person to receive and their preferences to the account and saving them.

[0145] The payment method in Fig. 6(b) is an example of a selection item. Selection items include, for example, a payment method, the presence or absence of administrator authority, an account type, a service type, or a preference of the authenticatee.

[0146] As described above, according to this embodiment, the authentication system 1 can quickly identify the person to be authenticated through facial authentication and allow the person to select an option, without any special device other than the authentication device 30 for facial authentication and the management device 10.

[0147] Second embodiment In the second embodiment, a method is described that allows a person to select a payment method while identifying the person to be authenticated when making a payment at a cash register in a supermarket or convenience store using face recognition. In the second embodiment, it is described that the combination of spoofing determination using a face image sequence and knowledge recognition by face input provides two effects: more reliable personal authentication and prevention of unintended authentication processing. In the second embodiment, the same parts as in the first embodiment are not described, and only the differences are described.

[0148] Fig. 8 is a diagram showing an example of a functional configuration of an authentication system 1 according to the second embodiment. The authentication system 1 in Fig. 8 is obtained by adding a spoofing determination unit 108 to the authentication system 1 in Fig. 3. The spoofing determination unit 108 is added in the management device 10.

[0149] The spoofing determination unit 108 calculates a non-biome score, which is a score indicating the likelihood of a face image being a non-biome, from the face image sequence acquired by the image acquisition unit 101. Using the non-biome score, the spoofing determination unit 108 can quantify how close a face image is to a biome (a real person) or a non-biome (a person printed or displayed on an artificial object such as paper or a display).

[0150] The authentication system 1 according to this embodiment performs registration processing and authentication processing.

[0151] [Registration process] The registration process can be performed in the same manner as described in the first embodiment. The registration device 20 is assumed to be a smartphone held by the person to be authenticated, but may be another terminal such as a terminal installed in a store.

[0152] [Authentication process] Fig. 9 is a flowchart showing the flow of authentication processing in the authentication system 1 according to the second embodiment. Fig. 10 is a diagram showing an example of a screen displayed on the authentication device 30. An example of the flow of authentication processing by the management device 10 and the authentication device 30 according to this embodiment will be described with reference to Figs. 9 and 10. Here, the authentication device 30 is assumed to be a device provided in a cash register terminal in a supermarket or a convenience store, but may be another terminal such as a smartphone held by the person to be authenticated.

[0153] The flow of authentication processing will be described with reference to Fig. 9. In Fig. 9, the same processes as those described in Fig. 7 are given the same reference numerals. Below, the differences between Fig. 9 and Fig. 7 will be described.

[0154] In step S901, the image capturing unit 301 captures a sequence of face images necessary for face authentication, face input recognition, and spoofing determination.

[0155] An example of capturing a face image sequence will be described with reference to FIG. 10(a). The device 1001 is an example of the authentication device 30, and includes an imaging device 1002 and a touch display 1003 (input device 34 and output device 35). After the person to be authenticated has finished registering the product he or she wishes to purchase, a message for selecting a payment method is displayed on the touch display 1003. Here, the person to be authenticated taps a button 1004 for making a payment using face authentication, which is used as a trigger to cause the image capturing unit 301 to start capturing an image. When capturing an image, as shown in FIG. 10(b), the device 1001 may display a guide 1011 for matching faces, and instruct the person to be authenticated to capture the face in a size suitable for face authentication and account recognition.

[0156] The image transmitting unit 302 transmits the sequence of face images captured by the image capturing unit 301 to the image acquiring unit 101 of the management device 10 .

[0157] The processes in steps S702 and S703 are the same as those in Fig. 7. In step S703, if the matching score of person A is equal to or greater than the second threshold, the face authentication unit 103 determines that the person to be authenticated is person A, and proceeds to step S902.

[0158] In step S902, the determination unit 105 refers to the member information (account table in FIG. 6(b)) stored in the storage unit 107 and counts the number of accounts linked to person A. If the number of accounts linked to person A is zero, the determination unit 105 proceeds to step S706. In step S706, the determination unit 105 determines that the selection of an account has failed. In this case, the management device 10 uses the touch display 1003 to prompt the person to be authenticated to make payment by a method other than facial authentication. The management device 10 may also display on the touch display 1003 that an account does not exist.

[0159] On the other hand, if the number of accounts linked to person A is one or more, the determining unit 105 advances the process to step S707. After step S707, the process advances to step S903.

[0160] In step S903, the determining unit 105 collates the face input recognized in step S707 with the membership information (account table in FIG. 6(b)) stored in the storage unit 107, and identifies the account selected by the person to be authenticated.

[0161] If the determination unit 105 succeeds in identifying the account, it determines that this account has been selected, and the process proceeds to step S905. At this time, the management device 10 may display information on the identified account on the touch display 1003, and may have the person to be authenticated correct any errors that may be present. The correction may be performed, for example, by the person to be authenticated touching the touch display 1003, or by the person to be authenticated making a specific face input, such as a head shake, toward the imaging device 1002.

[0162] If the determination unit 105 fails to identify the account, the process proceeds to step S904. In step S904, the management device 10 stops the authentication process. That is, if there is no payment method corresponding to the face input recognized in step S707 in the account table of Fig. 6(b), the management device 10 does not select a payment method. The management device 10 may inform the person to be authenticated of the failure to identify the account via the touch display 1003.

[0163] In this embodiment, the person to be authenticated needs to input some kind of face in order to perform authentication processing. Inputting a face has two effects: more reliable personal authentication and prevention of unintended authentication processing.

[0164] First, the effect of more reliable personal authentication will be described. In this embodiment, the face input registered by the person to be authenticated during the registration process is treated as information (secret information) known only to the person to be authenticated. Therefore, the face input performed by the person to be authenticated during the authentication process also serves as knowledge authentication, and the defense against presentation attacks by malicious third parties is enhanced. Here, there is a risk that the face input as secret information will be leaked by performing face input openly. However, the risk of face input leaking can be reduced by a method of increasing the complexity of the face input (for example, a face input in which multiple types of actions are performed consecutively is considered as one face input) or a method of allowing only face input that is difficult to see from a person behind to be used.

[0165] Next, we will explain the effect of preventing unintended authentication processing. Face input in authentication processing is performed by the person to be authenticated of his / her own will. Therefore, for example, it is possible to prevent some processing (e.g. payment) from being performed by the person to be authenticated accidentally looking into a payment terminal placed in a public space.

[0166] In step S905, the spoofing determination unit 108 calculates a non-biometric score, which is a score representing the likelihood of a person being a non-biometric person, based on the sequence of face images acquired by the image acquisition unit 101. Any of the known techniques described in the following Patent Documents 2, 3, and 4 can be applied to the spoofing determination.

[0167] (Patent Document 2) JP 2019-197426 A (Patent Document 3) JP 2017-191374 A (Patent Document 4) JP 2022-99130 A

[0168] As an example, a case where a spoofing classifier that calculates a non-living score based on an image sequence is trained will be described. First, a set of image sequences containing living bodies and a set of image sequences containing non-living bodies are collected as training data. Then, using these training data, a binary classification neural network is trained that classifies living bodies in the input image sequence as class 0 and non-living bodies as class 1. By setting the activation function of the output layer of this neural network to a sigmoid function, the output of the neural network becomes a scalar between 0 and 1. This output value is used as the non-living score.

[0169] When the input image sequence for spoofing judgment includes face input such as head turning, the accuracy of spoofing judgment generally improves. This is because the paper on which the face image is printed is flat, while a real face is three-dimensional. Therefore, when the orientation of the paper on which the face image is printed is changed and when the orientation of the real face is changed, a difference occurs in the variation of the distance between organ points, making spoofing judgment easier. In other words, some face input contributes to improving the accuracy of spoofing judgment and improving the reliability of personal authentication.

[0170] The spoofing determination unit 108 transmits the calculated non-biometric score to the determination unit 105 .

[0171] In step S906, the determining unit 105 determines whether the person to be authenticated is biometric based on the non-biometric score. If the non-biometric score calculated in step S905 exceeds the third threshold, the determining unit 105 determines that the person to be authenticated is not biometric, and proceeds to step S907, assuming that the person to be authenticated is highly likely to be an impersonator. In step S907, the management device 10 stops the authentication process and does not select a payment method. At this time, it is desirable not to display the reason for stopping the authentication process, in order to prevent information from being provided to a malicious attacker.

[0172] On the other hand, if the non-biometric score is equal to or smaller than the third threshold, deciding unit 105 determines that the person to be authenticated is biometric, and proceeds to step S908.

[0173] In step S908, the management device 10 performs a payment process. In the payment process, the management device 10 selects a payment method associated with the account selected in step S903, and makes a payment for the amount equivalent to the product selected by the authenticated person using the selected payment method.

[0174] The flow of the authentication process by the authentication device 30 of the second embodiment has been described above. However, the method of the authentication process is not limited to the above, and various modifications are possible.

[0175] For example, the flowchart in Fig. 9 describes a method of performing 1:N face authentication, face input recognition, and masquerade determination in that order. This processing order does not need to be fixed, and may be modified, for example, so that 1:N face authentication, face input recognition, and masquerade determination are performed in parallel. This modification can further reduce the time required for authentication processing.

[0176] Furthermore, the additional process of spoofing determination is not necessarily required, and can be omitted depending on the security level required by the system and the allowable amount of calculation.

[0177] As described above, the authentication system 1 of the second embodiment allows the person to be identified and to select a payment method when the person to be authenticated uses face recognition to make a payment at a cash register in a supermarket, convenience store, etc. Furthermore, by combining spoofing judgment using a face image sequence and knowledge authentication by face input, two effects are achieved: more reliable person authentication and prevention of unintended authentication processing.

[0178] (Third embodiment) In the third embodiment, an effect of distinguishing similar subjects from each other can be added to the face input. In the third embodiment, the same parts as in the second embodiment will not be described, and only the different points will be described.

[0179] The authentication system 1 according to the third embodiment performs registration processing and authentication processing.

[0180] [Registration process] Fig. 11 is a flowchart showing the flow of registration processing in the authentication system 1 according to the third embodiment. An example of the flow of registration processing by the management device 10 and the registration device 20 according to this embodiment will be described with reference to Fig. 11. In Fig. 11, the same processes as those described in Fig. 4 are given the same reference numerals. Below, differences between Fig. 11 and Fig. 4 will be described.

[0181] In the following, we consider a case where there are two persons α and β whose faces are very similar to each other, and the similarity between the facial features extracted from their facial photographs is very high, making it impossible to distinguish between the persons based on the facial features alone. It is assumed that person α has already completed the registration process, and has created two accounts, with face input a1 and face input a2 set in each account. The registration process that person β will now carry out will be described in detail below.

[0182] Steps S401 to S404 are the same as those in Fig. 4. After step S404, the process proceeds to step S1105.

[0183] In step S1105, the member information management unit 106 presents a list of face input options that the person to be authenticated can set to the person to be authenticated in the form of a list or the like via the output device 35 of the registration device 20. An example is shown in Fig. 5(c). For example, when the person to be authenticated selects the pull-down menu 503, a list of face input options that the person to be authenticated can select is displayed.

[0184] This face input list is created in the following procedure: It is assumed here that person β is about to create his or her first account.

[0185] First, the member information management unit 106 calculates the similarity between the facial feature amount stored in the general account table of Fig. 6(a) and the facial feature amount of the person to be authenticated (person β in this case) extracted in step S402. Then, the member information management unit 106 lists the person IDs of people whose similarity is equal to or greater than a fourth threshold value as a high similarity person ID list. The high similarity person ID list also includes the person himself (person β in this case).

[0186] Next, the member information management unit 106 searches the account table in FIG. 6(b) for records whose high similarity person ID list includes a person ID, and lists the face inputs linked to those records as a used face input list.

[0187] Next, the member information management unit 106 removes the face inputs included in the used face input list from the face inputs that can be recognized by the authentication device 30, and sets the result as the face inputs that can be set by the person to be authenticated.

[0188] For example, if person α is the only person whose facial features are similar to person β, then the person ID of person α and the person ID of person β are listed in a high similarity person ID list. Next, face input a1 and face input a2 are listed in a used face input list. Finally, if a set of face inputs that can be recognized by the authentication device 30 is A, then a set obtained by removing face input a1 and face input a2 from set A becomes the face inputs that the person to be authenticated can set.

[0189] The member information management unit 106 performs control to display options for face inputs from among the multiple face inputs, excluding face inputs associated with the account tables of other persons to be authenticated who have face features whose similarity to the face features extracted in step S402 is equal to or greater than a fourth threshold value.

[0190] However, when the face input list created by the above method is presented to person β, person β can know that a person who looks similar to person β has already used face input a1 and face input a2. If person β has malicious intent, he or she can impersonate another person (person α in this case) by using face input a1 or face input a2 in the authentication process described below. Therefore, the member information management unit 106 may simultaneously remove face inputs selected randomly from the face inputs that can be recognized by the authentication device 30, in addition to the face inputs included in the used face input list. This makes it possible to prevent person β from impersonating another person.

[0191] The subsequent steps S406 to S412 are similar to those in FIG.

[0192] [Authentication process] Fig. 12 is a flowchart showing the flow of authentication processing in the authentication system 1 according to the third embodiment. An example of the flow of registration processing by the management device 10 and authentication device 30 according to this embodiment will be described with reference to Fig. 12. In Fig. 12, the same processes as those described in Fig. 9 are given the same reference numerals. Below, differences between Fig. 12 and Fig. 9 will be described.

[0193] Steps S901 and S702 are the same as those in Fig. 9. After step S702, the process proceeds to step S1201.

[0194] In step S1201, the face authentication unit 103 acquires, as a candidate list, all of the people whose matching scores are equal to or greater than the fifth threshold among the matching scores of the N people calculated in step S702. The candidate list is interpreted as a list of people who may be the same person as the person to be authenticated. If the number of people on the candidate list is 0, the person to be authenticated is determined to be different from any of the N people, and the process proceeds to step S704. If the number of people on the candidate list is 1 or more, the process proceeds to step S1202.

[0195] In step S704, the face authentication unit 103 determines that identification of the person to be authenticated has failed.

[0196] In step S1202, the determining unit 105 refers to the member information (account table in FIG. 6(b)) stored in the storage unit 107, and counts the total number of accounts linked to any of the people in the candidate list. If the total number of accounts is 0, the process proceeds to step S706. If the total number of accounts is 1 or more, the process proceeds to step S707.

[0197] In step S706, the determination unit 105 determines that the selection of the account has failed. Step S707 is the same as in Fig. 9. After step S707, the process proceeds to step S1203.

[0198] In step S1203, the determination unit 105 compares the face input recognized in step S707 with the member information (account table in FIG. 6(b)) stored in the storage unit 107, and simultaneously identifies the person to be authenticated and the account selected by the person to be authenticated. This can be achieved by sequentially referring to records linked to any of the people in the candidate list in the account table, and searching for a record in which the same face input as the face input recognized in step S707 is registered.

[0199] If the determination unit 105 succeeds in identifying the authenticatee and the account, the process proceeds to step S905. At this time, the management device 10 may display information on the identified authenticatee and the account on a touch display installed in the authentication device 30, and have the authenticatee correct any errors. The correction may be performed, for example, by the authenticatee touching the touch display, or by the authenticatee performing a specific face input, such as turning his / her head, toward the imaging device.

[0200] If the determination unit 105 fails to identify either the person to be authenticated or the account, the process proceeds to step S1204. In step S1204, the management device 10 stops the authentication process. The management device 10 may inform the person to be authenticated of this through a touch display or the like provided on the authentication device 30.

[0201] Steps S905 to S908 are similar to those in FIG.

[0202] As described above, the authentication system 1 of the third embodiment can add an effect of distinguishing between similar persons to be authenticated, to the face input.

[0203] (Fourth embodiment) In the fourth embodiment, unlike the first to third embodiments, it is explained that even if the person to be authenticated has not registered membership information corresponding to an account table in advance, the person to be authenticated can be identified by facial recognition and some kind of selection can be made by the person to be authenticated at the same time and at high speed.

[0204] In the following, as an example, a case will be described in which a person to be authenticated simultaneously performs biometric authentication using face recognition and selects an in-flight meal at the entrance to a boarding gate at an airport.

[0205] 14(a) is a diagram showing an example of a device 1401. The device 1401 is an example of the authentication device 30. The authentication device 30 is provided at the entrance of a boarding gate at an airport. The authentication device 30 includes an imaging device 1402 and a touch display 1403 (which also serves as an input device 34 and an output device 35). A text box 1404 clearly shows the correspondence between face input and items.

[0206] [Registration process] It is assumed that the registration process is performed only in steps S401 and S402 of the flowchart in Fig. 4 described in the first embodiment. In other words, it is assumed that only member information corresponding to the general account table in Fig. 6(a) is stored in the storage unit 107.

[0207] [Authentication process] An example of the flow of authentication processing by the management device 10 and authentication device 30 according to this embodiment will be described with reference to FIG.

[0208] It is assumed that a selection table indicating the association between face inputs and selection items is stored in advance in the storage unit 107. An example of the selection table is shown in Fig. 13(a). In this example, a face input with the right eye closed is associated with in-flight meal set A, and a face input with the mouth open is associated with in-flight meal set B. Hereinafter, the selection items will be simply referred to as items.

[0209] Fig. 15 is a flowchart showing the flow of authentication processing in the authentication system 1 according to the fourth embodiment. An example of the flow of authentication processing by the management device 10 and authentication device 30 according to this embodiment will be described with reference to Fig. 15. In Fig. 15, the same processes as those described in Fig. 9 are given the same reference numerals. Below, differences between Fig. 15 and Fig. 9 will be described.

[0210] Steps S901 and S702 to S704 are the same as those in Fig. 9. In step S703, if the matching score of person A is equal to or greater than the second threshold, the face authentication unit 103 determines that the person to be authenticated is person A, and proceeds to step S707.

[0211] Step S707 is the same as in Fig. 9. After step S707, the process proceeds to step S1501.

[0212] In step S1501, the determination unit 105 checks the face input recognized in step S707 against the selection table of Fig. 13(a) stored in the storage unit 107, and identifies the item selected by the person to be authenticated. This can be achieved by sequentially referring to the records in the selection table and searching for a record in which the same face input as the face input recognized in step S707 is registered. The selection table is a table of multiple associations between face inputs and selection items.

[0213] If there is no corresponding record, the determination unit 105 advances the process to step S1502. At this time, the management device 10 may display on the touch display 1403 that the face input recognition has failed.

[0214] Furthermore, if a corresponding record exists, the determining unit 105 selects the selection item linked to that record (i.e., the in-flight meal selected by the person to be authenticated) and stores it in the storage unit 107. At this time, the management device 10 may display information on the selected selection item on the touch display 1403, and allow the person to correct any errors if there are any. Thereafter, the process proceeds to step S905.

[0215] In step S1502, the determination unit 105 prompts the person to be authenticated to select an in-flight meal. This can be achieved, for example, by arranging an A set of buttons 1405 and a B set of buttons 1406 corresponding to the selection items of the device 1401 on the touch display 1403 as shown in FIG. 14(b). If there is no selection item corresponding to the face input recognized in step S707 in the account table of FIG. 6(b), the determination unit 105 selects a selection item according to the operation of the person to be authenticated. The determination unit 105 stores the selected selection item in the storage unit 107. After that, the process proceeds to step S905.

[0216] Steps S905 to S907 are the same as those in Fig. 9. In step S906, if the non-biometric score is equal to or less than the third threshold, determining unit 105 determines that the person to be authenticated is highly likely to be biometric, and advances the process to step S1503.

[0217] In step S1503, the management device 10 performs post-processing such as opening the boarding gate and preparing the in-flight meal based on the identified person ID and the information on the selected option (i.e., the in-flight meal selected by the person to be authenticated).

[0218] As described above, the authentication system 1 of the fourth embodiment can perform face authentication and some selection simultaneously at high speed. Also, as described in the second embodiment, by having the person to be authenticated input his / her face, it is possible to improve the accuracy of spoofing determination.

[0219] The process of the authentication system 1 is not limited to the above, and various modifications are possible. For example, it can be modified so that it is not necessary to clearly indicate the correspondence between face input and items, as in the text box 1404 in Fig. 14(a). This can be realized by informing the person to be authenticated in advance that the face input of "nodding" corresponds to "yes" and the face input of "shaking the head" corresponds to "no."

[0220] Also, for example, if the person to be authenticated does not input his / her face, it can be modified so that a default item is considered to have been selected. For example, a selection table as shown in FIG. 13(b) is stored in the storage unit 107. If the person to be authenticated does not actively make a selection (which may include a case where the person has walked away from the authentication device 30), the default in-flight meal of "random" can be selected. This modification has the effect of allowing the person to omit selection when he / she feels burdened by making an active selection. Also, since a certain percentage of the people to be authenticated are expected to select "random," it becomes easy for the service provider to adjust the number of in-flight meal options.

[0221] In this example, the default is set to "automatic," but this may be further modified so that an item that the person to be authenticated is likely to select (for example, "Set A") is set as the default based on the past selection results of the person to be authenticated, etc. This modification increases the probability that the person to be authenticated will not need to perform face input, and has the effect of improving the convenience of being authenticated.

[0222] Also, for example, it can be modified so that an additional option is considered to have been selected only when the person to be authenticated inputs his / her face. For example, a selection table as shown in FIG. 13(c) is stored in the storage unit 107. If the person to be authenticated does not proactively make a selection (which may include a case where the person has walked away from the authentication device 30), a basic in-flight meal as a default menu can be selected. Also, if the person to be authenticated inputs a special face (here, "close the right eye"), a special in-flight meal menu requiring an additional fee can be selected. With this modification, only the person to be authenticated who requires an additional option needs to input his / her face, and therefore it is possible to prevent an additional option that is not intended by the person to be authenticated from being erroneously selected.

[0223] As described above, the authentication system 1 of the fourth embodiment can simultaneously and quickly identify the person to be authenticated through facial recognition and allow the person to make some selection, even if the person to be authenticated has not registered membership information corresponding to an account table in advance.

[0224] (Other embodiments) Although the above-mentioned embodiment has been described in detail, the present invention is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the gist described in the claims.

[0225] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0226] The above-described embodiments are merely illustrative examples, and the technical scope should not be interpreted as being limited thereby. That is, the present disclosure can be implemented in various forms without departing from its technical concept or main characteristics.

[0227] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A means for identifying a person representing the person to be authenticated based on facial feature values ​​extracted from one or more facial images including the face of the person to be authenticated; a recognition means for recognizing a face input made by the person to be authenticated from one or more face images including the face of the person to be authenticated; a selection means for selecting one of a plurality of selection items as a selection item corresponding to the person identified by the identification means, based on the face input recognized by the recognition means; 13. An information processing device comprising: (Configuration 2) The information processing device of configuration 1, wherein the facial input is a state of facial organs, a state of facial direction, a state of facial expression, a state of gaze direction, a state of a body part other than the face, a state of a specific object, a state of an attachment, a change in a facial organ, a change in a facial direction, a change in a facial expression, a change in a gaze direction, a change in a body part other than the face, a change in a specific object, or a change in an attachment. (Configuration 3) The information processing device according to configuration 1 or 2, characterized in that the selection means selects a selection item corresponding to the face input recognized by the recognition means based on a table of multiple correspondences between face inputs and selection items for the person identified by the identification means. (Configuration 4) 4. The information processing apparatus according to configuration 3, further comprising a registration means for registering a table of a plurality of associations between face inputs for each person and selection items. (Configuration 5) 5. The information processing apparatus according to configuration 4, wherein the registration means controls to display options for face input, and registers a table correlating a face input selected from the options for face input with a selection item. (Configuration 6) 6. The information processing apparatus according to configuration 5, wherein the registration means performs control so as to display options for some face inputs that are easily distinguishable from each other by a machine among a plurality of face inputs. (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the identification means extracts facial features based on a facial image quality score that quantifies how suitable the facial image is for facial recognition. (Configuration 8) The information processing device according to any one of configurations 3 to 6, characterized in that the selection means selects a default selection item when there is no selection item in the table corresponding to the face input recognized by the recognition means. (Configuration 9) The information processing device according to any one of configurations 3 to 6, characterized in that the selection means does not select a selection item when there is no selection item in the table corresponding to the face input recognized by the recognition means. (Configuration 10) The information processing device according to any one of configurations 3 to 6, characterized in that when there is no selection item in the table corresponding to the face input recognized by the recognition means, the selection means selects a selection item in response to an operation of the person to be authenticated. (Configuration 11) The method further includes a determination unit for determining whether or not the person to be authenticated is a living body based on one or more facial images including the face of the person to be authenticated, The information processing device according to any one of configurations 1 to 10, wherein the selection means selects one of the plurality of selection items when the determination means determines that the person to be authenticated is a living body. (Configuration 12) 12. The information processing apparatus according to configuration 11, wherein the selection means does not select an option when the determination means determines that the person to be authenticated is not a living body. (Configuration 13) 6. The information processing apparatus according to configuration 5, wherein the registration means registers facial feature amounts extracted from one or more facial images including a face of a person to be authenticated. (Configuration 14) The information processing device described in configuration 13, characterized in that the registration means controls to display options for face inputs among multiple face inputs, excluding face inputs associated with tables of other subjects having face features whose similarity to the extracted face features is equal to or greater than a threshold value. (Configuration 15) 15. The information processing device according to any one of configurations 1 to 14, wherein the selection items are a payment method, the presence or absence of administrator authority, an account type, a service type, or a preference of the person to be authenticated. (Configuration 16) 3. The information processing device according to claim 1, wherein the selection means selects a selection item corresponding to the face input recognized by the recognition means based on a table of multiple correspondences between face inputs and selection items. (Configuration 17) An information processing device according to any one of configurations 1 to 16, capturing the face image; An authentication system comprising: (Method 1) a step of identifying a person representing the person to be authenticated based on facial feature amounts extracted from one or more facial images including the face of the person to be authenticated; a recognition step of recognizing a face input made by the person to be authenticated from one or more face images including the face of the person to be authenticated; a selection step of selecting one of a plurality of selection items as a selection item corresponding to the person identified in the identification step based on the face input recognized in the recognition step; 13. A processing method for an information processing apparatus comprising: (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 16. [Explanation of symbols]

[0228] 101 image acquisition unit, 102 face feature extraction unit, 103 face authentication unit, 104 recognition unit, 105 decision unit, 106 member information management unit, 107 storage unit

Claims

1. A means for identifying a person representing a person to be authenticated, based on facial features extracted from one or more facial images including the face of the person to be authenticated, Recognition means for recognizing a facial input performed by the person to be authenticated from one or more facial images including the face of the person to be authenticated, Based on a table of multiple correspondences between facial input and selection items for a person identified by the identification means, a selection means selects one of the multiple selection items as the selection item corresponding to the person identified by the identification means, based on the facial input recognized by the recognition means. It has, An information processing apparatus characterized in that, in the table, the processing performed by the selection means differs depending on whether or not there is a selection item corresponding to the face input recognized by the recognition means.

2. The information processing apparatus according to claim 1, characterized in that the face input is the state of facial organs, the state of face orientation, the state of facial expression, the state of gaze direction, the state of body parts other than the face, the state of a specific object, the state of worn items, changes in facial organs, changes in face orientation, changes in facial expression, changes in gaze direction, changes in body parts other than the face, changes in a specific object, or changes in worn items.

3. The information processing device according to claim 1, further comprising a registration means for registering a table of multiple correspondences between facial inputs and selected items for each person.

4. The information processing device according to claim 3, characterized in that the registration means controls the display of facial input options and registers a table of correspondences between facial inputs selected from the facial input options and selected items.

5. The information processing apparatus according to claim 4, characterized in that the registration means controls to display a selection of facial inputs from among a plurality of facial inputs that are easily distinguishable by a machine.

6. The information processing apparatus according to claim 1, wherein the identifying means extracts facial features based on a facial image quality score that quantifies how suitable the facial image is for facial recognition.

7. The information processing apparatus according to claim 1, characterized in that the selection means selects a default selection item if there is no selection item in the table corresponding to the face input recognized by the recognition means.

8. The information processing apparatus according to claim 1, characterized in that the selection means does not select a selection item if there is no selection item in the table corresponding to the face input recognized by the recognition means.

9. The information processing apparatus according to claim 1, characterized in that, if there is no selection item in the table corresponding to the face input recognized by the recognition means, the selection means selects a selection item in accordance with the operation of the person being authenticated.

10. The system further includes determination means for determining whether or not the person to be authenticated is a living organism from one or more facial images, including the face of the person to be authenticated. The information processing apparatus according to claim 1, characterized in that the selection means selects one of the plurality of selection items when the determination means determines that the person to be authenticated is a living organism.

11. The information processing apparatus according to claim 10, characterized in that the selection means does not select an item if the determination means determines that the person to be authenticated is not a biological entity.

12. The information processing apparatus according to claim 4, characterized in that the registration means registers facial feature quantities extracted from one or more facial images, including the face of the person to be authenticated.

13. The information processing apparatus according to claim 12, characterized in that the registration means controls to display a selection of face inputs, excluding face inputs associated with a table of other authenticated persons that have face features whose similarity to the extracted face features is above a threshold, from among a plurality of face inputs.

14. The information processing device according to claim 1, characterized in that the selection items are a payment method, whether or not administrator privileges are present, account type, service type, or the preferences of the authenticated person.

15. The information processing device according to claim 1, characterized in that the selection means selects a selection item corresponding to the face input recognized by the recognition means based on a table of multiple correspondences between face input and selection items.

16. An information processing apparatus according to any one of claims 1 to 15, The device for capturing the aforementioned facial image and An authentication system characterized by having the following features.

17. A verification step to identify a person representing the person being verified based on facial features extracted from one or more facial images, including the face of the person being verified, A recognition step of recognizing the facial input performed by the person to be authenticated from one or more facial images, including the face of the person to be authenticated, Based on a table of multiple correspondences between facial input and selection items for the person identified in the aforementioned specific step, a selection step is performed in which one of the multiple selection items is selected as the selection item corresponding to the person identified in the aforementioned specific step, based on the facial input recognized in the aforementioned recognition step. It has, A processing method for an information processing device, characterized in that the processing in the selection step differs depending on whether or not there is a selection item in the table corresponding to the face input recognized in the recognition step.

18. A program for causing a computer to function as an information processing device as described in any one of claims 1 to 15.

Images