Authentication methods, authentication systems, terminals, and servers

Abstract

To provide an authentication method for verifying the identity of a person capable of reducing the time for completing a procedure, an authentication system, a terminal, and a server.SOLUTION: The authentication method for identity verification includes the steps of: acquiring first face information from an identity verification document of a user; instructing the user to make an authentication action; determining whether the instructed authentication action is made; comparing the first face information with second face information acquired by capturing an image of the user to determine whether they match; verifying that identity verification has been completed when it is determined that the indicated authentication action has been made and determined that the first face information and the second face information match. The second face information is acquired when the authentication action is being performed, and the authentication action includes a timing when the face faces forward.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to an authentication method, an authentication system, a terminal, and a server, and particularly relates to an authentication method, an authentication system, a terminal, and a server for identity verification.

Background Art

[0002] Conventionally, an authentication system for user authentication that uses a captured image acquired by having a user capture a face image is known (Patent Document 1).

[0003] In order to prevent "spoofing" that uses the face photo of an identity verification document such as a driver's license as a captured image, the authentication system of the above-mentioned Patent Document 1 determines whether the person in the user's captured image is the same person as the person in the photo of the identity verification document, and determines whether the user's captured image is an diverted use of the image from the identity verification document.

[0004] In the authentication system of the above-mentioned Patent Document 1, one or more motions are randomly extracted, an instruction of the motion is issued to the user, and a video of the user performing the motion is captured. When the capturing is completed, based on the captured video, the authentication system determines whether the user has performed the instructed motion, and determines whether the user captured in the image before the motion instruction and after the motion instruction is the same person.

Prior Art Literature

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0006] The inventors of this application conceived of a method to verify that the facial information provided belongs to the user attempting to register, that is, to verify the user's identity, when registering facial information for facial recognition. Therefore, the inventors of this application devised a method to perform user authentication based on the authentication system described in Patent Document 1, thereby verifying that the provided facial information belongs to the user attempting to register, that is, verifying the user's identity.

[0007] However, the inventors of the present invention noticed that the authentication system described in Patent Document 1 has a problem in that it requires a time period to acquire captured images before issuing instructions for operation, which increases the time required to complete authentication.

[0008] Therefore, this disclosure primarily aims to shorten the time required for the completion of an authentication procedure, which involves authentication methods, authentication systems, terminals, and servers for verifying identity. [Means for solving the problem]

[0009] The authentication method for this disclosure is: Executed by a computer An authentication method for verifying identity, The aforementioned computer, The system obtains first facial information from the user's identification document, instructs the user to perform an authentication action, determines whether the instructed authentication action was performed, compares the first facial information with second facial information obtained by imaging the user to determine if they match, and authenticates that the user's identity has been verified when it is determined that the instructed authentication action was performed and the first and second facial information match. The authentication operation is configured such that the user takes a first authentication operation in which the user takes a first pose, which is a pose other than facing forward, with the face facing in a different direction from the user, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face faces forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0010] The authentication system disclosed herein is an authentication system for verifying identity, comprising: a first facial information acquisition unit that acquires first facial information from the user's identity verification document; an action authentication unit that instructs the user to perform an authentication action and determines whether the instructed authentication action has been performed; a facial matching unit that compares the first facial information with second facial information acquired by imaging the user and determines whether they match; and an authentication unit that authenticates that identity verification has been performed when it has been determined that the instructed authentication action has been performed and that the first facial information and the second facial information match. The aforementioned authentication operation includes a first authentication operation in which the user takes a first pose, each of which is not a frontal pose and whose face is facing in a different direction from the other, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face is facing forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0011] The terminal disclosed herein is a terminal for performing authentication for identity verification, which acquires first facial information from the user's identity verification document, instructs the user to perform an authentication action, determines whether the instructed authentication action has been performed, compares the first facial information with second facial information obtained by imaging the user and determines if they match, and authenticates that identity verification has been performed when it is determined that the instructed authentication action has been performed and the first facial information and second facial information match. The authentication operation is configured such that the user takes a first authentication operation in which the user takes a first pose, which is a pose other than facing forward, with the face facing in a different direction from the user, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face faces forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0012] The server in this disclosure is a server for performing authentication for identity verification, which obtains first facial information from the user's identity verification document, instructs the user to perform an authentication action, determines whether the instructed authentication action has been performed, compares the first facial information with second facial information obtained by imaging the user to determine if they match, and authenticates that identity verification has been performed when it is determined that the instructed authentication action has been performed and the first facial information and the second facial information match. The authentication operation is configured such that the user takes a first authentication operation in which the user takes a first pose, which is a pose other than facing forward, with the face facing in a different direction from the user, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face faces forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure. [Effects of the Invention]

[0013] According to the present disclosure, there are provided an authentication method, an authentication system, a terminal, and a server for performing personal identification, which make it possible to reduce the time required until a procedure is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] [Figure 1] Functional block diagram of an authentication system and a registration system according to an embodiment [Figure 2] Block diagram of an authentication system and a registration system [Figure 3] Example of a pose determination database [Figure 4] Example of a motion instruction database [Figure 5] Example of a user database [Figure 6] Sequence diagram showing a process flow up to the start of a service according to the first embodiment [Figure 7] Diagram showing an example of (A) a document selection screen, (B) a personal identification number input screen, and (C) a reading execution screen [Figure 8] Examples of (A) a registration completion notification screen, (B) a first pose instruction screen, and (C) a pose authentication error screen [Figure 9] Examples of (A) a second pose instruction screen, (B) a matching execution screen, and (C) an authentication completion screen [Figure 10] Examples of (A) an authentication completion screen, (B) a registration completion screen, and (C) a notification screen notifying that user registration could not be performed [Figure 11] Time chart showing the flow of a user's authentication operation and instructions and the timing at which a captured image is captured when a sequence with sequence ID 1 is selected [Figure 12] Time chart showing the flow of a user's authentication operation and instructions and the timing at which a captured image is captured when a sequence with sequence ID 3 is selected [Figure 13] Sequence diagram showing a process flow up to the start of a service according to the second embodiment [Figure 14]Sequence diagram showing the processing flow until service commencement according to the third embodiment. [Figure 15] Modified versions of authentication and registration systems [Modes for carrying out the invention]

[0015] The first invention made to solve the aforementioned problem is: Executed by a computer An authentication method for verifying identity, The aforementioned computer, The system obtains first facial information from the user's identification document, instructs the user to perform an authentication action, determines whether the instructed authentication action was performed, compares the first facial information with second facial information obtained by imaging the user to determine if they match, and authenticates that the user's identity has been verified when it is determined that the instructed authentication action was performed and the first and second facial information match. The authentication operation is configured such that the user takes a first authentication operation in which the user takes a first pose, which is a pose other than facing forward, with the face facing in a different direction from the user, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face faces forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0016] According to this, authentication can be performed, and identity verification can be carried out using first and second facial information. Furthermore, since the second facial information, which is easier to match, is acquired while the authentication process is underway, an authentication method for identity verification can be provided that shortens the time required to complete the procedure.

[0017] Furthermore, in the second invention, when it is authenticated that the person's identity has been verified, at least one of the first facial information and the second facial information is recorded as facial information for facial recognition.

[0018] According to this, at least one of the first facial information and the second facial information can be used as facial information for facial recognition.

[0021] Also, the 3 In the invention, The aforementioned first pose and the aforementioned second pose This is a pair of faces that are symmetrically oriented with respect to the front of the body. Set as pose This will be the structure.

[0022] According to this, it is possible to obtain second facial information that is easier to match during the authentication process.

[0023] The following was done to solve the aforementioned problem. 4 The invention relates to an authentication system for identity verification, comprising: a first facial information acquisition unit that acquires first facial information from the user's identity verification document; an action authentication unit that instructs the user to perform an authentication action and determines whether the instructed authentication action has been performed; a facial matching unit that compares the first facial information with second facial information acquired by imaging the user and determines whether they match; and an authentication unit that authenticates that identity verification has been performed when it has been determined that the instructed authentication action has been performed and that the first facial information and the second facial information match. The aforementioned authentication operation includes a first authentication operation in which the user takes a first pose, each of which is not a frontal pose and whose face is facing in a different direction from the other, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face is facing forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0024] According to this, an authentication system can be provided that performs authentication operations and verifies the user's identity using first and second facial information. Since the second facial information is acquired while the authentication operation is in progress, the time required to complete the procedure can be shortened.

[0025] The following was done to solve the aforementioned problem. 5 The invention relates to a terminal for performing authentication for identity verification, which acquires first facial information from the user's identity verification document, instructs the user to perform an authentication action, determines whether the instructed authentication action has been performed, compares the first facial information with second facial information obtained by imaging the user to determine if they match, and authenticates that identity verification has been performed when it is determined that the instructed authentication action has been performed and the first facial information and second facial information match. The authentication operation is configured such that the user takes a first authentication operation in which the user takes a first pose, which is a pose other than facing forward, with the face facing in a different direction from the user, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face faces forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0026] According to this, authentication is performed using the first and second facial information, and the second facial information is acquired while the authentication is being performed. Therefore, a terminal can be provided that can shorten the time required to complete the procedure.

[0027] The sixth invention made to solve the aforementioned problem is: A server for performing authentication for identity verification acquires first facial information from the user's identity verification document, instructs the user to perform an authentication action, determines whether the instructed authentication action has been performed, compares the first facial information with second facial information obtained by imaging the user to determine if they match, and authenticates that identity verification has been performed when it is determined that the instructed authentication action has been performed and the first and second facial information match. The authentication operation is configured such that the user takes a first authentication operation in which the user takes a first pose, which is a pose other than facing forward, with the face facing in a different direction from the user, and a second authentication operation in which the user takes a second pose, and the transition from the first pose to the second pose is via a pose in which the face faces forward. The aforementioned second facial information is, between the first authentication operation and the second authentication operation The timing of when the face is facing forward Obtained by This will be the structure.

[0028] According to this, authentication is performed using the first and second facial information, and the second facial information is acquired while the authentication is in progress. Therefore, a server can be provided that can shorten the time required to complete the procedure.

[0029] The embodiments of this disclosure will be described below with reference to the drawings.

[0030] The authentication method according to the present invention is an authentication method for verifying identity, and is performed in an authentication system 1 that authenticates users when providing a service, during identity verification at the time of registration.

[0031] (First Embodiment) Authentication system 1 obtains a facial image from the user, verifies the user's identity using the authentication method according to the present invention, and performs facial authentication by comparing it with registered facial information. Authentication system 1 then grants permission to the authenticated user to receive a predetermined service.

[0032] Services available to users include, for example, the rental of buildings and parts thereof (e.g., rooms for holding events or meetings), access control, use of tools and equipment, rental of offices and vehicles (rental services), reservations for train cars and seats, sales of goods and tickets, use of various facilities including transportation (e.g., stadiums), medical examinations and treatments at hospitals, login to devices such as computers, and use of stations and parcel lockers. Furthermore, the services referred to here may also include certification services that determine whether or not another party (including individuals, corporations, and organizations) is authorized to provide services.

[0033] Other services that users may be able to access could include online exam services, sharing services for offices and vehicles, and matching services for personnel, skills, and spaces.

[0034] Businesses that provide services to users, i.e., service providers, may include individuals, corporations, and organizations.

[0035] Authentication system 1 functions as a registration system 101 for registering facial information of users who wish to use the service for the first time (i.e., new users). It is desirable that authentication system 1 has a function to verify the identity of the user when registering facial information, thereby preventing others from registering under false pretenses.

[0036] Therefore, in authentication system 1, during user registration, verification is performed to confirm that the person registering is indeed the person in question (i.e., identity verification). Furthermore, authentication system 1 also verifies that the user is not already registered, i.e., that duplicate registration has not occurred.

[0037] Identity verification is based on comparing the facial photograph on an identification document with a facial image of the person registering, and confirming that the person registering is not using another person's image. Authentication system 1 determines that both are the same person and confirms that the person registering is not using another person's image, at which point it performs authentication, deeming identity verification complete. In other words, authentication system 1 also functions as an authentication system that performs authentication for identity verification.

[0038] Identity verification documents may include driver's licenses, My Number cards, passports, residence cards, student IDs, and other forms of identification. It is desirable that these identity verification documents contain an integrated circuit chip (IC chip) on which personal information is stored. The information stored on the IC chip can be read via short-range wireless communication. For example, personal information stored on the IC chip of a driver's license may include name, photograph, date of birth, address, date of issuance, expiration date, and license number. This personal information may be provided to service providers as needed.

[0039] To verify that images are not being used without permission from others, a pose detection system is used in which registered users are instructed to assume a specific pose (posture), and their correctness is checked.

[0040] The following describes the details of the configuration of authentication system 1.

[0041] As shown in Figure 1, the authentication system 1 comprises a user terminal 2 carried by the user and an authentication server 3 operated and managed by an authentication service provider.

[0042] The user terminal 2 and the authentication server 3 are connected to each other via a communication network 5, enabling them to communicate with one another. The communication network 5 may be the so-called internet, or it may be a closed network. The user terminal 2 and the authentication server 3 may be connected to the communication network 5 by wire, or they may be connected to the communication network 5 wirelessly.

[0043] The authentication server 3 and user terminal 2 are connected to the provider server 7, which is managed by the service provider. The provider server 7 performs various processes to provide the various services offered by the service provider. The processes provided by the provider server 7 may include, for example, processes required for various online services such as online shopping, online reservations, and online examinations. The authentication system 1 performs facial recognition before the service is provided by the provider server 7, and the provider server 7 provides the service to the authenticated user.

[0044] User terminal 2 can consist of an information device with communication capabilities, such as a smartphone, tablet, or PC. The user uses the user terminal 2 they carry with them to communicate with the authentication server 3 and perform various processes for facial recognition and user registration.

[0045] In addition, the authentication system 1 may include an authentication device 2A as a user terminal 2. The authentication device 2A may be installed, for example, in various stores owned by the service provider and perform various processes such as unlocking doors and gates, facial recognition for electronic payments, and user registration.

[0046] As shown in Figure 2, the user terminal 2 includes a processor 11 (CPU, MPU, etc.), memory 12 such as volatile memory (RAM) or non-volatile memory (ROM), a touch panel display 13, a camera 14, a reader 15, a communication interface 16, and storage 17. In addition, the user terminal 2 may also include a speaker 18.

[0047] The reader 15 reads the data stored in the IC chip embedded in the identity verification document using near-field communication (NFC). The reader 15 may be a well-known NFC reader (or reader / writer).

[0048] As shown in Figure 1, the user terminal 2 includes, as a functional unit, a communication unit 21, a storage unit 22, a reading unit 23, an imaging unit 24, a notification unit 25, an operation authentication unit 26, and a face image acquisition unit 27.

[0049] The communication unit 21 includes a communication interface 16. The communication unit 21 communicates wirelessly or via wired connection with other devices (such as the authentication server 3) via the communication network 5 in accordance with a known communication protocol.

[0050] The storage unit 22 includes the memory 12 and storage 17 of the user terminal 2. The storage unit 22 stores information necessary for various processes performed on the user terminal 2.

[0051] The reading unit 23 includes at least one of the reading device 15 and the camera 14. The reading unit 23 reads the facial photograph (first facial information) from the identity verification document and transmits it to the authentication server 3. In this embodiment, the reading unit 23 reads data corresponding to the facial photograph from the IC chip mounted on the identity verification document via the reading device 15 and transmits it to the authentication server 3. Alternatively, the reading unit 23 may display a notification on the touch panel display 13 instructing the user to take an image of the identity verification document, read the facial photograph from the identity verification document (e.g., a driver's license), and transmit the facial photograph and personal information to the authentication server 3. In this embodiment, the reading unit 23 obtains the facial photograph and name from the IC chip of the identity verification document and transmits them to the authentication server 3.

[0052] The imaging unit 24 includes one or more cameras 14. The cameras 14 are located near the touch panel display 13 and are positioned to capture the face of the user operating the user terminal 2. The imaging unit 24 acquires still images and videos by capturing images of the user operating the user terminal 2.

[0053] The notification unit 25 includes at least one of the touch panel display 13 and the speaker 18, and provides various notifications to the user through the display on the touch panel and the sound emitted from the speaker 18.

[0054] The motion authentication unit 26 performs motion authentication by instructing the registered person to assume a predetermined pose (posture) and determining whether the instructed pose (posture) is correctly assumed. The motion authentication unit 26 is configured by the processor 11 of the user terminal 2 performing processing according to a predetermined program.

[0055] To perform this processing, the operation authentication unit 26 includes a pause instruction unit 31, a pause verification unit 32, and an operation verification unit 33.

[0056] The pause instruction unit 31 may be configured by the processor 11 of the user terminal 2 executing a predetermined program. The pause instruction unit 31 obtains the content to be notified from the pause judgment database 22A (pause judgment DB) stored in the storage unit 22 and issues a pause instruction to the user by sending a notification from the notification unit 25.

[0057] The pose determination database 22A records an identification number (hereinafter referred to as pose ID) corresponding to a pose (posture) and the corresponding instruction text. When the pose instruction unit 31 receives the pose ID to be instructed from the motion verification unit 33, it refers to the pose determination database 22A to obtain the instruction text corresponding to the pose ID and notifies the notification unit 25 of the obtained instruction text.

[0058] The pose matching unit 32 may be configured by the processor 11 of the user terminal 2 executing a predetermined program. After a pose instruction is given by the pose instruction unit 31, the pose matching unit 32 acquires the captured image from the imaging unit 24 in real time, extracts the user's face orientation, hand position, etc., using known methods, and determines whether the user has assumed a pose (posture) that matches the pose instruction.

[0059] The pose matching unit 32 can determine the orientation of the face (specifically, how many degrees it is facing left or right, how many degrees it is facing up or down, etc.) using, for example, a learning model constructed by machine learning. In addition, the pose matching unit 32 can extract the positions of the right hand, left hand, left shoulder, and right shoulder using, for example, a learning model constructed by machine learning, and determine whether the right hand is positioned above the right shoulder, or whether the left hand is positioned above the left shoulder, etc.

[0060] In this embodiment, the pause determination database 22A records conditions for determining whether a pause instruction has been met (hereinafter referred to as pause determination conditions) in association with a pause ID. When the pause matching unit 32 obtains the pause ID to be determined from the operation matching unit 33, it refers to the pause determination database 22A to obtain the pause determination conditions corresponding to the pause ID and determines whether the obtained pause determination is met. When it determines that the pause determination conditions have been met, the pause matching unit 32 notifies the operation matching unit 33 that the pause determination conditions have been met.

[0061] Figure 3 shows an example of a pose determination database 22A in which pose IDs and the content of the corresponding pose instructions are recorded. The pose determination database 22A records the pose ID, information related to the corresponding pose, text indicating the corresponding pose instruction, and conditions for determining that the pose (posture) corresponding to the pose instruction has been taken, in association with each other.

[0062] The motion verification unit 33 may be configured by the processor 11 of the user terminal 2 executing processing according to a predetermined program. When the motion verification unit 33 receives an instruction to start motion authentication from the authentication server 3, it retrieves a sequence of data from the motion instruction database 22B held in the storage unit 22. The sequence of data includes information relating to one or more pose instructions (posture instructions) and their order.

[0063] Figure 4 shows the operation instruction database 22B according to this embodiment. The operation instruction database 22B records multiple sequence data, each including a sequence identification number (sequence ID), a first pose identification number (first pose ID), and a second pose identification number (second pose ID). However, the sequence data recorded in the operation instruction database 22B always includes the first pose ID. In other words, the operation instruction database 22B may also contain sequence data that does not include the second pose ID (for example, the second pose ID is Null).

[0064] Upon receiving an instruction to start operation authentication, the operation verification unit 33 randomly selects a sequence from the operation instruction database 22B. Subsequently, the operation verification unit 33 retrieves the sequence data of the selected sequence from the operation instruction database 22B.

[0065] When a second pose ID is recorded in the sequence data, the motion matching unit 33 first issues a pose instruction to the pose instruction unit 31 corresponding to the first pose ID. When the pose matching unit 32 determines that the pose judgment conditions corresponding to the first pose ID are met, the motion matching unit 33 issues a pose instruction corresponding to the second pose ID. When the pose matching unit 32 determines that the pose judgment conditions corresponding to the first pose ID are met, it determines that a series of actions corresponding to the selected sequence have been performed (i.e., motion authentication has been performed) and notifies the authentication server 3 of the determination result. Furthermore, when motion authentication is completed, the pose matching unit 32 notifies the face image acquisition unit 27 that motion authentication has been completed.

[0066] If the second pose ID is not recorded in the sequence data, the motion matching unit 33 first issues a pose instruction corresponding to the first pose ID to the pose instruction unit 31. When the pose matching unit 32 determines that the pose judgment conditions corresponding to the first pose ID have been met, it determines that a series of actions corresponding to the selected sequence have been performed (i.e., motion authentication has been performed), and notifies the face image acquisition unit 27 and the authentication server 3 of the determination result.

[0067] When both the first pose ID and the second pose ID are recorded, the pose corresponding to the first pose ID (hereinafter referred to as the first pose) and the pose corresponding to the second pose ID (the second pose) are set to have different facial orientations.

[0068] As shown in Figures 3 and 4, in the sequence with sequence ID 1, the first pose is a pose (posture) with the face turned to the left, and in motion authentication, the pose instruction to turn to the left is given first. The second pose is a pose (posture) with the face turned to the right, and in the sequence with sequence ID 1, after the pose with the face turned to the left is confirmed, the instruction to turn to the right is given.

[0069] In the sequence with sequence ID 2, the first pose is with the face turned to the right, and the second pose is with the face turned to the left. In the sequence with sequence ID 3, the first pose is with the face turned downwards, and the second pose is with the face turned upwards. In the sequence with sequence ID 4, the first pose is with the face turned upwards, and the second pose is with the face turned downwards.

[0070] In this way, it is preferable that the first pose (first posture) and the second pose (second posture) are set so that when the user moves from the first pose to the second pose, the user's face is facing the imaging unit 24 (camera 14). It is preferable that the first pose (first posture) and the second pose (second posture) are set so that the orientation of the face is symmetrical with respect to the front of the body.

[0071] When the second pose ID is not recorded, the pose (posture) corresponding to the first pose ID is set so that the user faces forward to the touch panel of the user terminal 2 (i.e., the user faces forward to the imaging unit 24).

[0072] Specifically, as shown in Figures 3 and 4, in sequence ID 5, the first pose is set to a pose (posture) with the right hand raised, and in sequence ID 6, the first pose is set to a pose (posture) with the left hand raised.

[0073] The face image acquisition unit 27 may be configured by the processor 11 of the user terminal 2 executing a predetermined program and performing processing. The face image acquisition unit 27 acquires the user's face image from the images acquired by the imaging unit 24 when the user is facing the imaging unit 24 (camera 14) from the images acquired by the imaging unit 24 during the period from when the operation authentication starts until when the operation authentication is completed, that is, while the user is performing the authentication operation.

[0074] In this embodiment, the face image acquisition unit 27 receives a signal from the authentication server 3 instructing it to start motion authentication, gives a pose instruction for the first pose (i.e., starts motion authentication), and then, at predetermined intervals (or in real time), acquires captured images from the shooting unit and determines the orientation of the user's face until it is notified by the motion matching unit 33 (motion authentication unit 26) that motion authentication is complete. The face image acquisition unit 27 may use a learning model constructed by machine learning to determine the orientation of the user's face. When the face image acquisition unit 27 determines that the user's face is facing forward, it may save the captured image acquired by the shooting unit as a face image in the storage unit 22. The face image acquisition unit 27 may determine that the user's face is facing forward when the orientation of the user's face is within 5 degrees to the left or right of the front of the body.

[0075] Between the start of the operation authentication and the acquisition of the operation authentication result, if the face image acquisition unit 27 is able to acquire an image (second face information) of the user's face when it is facing forward, it transmits the acquired image to the authentication server 3.

[0076] If, between the start of motion authentication and the acquisition of the motion authentication result, the face image acquisition unit 27 is unable to acquire an image of the user's face facing forward from the imaging unit 24, the face image acquisition unit 27 instructs the motion authentication unit 26 to repeat the motion authentication by having the motion authentication unit 26 issue a pose instruction corresponding to the first pose again.

[0077] As shown in Figure 2, the authentication server 3 includes a processor 41 (CPU, MPU, etc.), memory 42 such as volatile memory (RAM) or non-volatile memory (ROM), storage 43, and a communication interface 44. In addition, the authentication server 3 may also include a display (not shown) and input devices such as a mouse or keyboard (not shown).

[0078] As shown in Figure 1, the authentication server 3 includes, as functional units, a communication unit 51, a storage unit 52, a face feature acquisition unit 53, a face matching unit 54, a face information registration unit 55, and a service start processing unit 56.

[0079] The communication unit 51 includes a communication interface 44. The communication unit 51 communicates wirelessly or via wired connection with other devices (such as user terminals 2) via the communication network 5 in accordance with known communication protocols.

[0080] The storage unit 52 includes memory 42 and storage 43. The storage unit 52 stores information necessary for various processes performed by the authentication server 3.

[0081] The memory unit 52 holds the user database 52A.

[0082] Figure 5 shows an example of the user database 52A. The user database 52A stores the identification number (user ID) of a registered user, the corresponding user's name, and the corresponding facial information, all associated with each other.

[0083] The facial information may include information relating to at least one of facial features and a facial image. In this embodiment, the facial information includes an identification number indicating the facial features (facial feature ID) and an identification number indicating the facial image (facial image ID).

[0084] In addition, the user database 52A stores the type of image corresponding to the facial image ID (either an identification document or a photograph) and the date and time the image was registered in the user database 52A, associated with the user ID. In the example shown in Figure 5, the type of image recorded is that of an identification document such as a driver's license or passport.

[0085] The facial feature acquisition unit 53 acquires the facial image from the identity verification document and the captured image acquired by the imaging unit 24 from the user terminal 2, and then acquires facial features from each image and outputs them to the facial matching unit 54. Facial features are data used to determine whether the people in the images are the same person, and may be quantities acquired based on feature points such as eyes, nose, mouth, and facial contours in the image.

[0086] In addition, if the facial feature acquisition unit 53 acquires only the facial image from the identity verification document from the user terminal 2, it acquires facial features from each image and outputs them to the facial matching unit 54.

[0087] Similarly, if the face feature acquisition unit 53 acquires only face images acquired by the authentication device 2A or captured images acquired by the user terminal 2, it acquires face features from each image and outputs them to the face matching unit 54.

[0088] The face matching unit 54 obtains the facial features of the face image on the identity verification document and the facial features of the captured image from the facial feature acquisition unit 53, and then determines whether the person in the face image and the captured image is the same person based on the two facial features. At this time, the face matching unit 54 may compare the two facial features and calculate a matching score. The matching score is a numerical value that indicates the degree of matching between the two facial features, and a higher number indicates a better match. The face matching unit 54 may determine that the person in the face image on the identity verification document and the captured image are the same person if the matching score is above a predetermined threshold. The face matching unit 54 outputs the determination result of whether the person in the face image on the identity verification document and the person in the captured image are the same person, along with their respective facial features, to the face information registration unit 55.

[0089] When the face matching unit 54 obtains only the face features corresponding to the face image on the identity verification document from the face feature acquisition unit 53, it refers to the user database 52A stored in the storage unit 52 and extracts the user ID of the user corresponding to the acquired face features. If the face matching unit 54 cannot obtain the corresponding user ID, it notifies the user terminal 2 that it will start the operation authentication. When it obtains the corresponding user ID, the face matching unit 54 determines that user registration has already been completed, notifies the user terminal 2 accordingly, and finishes the user registration process.

[0090] If the face matching unit 54 obtains only face features corresponding to the face image obtained by the authentication device 2A or the image captured by the user terminal 2 from the face feature acquisition unit 53, it refers to the user database 52A stored in the storage unit 52 and extracts the user ID of the user whose face features match the acquired face features (i.e., whose matching score is above a predetermined threshold). If the face matching unit 54 is able to extract the user ID of the corresponding user, it transmits information such as the user's name to the authentication device 2A or the user terminal 2. If it is unable to extract the user ID, it transmits a notification to the authentication device 2A or the user terminal 2 indicating that matching was not possible.

[0091] The facial information registration unit 55 performs authentication based on the determination result of the motion matching unit 33 (motion authentication unit 26) as to whether or not motion authentication has been performed, and the determination result of the facial matching unit 54 as to whether or not the facial photograph and the captured image match (i.e., they are photographs of the same person). When it is determined that motion authentication has been performed and that the facial photograph on the identity verification document matches the captured image, that is, when identity verification has been performed, the facial information registration unit 55 functions as an authentication unit that performs authentication for identity verification. Once authentication for identity verification is complete, the facial information registration unit 55 adds the data to the user database 52A held in the storage unit 52 and performs user registration.

[0092] Specifically, the facial information registration unit 55 acquires a new user ID not recorded in the user database 52A when the action matching unit 33 determines that a series of actions have been performed (a series of poses have been taken) and the facial matching unit 54 determines that it is the same person. Subsequently, the facial information registration unit 55 stores the facial photograph or captured image of the identity verification document in the storage unit 52 as the facial image corresponding to the user ID, and also stores the facial feature quantities obtained from the facial photograph or captured image in the storage unit 52. At this time, the facial information registration unit 55 associates the identification information (file name, etc.) of the facial image stored in the storage unit 52, the identification information (file name, etc.) of the facial feature quantities obtained from the facial photograph or captured image, and personal information such as the name with the newly acquired user ID, and records it in the user database 52A, adding the data.

[0093] When adding data to the user database 52A, the facial information registration unit 55 should record the type of image to be registered in the user database 52A (e.g., captured image, type of identity verification document) and the date and time of registration, along with the identification information of the facial image and facial features, and personal information such as the name, in association with the newly acquired user ID.

[0094] When recording to the user database 52A is complete and data addition is complete (i.e., user registration is complete), the facial information registration unit 55 instructs the service start processing unit 56 to start the processing necessary for providing the service provider's services.

[0095] However, if the facial information registration unit 55 receives input indicating a desire to receive the service provider's services without registering the information in the user terminal 2, it may instruct the service start processing unit 56 to start the processing necessary for providing the service provider's services without registering the data in the user database 52A.

[0096] The service start processing unit 56 performs the necessary processing (hereinafter referred to as pre-service start processing) for registered users before the service provider begins providing its services.

[0097] When the authentication server 3 has only acquired the facial photograph from the identity verification document (i.e., when the acquisition of the facial image by the imaging unit 24 has not yet been completed), the service start processing unit 56 may, during the pre-start processing, notify the user terminal 2 again to perform the necessary processing for facial authentication and motion authentication. Once the pre-start processing is complete, the service start processing unit 56 notifies the server providing the service provided by the service provider of the completion of the pre-start processing, along with the user ID of the user corresponding to the acquired facial photograph. If the service start processing unit 56 performs the pre-start processing without registering data in the user database 52A, the service provider may provide personal information obtained from the identity verification document instead of the user ID. This allows the user to use the services provided by the service provider after the authentication of identity verification is complete.

[0098] Next, regarding the operation of authentication system 1, we will first explain the registration process when a user uses a service provider's service for the first time.

[0099] Authentication System 1 executes a predetermined program (authentication program) to perform authentication for identity verification during registration. Figure 6 shows a time chart illustrating the process from authentication to completion of user registration.

[0100] When the registration process begins, the processor 11 of the user terminal 2 starts the process of reading the identity verification documents (S101).

[0101] When the processor 11 of the user terminal 2 starts the identity verification document reading process, it displays a document selection screen 61 for identity verification documents on the touch panel display 13. Figure 7(A) shows an example of the document selection screen 61. The document selection screen 61 is provided with one or more buttons 61A for selecting identity verification documents. In this embodiment, the document selection screen 61 is provided with two buttons 61A: one for selecting a driver's license as the identity verification document, and another for selecting a My Number Card.

[0102] When a user touches any of the buttons 61A, the processor 11 of the user terminal 2 displays a PIN entry screen 62, including a PIN entry field 62A, on the touch panel display 13. Figure 7(B) shows the PIN entry screen 62 when a driver's license is selected as the user's identification document.

[0103] When a PIN is entered, the processor 11 of the user terminal 2 displays a reading execution screen 63 on the touch panel display 13. Figure 7(C) shows an example of the reading execution screen 63. In the example shown in Figure 7(C), the touch panel display 13 displays a message 63A instructing the user to hold their identification document (driver's license) over the reading unit 23. When the reading unit 23 (reading device 15, first facial information acquisition unit) acquires a facial photograph and personal information such as the name from the IC chip of the identification document, the processor 11 of the user terminal 2 transmits data indicating the read facial photograph, personal information, and type of identification document to the authentication server 3 (S102).

[0104] However, the document selection screen 61 may be configured to allow the user to select a document that does not have an IC chip (for example, a student ID). In this case, the processor 11 of the user terminal 2 may issue a notification instructing the imaging unit 24 (camera 14) to take a picture of the user's face instead of the face image on the identity verification document. The processor 11 of the user terminal 2 may then send the face image acquired by the imaging unit 24 to the authentication server 3 instead of the face photograph on the identity verification document. Alternatively, before sending the acquired face image to the authentication server 3 in place of the face photograph on the identity verification document, the processor 11 of the user terminal 2 may accept input of personal information such as the user's name on the touch panel display 13 and send the acquired face image and personal information to the authentication server 3.

[0105] When the processor 41 of the authentication server 3 obtains data from the user terminal 2 showing the facial photograph and personal information of the identity verification document, it determines whether the person pictured in the facial photograph of the identity verification document is already registered and performs a double registration check (S104).

[0106] Specifically, the processor 41 of the authentication server 3 searches the user database 52A using facial features obtained from the facial photograph of the identity verification document, and determines whether or not facial features matching those obtained from the facial photograph are recorded.

[0107] When the user database 52A contains facial features that match those obtained from a facial photograph, the processor 41 of the authentication server 3 determines that the person in the facial photograph on the identity verification document is already registered (resulting in duplicate registration), and displays a registration notification screen 65 on the terminal to indicate that the person is already registered. This prevents duplicate registration of the facial image and personal information of a user who is already registered.

[0108] Figure 8(A) shows an example of the registered notification screen 65. In this embodiment, the registered notification screen 65 displays a pop-up 65A containing a message indicating that the user is registered and information about the user who was read (in this embodiment, the user's name and a photograph).

[0109] When the facial feature data corresponding to the facial photograph is not registered in the user database 52A (i.e., when it does not constitute duplicate registration), the processor 41 of the authentication server 3 and the processor 11 of the user terminal 2 send an instruction to start operation authentication (S104). As a result, the processor 11 (operation authentication unit 26) of the user terminal 2 starts the operation authentication process.

[0110] In the operation authentication process, first, the processor 11 of the user terminal 2 selects a sequence for operation authentication from the operation instruction database 22B (S105). The sequence selection should preferably be done randomly.

[0111] Next, the processor 11 of the user terminal 2 refers to the operation instruction database 22B and extracts the first pause ID and the second pause ID associated with the sequence ID determined by the authentication server 3 (S106).

[0112] The following describes an example where the first and second poses are set (the second pose field in the sequence ID is not null).

[0113] Next, the processor 11 of the user terminal 2 refers to the pose determination database 22A based on the extracted first pose ID and extracts text indicating a pose instruction corresponding to the first pose ID. Subsequently, the processor 11 of the user terminal 2 gives the user a pose instruction corresponding to the first pose by displaying the extracted text on the touch panel display 13, generating sound from the speaker 18 corresponding to the extracted text, etc. (S107).

[0114] Figure 8(B) shows an example of the first pose instruction screen 66 displayed on the touch panel display 13 of the user terminal 2 when a sequence with sequence ID 1 is selected and the first pose is a pose facing left. The first pose instruction screen 66 shown in the example preferably displays a real-time captured image 66A acquired by the imaging device (camera 14) along with text 66B instructing the user to take a second pose (i.e., a pose (posture) facing right).

[0115] Next, once the pose instruction corresponding to the first pose is completed, the processor 11 of the user terminal 2 refers to the pose determination database 22A based on the extracted first pose ID and obtains the pose determination conditions corresponding to the first pose. Subsequently, the processor 11 of the user terminal 2 determines, based on the image acquired by the imaging device (camera 14), whether or not the user has taken the first pose (i.e., whether or not the pose determination conditions corresponding to the first pose have been met) over a predetermined time (for example, 5 to 10 seconds) (S108).

[0116] Figure 8(C) shows the pose authentication error screen 67 that is displayed on the touch panel display 13 of the user terminal 2 when the user does not take the first pose (No in S108). As shown in Figure 8(C), the pose authentication error screen 67 should display text 67A instructing the user to take the first pose (a pose (posture) facing left).

[0117] In the example shown in Figure 8(C), the resume button 67B is displayed on the pause authentication error screen 67. When the user presses the resume button 67B, the processor 11 of the user terminal 2 again issues a pause instruction corresponding to the first pause via the touch panel display 13 or by sound emitted from the speaker 18 (S107).

[0118] When the processor 11 of the user terminal 2 determines that the user has taken a first pose, that is, that the pose determination conditions corresponding to the first pose have been met (cleared) (Yes in S108), it refers to the pose determination database 22A based on the extracted second pose ID and extracts text indicating a pose instruction corresponding to the second pose ID. Subsequently, the processor 11 of the user terminal 2 gives the user a pose instruction corresponding to the second pose by displaying the extracted text on the touch panel display 13, generating sound from the speaker 18 corresponding to the extracted text, etc. (S109).

[0119] Figure 9(A) shows an example of a second pose instruction screen 71 that is displayed on the touch panel display 13 of the user terminal 2 after a sequence with sequence ID 1 is selected and the user has taken a first pose. The second pose instruction screen 71 shown in the example preferably displays a real-time captured image 71A acquired by the imaging device (camera 14) along with text 71B instructing the user to take a second pose (i.e., a pose facing to the right).

[0120] Next, when the pose instruction corresponding to the second pose is completed, the processor 11 of the user terminal 2 refers to the pose determination database 22A based on the extracted second pose ID and obtains the pose determination conditions corresponding to the second pose. Subsequently, the processor 11 of the user terminal 2 determines, based on the image acquired by the imaging device (camera 14), whether or not the user has taken the first pose (i.e., whether or not the pose determination conditions corresponding to the first pose have been met) over a predetermined time (for example, 5 to 10 seconds) (S110).

[0121] If the processor 11 of the user terminal 2 determines that the user did not take a second pause (No in S110), the processor 11 of the user terminal 2 again issues a pause instruction corresponding to the first pause, either through a display on the touch panel display 13 or by sound emitted from the speaker 18. At this time, the screen displayed on the user terminal 2 may be the same as the pause authentication error screen 67 shown in Figure 8(C).

[0122] After the processor 11 of the user terminal 2 issues a pose instruction corresponding to the first pose (i.e., after starting the operation authentication), it attempts to acquire an image using the camera 14 (imaging device) at the moment the user is facing forward, until the result of determining whether or not the user has taken a second pose is obtained (until the result of operation authentication is obtained) (S111).

[0123] In this case, the processor 11 of the user terminal 2 may acquire video using the camera 14 between the start and completion of operation authentication, extract the timing when the user is facing forward from the video, and acquire an image of the user facing forward. Alternatively, the processor 11 of the user terminal 2 may acquire the user's posture by analyzing the real-time images captured by the camera 14 between the start and completion of operation authentication, and when it is possible to acquire the timing when the user is facing forward, it may take a picture with the camera 14 and acquire an image. The processor 11 of the user terminal 2 may also determine that the user is facing forward when the orientation of the user's face is within 5 degrees to the left or right of the shooting direction (optical axis) of the camera 14.

[0124] When the processor 11 of the user terminal 2 determines that the user has taken a second pose (Yes in S110), it instructs the user to take a first pose, and then determines whether the camera 14 has captured an image of the user facing forward (i.e., an image usable for face matching) between the time the determination result of whether the user has taken a second pose is obtained (S112). At this time, the processor 11 of the user terminal 2 may also determine whether the captured image is suitable for face matching (for example, if it is out of focus).

[0125] When the processor 11 of the user terminal 2 determines that it has obtained an image of the user facing forward (i.e., it has obtained an image suitable for face matching) (Yes in S112), it sends the image obtained by the camera 14 and a notification that the operation has been authenticated to the authentication server 3 (S113).

[0126] After the processor 11 of the user terminal 2 instructs the user to take a first pose, if it is unable to obtain an image of the user facing forward (No in S112) between the time the determination result of whether or not the user has taken a second pose is obtained, it will restart the operation authentication process from the pose instruction for the first pose (S107).

[0127] At this time, the processor 11 of the user terminal 2 also determines whether the image taken when the user is facing forward is suitable for face recognition (for example, if it is out of focus), and if it determines that it is not suitable, it should restart the motion authentication process from the pose instruction of the first pose.

[0128] When the processor 41 (face matching unit 54) of the authentication server 3 receives the captured image acquired by the imaging device from the user terminal 2 and a notification that the operation has been authenticated, it compares the captured image with the face photograph on the identity verification document (face matching) (S114). Specifically, the processor 41 of the authentication server 3 extracts face features from both the captured image and the face photograph on the identity verification document. Then, the authentication server 3 determines whether the person in the captured image and the person in the face photograph are the same person by comparing the two acquired face features. At this time, the processor 41 of the authentication server 3 may obtain a matching score for the two face features, and if the matching score is above a predetermined threshold, it may determine that the person in the captured image and the person in the face photograph on the identity verification document are the same person.

[0129] Figure 9(B) shows an example of a verification execution screen 72 that is displayed on the touch panel display 13 after the user terminal 2 has sent the captured image and a notification that the operation has been authenticated to the authentication server 3, until the face verification is completed. As shown in Figure 9(B), the verification execution screen 72 may also show the captured image and a loading animation 72A superimposed on each other.

[0130] The processor 41 of the authentication server 3 (face information registration unit 55, authentication unit) determines that authentication for identity verification has been performed when it has performed motion authentication (pose confirmation) (S110) and has determined that the person in the captured image and the person in the face photograph are the same person. Subsequently, the processor 41 of the authentication server 3 registers the user by adding the data to the user database 52A held in the storage unit 52 (S114).

[0131] Specifically, the processor 41 of the authentication server 3 acquires a new user ID and adds either the facial features obtained from the captured image or the facial features obtained from the facial photograph on the identification document, along with the name read from the identification document, to the user ID in the user database 52A. At the same time, the processor 41 of the authentication server 3 also adds either the captured image or the facial photograph on the identification document, along with the registration date and time, and the type of image to be added, to the user database 52A in the newly acquired user ID in the user database 52A.

[0132] In another embodiment, the processor 41 of the authentication server 3 sends a notification to the user terminal 2 indicating that authentication has been successful. Upon receiving the notification, the user terminal 2 displays an authentication completion screen 73 on the touch panel display 13. The authentication completion screen 73 displays text 73A indicating that authentication has been successful and a registration button 73B. When input is detected on the registration button 73B, the user terminal 2 sends a notification indicating that registration has been permitted. Upon receiving the notification that registration has been permitted, the authentication server 3 registers the user by associating either the captured image or the facial photograph from the identification document, along with the registration date and time, with the newly acquired user ID and adding them to the user database 52A.

[0133] Furthermore, the authentication completion screen 73 displayed on the user terminal 2 may also display a "Start using the service without registering" button along with (or instead of) the registration button 73B. When the user touches the "Start using the service" button, the service start processing unit 56 is instructed to start the processing necessary to provide the service provider's service, without registering data in the user database 52A. As a result, the user can receive the service without registering as a user.

[0134] Figure 9(C) shows an example of the authentication completion screen 73. The authentication completion screen 73 shown in Figure 9(C) displays text 73A indicating that authentication was successful, a registration button 73B, a facial image 73C read from the identity verification document, and an image 73D acquired by the camera 14 (imaging device).

[0135] Furthermore, in another embodiment, the processor 11 of the user terminal 2 displays a screen on the touch panel for selecting an image to be registered (or recorded) in the user database 52A, and accepts the user's selection of an image to be registered.

[0136] Figure 10(A) shows an example of an authentication completion screen 75 when accepting user selection of an image to register. The authentication completion screen 75 displays text 75A indicating that authentication was successful, a registration button 75B, a facial photograph 75C from the identification document, and an image 75D acquired by the camera 14 (imaging device), similar to the example shown in Figure 9(C). In the example in Figure 10(A), when the user touches either the facial photograph 75C or the image 75D, the thickness of the border (circle) indicating the image of the touched image increases, indicating that it has been selected. In the example of the authentication completion screen 75 shown in Figure 10(A), the facial photograph 75C from the identification document is selected. Subsequently, when the user touches the registration button 75B, information indicating the selected item is sent to the user terminal 2, and the processor 41 of the authentication server 3 adds either the selected image or the facial photograph from the identification document to the user database 52A, thereby registering the user.

[0137] In addition, the processor 41 of the authentication server 3 may compare the captured image and the facial photograph on the identification document based on various factors such as image quality, facial expression, background, and reflections of objects other than people, and store the one suitable for matching in the user database 52A. However, the storage location of these images is not limited to the authentication server 3; these images may also be stored on the user terminal 2, the provider server 7, etc.

[0138] Once the authentication server 3's processor 41 has finished adding the user to the user database 52A and user registration is complete, it sends a notification to the user terminal 2 indicating that registration is complete (S115). As a result, the user terminal 2's touch panel display 13 displays a registration completion screen 76 indicating that user registration is complete (S116).

[0139] Figure 10(B) shows an example of the registration completion screen 76. The registration completion screen 76 displays the facial photograph 76A from the identity verification document and the captured image 76B acquired by the camera 14 (imaging device), and may also display personal information read from the identity verification document. In the example of the registration completion screen 76 shown in Figure 10(B), text 76C and image 76D are displayed, showing information (name, registration date, registered image, and image type) recorded in the user database 52A, which includes some of the read personal information.

[0140] If the authentication server 3's processor 41 cannot determine that the person in the captured image and the person in the facial photograph are the same person, it notifies the user terminal 2 that identity verification could not be performed. As a result, a notification screen 77 informing the user that user registration could not be completed is displayed on the user terminal 2's touch panel display 13.

[0141] Figure 10(C) shows an example of a notification screen 77 that notifies the user that registration could not be completed. The notification screen 77 shown in Figure 10(C) displays a facial photograph 77A obtained from the identity verification document, an image captured 77B, and a notification 77C indicating that facial matching failed. In addition, the notification screen 77 shown in Figure 10(C) displays a back and resume button 77D to restart the registration process, and an exit button 77E to end the registration process.

[0142] Thus, in the authentication system 1, a system for authenticating identity is configured by a reading unit 23 that acquires a facial photograph (first facial information) from the user's identification document, an action authentication unit 26 that performs action authentication, a face matching unit 54 that compares the facial photograph with images captured during the authentication operation, and a face information registration unit 55 that acts as an authentication unit for authenticating identity. Furthermore, after authenticating identity, the authentication system 1 executes a program to implement a registration method for registering facial information, registers facial information for facial authentication, and also functions as a registration system 101 for user registration.

[0143] Once the registration process is complete, the processor 41 of the authentication server 3 executes pre-service startup processing as necessary (S117). Pre-service startup processing is the process required to provide services from the provider server 7 to registered users after the registration of facial information, and is determined appropriately according to the content of the provider's services.

[0144] For example, during pre-service processing, the authentication server 3 may transmit registered personal information (such as name) and information related to facial images to the provider server 7. Alternatively, during pre-service processing, the authentication server 3 may display a notification to the user terminal 2 prompting the user to take a facial image, and then perform facial authentication by comparing the acquired image with facial features stored in the user database 52A.

[0145] Once the pre-service processing is complete, the authentication server 3 sends a notification to the provider server 7 that the pre-service processing is complete (S118). Upon receiving the notification that the pre-service processing is complete, the provider server 7 provides the registered user with a service (e.g., online shopping, car rental service reservation, etc.) (S119).

[0146] Authentication system 1 also performs facial recognition processing (hereinafter referred to as "facial recognition processing") for registered users to receive services from service providers.

[0147] The facial recognition process is performed through the cooperation of the storage unit 52 and face matching unit 54 of the authentication server 3 and the imaging unit 24 of the user terminal 2. In other words, the storage unit 52, face matching unit 54, and imaging unit 24 constitute the authentication system 1 for facial recognition of the user.

[0148] In this embodiment, the authentication system 1 performs facial recognition based on facial features. Therefore, the facial recognition process is performed by the cooperation of the storage unit 52, the face matching unit 54, the imaging unit 24, and the facial feature acquisition unit 53. However, facial recognition may be performed based on any known method. Therefore, the facial feature acquisition unit 53 is not essential for the authentication system 1, and in cases where facial recognition is not based on facial features, the authentication system 1 can be composed of the storage unit 52, the face matching unit 54, and the imaging unit 24.

[0149] During facial recognition, a notification prompting the user to capture an image of their face is displayed on the touch panel display 13 of the user terminal 2. When the user captures an image of their face using the imaging unit 24 and performs a predetermined operation input on the touch panel display 13, the image acquired by the imaging unit 24 is transmitted to the authentication server 3.

[0150] When the captured image is transmitted to the authentication server 3, the face matching unit 54 compares the captured image with the face information stored in the user database 52A to determine whether or not there is a registered user who is the same person as the person shown in the captured image.

[0151] In this embodiment, when the captured image is transmitted to the authentication server 3, the face feature acquisition unit 53 acquires face features of the faces captured in the captured image. Subsequently, the face matching unit 54 uses the face features acquired by the face feature acquisition unit to determine whether or not there is a registered user who is the same person as the person captured in the image.

[0152] When the face matching unit 54 determines that a registered user exists who is the same person as the person captured in the image, it sends a notification to the user terminal 2 indicating that the user has been authenticated, and also notifies the provider server 7 of the user ID of the registered user. As a result, a notification indicating that face authentication has been performed is displayed on the user terminal 2, and the user becomes able to use the services provided by the provider server 7.

[0153] When the face matching unit 54 determines that no matching person exists in the captured image for a registered user, it sends a notification to the user terminal 2 indicating that the user has been authenticated. As a result, the user terminal 2 displays a notification indicating that face authentication was not successful.

[0154] The facial recognition process is also performed on the authentication device 2A, which acts as the user terminal 2. Similar to the case where the user terminal 2 is a smartphone, when the imaging unit 24 of the authentication device 2A acquires an image including the user's face, it is transmitted to the authentication server 3. The facial matching unit 54 then determines whether the person in the image matches the registered user. If the same person exists, various displays and processes are performed, such as a notification indicating that facial recognition has been successful on the authentication device 2A.

[0155] Next, we will explain the effects of the authentication system 1 configured in this way.

[0156] In authentication system 1, when the facial photograph (first facial information) read from the identity verification document matches the captured image (second facial image), and when it is determined that the user has assumed a pose (posture) in accordance with the pose instructions, the facial information is added to the user database 52A and user registration is performed. As shown in Figure 6, the captured image is acquired by the imaging device (imaging unit 24) while the user is performing an action (authentication action) in accordance with the pose instructions.

[0157] As shown in Figure 6, while the user performs an action (authentication action) according to the pose instruction, an image including the user's face is acquired, and facial recognition is performed using that image. Therefore, since the user does not need to separately capture an image of their face when registering facial information to be used for facial recognition, the time required to complete the registration procedure can be shortened.

[0158] Figure 11 shows the expected sequence of actions a user might take when sequence ID 1 (see Figure 4) is selected and an action command is issued.

[0159] In the sequence with sequence ID 1, the user is first instructed to assume a pose facing left (first pose). Once it is confirmed that the user is facing left, the user is then instructed to assume a pose facing right (second pose).

[0160] As shown in Figure 11, between the time the user takes a pose facing left (first pose) and the time they take a pose facing right (second pose), there is a moment when the user's face is facing forward (see the frame labeled "Shooting Execution" in Figure 11). This allows the camera 14 to take a picture at that moment, obtaining an image of the user's face facing forward, thus enabling proper matching with the facial photograph read from the identity verification document during registration.

[0161] Figure 12 shows the expected sequence of actions a user might take when sequence ID 3 (see Figure 4) is selected and an action command is issued.

[0162] In sequence ID 3, the user is first instructed to assume a downward-looking pose (first pose). Once it is confirmed that the user is looking downwards, the user is then instructed to assume an upward-looking pose (second pose).

[0163] As shown in Figure 12, the period between the user taking a downward-looking pose (first pose) and taking an upward-looking pose (second pose) includes a moment when the user's face is facing forward (see the frame labeled "Shooting Execution" in Figure 12). This allows for the acquisition of an image of the user's face facing forward, enabling proper matching with the facial photograph read from the identity verification document during registration.

[0164] As shown in Figures 3 and 4, sequences with sequence IDs 1 to 4 involve the authentication process of taking two consecutive poses (postures) with different facial orientations. Furthermore, these two poses (postures) are configured so that the face faces symmetrically with respect to the front of the body. Therefore, as can be seen from Figures 11 and 12, the authentication process includes a moment when the user's face is facing forward. This makes it easy to obtain an image of the user's face facing forward, enabling proper matching with the facial photograph read from the identity verification document during registration.

[0165] As shown in Figures 3 and 4, sequences with sequence IDs 5 and 6 instruct the user to raise either their left or right hand. When raising either the left or right hand, the user is expected to assume the corresponding pose (posture) while facing user terminal 2. Therefore, the authentication process includes a moment when the user's face is facing forward. Thus, it is possible to acquire an image of the user's face facing forward, enabling proper matching with the facial photograph read from the identity verification document during registration.

[0166] (Second Embodiment) The authentication system 1 according to the second embodiment differs from the authentication system 1 according to the first embodiment in the sequence of registration processing, but the other configurations are the same, so a description is omitted.

[0167] Figure 13 shows a registration process sequence according to the second embodiment, where the selected sequence includes two pauses. When a sequence is selected on the user terminal 2, a pause instruction is issued.

[0168] As shown in Figure 6, in the first embodiment, a pause instruction for the first pose is given (S017), and when the corresponding first pose is taken (Yes in S108), a pause instruction for the second pose is given (S109). On the other hand, as shown in Figure 13, in the second embodiment, the processor 11 of the user terminal 2 sequentially gives a pause instruction for the first pose (S017) and a pause instruction for the second pose (S109), and then, after a predetermined time (for example, 5 to 10 seconds) has elapsed, it determines whether the pause condition has been met (S201).

[0169] However, the posing instructions for the first pose and the posing instructions for the second pose may be given sequentially or simultaneously.

[0170] The processor 11 of user terminal 2 issues multiple pause instructions and comprehensively determines whether the pause condition is met based on the results of determining whether the corresponding pauses were taken. For example, the processor 11 of user terminal 2 may determine that the pause condition is met when all the pauses corresponding to each pause instruction have been taken, or the processor 11 of user terminal 2 may determine that the pause condition is met when the proportion of pauses corresponding to the pause instructions that have been taken is above a certain level. In addition, the processor 11 of user terminal 2 may determine that the pause condition is met when at least some of the pauses corresponding to the pause instructions have been taken.

[0171] In this embodiment, the processor 11 of the user terminal 2 determines that the pause condition is met when either the first pause is taken in response to the instruction for a first pause (hereinafter referred to as the first pause condition) or the second pause is taken in response to the instruction for a second pause (hereinafter referred to as the second pause condition) is met. Alternatively, the processor 11 of the user terminal 2 may also determine that the pause condition is met when both the first pause condition and the second pause condition are met.

[0172] After the first pose instruction (S107) is given, and until it is determined that the pose condition has been met (S112), the processor 11 of the user terminal 2 uses the camera 14 to acquire images of the user facing forward (S111).

[0173] When it is determined that the pose condition has been met, the processor 11 of the user terminal 2 determines whether the image captured by the camera 14 is an image in which the user is facing forward and is suitable for face matching (S112). If it is determined that the image is not suitable for face matching (No in S112), the processor 11 of the user terminal 2 issues another pose instruction (first pose instruction and second pose instruction) (S107). If it is determined that the image is suitable for face matching (Yes in S112), the processor 11 of the user terminal 2 sends a notification that the operation has been authenticated and the captured image to the authentication server 3, similar to the first embodiment (S113).

[0174] When the processor 41 of the authentication server 3 receives notification that the operation has been authenticated and the captured image, it extracts facial features from the facial photograph (first facial information) of the identity verification document and the captured image (second facial information), respectively, performs facial matching, and determines whether or not they are the same person (S114). If it is determined that they are the same person, the processor 41 of the authentication server 3 registers the user by adding at least one of the facial photograph or captured image of the identity verification document, its type, at least one of the extracted facial features, and personal information such as the name to the user database 52A (S114).

[0175] Once the authentication server 3's processor 41 has finished adding the user to the user database 52A and user registration is complete, it sends a notification to the user terminal 2 indicating that registration is complete, similar to the first embodiment (S115). As a result, a registration completion screen 76 indicating that user registration is complete is displayed on the user terminal 2's touch panel display 13 (S116).

[0176] In the authentication system 1 according to the second embodiment, the captured image is also acquired by the imaging device (imaging unit 24) while the user is performing an action (authentication action) according to the pose instruction. Therefore, when registering facial information to be used for facial recognition, the user does not need to separately capture an image of their own face, thus shortening the time required to complete the registration procedure.

[0177] (Third embodiment) The authentication system 1 according to the third embodiment differs from the authentication system 1 according to the first embodiment in the sequence of registration processing, but the other configurations are the same, so a detailed explanation is omitted.

[0178] Figure 13 shows a registration process sequence according to the second embodiment, where the selected sequence includes two pauses. When a sequence is selected on the user terminal 2, a pause instruction is issued.

[0179] Figure 14 shows a registration process sequence according to the third embodiment, where the selected sequence includes two pauses. When a sequence is selected on the user terminal 2, a pause instruction is issued.

[0180] As shown in Figure 14, in the third embodiment, the processor 11 of the user terminal 2 sequentially issues a pause instruction for the first pose (S105) and a pause instruction for the second pose (S106). After issuing the pause instruction for the first pose (S105) and after issuing the pause instruction for the second pose (S106), the processor 11 of the user terminal 2 acquires an image captured by the camera 14 at the moment the user is facing forward (S111).

[0181] Next, the processor 11 of the user terminal 2 determines whether the image captured by the camera 14 is an image in which the user is facing forward and is suitable for face recognition (S112). If it is determined that the image is not suitable for face recognition (No in S112), the processor 11 of the user terminal 2 issues another pose instruction (first pose instruction and second pose instruction) (S107, S109). If it is determined that the image is suitable for face recognition, the processor 11 of the user terminal 2 transmits the captured image to the authentication server 3 (S301).

[0182] When the processor 41 of the authentication server 3 receives the captured image from the processor 11 of the user terminal 2, it extracts facial features from the facial photograph on the identity verification document and the captured image, performs facial matching, and determines whether or not they are the same person (S302). The processor 41 of the authentication server 3 sends the result of the facial matching to the user terminal 2 (S303).

[0183] When the processor 11 of the user terminal 2 receives the results of face matching from the authentication server 3, it determines whether the first pose (first posture) and the second pose (second posture) have been taken, and whether the face photograph (first face information) and captured image (second face information) on the identity verification document match, that is, whether the conditions required for registration (hereinafter, registration conditions) have been met (cleared) (S304).

[0184] When the processor 11 of user terminal 2 determines that the conditions required for registration (hereinafter referred to as registration conditions) have been met (cleared) (Yes in S304), it sends a registration permission notification to the authentication server 3 (S305). When the processor 11 of user terminal 2 determines that the registration conditions have not been met (No in S304), it issues another pause instruction (first pause instruction and second pause instruction) (S107).

[0185] When the authentication server 3's processor 41 receives a registration permission notification, it adds at least one of the facial photograph or captured photograph from the identity verification document, its type, at least one of the extracted facial feature quantities, and personal information such as the name to the user database 52A and performs user registration (S306).

[0186] Once the authentication server 3's processor 41 has finished adding the user to the user database 52A and user registration is complete, it sends a notification to the user terminal 2 indicating that registration is complete, similar to the first embodiment (S115). As a result, a registration completion screen 76 indicating that user registration is complete is displayed on the user terminal 2's touch panel display 13 (S116).

[0187] In the authentication system 1 according to the third embodiment, the captured image is also acquired by the camera 14 (imaging device, imaging unit 24) while the user is performing an action (authentication action) according to the pose instruction. Therefore, when registering facial information to be used for facial recognition, the user does not need to separately capture an image of their own face, thus shortening the time required to complete the registration procedure.

[0188] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments.

[0189] In the first to third embodiments described above, the operation instruction database 22B and the pause determination database 22A are described in an example where they are stored in the storage unit 22 of the user terminal 2, respectively. However, as shown in Figure 15, they may also be stored in the storage unit 52 of the authentication server 3. Furthermore, in the first to third embodiments described above, the operation authentication unit 26 is provided in the user terminal 2, but the operation authentication unit 26 may be provided in the authentication server 3.

[0190] Figure 4 shows an example where the motion instruction database 22B contains only a sequence of instructions for one or two poses, but the motion instruction database 22B may also contain sequences of two or more poses. However, in order to obtain an image suitable for facial recognition, it is preferable that the motion instruction database 22B contains a sequence in which the face faces forward at the moment when a series of poses (postures) are taken (i.e., when the authentication operation is performed).

[0191] In the above embodiment, the processor 11 of the user terminal 2 was configured to acquire an image captured when the user was facing forward. However, if the processor 11 of the user terminal 2 cannot acquire an image captured when the user is facing forward, it may generate an image captured when the user is facing forward from multiple images. In this case, the processor 11 of the user terminal 2 may generate an image captured when the user is facing forward using an image captured when the user is facing in a direction symmetrical to the front. [Industrial applicability]

[0192] The authentication method, authentication system, terminal, and server relating to this disclosure are authentication methods, authentication systems, terminals, and servers for verifying identity, and are useful as authentication methods, authentication systems, terminals, and servers for shortening the time required to complete the procedure. [Explanation of Symbols]

[0193] 1: Authentication System 2: User terminal 2A: Authentication device 3: Authentication Server 5: Communication Network 7: Provider Server 11: Processor 12: Memory 13: Touch panel display 14: Camera 15: Reader 16: Communication Interface 17: Storage 18: Speaker 21: Communications Department 22: Storage section 22A: Pose detection database 22B: Operation Instruction Database 23: Reading Section 24: Imaging Department 25:Notification Department 26: Operation Authentication Unit 27: Face Image Acquisition Unit 31: Pose instruction section 32: Pose matching unit 33: Operation Verification Unit 41: Processor 42: Memory 43: Storage 44: Communication Interface 51: Communications Department 52: Storage section 52A: User Database 53: Facial Feature Acquisition Unit 54: Face Recognition Unit 55: Facial Information Registration Section (Authentication Section) 56: Service Startup Processing Unit 61: Document Selection Screen 61A: Button 62: PIN entry screen 62A: PIN entry field 63: Read execution screen 63A: Message 65: Registered Notification Screen 65A: Pop-up 66: First pose instruction screen 66A: Captured image 66B: Text 67: Pause Authentication Error Screen 67A: Text 67B: Resume button 71: Second pose instruction screen 71A: Captured image 71B: Text 72: Verification execution screen 72A: Loading animation 73: Authentication completion screen 73A: Text 73B: Register button 73C: Face image 73D: Captured image 75: Authentication completion screen 75A: Text 75B: Register button 75C: Face photo 75D: Captured image 76: Registration completion screen 76A: Face photo 76B: Captured image 76C: Text 76D: Image 77: Notification screen 77A: Face photo 77B: Captured image 77C :Notification 77D: Resume button 77E: Exit button 101: Registration System

Claims

1. An authentication method for verifying identity performed by a computer, The aforementioned computer, First facial information is obtained from the user's identification documents. The system instructs the user to perform an authentication action and determines whether the instructed authentication action has been performed. The first facial information and the second facial information obtained by imaging the user are compared to determine if they match. The system is configured to authenticate that identity verification has been performed when it is determined that the instructed authentication operation has been performed and that the first facial information and the second facial information match. The aforementioned authentication process includes a first authentication process in which the person takes a first pose, each of which is not a frontal pose and whose face is facing in a different direction from the other person, and a second authentication process in which the person takes a second pose. The transition from the first pose to the second pose involves a pose where the face is facing forward. The second facial information is acquired by an authentication method in which the face is facing forward between the first and second authentication operations.

2. The authentication method according to claim 1, wherein when identity verification is authenticated, at least one of the first facial information and the second facial information is recorded as facial information for facial authentication.

3. The authentication method according to claim 1, wherein the first pose and the second pose are set to two poses in which the face is facing in a symmetrical direction with respect to the front of the body.

4. An authentication system for verifying identity, A first facial information acquisition unit that acquires first facial information from the user's identity verification documents, An operation authentication unit that instructs the user to perform an authentication operation and determines whether the instructed authentication operation has been performed, A face matching unit that compares the first face information with the second face information obtained by imaging the user and determines whether they match, The system includes an authentication unit that determines that the instructed authentication operation has been performed and that the first facial information and the second facial information match, and that identity verification has been performed. The aforementioned authentication process includes a first authentication process in which the person takes a first pose, each of which is not a frontal pose and whose face is facing in a different direction from the other person, and a second authentication process in which the person takes a second pose. The transition from the first pose to the second pose involves a pose where the face is facing forward. The second facial information is acquired by an authentication system at the moment when the face is facing forward between the first and second authentication operations.

5. A terminal for performing authentication for identity verification, First facial information is obtained from the user's identification documents. The system instructs the user to perform an authentication action and determines whether the instructed authentication action has been performed. The first facial information and the second facial information obtained by imaging the user are compared to determine if they match. The system is configured to authenticate that identity verification has been performed when it is determined that the instructed authentication operation has been performed and that the first facial information and the second facial information match. The aforementioned authentication process includes a first authentication process in which the person takes a first pose, each of which is not a frontal pose and whose face is facing in a different direction from the other person, and a second authentication process in which the person takes a second pose. The transition from the first pose to the second pose involves a pose where the face is facing forward. The second facial information is acquired by a terminal at the moment when the face is facing forward between the first and second authentication operations.

6. A server for performing authentication for identity verification, First facial information is obtained from the user's identification documents. The system instructs the user to perform an authentication action and determines whether the instructed authentication action has been performed. The first facial information and the second facial information obtained by imaging the user are compared to determine if they match. The system is configured to authenticate that identity verification has been performed when it is determined that the instructed authentication operation has been performed and that the first facial information and the second facial information match. The aforementioned authentication process includes a first authentication process in which the person takes a first pose, each of which is not a frontal pose and whose face is facing in a different direction from the other person, and a second authentication process in which the person takes a second pose. The transition from the first pose to the second pose involves a pose where the face is facing forward. The second facial information is acquired by a server at the moment when the face is facing forward between the first and second authentication operations.

Images