Facial recognition server, facial recognition system, facial recognition method, and facial recognition program

Abstract

To allow for preventing double registration of face information of a user for use in face authentication.SOLUTION: A face authentication server 2 acquires face information of a user as first face information for collation from a user terminal 3, and executes first collation processing of collating the first face information for collation with a plurality of pieces of registered face information stored in a storage device 13 to calculate collation scores indicating similarity between the first face information for collation and the registered face information. If a collation score equal to or higher than a prescribed threshold is calculated in the first collation processing, the server causes the user terminal 3 to display a registering operation screen 66 for prompting the user to execute an operation of registering the face information of the user, wherein the registering operation screen 66 includes alert information showing that the face information of the user has been registered, and when the operation of registering the face information of the user is executed on the registering operation screen 66, the face information of the user is stored in the storage device as the registered face information.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a face authentication server, a face authentication system, a face authentication method, and a face authentication program that register information regarding a user's face image as face information for face authentication.

Background Art

[0002] In recent years, services that utilize face authentication technology (hereinafter referred to as face authentication services) have become widespread in order to manage access to facilities and conduct payment procedures for goods and services. In face authentication technology, face matching is performed by comparing face information of a user (i.e., the person to be authenticated) obtained as an authentication target with information regarding a face image registered in advance (hereinafter referred to as a registered image). Thereby, it is possible to determine whether the user is a legitimate registered person.

[0003] Regarding information regarding a user's registered image, for example, a telework management system is known in which a plurality of types of feature amounts with different face orientations and the like for the same person are prepared in order to increase the recognition rate of the same person (see Patent Document 1).

[0004] Also, a plurality of face images are extracted from an image captured by one camera, any one of the plurality of face images is selected as a reference face image, the similarity between the reference face image and the plurality of face images is obtained and compared for each of the plurality of face images, and among the comparison results, face images having a plurality of predetermined similarities are registered in a face identification dictionary that identifies the face image of the person of the reference face image. An information processing apparatus is known (see Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

[0006] Incidentally, with the spread of facial recognition services, there is an increasing trend for users themselves to register their facial images as described above, without involving a third party such as the administrator of the facial recognition system. Furthermore, there are cases where users register their facial images linked to information other than their name, which would otherwise easily identify them (i.e., the individual user) (for example, the user's nickname).

[0007] In such cases, users may unintentionally re-register information about their face image (i.e., a face image similar to the registered image) with the same facial recognition service (for example, the same shopping site) even if information about their face image has already been registered. Such duplicate registration of information about similar face images contributes little to improving the recognition rate of the same person, while complicating the management of information about registered images and potentially hindering the smooth processing of facial recognition.

[0008] However, the prior art described in Patent Documents 1 and 2 above does not take any special consideration to prevent the duplicate registration of facial information for user facial recognition.

[0009] Therefore, the main purpose of this disclosure is to provide a facial recognition server, a facial recognition system, a facial recognition method, and a facial recognition program that can easily prevent duplicate registration of user facial information used for facial recognition. [Means for solving the problem]

[0010] The facial recognition server of this disclosure is a facial recognition server that registers information relating to a user's facial image as facial information for facial recognition, comprising: a processor that performs a facial registration process for registering the user's facial information; a storage device that stores information relating to multiple facial images of different people that have been registered in the past as registered facial information; and a communication device for communicating with a user terminal used by the user, wherein the processor, in the facial registration process, obtains the user's facial information from the user terminal as first matching facial information, and performs a first matching process that matches the first matching facial information with a plurality of registered facial information to calculate a matching score indicating the similarity of the facial information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to cause the user to perform a registration operation for the user's facial information, the registration operation screen includes alert information indicating that the user's facial information has already been registered, and when the registration operation for the user's facial information is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The system is configured to store the registered facial information in the storage device.

[0011] The facial recognition system disclosed herein is a facial recognition system that registers information relating to a user's facial image as facial information for facial recognition, comprising: a user terminal used by the user; a facial recognition server that performs a facial registration process to register the user's facial information; a storage device that stores information relating to multiple different people's facial images previously registered as registered facial information; and a communication device for communicating with the user terminal used by the user. The facial recognition server, in the facial registration process, obtains the user's facial information from the user terminal as first matching facial information, performs a first matching process that compares the first matching facial information with a plurality of registered facial information, calculates a matching score indicating the similarity of the facial information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to prompt the user to perform a registration operation for the user's facial information, the registration operation screen includes alert information indicating that the user's facial information has already been registered, and when the user's facial information registration operation is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The system is configured to store the registered facial information in the storage device.

[0012] The facial recognition method disclosed herein is a facial recognition method using a facial recognition server that registers information relating to a user's facial image as facial information for facial recognition, wherein the method obtains the user's facial information from a user terminal used by the user as first matching facial information, performs a first matching process that matches the first matching facial information with a plurality of registered facial information which is information relating to the facial images of a plurality of different people, calculates a matching score indicating the similarity of the facial information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to prompt the user to perform a registration operation for the user's facial information, the registration operation screen includes alert information indicating that the user's facial information has already been registered, and when the registration operation for the user's facial information is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The system is configured to store the registered facial information in a storage device.

[0013] The facial recognition program of this disclosure is a facial recognition program that causes a computer to perform information processing to register information about a user's facial image as facial information for facial recognition, wherein the information processing includes: obtaining the user's facial information from a user terminal used by the user as first matching facial information; performing a first matching process to compare the first matching facial information with a plurality of registered facial information which is information about the facial images of a plurality of different people, thereby calculating a matching score indicating the similarity of the facial information for each; if a matching score of a predetermined threshold or higher is calculated in the first matching process, displaying a registration operation screen on the user terminal to cause the user to perform a registration operation for the user's facial information; the registration operation screen includes alert information indicating that the user's facial information has already been registered; and if the user's facial information registration operation is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The configuration includes a procedure for storing the registered facial information in a storage device. [Effects of the Invention]

[0014] According to the present disclosure, it is possible to easily prevent double registration of user face information used for face authentication.

Brief Description of the Drawings

[0015] [Figure 1] Overall configuration diagram of the face authentication system according to the first embodiment [Figure 2] Functional block diagram of the face authentication server shown in FIG. 1 [Figure 3] Explanatory diagram showing an example of the registered user database shown in FIG. 2 [Figure 4] Explanatory diagram showing an example of the collation score database shown in FIG. 2 [Figure 5] Functional block diagram of the user terminal shown in FIG. 1 [Figure 6] Flowchart showing the flow of face registration processing in the face authentication system according to the first embodiment [Figure 7] Sequence diagram showing an example of the operations of the face authentication server and the user terminal in the face registration processing according to the first embodiment [Figure 8] Explanatory diagram showing an example of the screen displayed on the user terminal in the face registration processing according to the first embodiment ((A) Face capture screen, (B) Registration operation screen, (C) Registration completion screen) [Figure 9] Flowchart showing a modified example of the flow of face registration processing shown in FIG. 6 [Figure 10] Sequence diagram showing a modified example of an example of the operations of the face authentication server and the user terminal shown in FIG. 7 [Figure 11] Explanatory diagram showing an example of the screen displayed on the user terminal in the face registration processing according to the modified example [Figure 12] Flowchart showing the flow of face registration processing in the face authentication system according to the second embodiment [Figure 13] Sequence diagram showing an example of the operations of the face authentication server and the user terminal in the face registration processing according to the second embodiment [Figure 14] Explanatory diagram showing an example of the registered user database according to the second embodiment [Figure 15]Flowchart showing the flow of facial recognition processing in the facial recognition system according to the second embodiment. [Figure 16] Sequence diagram showing an example of the operation of the facial recognition server and user terminal in the facial recognition process according to the second embodiment. [Figure 17] Flowchart showing the flow of face registration processing in the face recognition system according to the third embodiment. [Figure 18] Sequence diagram showing an example of the operation of the facial recognition server and user terminal in the facial registration process according to the third embodiment. [Figure 19] Diagram illustrating an example of a registered user database according to the third embodiment. [Figure 20] Flowchart showing the flow of face registration processing in the face recognition system according to the fourth embodiment. [Figure 21] Sequence diagram showing an example of the operation of the facial recognition server and user terminal in the facial registration process according to the fourth embodiment. [Figure 22] This diagram illustrates an example of the screen displayed on the user terminal during the face registration process according to the fourth embodiment ((B1) Registration / Update Operation Screen, (B2) User ID Input Screen). [Modes for carrying out the invention]

[0016] The first invention made to solve the aforementioned problem is a face recognition server that registers information relating to a user's face image as face information for face recognition, comprising: a processor that performs a face registration process for registering the user's face information; a storage device that stores information relating to multiple different people's face images that have been registered in the past as registered face information; and a communication device for communicating with a user terminal used by the user, wherein the processor, in the face registration process, obtains the user's face information from the user terminal as first matching face information, and performs a first matching process that matches the first matching face information with a plurality of registered face information to calculate a matching score indicating the similarity of the face information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to cause the user to perform a registration operation for the user's face information, the registration operation screen includes alert information indicating that the user's face information has already been registered, and when the registration operation for the user's face information is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The system is configured to store the registered facial information in the storage device.

[0017] According to this, if the matching score between the user's facial information and previously registered facial information is above a predetermined threshold (i.e., the user's face is similar to the face of a registered person), the user is prompted to perform a facial information registration operation (i.e., the user is asked to decide whether or not to register their facial information), thus easily preventing duplicate registration of the user's facial information used for facial recognition. Furthermore, by applying a re-matching process to the information regarding the user's re-captured facial image and registering it as registered facial information, it becomes possible to reduce the similarity between the newly registered user's facial image information and the existing registered facial information (i.e., information regarding the facial images of other people who resemble the user), thereby improving the accuracy of the user's facial recognition process using the registered facial information.

[0020] Also, the 2 The invention is A face recognition server that registers information about a user's face image as face information for face recognition, comprising: a processor that performs a face registration process for registering the user's face information; a storage device that stores information about multiple face images of different people that have been registered in the past as registered face information; and a communication device for communicating with a user terminal used by the user, wherein the processor, in the face registration process, obtains the user's face information from the user terminal as first matching face information, performs a first matching process that matches the first matching face information with a plurality of the registered face information, calculates a matching score indicating the similarity of the face information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to prompt the user to perform a registration operation for the user's face information, and the registration operation screen includes alert information indicating that the user's face information has already been registered. When the user's face information registration operation is performed on the registration operation screen, a flag indicating the existence of other similar face information is added to the user's face information newly stored in the storage device as registered face information, and to the registered face information stored in the storage device for which a matching score equal to or greater than the threshold was calculated in the first matching process. Furthermore, in the first matching process, the flag is added only to the registered face information for which the highest matching score was calculated among the registered face information for which a matching score equal to or greater than the threshold was calculated. This will be the structure.

[0021] According to this, even if there is registered facial information for facial images of other people that are similar to the user's facial image, the registered facial information for the user's facial image and the registered facial information for facial images of other people can be easily distinguished by a flag indicating the existence of similar facial information elsewhere. Furthermore, a predetermined flag can be appropriately added to registered facial information relating to facial images of other individuals that are similar to the user's facial image.

[0024] Also, the 3 The invention is configured such that the processor performs a user face recognition process, and in the face recognition process, it calculates a matching score indicating the similarity of face information by comparing information about the user's face image generated by photography with the registered face information stored in the storage device, and if the face recognition process is not performed normally because a matching score of a threshold or higher is calculated for multiple registered face information, it determines whether the flag is attached to the registered face information for which the matching score of a threshold or higher has been calculated, and if it is determined that the flag is attached to the registered face information, it corrects the threshold to increase the value of the registered face information to which the flag is attached.

[0025] According to this, even if there is registered facial information for facial images of other people that are similar to the user's facial image, the accuracy of the user's facial recognition process using the registered facial information can be improved by increasing the threshold for the matching process related to that registered facial information.

[0026] Also, the 4 The invention is configured such that, when the processor determines that the flag has been added to the registered face information, it obtains the difference between the matching score and the threshold for the registered face information, and in the correction of the threshold, it adds a value greater than the difference to the threshold.

[0027] This makes it possible to appropriately adjust the threshold for the matching process, and as a result, improve the accuracy of user facial recognition processing using registered facial information.

[0028] Furthermore, the fifth invention is,A face recognition server that registers information about a user's face image as face information for face recognition, comprising: a processor that performs a face registration process for registering the user's face information; a storage device that stores information about multiple face images of different people that have been registered in the past as registered face information; and a communication device for communicating with a user terminal used by the user, wherein the processor, in the face registration process, obtains the user's face information from the user terminal as first matching face information, performs a first matching process that matches the first matching face information with multiple registered face information, calculates a matching score indicating the similarity of the face information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to prompt the user to perform a registration operation for the user's face information, the registration operation screen includes alert information indicating that the user's face information has already been registered, and when the registration operation for the user's face information is performed on the registration operation screen, stores the user's face information in the storage device as registered face information. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, an update operation screen is displayed on the user terminal to prompt the user to perform an update operation of the user's face information. If the user's face information update operation is performed on the update operation screen, an input operation screen is displayed on the user terminal to prompt the user to input information for identifying the user. If the information for identifying the user entered from the input operation screen matches the information for identifying the user associated with the registered face information corresponding to the matching score equal to or greater than the predetermined threshold, the user's face information is stored in the storage device as the registered face information.

[0029] According to this, if a user understands that their facial information has already been registered and wishes to update it, the system can appropriately respond to that user's request.

[0030] Also, the 6 The invention relates to a facial recognition system that registers information relating to a user's face image as facial information for facial recognition, comprising: a user terminal used by the user; a facial recognition server that performs a facial registration process to register the user's facial information; a storage device that stores information relating to multiple different people's facial images previously registered as registered facial information; and a communication device for communicating with the user terminal used by the user. The facial recognition server, in the facial registration process, acquires the user's facial information from the user terminal as first matching facial information, performs a first matching process that compares the first matching facial information with a plurality of registered facial information, calculates a matching score indicating the similarity of the facial information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to prompt the user to perform a registration operation for the user's facial information, the registration operation screen includes alert information indicating that the user's facial information has already been registered, and when the registration operation for the user's facial information is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The system is configured to store the registered facial information in the storage device.

[0031] According to this, if the matching score between the user's facial information and previously registered facial information exceeds a predetermined threshold, the user is prompted to perform a facial information registration operation, thus easily preventing duplicate registration of the user's facial information used for facial authentication. Furthermore, by processing the re-captured user's facial image again to create registered facial information, it becomes possible to reduce the similarity between the newly registered user's facial image and the existing registered facial information, thereby improving the accuracy of user facial recognition processing using registered facial information.

[0032] Also, the 7 The invention relates to a facial recognition method using a facial recognition server that registers information relating to a user's face image as facial information for facial recognition, wherein the method obtains the user's face information from a user terminal used by the user as first matching facial information, performs a first matching process that compares the first matching facial information with a plurality of registered facial information which is information relating to the face images of a plurality of different people, calculates a matching score indicating the similarity of the facial information, and if a matching score of a predetermined threshold or higher is calculated in the first matching process, displays a registration operation screen on the user terminal to prompt the user to perform a registration operation for the user's facial information, the registration operation screen includes alert information indicating that the user's facial information has already been registered, and when the registration operation for the user's facial information is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The system is configured to store the registered facial information in a storage device.

[0033] According to this, if the matching score between the user's facial information and previously registered facial information exceeds a predetermined threshold, the user is prompted to perform a facial information registration operation, thus easily preventing duplicate registration of the user's facial information used for facial authentication. Furthermore, by processing the re-captured user's facial image again to create registered facial information, it becomes possible to reduce the similarity between the newly registered user's facial image and the existing registered facial information, thereby improving the accuracy of user facial recognition processing using registered facial information.

[0034] Also, the 8The invention relates to a facial recognition program that causes a computer to perform information processing to register information relating to a user's facial image as facial information for facial recognition, wherein the information processing includes: acquiring the user's facial information from a user terminal used by the user as first matching facial information; performing a first matching process to compare the first matching facial information with a plurality of registered facial information which is information relating to the facial images of a plurality of different people, thereby calculating a matching score indicating the similarity of the facial information for each; if a matching score of a predetermined threshold or higher is calculated in the first matching process, a registration operation screen is displayed on the user terminal to cause the user to perform a registration operation for the user's facial information; the registration operation screen includes alert information indicating that the user's facial information has already been registered; and if the registration operation for the user's facial information is performed on the registration operation screen, A re-capture screen is displayed on the user terminal to prompt the user to re-capture their face image. If the user re-captures their face image, information regarding the re-captured face image of the user is obtained from the user terminal as second matching face information. A second matching process is performed to compare the second matching face information with a plurality of registered face information to calculate a second matching score indicating the similarity of the face information. If the second matching process does not calculate a matching score that is equal to or greater than the threshold, information regarding the re-captured face image of the user is obtained. The configuration includes a procedure for storing the registered facial information in a storage device.

[0035] According to this, if the matching score between the user's facial information and previously registered facial information exceeds a predetermined threshold, the user is prompted to perform a facial information registration operation, thus easily preventing duplicate registration of the user's facial information used for facial authentication. Furthermore, by processing the re-captured user's facial image again to create registered facial information, it becomes possible to reduce the similarity between the newly registered user's facial image and the existing registered facial information, thereby improving the accuracy of user facial recognition processing using registered facial information.

[0036] The embodiments of this disclosure will be described below with reference to the drawings.

[0037] (First Embodiment) Figure 1 is an overall configuration diagram of the facial recognition system 1 according to the first embodiment of this disclosure. Figure 2 is a functional block diagram of the facial recognition server 2 shown in Figure 1. Figure 3 is an explanatory diagram showing an example of the registered user DB 31 shown in Figure 2. Figure 4 is an explanatory diagram showing an example of the matching threshold DB 32 shown in Figure 2. Figure 5 is a functional block diagram of the user terminal 3 shown in Figure 1.

[0038] As shown in Figure 1, the facial recognition system 1 includes a facial recognition server 2 operated by a provider of facial recognition services. The facial recognition system 1 may also include multiple user terminals 3, each used by multiple users who are to be authenticated by the facial recognition service.

[0039] The facial recognition server 2 and each user terminal 3 can communicate with each other via a publicly known communication network 5 such as the Internet. However, the facial recognition server 2 and each user terminal 3 may also send and receive necessary information and data to and from each other via other information processing devices (i.e., indirectly) not shown.

[0040] The facial recognition server 2 consists of computers that perform the necessary processes to provide the facial recognition service. The facial recognition server 2 can perform information processing (hereinafter referred to as "facial registration processing") to register the facial information of users used for facial recognition (i.e., information about facial images) in a database. In addition, the facial recognition server 2 can perform information processing (hereinafter referred to as "facial recognition processing") to authenticate the facial image of the user to be authenticated based on the facial information of multiple people (i.e., multiple users whose facial information has already been registered) registered in the database.

[0041] As shown in Figure 2, the facial recognition server 2 includes a communication unit 11 (an example of a communication device), a control unit 12 (an example of a processor), and a storage unit 13 (an example of a storage device).

[0042] The communication unit 11 communicates wirelessly or via wired connection with other devices (in this case, user terminal 3, etc.) via the communication network 5 in accordance with a known communication protocol.

[0043] In the control unit 12, the image acquisition unit 21 acquires a captured image including the user's face from the user terminal 3 when the user registers their own face information during the face registration process. The image acquisition unit 21 can also acquire a captured image including the user's face from the user terminal 3 when the user performs face authentication during the face authentication process.

[0044] The face detection unit 22 detects a person's face from the captured image acquired from the user terminal 3 and obtains the user's face image by extracting the region of the detected face.

[0045] The feature generation unit 23 obtains the user's facial features by detecting facial feature points from the acquired user's face image.

[0046] The face registration unit 24 can store the acquired user face features (an example of matching face information) in the storage unit 13 along with the corresponding face image and user-related information during the face registration process (i.e., register them in the registered user DB 31).

[0047] As will be explained in more detail later, the face registration unit 24 can determine whether there is a possibility of duplicate registration of the user's face information during the face registration process (i.e., whether the user's face is similar to the face of a person who has already been registered). If the face registration unit 24 determines that there is a possibility of duplicate registration of face information, it will prompt the user to be registered to decide whether or not to register that face information, and will only execute the registration if the user requests it. In addition, if the face registration unit 24 determines that there is a possibility of such duplicate registration, it can add a predetermined flag to the information of the user to be registered and to the face information of a person who is similar to the user.

[0048] As shown in Figure 3, the Registered User DB31 is a database containing facial information and related information for multiple registered users (i.e., multiple individuals). Each user's facial information includes the user's facial image and facial features obtained from that image. However, in the facial recognition system 1, either the facial image or facial features may be used as the facial information for each user. Each user's related information includes the user ID (identifier), the date and time of registration in the Registered User DB31, information about the registration site (e.g., URL), and the user's name, gender, date of birth, age, nickname, and telephone number. The registration site information includes information about the site that prompted the user to register their facial information, or in other words, information about the site (e.g., shopping site) where the user wishes to use the facial recognition service.

[0049] However, in the registered user DB31, information that easily identifies an individual user (in this case, the user's name and phone number) may be omitted. For example, each user's facial information may be associated only with information that does not identify the individual user (such as information from the registration site or a nickname).

[0050] In the facial recognition process, the facial recognition unit 25 determines whether the user is a legitimate registrant by comparing the facial information of the user to be authenticated with the facial information of the person stored in the storage unit 13 (i.e., registered in the registered user DB 31).

[0051] As will be explained in more detail later, the facial recognition unit 25 is capable of correcting the matching threshold used in the matching process of facial information (in this case, facial features) during the facial recognition process. In particular, the facial recognition unit 25 can correct the matching threshold based on the results of the matching process of facial information in the facial registration process.

[0052] The GUI control unit 26 transmits various GUI (Graphical User Interface) screens to the user terminal 3, causing those GUI screens to be displayed on the user terminal 3's display. The GUI control unit 26 can also acquire input information based on the user's input operations on each GUI screen. Furthermore, the GUI control unit 26 can control the display of the GUI screens based on that user input information.

[0053] The storage unit 13 stores the matching threshold DB 32. As shown in Figure 4, the matching threshold DB 32 is a database containing information on matching thresholds for matching scores calculated in face information matching for each user (i.e., a registered person) registered in the registered user DB 31. The matching score indicates the similarity between two sets of face information being matched. The matching threshold is used to determine whether the face information to be matched is similar to each other.

[0054] For example, in facial recognition processing, if the matching score between the facial information of the user being matched and the facial information of a person already registered in the registered user DB31 is equal to or greater than the matching threshold set for that person, it is determined that the facial information of the user being matched and the facial information of the registered person are similar (or that the facial images of both are similar to each other). On the other hand, in facial recognition processing, if the matching score is less than the matching threshold, it is determined that the facial information of both is not similar to each other.

[0055] The computer comprising the facial recognition server 2 may be equipped with one or more known hardware such as a processor (CPU, MPU, etc.), memory, a display, input devices, a network interface, and storage.

[0056] The communication unit 11 may include a communication device equipped with an antenna, a communication circuit, and the like.

[0057] At least some of the functions of each of the parts 21 to 26 in the control unit 12 can be realized by one or more processors executing a predetermined control program. Furthermore, the control unit 12 can comprehensively control the operation of the facial recognition server 2.

[0058] The memory unit 13 includes a storage device for storing data and information necessary for processing by the facial recognition server 2. The memory unit 13 does not need to be integrated with the facial recognition server 2; it may be an external storage device connected to the facial recognition server 2 via a network or the like.

[0059] Note that in Figure 1, for the sake of explanation, only one facial recognition server 2 is shown. However, in the facial recognition system 1, at least some of the functions of the facial recognition server 2 described above may be realized by the cooperation of multiple computers.

[0060] User terminal 3 includes information devices with communication capabilities, such as smartphones, tablet devices, or notebook PCs, carried by the user. However, information terminals installed in stores or public facilities (e.g., kiosk terminals) may also be used as user terminal 3.

[0061] As shown in Figure 5, the user terminal 3 has a communication unit 41, an imaging unit 42, an operation unit 43, a display unit 44, a control unit 45, and a storage unit 46.

[0062] The communication unit 41 has the same configuration as the communication unit 11 described above and communicates wirelessly or via wired connection with other devices (in this case, the facial recognition server 2, etc.) via the communication network 5 according to a known communication protocol. The imaging unit 42 includes a camera equipped with known hardware and can acquire (generate) an image including the user's face by taking a picture with the camera.

[0063] The operation unit 43 includes known input devices such as a touch panel display and input buttons that allow user input operations on the user terminal 3.

[0064] The display unit 44 includes known display devices such as liquid crystal displays. However, a touch panel display used as an input device may also function as a display device.

[0065] The control unit 45 includes one or more processors, and these processors execute predetermined control programs to perform processing related to the facial recognition service used by the user (for example, processing to register the user's facial information and processing to perform facial recognition of the user). In addition, the control unit 45 can comprehensively control the operation of the user terminal 3.

[0066] The storage unit 46 includes a storage device for storing data and information necessary for processing the user terminal 3. The storage unit 46 stores a facial recognition application 51, which is one of the control programs for the user terminal 3 to execute processing related to the facial recognition service. The facial recognition application 51 is installed on the user terminal 3 in advance.

[0067] Furthermore, the memory unit 46 stores user information 52 used when using the facial recognition service. User information 52 may include information about the user using the user terminal 3 (for example, the user's name, gender, age, nickname, and telephone number) and a captured image including the user's face generated by the shooting unit 42 (i.e., camera shooting). At least a portion of the information contained in user information 52 may be used as information to identify the user.

[0068] However, in the user information 52, as with the registered user DB 31 described above, information that makes it easy to identify an individual user may be omitted. Also, the user information 52 does not need to be permanently stored in the storage unit 46; for example, it may be temporary information entered by the user on a GUI screen.

[0069] A service provider offering any service can partner with a facial recognition service provider to allow customers to use facial recognition system 1 when providing them with a specified service. In other words, users of facial recognition system 1 include customers of such service providers. Services provided to users include, for example, the sale of goods and tickets, use of various facilities including transportation, medical examinations and treatments at hospitals, guarantees or provision of credit related to credit sales, and permission to log in to specific websites. Facial recognition system 1 may include an information processing device managed by such service provider and which works in cooperation with facial recognition server 2 to perform necessary processing.

[0070] Next, the face registration process performed by the face recognition system 1 according to the first embodiment will be described.

[0071] Figure 6 is a flowchart showing the flow of the face registration process in the face recognition system 1. Figure 7 is a sequence diagram showing an example of the operation of the face recognition server 2 and user terminal 3 during the face registration process. Figure 8 is an explanatory diagram showing an example of the screen displayed on the user terminal 3 during the face registration process.

[0072] Note that in Figures 6 and 7, the same codes (ST101 to ST108) are assigned to the corresponding processes, except for step ST109. However, Figure 7 shows the case where a matching score exceeding the matching threshold is detected in step ST103 of Figure 6 (Yes in ST103), and where the user performs a registration operation in step ST106 (Yes in ST106).

[0073] The user can register their facial information with the facial recognition system 1 by following the instructions of the facial recognition application 51 running on the user terminal 3.

[0074] As shown in Figures 6 and 7, first, the user takes a picture of their face using the camera provided by the user terminal 3 (ST101).

[0075] In step ST101, the user terminal 3's display shows a face capture screen 61 as a GUI screen prompting the user to capture their face, as shown in Figure 8(A), for example. On the face capture screen 61, the user places their face within the capture area 62 and presses the capture button 63 to perform a face capture. Furthermore, on the face capture screen 61, the user presses the face image registration button 64, which sends a face registration request containing the user's face image to the face authentication server 2.

[0076] When the face recognition server 2 receives the face registration request, it executes a matching process (an example of the first matching process) (ST102). In this matching process, information about the user's face image received from the user terminal 3 (an example of the first matching face information) is compared with the face information of multiple individuals registered in the registered user DB 31 (an example of registered face information). In this embodiment, the face recognition server 2 compares the face features obtained from the user's face image with the face features of multiple individuals registered in the registered user DB 31.

[0077] Furthermore, in step ST102, the facial recognition server 2 calculates a matching score indicating the similarity between the user's facial information and the facial information of each registered person using a known method. In this embodiment, a higher matching score indicates a higher degree of similarity (i.e., a higher probability that the user and the registered person are the same person).

[0078] Next, the facial recognition server 2 compares the matching score calculated in step ST102 with the matching threshold set for the corresponding person (see Figure 4) to determine whether the calculated matching score is equal to or greater than the matching threshold (ST103). If the facial recognition server 2 calculates a matching score equal to or greater than the matching threshold for any person, it extracts the relevant information for that person from the registered user DB31 (see Figure 3) (ST104).

[0079] Subsequently, the facial recognition server 2 sends a command to the user terminal 3 to display a registration screen in order to have the user perform the operation to register the user's facial information (i.e., to have the user decide whether or not to register their facial information). This command to display a registration screen may include relevant information about the person to be matched, which was extracted in step ST104.

[0080] When the user terminal 3 receives the command to display the registration operation screen, the user terminal 3's display shows the registration operation screen 66, for example, as shown in Figure 8(B) (ST105). The registration operation screen 66 includes alert information indicating the possibility of duplicate registration (i.e., the user's face information has already been registered) for the user's face image (or face information). The registration operation screen 66 displays alert messages such as "Duplicate registration detected" and "Have you already registered a face image?" to alert the user. The alert information may also include at least some of the relevant information of the person to be matched, extracted in step ST104 (for example, a message such as "Have you already registered face information with the nickname ○○?") to help the user recall the possibility of duplicate registration. In this embodiment, the registration operation screen 66 is smaller in size than the face capture screen 61 and is displayed as a pop-up overlapping the face capture screen 61. The registration operation screen 66 may also be displayed in place of the face capture screen 61.

[0081] On the registration operation screen 66, when the user presses the image registration button 67 (i.e., the user performs the operation to register face information) (ST106, however, ST106 is Yes in Figure 6), the user terminal 3 sends a registration command to the face authentication server 2 in order to register the user's face information in the registered user DB 31. If the user presses the cancel button 68 on the registration operation screen 66 (No in ST106 as shown in Figure 6), the face registration process ends without the user's face information being registered. For example, a user can press the cancel button 68 if they recall having previously registered their face information.

[0082] When the facial recognition server 2 receives the registration command, it registers the facial information of the user to be registered in the registered user DB 31 (ST107). At this time, even if facial information of a person similar to the facial information of the newly registered user has already been registered, the user's facial information will be registered (forced registration) after the user makes a decision (i.e., performs the registration operation in step ST106). Subsequently, the facial recognition server 2 sends a command to the user terminal 3 to display a registration completion screen in order to inform the user that the registration of the user's facial information is complete.

[0083] When user terminal 3 receives a command to display the registration completion screen, it displays the registration completion screen 71 on its display, for example, as shown in Figure 8(C) (ST108). When the user presses the OK button 69 on the registration completion screen 71, the face registration process is completed.

[0084] On the other hand, if the face recognition server 2 does not calculate a matching score that is equal to or greater than the matching threshold during the matching process (No in step ST103 shown in Figure 6), the possibility of duplicate registration is not (or is small), so the user's face information is registered in the registered user DB31 (normal registration) (ST109). This completes the face registration process.

[0085] Furthermore, the facial recognition server 2 may be configured to perform only facial registration processing (i.e., function as a facial registration server). In that case, the registered facial information obtained through the facial registration processing (i.e., the information registered in the registered user DB 31) is used by other information processing devices capable of performing facial recognition processing.

[0086] In the example of the face registration process described above, even if face information of a person similar to the user's face information (in this case, face image or face features) is already registered in the registered user DB31, if the user performs a registration operation, the user's face information may still be forcibly registered.

[0087] However, as shown below, if facial information similar to the user's facial information has already been registered, the facial recognition server 2 may be configured to request the user to retake a photo of their face.

[0088] Thus, in the facial recognition system 1, if the matching score between the facial information of the user to be registered and the facial information of any person in the registered user DB 31 (i.e., previously registered facial information) is equal to or greater than the matching threshold, the system prompts the user to perform a facial information registration operation (i.e., prompts the user to decide whether or not to register the facial information). As a result, the facial recognition system 1 can easily prevent duplicate registration of the user's facial information used for facial recognition.

[0089] Figure 9 is a flowchart showing a modified version of the face registration process shown in Figure 6. Figure 10 is a sequence diagram showing a modified example of the operation of the face authentication server 2 and user terminal 3 shown in Figure 7. Figure 11 is an explanatory diagram showing an example of the screen displayed on the user terminal 3 during the face registration process related to the modified version. Note that Figure 10 shows the case where all matching scores calculated in step ST106c of Figure 9 are below the matching threshold.

[0090] In the modified examples shown in Figures 9 and 10, after the user performs the facial information registration operation in step ST106, re-shooting (ST106a), second matching process (ST106b), and determination of the calculated matching score (ST106c) are performed.

[0091] More specifically, after step ST106, when the facial recognition server 2 receives a command to register the user's facial information, it sends a request to the user terminal 3 to retake the image (see Figure 10).

[0092] When user terminal 3 receives the reshoot request, it displays a reshoot screen 81 on its display (ST108) to prompt the user to reshoot their face, as shown in Figure 11, for example. The reshoot screen 81 displays a message prompting the user to reshoot, such as "Please take another photo of your face." In this embodiment, the reshoot screen 81 is smaller than the face-shooting screen 61, similar to the registration operation screen 66 described above, and is displayed as a pop-up overlapping the face-shooting screen 61.

[0093] In response to the message on the re-shooting screen 81, the user can re-shoot their face on the face shooting screen 61 (see Figure 8(A)) in the same way as in step ST101 (ST106a). Furthermore, when the user presses the face image registration button 64, the user's face image generated by the re-shooting is sent to the face authentication server 2.

[0094] When the facial recognition server 2 receives the recaptured user's facial image, it performs a second matching process (an example of a second matching process) based on the recaptured facial image, similar to the matching process in step ST102 (ST106b). In this matching process, information about the recaptured user's facial image (an example of second matching facial information) is compared with the facial information of multiple individuals registered in the registered user DB31 (an example of registered facial information).

[0095] The facial recognition server 2 compares the matching score calculated in step ST106b with the matching threshold set for each person to be matched (see Figure 4) to determine whether the calculated matching score is equal to or greater than the matching threshold. If, in the matching process for each person, the facial recognition server 2 does not calculate a matching score equal to or greater than the matching threshold (ST106c), it registers the face information of the target user in the registered user DB31 (ST107).

[0096] On the other hand, if the facial recognition server 2 calculates a matching score that is equal to or higher than the matching threshold in any of the matching processes for each person in step ST106c (Yes in step ST106c shown in Figure 9), it can again request the user to take a photo again in step ST106a.

[0097] This modified face registration process allows for a reduction in the similarity between newly registered user face information and existing person face information by re-applying information about the user's re-captured face image to the registered face information through a second matching process. As a result, the accuracy of the user's face recognition process can be improved.

[0098] (Second Embodiment) Next, a facial recognition system 1 according to the second embodiment of this disclosure will be described with reference to Figures 12 to 16. Regarding the facial recognition system 1 according to the second embodiment, components similar to those in the first embodiment are denoted by the same reference numerals and detailed descriptions are omitted. Furthermore, regarding the facial recognition system 1 according to the second embodiment, matters not specifically mentioned below are the same as in the first embodiment.

[0099] Figure 12 is a flowchart showing the flow of the face registration process in the face recognition system 1 according to the second embodiment. Figure 13 is a sequence diagram showing an example of the operation of the face recognition server 2 and user terminal 3 in the face registration process. Figure 14 is an explanatory diagram showing an example of a registered user database.

[0100] Note that in Figures 12 and 13, the same codes (ST201 to ST209) are assigned to the corresponding processes, except for step ST210. However, Figure 13 shows the case where a matching score exceeding the matching threshold is detected in step ST203 of Figure 12, and the user performs a registration operation in step ST206.

[0101] As shown in Figures 12 and 13, in the face registration process according to the second embodiment, steps ST201 to 206 are executed, which are the same as steps ST101 to ST106 shown in Figures 6 and 7, respectively.

[0102] Next, when the facial recognition server 2 receives a registration command from the user terminal 3, it adds a flag (hereinafter referred to as the multi-match flag) to the target of the matching process in step ST202 to indicate that similar facial information already exists (ST207). More specifically, the multi-match flag is added to the facial information of the user to be registered (i.e., newly stored in the storage unit 13) and to the facial information of the person for whom a score equal to or greater than the matching threshold was calculated in step ST203. In this embodiment, the multi-match flag is set to "1" (i.e., ON) if similar facial information already exists, and to "0" (i.e., OFF) if similar facial information already exists.

[0103] For example, in step ST203, let's consider the case where the matching score for the facial information of person AAA (i.e., the person with user ID "AAA") in the registered user DB31 is determined to be equal to or greater than the matching threshold. In this case, as shown in Figure 14, the multi-match flag for person AAA in the registered user DB31 is set to "1" (i.e., it is changed from "0" to "1").

[0104] Subsequently, the facial recognition server 2 registers the facial information of the user to be registered in the registered user DB 31 (ST208), similar to step ST107 shown in Figures 6 and 7. At this time, the multi-match flag of the newly registered user facial information in the registered user DB 31 is set to "1", similar to person AAA shown in Figure 14.

[0105] In step ST203, if the matching score for the facial information of multiple individuals in the registered user DB31 is determined to be equal to or greater than the matching threshold, the multi-match flag for those individuals is set to "1". However, if the matching score for the facial information of multiple individuals is determined to be equal to or greater than the matching threshold, the multi-match flag may be added only to the facial information of the individual with the highest matching score among those individuals. This ensures that the multi-match flag is appropriately added to the facial information of individuals (i.e., prevents the excessive addition of the multi-match flag).

[0106] Subsequently, in the face registration process according to the second embodiment, steps ST209 and ST210 are executed, which are the same as steps ST108 and ST109 shown in Figures 6 and 7, respectively.

[0107] Next, the facial recognition process performed by the facial recognition system 1 according to the second embodiment will be described.

[0108] Figure 15 is a flowchart showing the flow of the facial recognition process in the facial recognition system 1 according to the second embodiment. Figure 16 is a sequence diagram showing an example of the operation of the facial recognition server 2 and user terminal 3 in the facial recognition process.

[0109] Note that in Figures 15 and 16, the same codes (ST301 to ST308) are assigned to the corresponding processes, respectively. However, Figure 16 shows the case where the matching fails in step ST303 of Figure 15, the multi-match flag exists in step ST304, and the re-match is successful in step ST307.

[0110] As described above, users who have registered their facial information can use the facial recognition service provided by the facial recognition system 1 by following the instructions of the facial recognition application 51 executed on the user terminal 3.

[0111] As shown in Figures 15 and 16, first, the user takes a picture of their face using the camera provided by the user terminal 3 (ST301).

[0112] In step ST301, a screen similar to the face capture screen 61 (see Figure 8(A)) described above is displayed, and the user can capture their face on that screen. When the user performs an operation to request face authentication, a face authentication request including the user's face image is sent to the face authentication server 2.

[0113] When the face recognition server 2 receives the face registration request, it performs a matching process (ST302). In this matching process, information about the user's face image received from the user terminal 3 is compared with the face information of multiple individuals registered in the registered user DB31. In this embodiment, the face recognition server 2 performs the matching using face features, similar to the face registration process described above.

[0114] Next, the facial recognition server 2 determines whether the matching process has failed. For example, if the facial recognition server 2 finds that there are multiple individuals in the registered user DB 31 whose matching score is above the matching threshold calculated in step ST302, it determines that the matching process has failed (i.e., the facial recognition process is not performed correctly) (ST303). If the matching process has failed (Yes in ST303 in Figure 15), the facial recognition server 2 further determines whether a multi-match flag has been added to the facial information of the individual whose matching process failed (ST304).

[0115] Therefore, if the multi-match flag is attached to the person's facial information (Yes in ST304 in Figure 15), the facial recognition server 2 corrects the matching score for that person (ST305). For example, the facial recognition server 2 can correct the matching threshold by adding a predetermined value to the current matching threshold set for the target person (see Figure 4) to increase the matching threshold.

[0116] Subsequently, the face recognition server 2 performs a re-matching process using the corrected matching threshold (ST306). At this time, the same matching process as in step ST302 is performed again. The face recognition server 2 then determines that the re-matching process was successful if there are no multiple facial records in the registered user DB31 for a person whose matching score is equal to or greater than the matching threshold calculated in step ST306 (ST307, however, in Figure 15, ST307 is Yes). Next, the face recognition server 2 sends a command to the user terminal 3 to display a face recognition completion screen in order to inform the user that face recognition of the user has been completed.

[0117] When user terminal 3 receives the command to display the facial recognition completion screen, it displays a facial recognition completion screen on its display to the user indicating that facial recognition has been successfully completed (ST308). The facial recognition process ends when the user confirms this facial recognition completion screen.

[0118] Alternatively, if the re-matching process in step ST307 fails (No in ST307 in Figure 15), the system can return to step ST305 and re-correct the matching score. In this case, the facial recognition server 2 can gradually increase the matching score by repeatedly correcting the matching score in step ST305.

[0119] Thus, according to the facial recognition system 1 of the second embodiment, even if facial information relating to the facial image of another person similar to the facial image of the user to be registered exists in the registered user DB31, the user's facial information and the facial information of the other person can be easily distinguished by the multi-match flag, which indicates that similar facial information exists elsewhere. Furthermore, in the facial matching process, if the multi-match flag is attached to the facial information of the person to be matched, the matching score of that person is corrected, thereby making it possible to avoid matching failures caused by the existence of multiple facial information of a person that exceeds the matching threshold.

[0120] (Third embodiment) Next, a facial recognition system 1 according to the third embodiment of this disclosure will be described with reference to Figures 17 to 19. With respect to the facial recognition system 1 according to the third embodiment, components similar to those in the first or second embodiment are denoted by the same reference numerals and detailed descriptions are omitted. Furthermore, with respect to the facial recognition system 1 according to the third embodiment, matters not specifically mentioned below are the same as in the first or second embodiment.

[0121] Figure 17 is a flowchart showing the flow of the face registration process in the face recognition system 1 according to the third embodiment. Figure 18 is a sequence diagram showing an example of the operation of the face recognition server 2 and user terminal 3 in the face registration process. Figure 19 is an explanatory diagram showing an example of a registered user database. Figure 20 is an explanatory diagram showing an example of a matching score database.

[0122] As shown in Figures 17 and 18, in the face registration process according to the third embodiment, steps ST401 to 406 are executed, which are the same as steps ST201 to ST206 shown in Figures 12 and 13, respectively.

[0123] Next, when the facial recognition server 2 receives a registration command from the user terminal 3, it calculates the difference (or the absolute value of the difference) between the matching score calculated for the facial information of the person to be matched in step ST402 and the corresponding matching threshold, and adds a value greater than that difference to the matching threshold, thereby registering the corrected matching threshold value in the registered user DB 31 (ST407).

[0124] For example, consider the case where, in step ST402, a matching score of person AAA was calculated to be equal to or greater than the matching threshold, and in step ST407, it was determined that the difference between the matching score and the matching threshold was 59. In this case, as shown in Figure 19, the correction value for the matching threshold of person AAA in the registered user DB31 is 60, which is the difference of 59 plus 1. When the face matching process is executed, the matching threshold is corrected by adding this correction value to the current matching threshold (see Figure 4).

[0125] Subsequently, the facial recognition server 2 registers the facial information of the user to be registered in the registered user DB 31 (ST408), similar to step ST208 shown in Figures 12 and 13.

[0126] Subsequently, in the face registration process according to the third embodiment, steps ST409 and ST410, which are the same as steps ST209 and ST210 shown in Figures 12 and 13, are executed, respectively.

[0127] Next, the facial recognition process performed by the facial recognition system 1 according to the third embodiment will be described.

[0128] In the face registration process according to the third embodiment, steps substantially the same as steps ST301 to ST308 shown in Figures 15 and 16 are performed. However, in the face registration process according to the third embodiment, the correction of the matching score in step ST305 is performed based on the difference between the matching score calculated in step ST407 (see Figures 17 and 18) and the matching threshold (more precisely, by adding a value larger than that difference (see the correction value of the matching threshold for person AAA in Figure 19)).

[0129] Thus, according to the facial recognition system 1 of the third embodiment, the matching threshold in the facial recognition process can be appropriately corrected based on the difference between the matching score and the matching threshold. As a result, the facial recognition system 1 can improve the accuracy of the user's facial recognition process.

[0130] (Fourth Embodiment) Next, a facial recognition system 1 according to the fourth embodiment of this disclosure will be described with reference to Figures 20 to 22. With respect to the facial recognition system 1 according to the fourth embodiment, components that are the same as those in any of the first to third embodiments will be denoted by the same reference numerals as the corresponding components and detailed descriptions will be omitted. Furthermore, with respect to the facial recognition system 1 according to the fourth embodiment, matters not specifically mentioned below will be the same as in any of the first to third embodiments.

[0131] Note that in Figures 20 and 21, the same codes (ST501-ST505, ST507-ST511) are assigned to the corresponding processes, with the exception of steps ST506, ST512, and ST513. However, Figure 21 shows the case where a matching score above the matching threshold is detected in step ST503 of Figure 20 (Yes in ST503), no registration operation is performed by the user in step ST506 (No in ST506), an update operation is performed by the user in step ST507 (Yes in ST507), and the user ID is valid in step ST509 (Yes in ST509). Also, the screens shown in (B1) and (B2) of Figure 22 are displayed in place of the screen shown in (B) of Figure 8.

[0132] For example, in the facial recognition system 1 according to the first embodiment described above, the facial registration process is primarily intended for cases where each user who is to be authenticated by the facial recognition service is unaware of duplicate registration of their own facial information (see steps ST106 to ST107 in Figure 6, etc.). In contrast, the facial recognition system 1 according to the fourth embodiment can also handle updates of the user's facial information.

[0133] As shown in Figures 20 and 21, in the face registration process according to the fourth embodiment, steps ST501 to 506 are executed, which are the same as steps ST101 to ST106 shown in Figures 6 and 7, respectively.

[0134] In step ST505, when the user terminal 3 receives a command from the face recognition server 2 to display the registration operation screen, the user terminal 3's display shows, for example, the registration / update operation screen 166 (an example of an update operation screen) as shown in Figure 22(B1). In this embodiment, the registration / update operation screen 166 is smaller in size than the face capture screen 61, similar to the registration operation screen 66 described above, and is displayed as a pop-up overlapping the face capture screen 61.

[0135] The registration / update operation screen 166 includes alert information indicating the possibility of duplicate registration, similar to the registration operation screen 66 shown in Figure 8(B) above. The registration / update operation screen 166 also includes an image registration button 167, similar to the image registration button 67 described above, for the user to perform the face information registration operation. However, unlike the registration operation screen 66 described above, the registration / update operation screen 166 includes an image update button 168 for the user to perform the face information update operation.

[0136] On the registration / update operation screen 166, the user can press the image update button 168 to update face information (No in ST506 and Yes in ST507). As a result, the user terminal 3's display shows, for example, a user ID input screen 171 (an example of an input operation screen for the user to input information to identify the user) which prompts the user to enter a user ID, as shown in Figure 22(B2). In this embodiment, the user ID input screen 171 is smaller than the face capture screen 61, similar to the registration / update operation screen 166 described above, and is displayed as a pop-up overlapping the face capture screen 61.

[0137] Therefore, when a user enters their ID in the ID input field 172 on the user ID input screen 171 and then presses the OK button 173 (ST508), the user terminal 3 sends a verification request for the entered user ID (hereinafter referred to as the verification target ID) to the facial recognition server 2. At this time, the facial recognition server 2 also receives information about the verification target ID. The user can cancel the input of their user ID by pressing the cancel button 174 on the user ID input screen 171.

[0138] When the facial recognition server 2 receives a user ID verification request from the user terminal 3, it executes a process to verify the validity of the ID to be verified (hereinafter referred to as the ID verification process) (ST509). In the ID verification process, the facial recognition server 2 determines whether there is a user ID that matches the ID to be verified among the user IDs of one or more people for whom a matching score of or higher than the matching threshold was calculated in step ST503. At this time, the facial recognition server 2 can determine whether each user ID associated with the facial information of one or more people for whom a matching score of or higher than the matching threshold was calculated matches the ID to be verified by referring to the registered user DB 31 (see Figure 3).

[0139] Therefore, if a user ID matching the ID to be verified exists, the face recognition server 2 determines that the ID to be verified is valid (Yes in ST509) and updates the face information associated with that ID (i.e., the face information of the user whose user ID was determined to be valid) (ST510). At this time, the face information registered in the registered user DB31 for the target user is replaced with the new face information obtained based on the face capture in step ST501 (i.e., stored in the storage unit 13). Subsequently, the face recognition server 2 sends a command to the user terminal 3 to display a registration completion screen in order to inform the user that the registration (including updating) of the user's face information is complete (ST511).

[0140] On the other hand, if the facial recognition server 2 finds that no user ID matches the ID to be verified, it determines that the ID to be verified is invalid (No in ST509) and terminates the facial information update process. However, if the facial recognition server 2 determines that the ID to be verified is invalid (No in ST509), it may request the user terminal 3 to re-enter the user ID. At this time, a screen (including a message, etc.) prompting the user to re-enter the user ID is displayed on the user terminal 3's display. After the user re-enters the user ID, the same process as described above is executed again for the re-entered user ID (i.e., the new ID to be verified). This prevents the facial information update process from being immediately terminated if the user temporarily enters an incorrect user ID.

[0141] Furthermore, if the user presses the cancel button 169 on the registration / update operation screen 166 (No in ST507), the process will end without updating the user's facial information.

[0142] Furthermore, if, on the registration / update operation screen 166, the user presses the image registration button 167 instead of the image update button 168 (Yes in ST506), the face recognition server 2 registers the face information of the user to be registered in the registered user DB 31, similar to step ST107 shown in Figure 6 (ST512).

[0143] Furthermore, if the face recognition server 2 does not calculate a matching score equal to or greater than the matching threshold during the matching process (No in ST503), it registers the user's face information in the registered user DB 31 (ST513), similar to step ST109 shown in Figure 6.

[0144] Furthermore, Figure 22(B) shows an example of a registration / update operation screen 166 in which the user can selectively register and update facial information. However, the user terminal 3 may display separate screens for the user to perform facial information registration or update operations. In this embodiment, the system is configured to determine whether or not to update facial information based on the validity of the user ID entered as information for identifying the user. However, other information included in the user information 52 described above may be used instead of the user ID.

[0145] Thus, according to the facial recognition system 1 of the fourth embodiment, when a user understands that their facial information has already been registered and wishes to update that facial information, the system can appropriately respond to the user's request.

[0146] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0147] The facial recognition server, facial recognition system, facial recognition method, and facial recognition program related to this disclosure are useful as a facial recognition server, facial recognition system, facial recognition method, and facial recognition program for registering user facial information used for facial recognition, and can prevent duplicate registration of user facial information used for facial recognition. [Explanation of Symbols]

[0148] 1: Facial recognition system 2: Face recognition server 3: User terminal 5: Communication Network 11: Communications Department 12: Control Unit 13: Storage part 21: Image acquisition unit 22: Face detection unit 23: Feature generation unit 24: Face Registration Section 25: Facial Recognition Section 26: GUI Control Unit 41: Communications Department 42: Photography Department 43:Operation unit 44: Display section 45: Control Unit 46: Storage part 51: Facial recognition application 52: User Information 61: Face capture screen 62: Shooting area 63: Shoot button 64: Face Image Registration Button 66: Registration Operation Screen 67: Image registration button 68: Cancel button 69: OK button 71: Registration completion screen 81: Reshoot screen 166: Update operation screen 167: Image registration button 168: Image update button 169: Cancel button 171: User ID input screen 172: ID input field 173: OK button 174: Cancel button

Claims

1. A facial recognition server that registers information about a user's facial image as facial information for facial recognition, A processor that performs a face registration process to register the user's face information, A memory device that stores information about the facial images of multiple different individuals that have been registered in the past as registered facial information, A communication device for communicating with the user terminal used by the aforementioned user. Equipped with, The processor, in the face registration process, The user's facial information is obtained from the user terminal as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, a re-capture screen is displayed on the user terminal to prompt the user to re-capture the facial image. If the user's facial image is recaptured, information regarding the recaptured user's facial image is obtained from the user terminal as second matching facial information. By performing a second matching process that matches the second matching face information with a plurality of registered face information, a second matching score indicating the similarity of the face information is calculated. If the second matching process does not calculate a matching score equal to or greater than the threshold, the face authentication server stores the information regarding the re-captured user's face image as registered face information in the storage device.

2. A face recognition server that registers information relating to a user's face image as face information for face recognition, A processor that performs a face registration process to register the user's face information, A memory device that stores information about the facial images of multiple different individuals that have been registered in the past as registered facial information, A communication device for communicating with the user terminal used by the aforementioned user. Equipped with, The processor, in the face registration process, The user's facial information is obtained from the user terminal as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. A face authentication server that, when a user's face information registration operation is performed on the registration operation screen, adds a flag indicating the existence of other similar face information to the user's face information newly stored in the storage device as registered face information and to the registered face information stored in the storage device for which a matching score equal to or greater than the threshold has been calculated in the first matching process, and adds the flag only to the registered face information for which the highest matching score has been calculated among the registered face information for which a matching score equal to or greater than the threshold has been calculated in the first matching process.

3. The aforementioned processor, The facial recognition process for the user is performed, and in the facial recognition process, By comparing the information regarding the user's face image generated by the photograph with the registered face information stored in the storage device, the matching score indicating the similarity of the face information is calculated. If, in the calculation of the matching score, a matching score equal to or greater than the threshold is calculated for multiple registered facial information, and the facial recognition process fails to be performed normally, it is determined whether or not the flag is attached to the registered facial information for which the matching score equal to or greater than the threshold was calculated. The face authentication server according to claim 2, which, when it is determined that the flag has been added to the registered face information, corrects the threshold for the registered face information to which the flag has been added to increase.

4. The aforementioned processor, If it is determined that the registered face information has the flag attached, the face authentication server according to claim 3 obtains the difference between the matching score and the threshold for the registered face information, and in the correction of the threshold, adds a value greater than the difference to the threshold.

5. A face recognition server that registers information relating to a user's face image as face information for face recognition, A processor that performs a face registration process to register the user's face information, A memory device that stores information about the facial images of multiple different individuals that have been registered in the past as registered facial information, A communication device for communicating with the user terminal used by the aforementioned user. Equipped with, The processor, in the face registration process, The user's facial information is obtained from the user terminal as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, the user's facial information is stored in the storage device as the registered facial information. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, an update operation screen is displayed on the user terminal to prompt the user to perform an update operation on the user's facial information. When the user's facial information update operation is performed on the update operation screen, an input operation screen is displayed on the user terminal to prompt the user to input information for identifying the user. If the information for identifying the user entered from the input operation screen matches the information for identifying the user associated with the registered face information corresponding to the matching score above a predetermined threshold, A face recognition server that stores the user's face information as registered face information in the storage device.

6. A facial recognition system that registers information about a user's facial image as facial information for facial recognition, The user terminal used by the aforementioned user, A face recognition server that performs a face registration process to register the face information of the aforementioned user, A memory device that stores information about the facial images of multiple different individuals that have been registered in the past as registered facial information, A communication device for communicating with the user terminal used by the aforementioned user. Equipped with, The aforementioned facial recognition server, in the facial registration process, The user's facial information is obtained from the user terminal as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, a re-capture screen is displayed on the user terminal to prompt the user to re-capture the facial image. If the user's facial image is recaptured, information regarding the recaptured user's facial image is obtained from the user terminal as second matching facial information. By performing a second matching process that matches the second matching face information with a plurality of registered face information, a second matching score indicating the similarity of the face information is calculated. A face recognition system that, if a matching score equal to or greater than the threshold is not calculated in the second matching process, stores the information regarding the re-captured user's face image as registered face information in the storage device.

7. A facial recognition method using a facial recognition server that registers information about a user's facial image as facial information for facial recognition, The user's facial information is obtained from the user terminal used by the user as first matching facial information. By performing a first matching process that compares the first matching face information with multiple registered face information, which is information about the face images of multiple different people, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, a re-capture screen is displayed on the user terminal to prompt the user to re-capture the facial image. If the user's facial image is recaptured, information regarding the recaptured user's facial image is obtained from the user terminal as second matching facial information. By performing a second matching process that matches the second matching face information with a plurality of registered face information, a second matching score indicating the similarity of the face information is calculated. A face recognition method in which, if a matching score equal to or greater than the threshold is not calculated in the second matching process, the information relating to the re-captured user's face image is stored in a storage device as registered face information.

8. A facial recognition program that causes a computer to perform information processing to register information related to a user's facial image as facial information for facial recognition, The aforementioned information processing includes, The user's facial information is obtained from the user terminal used by the user as first matching facial information. By performing a first matching process that compares the first matching face information with multiple registered face information, which is information about the face images of multiple different people, a matching score indicating the similarity of the face information is calculated for each. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, a re-capture screen is displayed on the user terminal to prompt the user to re-capture the facial image. If the user's facial image is recaptured, information regarding the recaptured user's facial image is obtained from the user terminal as second matching facial information. By performing a second matching process that matches the second matching face information with a plurality of registered face information, a second matching score indicating the similarity of the face information is calculated. A face recognition program that includes a step of storing information about the re-captured user's face image as registered face information in a storage device if a matching score equal to or greater than the threshold is not calculated in the second matching process.

9. A facial recognition system that registers information relating to a user's facial image as facial information for facial recognition, The user terminal used by the aforementioned user, A face recognition server that performs a face registration process to register the face information of the aforementioned user, A memory device that stores information about the facial images of multiple different individuals that have been registered in the past as registered facial information, A communication device for communicating with the user terminal used by the aforementioned user. Equipped with, The aforementioned facial recognition server, in the facial registration process, The user's facial information is obtained from the user terminal as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. A face recognition system that, when a user's face information registration operation is performed on the registration operation screen, adds a flag indicating the existence of other similar face information to the user's face information newly stored in the storage device as registered face information and to the registered face information stored in the storage device for which a matching score equal to or greater than the threshold has been calculated in the first matching process, and adds the flag only to the registered face information for which the highest matching score has been calculated among the registered face information for which a matching score equal to or greater than the threshold has been calculated in the first matching process.

10. A facial recognition method using a facial recognition server that registers information relating to a user's facial image as facial information for facial recognition, The user's facial information is obtained from the user terminal used by the user as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. A face authentication method in which, when a user's face information registration operation is performed on the registration operation screen, a flag indicating the existence of other similar face information is added to the user's face information newly stored in the storage device as registered face information and to the registered face information stored in the storage device for which a matching score equal to or greater than the threshold was calculated in the first matching process, and the flag is added only to the registered face information for which the highest matching score was calculated among the registered face information for which a matching score equal to or greater than the threshold was calculated in the first matching process.

11. A facial recognition program that causes a computer to perform information processing to register information relating to a user's facial image as facial information for facial recognition, The aforementioned information processing includes, The user's facial information is obtained from the user terminal used by the user as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. A face authentication program that, when a user's face information registration operation is performed on the registration operation screen, adds a flag indicating the existence of other similar face information to the user's face information newly stored in the storage device as registered face information and to the registered face information stored in the storage device for which a matching score equal to or greater than the threshold has been calculated in the first matching process, and adds the flag only to the registered face information for which the highest matching score has been calculated among the registered face information for which a matching score equal to or greater than the threshold has been calculated in the first matching process.

12. A facial recognition system that registers information relating to a user's facial image as facial information for facial recognition, The user terminal used by the aforementioned user, A face recognition server that performs a face registration process to register the face information of the aforementioned user, A memory device that stores information about the facial images of multiple different individuals that have been registered in the past as registered facial information, A communication device for communicating with the user terminal used by the aforementioned user. Equipped with, The aforementioned facial recognition server, in the facial registration process, The user's facial information is obtained from the user terminal as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, the user's facial information is stored in the storage device as the registered facial information. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, an update operation screen is displayed on the user terminal to prompt the user to perform an update operation on the user's facial information. When the user's facial information update operation is performed on the update operation screen, an input operation screen is displayed on the user terminal to prompt the user to input information for identifying the user. A facial recognition system that, when the information for identifying the user entered from the input operation screen matches the information for identifying the user associated with registered facial information corresponding to a matching score above a predetermined threshold, stores the user's facial information in the storage device as registered facial information.

13. A facial recognition method using a facial recognition server that registers information relating to a user's facial image as facial information for facial recognition, The user's facial information is obtained from the user terminal used by the user as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, the user's facial information is stored in the storage device as the registered facial information. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, an update operation screen is displayed on the user terminal to prompt the user to perform an update operation on the user's facial information. When the user's facial information update operation is performed on the update operation screen, an input operation screen is displayed on the user terminal to prompt the user to input information for identifying the user. A facial recognition method in which, if the information for identifying the user entered from the input operation screen matches the information for identifying the user associated with registered facial information corresponding to a matching score above a predetermined threshold, the user's facial information is stored in the storage device as the registered facial information.

14. A facial recognition program that causes a computer to perform information processing to register information relating to a user's facial image as facial information for facial recognition, The aforementioned information processing includes, The user's facial information is obtained from the user terminal used by the user as first matching facial information. By performing a first matching process that matches the first matching face information with a plurality of registered face information, a matching score indicating the similarity of the face information is calculated. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, a registration operation screen is displayed on the user terminal to prompt the user to perform a registration operation for the user's facial information, and the registration operation screen includes alert information indicating that the user's facial information has already been registered. When the user's facial information registration operation is performed on the registration operation screen, the user's facial information is stored in the storage device as the registered facial information. If a matching score equal to or greater than a predetermined threshold is calculated in the first matching process, an update operation screen is displayed on the user terminal to prompt the user to perform an update operation on the user's facial information. When the user's facial information update operation is performed on the update operation screen, an input operation screen is displayed on the user terminal to prompt the user to input information for identifying the user. If the information for identifying the user entered from the input operation screen matches the information for identifying the user associated with the registered face information corresponding to the matching score above a predetermined threshold, A facial recognition program that stores the user's facial information as registered facial information in the storage device.

Images