Facial Recognition System

Abstract

By making various facial recognition systems available in a unified and cross-sectional manner, facial recognition will become more familiar and easier to use. [Solution] This problem is solved by a facial recognition system that includes multiple types of facial recognition engines with different facial template generation algorithms, an administration server that accepts images of users' faces or scan data of their three-dimensional shapes (hereinafter referred to as "facial images, etc.") and distributes the facial images, etc. to each of the facial recognition engines, and a user terminal on which users take photos of the facial images, etc., and the administration server or the user terminal has a pre-check means that is a means for provisionally evaluating the data quality of the taken facial images, etc. separately from the quality check in each of the facial recognition engines.

Description

[Technical Field]

[0001] The present invention relates to a face authentication system. [Background technology]

[0002] With the recent spread of digital transformation (DX), the range of applications for facial recognition technology is expanding. The main advantages of facial recognition are the convenience and improved security of the authentication process. Unlike knowledge authentication or possession authentication, facial recognition is a form of biometric authentication, and does not require memorization of a passcode or the carrying of any personal items. Furthermore, since the shapes of people's faces vary widely, it is difficult for others to imitate them. Furthermore, contactless use makes it hygienic and particularly effective as a measure against infectious diseases. Furthermore, technology that takes into account changes in environmental illuminance and age has reached a practical level, ensuring stable authentication accuracy. These characteristics make facial recognition promising for application in a variety of fields. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6839313 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, conventional facial recognition systems require users to take and register a face for each device (smartphone, etc.), facility, or organization where the system is installed, and there are various ways to start using the system. As a result, users must receive explanations and instructions, and must research how to set up the system themselves, each time they want to use facial recognition. This hassle makes facial recognition difficult to use and hinders its widespread adoption.

[0005] In view of these issues, the present invention aims to make face recognition more accessible and easier by enabling the use of various face recognition systems in a unified and cross-sectional manner. [Means for solving the problem]

[0006] In order to solve the above problems, the face recognition system of the present invention comprises a plurality of types of face recognition engines with different face template generation algorithms, a management server that accepts images of users' faces or scan data of their three-dimensional shapes (hereinafter referred to as "face images, etc.") and distributes the face images, etc. to each of the face recognition engines, and a user terminal that allows users to take pictures of their own face images, etc.

[0007] By installing a management server that wraps various facial recognition engines and functions as an abstraction (intermediate) layer, and by providing a user terminal that is a common mechanism for users to upload facial images, etc., users will be able to use facial recognition engines of different types and purposes in a unified and cross-sectional manner.

[0008] In this case, each of the multiple types of face recognition engines includes a camera that captures the face of an oncoming person, a face database in which face template data is stored, and an authentication means that compares the face captured by the camera with the face template data, and the multiple types of face recognition engines may include a cloud type in which the camera and the face database are communicatively connected via the Internet, and an edge type in which the face database and the authentication means are located in the same housing as the camera or on the same local network as the camera.

[0009] Furthermore, it is desirable that the management server or the user terminal has a pre-check means for evaluating the data quality of the captured facial images, etc. before distributing the facial images, etc. to each of the facial recognition engines. The management server centrally checks the quality of the facial images, etc. on behalf of various facial recognition engines, simplifying the process of confirming the quality of the facial images, etc. This minimizes the number of transfers and transactions of facial images, etc., and shortens the turnaround time after a user uploads a facial image, etc.

[0010] In this case, it is desirable that the pre-check means at least performs face detection on the captured face image, etc., and evaluates the data quality of the face image, etc. By evaluating the quality by imitating the process of generating an actual face template performed by each face recognition engine, it is possible to reduce the discrepancy between the evaluation result and whether or not the face is acceptable to each face recognition engine.

[0011] Alternatively, the management server may transfer the received face image, etc. to a precheck engine that is one of the face recognition engines, and receive from the precheck engine information indicating whether generation of a face template was successful or information indicating the data quality of the face image, etc. Instead of the management server having its own precheck means for quality evaluation, it may also use one of the face recognition engines as the precheck means.

[0012] Alternatively, the management server may distribute the received facial images, etc. to each of the facial recognition engines and receive information from these facial recognition engines indicating whether the generation of a facial template was successful. This prevents discrepancies between the evaluation results of the facial images, etc. and whether they are accepted by each facial recognition engine. In this case, no pre-check is performed.

[0013] Furthermore, in the facial authentication system of the present invention, the management server stores information regarding the user's payment method, and among the individual services provided using each facial authentication engine, for services involving payment by the payment method, it is desirable that authentication by another authentication method be additionally required or may be required in addition to facial authentication by the facial authentication engine used for that service.Facial authentication is biometric authentication and can be said to have higher authentication strength than other authentication methods, but the probability of false positives (false acceptance rate) is not zero.In payment processing where attempts of abuse are naturally expected, the security of payments can be further improved by incorporating facial authentication as part of MFA (Multi-Factor Authentication).

[0014] Furthermore, in the face authentication system of the present invention, it is desirable that the management server holds a user ID, which is unique identification information linked to the face image, etc., and that the individual services provided using each face authentication engine include an independent service that is a service having an information processing system using its own user account, and when face authentication is successful by the face authentication engine used in the independent service, the user ID or information that can uniquely identify the user account of the independent service is transmitted to the information processing system of the independent service. This allows the face authentication system of the present invention to be flexibly linked to existing systems.

[0015] Furthermore, it is desirable that the user terminal has a service registration means for registering the services that the user intends to use among the individual services provided using each facial recognition engine, and that the management server, upon receiving the user's facial image, distributes the facial image to at least the facial recognition engines used for the services registered by the user. Registering a facial image to a facial recognition engine for a service that the user does not use not only wastes resources but also increases unnecessary security risks. By allowing the user to select services using a unified system, such problems can be alleviated.

[0016] In this case, it is desirable that the management server retains the facial image etc. received from the user for at least a certain period of time, and when a service is registered by the service registration means after receiving the facial image etc., the management server distributes the retained facial image etc. to the face recognition engine used for the registered service. This allows the user to register for a service more easily without having to take a picture of their face each time they select a service.

[0017] In addition, when a user cancels registration for any service using the service registration means, it is desirable that the face template for that user generated by the face recognition engine used for that service be deleted or invalidated in order to prevent unnecessary resource consumption and remaining security risks.

[0018] In this case, the user terminal is owned by the user, and it is desirable that, after distributing the facial image, etc. to each facial recognition engine, when the user takes another photograph of the facial image, etc., the management server redistributes the new facial image, etc. to at least the facial recognition engines used for the services to which the user is registered. This is because a person's face is not permanently immutable, and the accuracy of the registered facial image, etc. may be insufficient. According to the present invention, even in such cases, the user can quickly re-register the facial image, etc., without the cooperation of the administrator of each service, etc.

[0019] In this case, the facial authentication system of the present invention preferably further includes a camera provided for each service that captures the face of an oncoming person, and when authentication of a user whose face is captured by that camera fails, any of the camera devices displays text or emits audio prompting the user to confirm whether the user is registered with that service. In order to sufficiently reduce the risk of false positives in facial authentication, it is effective to break down services into as small a range as possible. On the other hand, if services are broken down into extremely small segments, such as when users are prompted to select stores within the same chain for each store, the actual registration range may be narrower than the user perceives. In such cases, the cause of authentication failure can be quickly identified by having the user check the registration status of the service.

[0020] Furthermore, it is desirable that the user terminal has a service registration means for registering services to be used by the user from among the individual services provided using each of the facial recognition engines, and that the services include services that cannot be registered without an invitation operation by the service provider, a passcode or link notified to the user by the service provider, or an approval operation by the service provider, and services that can be registered without the invitation operation, passcode, link, or approval operation. This is because, among services that use facial recognition, there are services that are provided to general consumers and the entire public, and services that are provided only to contractors and certain related parties.

[0021] Furthermore, the face recognition system of the present invention may be configured such that the management server holds personal information, information on payment methods, information on tickets, or information set for each user by a provider of an individual service provided using each face recognition engine (hereinafter, such information will be collectively referred to as "user information"), and the user information includes information that is referenced when providing a specific service but is not referenced when providing other services. This is because as the number of available services increases, the number of types of data used only for that service also increases.

[0022] The user terminal may also be configured to transmit the facial images, etc., of the user's face captured from multiple directions to the management server, and the facial images, etc. may include images that are unnecessary for any of the facial recognition engines, because, depending on the type of facial recognition engine, not only a frontal image of the face but also an image of the face at an angle or with the head tilted may be required. [Effects of the Invention]

[0023] In this way, the face authentication system of the present invention makes face authentication more familiar and easier to use. [Brief explanation of the drawings]

[0024] [Figure 1]FIG. 2 is a block diagram showing an overview of a face authentication system F. [Figure 2] FIG. 2 is a layout diagram showing the functional configuration of a face authentication system F. [Figure 3] FIG. 2 is an image diagram illustrating the types of face recognition engine 20. [Figure 4] This is a use case diagram of the face authentication system F from the user's perspective. [Figure 5] FIG. 10 is a sequence diagram showing a process in which a user registers his or her own face in the face authentication system F. [Figure 6] FIG. 10 is a sequence diagram showing a modified example of the pre-check process for the face image P. [Figure 7] FIG. 2 is a schematic diagram showing how a user selects a service. [Figure 8] FIG. 10 is a sequence diagram showing the process of registering and deregistering a service in the face authentication system F. [Figure 9] Fig. 9(a) is a sequence diagram showing the registration process of a service that requires approval from the service provider, and Fig. 9(b) is a sequence diagram showing the registration process of a service that requires a prior invitation from the service provider. [Figure 10] FIG. 2 is a sequence diagram showing the face recognition process in each face recognition engine 20. [Figure 11] FIG. 10 is a sequence diagram showing another embodiment of the face authentication system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The facial recognition system F described below is a system that unifies facial recognition engines used in various services and provides users with a unified UI for using these engines. Below, this feature of the facial recognition system F and other associated features will be described using an embodiment as an example.

[0026] In addition, the term "service" as used herein refers to, for example, sales services, food and beverage services, transportation services, accommodation services, real estate services, payment services, childcare / nursing care services, entertainment services, etc. provided by businesses or public institutions. The services of the present invention are not limited to comprehensive services, but also include partial services, such as payment services within food and beverage services provided by businesses. Furthermore, the term is not limited to external services, but also includes services such as locking and unlocking provided by access control systems within a company or at home. In other words, the meaning of service here is extremely broad.

[0027] In addition, the term "face recognition engine" as used in this invention refers to a system that identifies an individual from input face data, and includes on-premise engines provided as independent products, cloud service engines, engines built into devices such as smartphones, or engines that combine these.

[0028] <System Overview> FIG. 1 is a block diagram showing an overview of a face authentication system F. The face authentication system F of this embodiment is mainly composed of a management server 10, multiple face authentication engines 20, a user application 30, and a service provider tool 39. The management server 10 holds data related to users of the face authentication system F and transmits and receives data to and from each face authentication engine 20. The user application 30 is a UI that allows users of the face authentication system F to take photos of their own faces and input personal information. The service provider tool 39 is a UI that allows each service provider to input and set data required for their service. In the face authentication system F, the service provider does not directly operate the face authentication engine 20 used in their service, but operates it via the management server 10 by using the service provider tool 39.

[0029] <Functional configuration> Fig. 2 is a layout diagram showing the functional configuration of the face recognition system F. As shown in Fig. 2, each face recognition engine 20 and each user application 30 of the face recognition system F is communicably connected to the management server 10. Although the service provider tool 39 is omitted from Fig. 2, the service provider tool 39 is also communicably connected to the management server 10. In this embodiment, each face recognition engine 20, each user application 30, and each service provider tool 39 transmits and receives data to and from the management server 10 via the Internet.

[0030] (User app) The user application 30 is a dedicated application installed on a user's smartphone (user terminal). The user application 30 includes a registration form 32 into which the user enters user information 111, including personal information and information related to payment methods, a face registration program 33 that uploads a face image P captured by the smartphone camera 31 to the management server 10, and a service addition button 34 that the user uses to register for a service that uses face recognition. The user application 30 is not limited to a smartphone application, but may also be dedicated software installed on the user's PC or a web application. In this embodiment, the face image P is described as a typical digital photograph image. However, depending on the type of face recognition engine 20 used, the user application 30 may also acquire scan data that captures the three-dimensional shape of the face. In the present invention, acquiring scan data of the user's face is also referred to as "capturing."

[0031] (Facial recognition engine) Each face recognition engine 20 includes a camera 21, which is a photographing device that photographs the face of an oncoming person, and a screen 22, which is a general-purpose touch panel. In this embodiment, the screen 22 displays the face being photographed by the camera 21, as well as messages and input forms for the user. Each face recognition engine 20 also includes a face database 25 that stores a face template T, which is biometric data for user face recognition, and an authentication program 26 (authentication means) that compares a face photographed by the camera 21 with the face template T. The face template T here refers to model data for comparison in which features are extracted from a user's face image P photographed by a user application 30, and may also be referred to as a face vector, feature amount, face model, or face map. A face recognition engine 20 suited to the characteristics and environment of the service provided using the engine is individually selected, and therefore the face recognition engine 20 of this embodiment includes a mixture of various products and services with different algorithms for generating the face template T.

[0032] The face recognition engine 20 of this embodiment is broadly divided into two types depending on its application, scale of use, etc. FIG. 3 is an image diagram illustrating the types of face recognition engine 20. The face recognition engine 20 of this embodiment includes a so-called edge-type face recognition engine in which the authentication program 26 and face database 25 are also built into a housing in which the camera 21 and the screen 22 are mounted, as shown in FIG. 3(a), and a so-called cloud-type face recognition engine in which a front end such as the camera 21 and the screen 22 is connected to a back end such as the authentication program 26 and the face database 25 via the Internet, as shown in FIG. 3(b). The edge-type in this embodiment also includes a device storing the authentication program 26 and the face database 25 connected to the same local network as the camera 21. The "local network" here refers to a LAN to which the camera 21 is connected or a network area network (PAN) connected to the camera 21. The cloud-type in this embodiment also includes a device in which the authentication program 26 is located in the same housing as the camera 21 and remotely accesses the face database 25 on the cloud.

[0033] (Management Server) 2, the explanation will be continued. The management server 10 has a user database 11 in which user account data including the above-mentioned user information 111 and face images P is registered, and a face recognition engine database 12 in which identification information, address information, product name and version, and other information necessary for managing each face recognition engine 20 are registered. The user database 11 in this embodiment also includes a user ID 113 (described later) that is unique identification information linked to the user account.

[0034] The management server 10 also has a pre-check program 16 that checks the data quality of facial images P uploaded from the user application 30, and a distribution program 17 that distributes facial images P that pass the quality check to each face recognition engine 20. Facial images P that pass the quality check are registered in the user database 11. When each face recognition engine 20 receives a facial image P from the management server 10, the authentication program 26 extracts features from the facial image P using the same algorithm as used during authentication, and generates a face template T. The generated face template T is registered in the face database 25.

[0035] The management server 10 of this embodiment is a virtual server constructed using a cloud hosting service. The management server 10 is not limited to a cloud server and may be an on-premise server installed within the operator's LAN. The management server 10 may also be composed of a group of multiple servers divided physically or functionally. The functions and data of the management server 10 shown in FIG. 2 are only a small portion of the functions and data possessed by the management server 10. The various functions possessed by the management server 10 do not all need to be developed in-house; they can also be implemented using external services. For example, the management server 10 of this embodiment includes a payment program 18 for processing a user's payment request and a PIN code 112 for identity verification at the time of payment, but the actual payment processing may be delegated to an external service that specializes in these functions. The functions of the management server 10 include the functions delegated to the external service.

[0036] (Use Case) FIG. 4 is a use case diagram of the facial recognition system F from the user's perspective. The facial recognition system F has two interfaces: a user application 30 for inputting the user's user information 111, photographing the user's face, and selecting (registering) the services the user will use; and a facial recognition engine 20 (camera 21) for photographing the user's face at the site where each service is provided. The distribution of facial images P to each facial recognition engine 20 and coordination with each service provider for registering the service to be used are performed via the management server 10. By providing the management server 10 that wraps various facial recognition engines 20 and services and the user application 30 as a single UI, the user can use facial recognition for various services in a unified and cross-sectional manner. This makes the facial recognition system F of this embodiment more accessible and convenient.

[0037] <Facial image pre-check function> 5 is a sequence diagram showing a process in which a user registers his or her own face in the face authentication system F. Hereinafter, with reference to FIG. 5, a pre-check function of a face image P in the process in which a user registers his or her face will be described.

[0038] (Face Registration 1) As shown in FIG. 5(a), when a user registers his or her face in the face authentication system F, the user inputs the necessary user information 111 into the registration form 32 of the user application 30 and takes a picture of his or her face with the camera 31. The input personal information 111 and face image P are transferred to the management server 10. The management server 10 evaluates, using the pre-check program 16, whether the data quality of the captured face image P is at a level that allows each face authentication engine 20 to generate a face template T. If the quality of the face image P is confirmed to meet this level, the management server 10 registers the face image P in the user database 11 and notifies the user that registration is complete. On the other hand, if the quality of the face image P is evaluated as insufficient, the management server 10 requests the user to take a picture of his or her face again (not shown). The face image P registered in the user database 11 is retained permanently or for a certain period of time and is distributed to each face authentication engine 20 at the time of service registration, which will be described later. In this manner, in this embodiment, the management server 10 centrally checks the quality of the face image P on behalf of various face authentication engines 20, thereby simplifying the process of checking the quality of the face image P. This minimizes the number of transfers and transactions of the facial image P, and shortens the turnaround time after the user uploads the facial image P.

[0039] Here, the precheck program 16 of this embodiment performs preprocessing, face detection, facial landmark detection, face alignment and cropping, noise removal, image normalization, feature extraction (generation of a face template), and even saving of the face template on the captured face image P, and determines that the data quality of the face image P is at an acceptable level upon completion of a full range of face template generation processes. In this way, in this embodiment, the quality of the face image P is evaluated by simulating the generation process of an actual face template T performed by each face recognition engine 20, thereby reducing the discrepancy between the evaluation result and whether or not the face is acceptable to each face recognition engine 20. The degree of checking by the precheck program 16 is adjustable, but it is desirable to at least check whether or not a face can be detected.

[0040] In the face authentication system F of this embodiment, the management server 10 has the pre-check program 16, but the pre-check program 16 can also be implemented as a function of the user application 30. In this case, a quality check of the face image P is performed before the face image P is transferred to the management server 10. Alternatively, a mechanism may be considered in which a simple check of the face image P is performed on the user application 30 side, and then a more detailed check is performed on the management server 10 side.

[0041] As mentioned above, the face recognition engine 20 of the face recognition system F includes a variety of products and services with different algorithms for generating face templates T. Depending on the type of face recognition engine 20, images of the face at an angle or with the head tilted may be required in addition to a frontal image of the face. Therefore, the face image P of the user captured according to the guidance of the user application 30 may include images that are unnecessary for one of the face recognition engines 20.

[0042] (Re-registering face image) FIG. 5(b) is a sequence diagram showing the process when a user re-registers a face image P. A user of the face authentication system F can re-register his or her own face image P at any time by using the user application 30. This is because a person's face is not permanently unchanged, and the accuracy of the registered face image P may be insufficient. Here, if the user has not yet selected a service, the re-registration process is the same as the face registration 1 described above. On the other hand, if the user has already registered for a service, after pre-checking the face image P, the new face image P is also redistributed to the face authentication engines 20 used in the services for which the user has registered, and the face template T is updated in each face authentication engine 20.

[0043] (Face Registration 2) FIG. 6 is a sequence diagram illustrating a modified example of the precheck process for a facial image P. In the example of FIG. 5, the management server 10 checks the quality of the facial image P using its own precheck program 16. However, the quality check for the facial image P can also be performed using one of the facial recognition engines 20. In the example of FIG. 6(a), the facial recognition engine 20 with the strictest quality standards for the facial image P is used as the precheck engine 20′. In this case, the management server 10 transfers the facial image P received from the user application 30 to the precheck engine 20′ and receives the check result from the precheck engine 20′. The check result may be information indicating whether the precheck engine 20′ successfully generated a face template T or information indicating the data quality of the facial image P. The process after the facial image P passes the quality check is the same as in the example of FIG. 5(a). If the precheck engine 20′ evaluates the data quality of the facial image P as insufficient, it requests the user to re-take a photograph of their face (not shown).

[0044] (Face Registration 3) FIG. 6(b) shows an example in which a face image P is directly distributed to each face recognition engine 20, without performing a pre-check, and a face template T is created. In this example, the management server 10 transfers the face image P received from the user application 30 to each face recognition engine 20 as is, and receives the generation results of the face template T from each face recognition engine 20. The process after all face recognition engines 20 have successfully generated the face template T is the same as the example in FIG. 5(a). Here, if any of the face recognition engines 20 fails to generate the face template T due to the data quality of the face image P, the user is requested to take a picture of their face again (not shown).

[0045] The data quality check by the pre-check program 16 of this embodiment differs from the algorithm for generating the actual face template T executed in each face recognition engine 20. While setting the pass standard by the pre-check program 16 as strict as possible can prevent failure in the subsequent generation process of the face template T, this cannot be guaranteed. The example of FIG. 6(b) reliably prevents discrepancies between the evaluation result of the face image P and the acceptance / non-acceptance by each face recognition engine 20. As will be described in detail later, in the face recognition system F of this embodiment, after the face image P is registered in the management server 10, the user selects the service to be used, and the face image P is transferred to the face recognition engine 20 used in the selected service. In other words, when the user first photographs his or her face, the service (face recognition engine 20) to be used by the user is unknown. Therefore, the example of FIG. 6(b) can be used in cases where the service to be used by the user is known in advance, the number of services available to the user is limited, or the user re-photographs the face image P after registering the service.

[0046] <Service registration function> 7 is a schematic diagram showing how a user selects a service. As shown in FIG. 7(a), the user application 30 is provided with a service addition button 34 (service selection means) that allows the user to register a service that the user will use. The management server 10 of this embodiment distributes the user's face image P only to the face recognition engine 20 used in the service registered by the user. This is because distributing the face image P to the face recognition engine 20 of a service that the user does not use not only wastes resources but also increases unnecessary security risks.

[0047] Tapping the Add Service button 34 in the user application 30 displays a list of services that the user can add. For example, if Service A (341) is selected, an input component 342 appears, allowing the user to enter information specific to the service, such as a membership number or tickets held. The input component 342 is not limited to a text box, but may also be a drop-down list, radio button, check box, or the like. The data entered here is saved in the user database 11 as user information 111. Depending on the selected service, the input component 342 may not be displayed. In other words, the user database 11 contains information that is referenced when providing a specific service but not referenced in other services. Among services that use facial recognition, there are those provided to general consumers or the entire public and those provided only to subscribers or certain related parties, as will be described in more detail later. If the service is intended for a specific individual, an input component 342 may appear for entering a passcode or other information previously provided by the service provider.

[0048] When the user enters the necessary information and taps the registration button 343, the management server 10 distributes the user's stored facial image P to the face recognition engine 20 of service A. The face recognition engine 20, to which the facial image P has been distributed, generates and registers a face template T from the facial image P. This prepares the user to use face recognition of service A.

[0049] Here, if service A is a service that requires approval from the service provider, an application button 344 is displayed instead of the registration button 343. When the user inputs the necessary information and taps the application button 344, an approval request is sent from the management server 10 to the service provider.

[0050] Alternatively, if service A is a service having an information processing system that uses its own user account, a sign-in form 345 for service A may be displayed to associate the face template T of face authentication system F with the user account of service A. Alternatively, the user may be redirected to a sign-in screen for service A. By signing in through this sign-in form 345, the identification information of the user is associated with that of service A. In this case, the face authentication system F may have identification information of the user account of service A, or the service A may have the user ID 113 of face authentication system F. When face authentication is successful in face authentication engine 20 of service A, the user ID 113 or information that can uniquely identify the user account of service A is sent to the system of service A.

[0051] The OAuth framework can also be used for this inter-service collaboration process. The OAuth protocol specifications are publicly known, so a detailed explanation will be omitted here, but the general flow is as follows. In the following example, the OAuth access token is information that can uniquely identify the user account of Service A. (1) When a user of face authentication system F selects service A in user application 30, the user is redirected to the authorization server (sign-in screen) of service A. (2) The user signs in to Service A at the redirect destination and allows the authorization server to link Service A with Facial Recognition System F. (3) As a result, the authorization server issues an authorization code to the face authentication system F, and the face authentication system F sends an access token request to the authorization server along with the issued authorization code. (4) The authorization server issues an access token to the face authentication system F. (5) When the user is successfully face-authenticated by the face authentication engine 20 at the site where the service A is provided, a message including an access token is sent from the face authentication system F or the face authentication engine 20 to the system of the service A. (6) This allows Service A to identify which user's face has been successfully recognized in Service A's system.

[0052] Note that this cooperation operation may be performed on the service A side, rather than the user application 30. Fig. 7(b) shows the face recognition usage setting screen 90 on the service A side. By signing in here with the user account of the face recognition system F, the identification information of both parties is associated, and the service A is registered on the user application 30.

[0053] (Service Registration 1) FIG. 8 is a sequence diagram showing the process by which a user registers or cancels service registration in the face authentication system F. When a user registers a service, the user taps the Add Service button 34 in the user application 30 and selects the service from the displayed list of services. This selection also includes inputting required information specific to the service. When the user selects a service, the management server 10 distributes the user's face image P to the face authentication engine 20 used for that service. The face authentication engine 20 generates a face template T from the face image P distributed from the management server 10 and registers it in the face database 25. Upon receiving a success code for the face template T from the face authentication engine 20, the management server 10 notifies the user application 30 that the service registration is complete. On the other hand, if the face authentication engine 20 fails to generate the face template T due to the data quality of the face image P, the management server 10 requests the user to re-take a face photograph (not shown).

[0054] (Service cancellation) When a user wants to cancel registration for a service, the user taps the add service button 34 of the user application 30 and taps the service to which the user has registered from the list of services that is displayed. This displays a message asking whether to cancel registration for that service. The cancellation process begins when the user responds to this message to continue with the cancellation. When the user performs an operation to cancel registration for a service, the management server 10 sends an instruction to delete the user's face template T to the face recognition engine 20 used for that service. The face recognition engine 20 deletes the user's face template T in accordance with the instruction from the management server 10. Here, the face recognition engine 20 may disable the face template T so that it cannot be used without deleting it. Upon receiving a code indicating success in the deletion process from the face recognition engine 20, the management server 10 notifies the user application 30 that the service cancellation has been completed.

[0055] (Service Registration 2) FIG. 9(a) is a sequence diagram showing the registration process for a service that requires approval from a service provider. When registering a service that requires approval from a service provider, the user first taps the Add Service button 34 in the user application 30, selects the service from the list of services displayed, enters the required information specific to the service, and taps the Apply button 344. When the Apply button 344 is tapped, the management server 10 sends an approval request to the service provider (service provider tool 39). The management server 10 then waits for the service provider's approval and distributes the user's facial image P to the face recognition engine 20 used for the service. The service provider's approval operation may not simply approve the application but may also set service-specific information for the user. For example, if the service is a locking and unlocking service for each facility in an apartment building, this information may include the user's room number. The data set by the service provider is registered in the user database 11 as user information 111. The subsequent process is the same as in Service Registration 1 described above, except that a registration completion notification is sent not only to the user but also to the service provider.

[0056] (Service Registration 3) FIG. 9(b) is a sequence diagram showing the registration process for a service that requires a prior invitation from the service provider. When registering a service that requires a prior invitation, the service provider first performs an invitation operation to a specific user from the service provider tool 39 and sends an invitation. The user receives the invitation and selects the service. The subsequent process is the same as in service registration 2 described above. The registration invitation sent to the user may include a link or passcode for selecting the service. The passcode here refers to a numeric string, a character string, or a one-dimensional or two-dimensional code such as a barcode that can be read by a scanner. Alternatively, the service provider's invitation operation may correspond to a prior approval operation for the user, and the registration invitation may simply be a notification thereof.

[0057] <Facial Recognition Process> 10 is a sequence diagram showing the face recognition process in each face recognition engine 20. Each pattern of the face recognition process will be described below with reference to FIG.

[0058] When a user photographs their face with the camera 21 of the facial recognition engine 20 at the service provider site, the authentication program 26 compares the face with a face template T. If authentication is successful, the intended request is accepted. On the other hand, if facial recognition fails, a message prompting the user to confirm whether the service is registered is displayed on the screen 22. As described above, in the facial recognition system F of this embodiment, the user must pre-register the service that uses facial recognition. While it is effective to divide the services into as small a number of categories as possible to sufficiently reduce the risk of false positives (false acceptance rates) in facial recognition, if the services are divided into extremely small categories, such as when users are required to select individual stores within the same chain, the actual registered range may be narrower than the user recognizes. In such cases, the cause of authentication failure can be quickly identified by prompting the user to check the registration status of the service. Note that the method of prompting the user to confirm is not limited to displaying a message on the screen 22; for example, a voice message to that effect may be emitted.

[0059] If the service for which facial recognition was successful involves a user payment process, the facial recognition engine 20 further requests the input of a PIN code 112. The user enters a PIN code using the numeric keypad displayed on the screen 22, and if it matches the registered PIN code 112, the payment process is carried out. If the entered PIN code does not match, the user is requested to re-enter it multiple times, and if it still does not match, the payment process fails. Facial recognition is biometric authentication and can be said to have higher authentication strength than other authentication methods, but as mentioned above, the probability of false positives is not zero. In payment processes where attempts at abuse are naturally expected, the security of payments can be further enhanced by incorporating facial recognition as part of multi-factor authentication.

[0060] <Other embodiments> (Service Selection 4) FIG. 11 is a sequence diagram illustrating another embodiment of the face authentication system of the present invention. In the above embodiment, the management server 10 is mainly provided with a pre-check program 16, and one face image P is distributed to multiple face authentication engines 20. However, the pre-check program 16 and the standardization of face images P are not essential elements of the face authentication system of the present invention, and they can be omitted. In this case, as shown in FIG. 11, for example, a face image is required each time a service is selected. This configuration can be adopted when the number of face authentication engines 20 used is small or when the specifications of the face images P required by these engines are significantly different. Alternatively, the mechanism of the above embodiment can be used for face authentication engines 20 with similar required specifications for the face image P, and the mechanism of FIG. 11 can be adopted for services requiring a unique face image P.

[0061] Although the embodiments of the present invention have been described above, the scope of the present invention is not limited to these, and various modifications can be made without departing from the spirit of the invention. [Explanation of symbols]

[0062] F: Facial recognition system, P: Facial image, T: Facial template, 10: Management server, 11: User database, 111: User information, 112: PIN code, 113: User ID, 12: Facial recognition engine database, 16: Pre-check program (pre-check means), 17: Distribution program, 18: Payment program, 20: Facial recognition engine, 20': Pre-check engine, 21: Camera (photographing device), 22: Screen, 25: Face database, 26: Authentication program (authentication means), 30: User application (user terminal), 31: Camera, 32: Registration form, 32: Face registration program, 34: Add service button (service selection means), 341: Service A, 342: Input component, 343: Registration button, 344: Application button, 345: Sign-in form, 39: Service provider tool, 90: Facial recognition usage setting screen 90, 91: Sign-in form, 92: Collaboration start button

Claims

1. Multiple types of face recognition engines with different face template generation algorithms, a management server that receives a user's face image or scan data of its three-dimensional shape (hereinafter referred to as "face image, etc.") and distributes the face image, etc. to each of the face recognition engines; a user terminal with which a user takes a photograph of the face image, etc., the management server or the user terminal has a pre-check means for provisionally evaluating data quality of the captured face image or the like separately from a quality check in each face recognition engine, the management server holds a user ID, which is unique identification information linked to the facial image, etc.; Among the individual services provided using each of the face recognition engines, there is an independent service which is a service having an information processing system using a unique user account, When face authentication is successful by the face recognition engine used in the independent service, the user ID or information capable of uniquely identifying the user account of the independent service is transmitted to the information processing system of the independent service. Facial recognition system.

2. the pre-check means performs at least face detection on the photographed face image, etc., and evaluates the data quality of the face image, etc.; The face authentication system according to claim 1 .

3. As the pre-check means, a pre-check engine is used, which is one of the face recognition engines less than the total number of the face recognition engines, The management server transfers the received face image, etc. to the pre-check engine, and receives information indicating whether generation of a face template was successful or not, or information indicating data quality of the face image, etc., from the pre-check engine. The face authentication system according to claim 1 .

4. The plurality of types of face recognition engines each include: A photographing device for photographing the face of an oncoming person; a face database in which face template data is stored; an authentication means for comparing a face photographed by the photographing device with the face template data, The plurality of types of face recognition engines include: a cloud-based system in which the photographing device and the face database are communicably connected via the Internet; an edge type in which the face database and the authentication means are arranged in the same housing as the image capture device or in the same local network as the image capture device; The face authentication system according to claim 1 .

5. The management server stores information about the user's payment method, Among the individual services provided using each of the facial recognition engines, in a service that involves payment by the payment means, in addition to face recognition by the facial recognition engine used for that service, authentication by another authentication means is or may be required. The face authentication system according to claim 1 .

6. the user terminal has a service registration means for allowing a user to register a service that the user wishes to use from among the individual services provided using each of the face recognition engines; When the management server receives the facial image etc. of the user, the management server distributes the facial image etc. to at least the facial recognition engine used for the service for which the user has registered. The face authentication system according to claim 1 .

7. The management server stores the face image etc. received from the user for at least a certain period of time, When a service is registered by the service registration means after receiving the facial image etc., the management server distributes the stored facial image etc. to the face recognition engine used for the registered service. The face authentication system according to claim 6 .

8. When a user cancels registration for any service using the service registration means, the face template of the user generated by the face recognition engine used for that service is deleted or invalidated. The face authentication system according to claim 6 .

9. the user terminal is owned by the user; When the user takes a photograph of the face image etc. again after distributing the face image etc. to each face recognition engine, the management server redistributes the new face image etc. to at least the face recognition engines used for the service to which the user is registered. The face authentication system according to claim 6 .

10. Further, a photographing device is provided for each service to photograph the face of an oncoming person, When any of the photographing devices fails to authenticate a user whose face has been photographed with the photographing device, the photographing device displays text or emits sound to prompt the user to confirm whether the service is registered. The face authentication system according to claim 6 .

11. the management server holds personal information, information on payment methods, information on tickets, or information set for each user by a provider of an individual service provided using each of the face recognition engines (hereinafter, such information will be collectively referred to as "user information"); The user information includes information that is referenced when providing a specific service but is not referenced when providing other services. The face authentication system according to claim 1 .

12. The user terminal transmits the facial images, etc., of the user's face taken from multiple directions to the management server; The facial images etc. include any images unnecessary for the facial recognition engine. The face authentication system according to claim 1 .

Images