Facial recognition system, facial recognition method, and facial recognition program

Abstract

To provide a face authentication system capable of stably and efficiently operating a face authentication function.SOLUTION: In face authentication, a face authentication unit 1 picks up an image of a target of face authentication with a camera 11, a face matching server 6 executes a series of collation processing between a piece of facial feature data of the target person generated from the facial image data of the target person and facial feature data of registrants stored in groups. In face authentication, especially in face image extraction processing, the face authentication unit 1 sets a cutout area in the image data of the target person and cuts out facial image data of the target person. A face area included in the image data of the target is detected, and a rectangular face frame surrounding the face area is set. When a size of the face frame is within a prescribed proper range, a cutout area centered around the face frame and expanded vertically and horizontally by an integral number of times is set. When the entire cutout area is not included in the image data of the target, a cutout image of the image data of the target person is superimposed on a background image of a same size as the cutout area to generate a piece of face image data of the target person.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a face recognition system, a face recognition method, and a face recognition program that execute face recognition processing based on image data obtained by photographing a person who is to be subjected to face recognition. [Background technology]

[0002] Conventionally, a known face matching technology comprises at least a feature extraction means, a registrant feature storage means, and a matching means in a first device, and a registrant face image storage means in a second device stored in a location with a higher security level than the first device; when the first device needs to use the registrant's face image, a data communication path is established between the first device and the second device, the registrant's face image is obtained from the second device via the data communication path, and the registrant's face image data is used in a work memory without being stored in a non-volatile recording medium, making it possible to protect the registrant's face image as much as possible even if the face matching system is stolen or attacked (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5353147 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, a face authentication device that has an imaging means and facial feature amount data of all registered persons and executes face matching processing is installed in a physical space, and the facial image data of registered persons is managed in a cyberspace such as an image server. Therefore, no consideration is given to a configuration in which an authentication server is provided, that is, a mechanism for a service provider to stably and efficiently operate a face authentication function by storing both the feature amount data and the facial image data on a cloud server or the like.

[0005] Therefore, a main object of the present invention is to provide a face recognition system, a face recognition method, and a face recognition program that can stably and efficiently operate a face recognition function for each face recognition service provider. [Means for solving the problem]

[0006] The face authentication system of the present invention is a face authentication system that registers face image data of a user and performs face authentication to determine whether the subject is the user or not based on face image data obtained by photographing the subject, and includes a plurality of face authentication machines having cameras that photograph the subject, and a face authentication server device connected to the plurality of face authentication machines via a network, and the face authentication server device includes a face image management unit that accumulates the face image data of the user during the registration, a plurality of face image matching units that perform a matching process between the face feature amount data of the subject generated from the face image data of the subject acquired by the face authentication machine and the face feature amount data of the user generated from the face image data of the user, and a facial image matching unit that performs a matching process to determine whether the subject is the user or not based on the face image matching unit. and an authentication result sending unit that sends an authentication result indicating the result to the face authentication device, and during the face authentication, in a face image extraction process in which a cut-out area is set in the image data of the subject and the face image data of the subject is cut out, the face authentication device detects a face area included in the image data of the subject and sets a rectangular face frame surrounding the face area, and if the size of the set face frame is within a predetermined appropriate range, sets the cut-out area expanded by an integer multiple in the vertical and horizontal directions centered on the face frame, and if the entire cut-out area is not included in the image data of the subject, generates the face image data of the subject by superimposing the cut-out image of the image data of the subject on a background image of the same size as the cut-out area.

[0007] Furthermore, a face authentication method of the present invention is a face authentication method that registers face image data of a user, and executes face authentication to determine whether or not the subject is the user by a server device based on face image data obtained by photographing a subject, wherein, during the registration, a face image management unit included in the server device stores face image data of the user, and during the face authentication, a plurality of face authentication machines connected to the server device via a network photograph the subject with a camera, and a plurality of face image matching units included in the server device execute a matching process between face feature amount data of the subject generated from the face image data of the subject acquired from the face authentication machine and face feature amount data of the user, thereby performing authentication indicating whether or not the subject is the user. The result is sent to the facial recognition device, and during the facial recognition, the facial recognition device detects the facial area included in the image data of the subject and sets a rectangular face frame surrounding the facial area in a facial image extraction process that sets a cut-out area in the image data of the subject and cuts out the facial image data of the subject.If the size of the set face frame is within a predetermined appropriate range, the cut-out area is set by expanding the face frame vertically and horizontally by an integer multiple, and if the image data of the subject does not include the entire cut-out area, the cut-out image of the image data of the subject is superimposed on a background image of the same size as the cut-out area to generate the facial image data of the subject.

[0008] A face recognition program according to the present invention causes a computer to execute the face recognition method. Effect of the Invention

[0009] According to the present invention, even if the image data of a subject is outside the set cutout area, appropriate face image data can be generated from the image data. This allows the face recognition function to be operated stably and efficiently. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an overall configuration diagram of a face authentication system according to an embodiment of the present invention. [Diagram 2] An overview of match groups [Diagram 3] An explanatory diagram showing an overview of the face matching inquiry performed by the face recognition device 1. [Figure 4] An explanatory diagram showing an overview of a face matching unit [Diagram 5] Block diagram showing the schematic configuration of face recognition device 1 [Figure 6] A block diagram showing the schematic configuration of the management terminal 2. [Figure 7] A block diagram showing the schematic configuration of the face management server 5. [Figure 8] A block diagram showing a schematic configuration of the face matching server 6. [Figure 9] An explanatory diagram showing an overview of the face image extraction process performed by the face recognition device 1. [Figure 10] An explanatory diagram showing an overview of the face extraction process performed by the face recognition device 1. [Figure 11] An explanatory diagram showing an overview of the face extraction process performed by the face recognition device 1. [Figure 12] FIG. 1 is an explanatory diagram showing an overview of a face image extraction process; [Figure 13] An explanatory diagram showing an overview of face recognition inquiry based on matching group information [Figure 14] An explanatory diagram showing the login screen displayed on the management terminal 2. [Figure 15] FIG. 13 is an explanatory diagram showing the screen displayed on the management terminal 2 when registering, referencing, updating, and deleting a comparison group. [Figure 16] FIG. 13 is an explanatory diagram showing the screen displayed on the management terminal 2 when registering, referencing, updating, and deleting a comparison group. [Figure 17] A sequence diagram showing the steps for registering, referencing, and updating a matching group [Figure 18] An explanatory diagram showing the overview of registration, reference, and update processes for match groups [Figure 19] Sequence diagram showing the procedure for the deletion process for a matching group [Figure 20] An overview of the deletion process for match groups [Figure 21]Sequence diagram showing the procedure for bulk registration processing for match groups [Figure 22] FIG. 13 is an explanatory diagram showing a screen displayed on the management terminal 2 when registering, referencing, updating, and deleting associations between the face authentication device 1 and a matching group. [Diagram 23] FIG. 13 is an explanatory diagram showing a screen displayed on the management terminal 2 when registering, referencing, updating, and deleting associations between the face authentication device 1 and a matching group. [Figure 24] A sequence diagram showing the procedure of a registration process for association between the face authentication device 1 and a matching group. [Diagram 25] FIG. 1 is an explanatory diagram showing an overview of a registration process for association between a face authentication device 1 and a matching group. [Figure 26] A sequence diagram showing the steps of a reference process for association between the face recognition device 1 and a matching group. [Figure 27] A sequence diagram showing the procedure for updating the association between the face recognition device 1 and the matching group. [Figure 28] A sequence diagram showing the procedure for deleting an association between the face authentication device 1 and a matching group. [Figure 29] FIG. 13 is an explanatory diagram showing a screen displayed on the management terminal 2 when registering, referencing, or deleting an association between the face matching server 6 and a matching group. [Diagram 30] FIG. 13 is an explanatory diagram showing a screen displayed on the management terminal 2 when registering, referencing, or deleting an association between the face matching server 6 and a matching group. [Diagram 31] A sequence diagram showing the procedure of a registration process regarding the association between the face matching server 6 and a matching group. [Diagram 32] A sequence diagram showing the procedure of a reference process regarding the association between the face matching server 6 and a matching group. [Diagram 33] A sequence diagram showing the procedure for deleting an association between the face matching server 6 and a matching group. [Diagram 34] FIG. 13 is an explanatory diagram showing an overview of a deletion process for an association between a face matching server 6 and a matching group. [Diagram 35]FIG. 13 is an explanatory diagram showing the screen displayed on the management terminal 2 when registering, referencing, updating, and deleting a user. [Diagram 36] FIG. 13 is an explanatory diagram showing the screen displayed on the management terminal 2 when registering, referencing, updating, and deleting a user. [Figure 37] A sequence diagram showing the steps of the user registration process [Figure 38] FIG. 1 is an explanatory diagram showing an overview of a registration process for a user. [Figure 39] A sequence diagram showing the steps for bulk registration processing for users [Diagram 40] A sequence diagram showing the steps of the user reference process [Diagram 41] Sequence diagram showing the procedure for updating users [Diagram 42] An explanatory diagram showing an overview of updating processing related to users [Diagram 43] Sequence diagram showing the procedure for deleting users [Diagram 44] An explanatory diagram showing an overview of a deletion process for a user [Diagram 45] Description showing an overview of face image extraction processing [Figure 46] FIG. 1 is an explanatory diagram showing an overview of an image quality check process. [Figure 47] FIG. 1 is an explanatory diagram showing an overview of a facial feature amount calculation process; [Figure 48] An explanatory diagram showing an overview of face feature registration processing (feature designation) [Figure 49] An explanatory diagram showing an overview of face feature registration processing (image designation) [Figure 50] FIG. 1 is an explanatory diagram showing an overview of a facial feature amount deletion process; [Figure 51] An explanatory diagram showing an overview of face registration [Figure 52] FIG. 1 is an explanatory diagram showing an overview of face information synchronization processing; [Figure 53] FIG. 13 is an explanatory diagram showing an overview of a process for copying face feature amount data; [Figure 54] An explanatory diagram showing an overview of face recognition processing [Figure 55] An explanatory diagram showing an overview of the authentication log reference process [Figure 56] FIG. 13 is an explanatory diagram showing an authentication log reference screen displayed on the management terminal 2. [Figure 57] An explanatory diagram showing an overview of the linkage of face recognition device 1 with external applications [Figure 58] An explanatory diagram showing an overview of the operation status monitoring function in this system [Figure 59] An explanatory diagram showing the backup screen and restore screen displayed on the management terminal 2. [Figure 60] FIG. 13 is an explanatory diagram showing an overview of a face image extraction process according to a modified example of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The first invention made to solve the above-mentioned problems is a face authentication system that registers face image data of a user and performs face authentication to determine whether the subject is the user or not based on face image data obtained by photographing the subject, the face authentication system comprising: a plurality of face authentication devices each having a camera that photographs the subject; and a face authentication server device connected to the plurality of face authentication devices via a network, the face authentication server device including a face image management unit that accumulates the face image data of the user during the registration; a plurality of face image matching units that execute a matching process between face feature amount data of the subject generated from the face image data of the subject acquired by the face authentication device and face feature amount data of the user generated from the face image data of the user; and an authentication result sending unit that sends an authentication result indicating whether or not the person in question is a user to the face authentication device. During the face authentication, the face authentication device detects a face area included in the image data of the subject and sets a rectangular face frame surrounding the face area in a face image extraction process that sets a cut-out area in the image data of the subject and cuts out the face image data of the subject. If the size of the set face frame is within a predetermined appropriate range, the cut-out area is set by expanding the face frame vertically and horizontally by an integer multiple, and if the image data of the subject does not include the entire cut-out area, the cut-out image of the image data of the subject is superimposed on a background image of the same size as the cut-out area to generate the face image data of the subject.

[0012] With this, even if the image data of the subject is outside the set cutout area, appropriate face image data can be generated from the image data, which allows the face recognition function to be operated stably and efficiently.

[0013] A second invention is a face authentication method for registering face image data of a user, and executing face authentication to determine whether the subject is the user or not by a server device based on face image data obtained by photographing a subject, wherein, during the registration, a face image management unit included in the server device stores the face image data of the user, and during the face authentication, a camera is used to photograph the subject in a plurality of face authentication machines connected to the server device via a network, and a plurality of face image matching units included in the server device execute a matching process between face feature amount data of the subject generated from the face image data of the subject obtained from the face authentication machine and face feature amount data of the user, and an authentication result indicating whether the subject is the user or not is obtained. to the face recognition device, and during the face recognition, the face recognition device detects a face area included in the image data of the subject and sets a rectangular face frame surrounding the face area in a face image extraction process for setting a cut-out area in the image data of the subject and cutting out the face image data of the subject, and if the size of the set face frame is within a predetermined appropriate range, sets the cut-out area expanded vertically and horizontally by an integer multiple with the face frame as the center, and if the entire cut-out area is not included in the image data of the subject, superimposes the cut-out image of the image data of the subject on a background image of the same size as the cut-out area to generate the face image data of the subject.

[0014] According to this, like the first invention, the face recognition function can be operated stably and efficiently.

[0015] A third invention is a face recognition program for causing a computer to execute the face recognition method.

[0016] According to this, like the first invention, the face recognition function can be operated stably and efficiently.

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] FIG. 1 is a diagram showing the overall configuration of a face authentication system according to this embodiment.

[0019] This face authentication system includes a face authentication device 1, a management terminal 2 (terminal device, face authentication management device), and a face authentication server 3 (cloud server). The face authentication server 3 includes a face management server 5 (face image management unit) and a face matching server 6 (face image matching unit).

[0020] The face authentication device 1, management terminal 2, face management server 5, and face matching server 6 are connected via a network such as the Internet. A face authentication system is constructed for each face authentication service provider (business operator). Multiple face authentication devices 1 are installed at each location where face authentication is required, such as the entrances and exits of buildings and rooms. A required number of face matching servers 6 are provided according to the number of face authentication devices 1, etc.

[0021] The face authentication device 1 is equipped with a camera 11, which captures an image of a user. The face authentication device 1 also has a display 12, which displays the face authentication result acquired from the face matching server 6, and notifies the user of the face authentication result.

[0022] The management terminal 2 is operated by an administrator, and is configured as a PC, on which a management application is installed that manages the operations of the face authentication device 1, the face management server 5, and the face matching server 6. This management application allows the administrator to perform various management tasks. The management application is implemented as a Web application.

[0023] The face management server 5 centrally manages user information. Specifically, it accumulates and manages information such as the first and last names of registered users and face image data.

[0024] During face authentication, the face matching server 6 acquires face image data of the person to be subjected to face authentication from the face authentication device 1, generates face feature data of the person from the face image data, and performs face matching by comparing the face feature data of the person to the face feature data of a registered person (registered user) stored in the device, thereby performing face authentication to determine whether the person is a registered person or not.

[0025] Furthermore, prior to face authentication, when a user is registered, the face matching server 6 acquires face image data of the user from the management terminal 2, generates face feature amount data of the user from the face image data, and stores the data in the face matching server 6. It is also possible to acquire a photographed image of the user from the management terminal 2, and have the face matching server 6 acquire face image data from the photographed image.

[0026] In this embodiment, the face management server 5 (face image management unit) and the face matching server 6 (face image matching unit) are located in physically different information processing devices, but they may also be located in a single information processing device.

[0027] In addition, in this embodiment, the management terminal 2 and the face management server 5 are provided, but the management terminal 2 and the face management server 5 can be configured as a single information processing device. For example, by installing a management application in the face management server 5, the face management server 5 can also function as the management terminal 2.

[0028] In addition, in the present embodiment, facial feature matching is performed, but the facial matching is not limited to facial feature matching, and a matching method to which machine learning etc. is applied may be adopted. Furthermore, the present embodiment can be applied to biometric authentication other than facial authentication.

[0029] Next, the comparison group will be described with reference to Fig. 2, which is an explanatory diagram showing an overview of the comparison group.

[0030] In this embodiment, users are grouped according to the installation location of the face authentication device 1 where the users undergo face authentication, and a matching group is set for each user. Similarly to the users, the face authentication devices 1 are grouped according to the installation location of the face authentication device 1, and a matching group is set for each face authentication device 1. Furthermore, the face matching servers 6 corresponding to the face authentication devices 1, i.e., the face matching servers 6 that accept inquiries (requests) for face matching from the face authentication devices 1, are grouped in the same way as the face authentication devices 1, and a matching group is set for each face matching server 6.

[0031] For example, in a face authentication device 1 installed at the entrance of a multi-tenant building in which multiple companies are located, users belonging to multiple companies undergo face authentication. On the other hand, information on users belonging to different organizations cannot be managed in the same database. In such a case, therefore, a matching group is formed for each of the multiple companies, and the registered information of users is stored in matching group units, and the face authentication device 1 makes a face matching inquiry to the face matching server 6 of the matching group for each company.

[0032] A user can only be authenticated by a face authentication device 1 that corresponds to the matching group to which the user belongs. Therefore, even if the user is already registered, he or she cannot be authenticated by a face authentication device 1 that does not belong to the user's matching group, for example, a face authentication device 1 in a building that the user is not permitted to enter. For example, if a user who works at one business office visits another business office, he or she cannot be authenticated by the face recognition device 1 at that business office.

[0033] Next, a description will be given of a face matching inquiry from the face authentication device 1 to the face matching server 6. FIG.

[0034] The face authentication device 1 makes an inquiry about face matching to a face matching server 6 that belongs to the same matching group as the device itself, and sends a request for face matching to the face matching server 6 that is the inquiry destination (request destination). The face authentication device 1 also holds the network address (IP address) of the face matching server 6 as matching group information, and makes an inquiry about face matching to the face matching server 6 that corresponds to the device itself based on this network address.

[0035] The face authentication inquiry method includes a simultaneous inquiry shown in FIG. 3(A), a random inquiry shown in FIG. 3(B), and a sequential inquiry (switching inquiry destination) shown in FIG. 3(C).

[0036] As shown in Fig. 3(A), in the case of a simultaneous inquiry, a facial matching inquiry is made simultaneously to the face matching servers 6 corresponding to each of the multiple matching groups. For example, when the face authentication device 1 is installed at the entrance of a multi-tenant building and users of multiple matching groups undergo face authentication, a facial matching request is sent simultaneously to the face matching servers 6 of each group.

[0037] In order to reduce the load on one face matching server 6, it is also possible to split and store the facial feature data of users belonging to one matching group among multiple face matching servers 6. In this case, too, a face matching inquiry is made simultaneously to all face matching servers 6 in the same matching group as the face authentication device 1 and that have different registered contents.

[0038] As shown in FIG. 3(B), in the case of random inquiry, a face matching server 6 to be queried for face matching is randomly selected from among multiple face matching servers 6 in the same matching group as the face authentication device 1. The multiple face matching servers 6 have the same registered contents. In other words, the same user is the target of matching, and the facial feature amount data of the same user is stored. In this inquiry method, inquiries from the face authentication device 1 are distributed to each face matching server 6, so the load on each face matching server 6 can be reduced and the load can be distributed.

[0039] As shown in FIG. 3(C), in the case of sequential inquiry, the inquiry destination for face matching is selected sequentially from among multiple face matching servers 6 in the same matching group as the face authentication device 1. Specifically, an order (priority) is assigned to the face matching servers 6, and the face matching servers 6 are selected according to that order, and if there is no response from the face matching servers 6 with higher ranks, the transmission destination is switched to the face matching server 6. That is, the face matching server 6 with the first rank in the matching group is selected, and a face authentication request is made to that face matching server 6. Then, if there is no response from the face matching server 6, another face matching server 6 with the next rank in the same matching group is selected, and a face authentication request is made to that face matching server 6. This type of control allows for redundancy (backup).

[0040] Next, the face matching unit will be described with reference to Fig. 4, which is an explanatory diagram showing an overview of the face matching unit.

[0041] The face matching server 6 can be provided with a plurality of face matching units for performing face matching. The face matching units have a face matching process for executing a face matching program, and a face matching database in which information (facial feature amount data) of users to be compared for face matching is registered.

[0042] This face matching unit is provided for each matching group, and performs face matching in response to a request for face matching from a face authentication device 1 that belongs to the matching group. Therefore, one face matching server 6 can handle a plurality of matching groups.

[0043] Here, the face recognition device 1 holds the network address (IP address) of the face matching server 6 as matching group information, and can make a face matching inquiry to the face matching server 6 corresponding to the device itself based on this network address, but in this embodiment, it also holds a network address (IP address) for each face matching unit, and can make a face matching inquiry to the face matching unit corresponding to the device itself based on this network address.

[0044] In addition, since a face matching database is provided for each matching group, the face feature amount data of the user can be stored for each matching group. The face matching process executes a matching process between the face feature amount data stored for each group and the face feature amount data generated from the face image data acquired from the face authentication device 1 of the same group.

[0045] Here, there are various modes of the correspondence relationship between the face authentication device 1 and the face matching unit.

[0046] In the example shown in FIG. 4(A), the face matching server 6 is provided with one face matching unit of the matching group to which the face authentication device 1 belongs, and the face authentication device 1 and the face matching server 6 have a one-to-one correspondence.

[0047] In the example shown in FIG. 4(B), the face authentication device 1 and the face matching server 6 correspond one-to-one, but the face matching server 6 is provided with two face matching units for the matching group to which the face authentication device 1 belongs. In this case, in the case of a face authentication device 1 installed at the entrance of a multi-tenant building, information of users who belong to different organizations (e.g., user A, user B) can be managed in different matching databases.

[0048] In the example shown in Fig. 4(C), the face authentication device 1 and the face matching server 6 correspond in a one-to-two relationship, and the face authentication device 1 makes face matching inquiries (matching requests for user X) to the two face matching servers 6. In this case, by providing different face matching servers 6 with the same face matching unit, i.e., two or more face matching units that perform face matching using the facial feature amount data of the same user, it is possible to distribute the load on the face matching server 6 and handle vital matters.

[0049] Next, a description will be given of a schematic configuration of the face authentication device 1. Fig. 5 is a block diagram showing a schematic configuration of the face authentication device 1.

[0050] The face authentication device 1 includes a camera 11 (imaging unit), a display 12 (output unit), a communication unit 13, a storage unit 14, and a control unit 15.

[0051] The camera 11 constantly captures a predetermined shooting area, and when a person enters the shooting area, the person is photographed, and a photographed image of the person to be subjected to face recognition can be obtained. In order to save power, a human sensor (not shown) may be provided so that the camera 11 is activated when it detects the approach of a person.

[0052] The display 12 displays the face recognition result so that the person undergoing face recognition can check the face recognition result. Note that a speaker may be provided as an output unit that outputs the recognition result of the subject, so that the face recognition result is notified by voice.

[0053] The communication unit 13 (face image transmission unit, authentication result reception unit) communicates with the face matching server 6 via the network. In this embodiment, it transmits face image data to the face matching server 6. It also receives the authentication result of the user from the face matching server 6. The communication unit 13 also communicates with the face management server 5 via the network. In this embodiment, it receives matching group information and the like from the face management server 5.

[0054] The storage unit 14 stores collation group information, a control program executed by a processor constituting the control unit 15, and the like.

[0055] The control unit 15 includes a startup processing unit 21, a face image extraction unit 22, a face matching request unit 23, an authentication result notification unit 24, an authentication result adjustment unit 25, and an operation status monitoring unit 26. The control unit 15 is configured with a processor, and each unit of the control unit 15 is realized by causing the processor to execute a program stored in the storage unit 14.

[0056] When the face authentication device 1 is started up, the startup processing unit 21 acquires the matching group information from the face management server 5 and stores it in the storage unit 14. Note that the acquisition of the matching group information is not limited to this, and the matching group information may be distributed from the face management server 5 to the target face authentication device 1 along with an update of the matching group information.

[0057] The face image extraction unit 22 acquires a photographed image of a person to be subjected to face authentication from the camera 11 (camera image capture), detects the person's face from the photographed image (face detection), judges whether the size of the detected face is appropriate (face size check), cuts out the face area from the photographed image (face cutout), and acquires face image data of the person. Note that the face image data may be data of only the image of the face area, but may also be a combination of the photographed image data and position information of the face area on the photographed image data (face frame information).

[0058] The face matching request unit 23 transmits a face matching request to the face matching server 6 that is in the matching group matching the own device.

[0059] The authentication result notification unit 24 displays the face authentication result acquired from the face matching server 6 on the display 12, and notifies the user of the face authentication result.

[0060] Authentication result adjustment unit 25 performs control for linking the user's face matching result acquired from face matching server 6 with an external system, a management server 8 (external device) shown in Fig. 57 described later. This authentication result adjustment unit 25 is configured as a connection application, and notifies the user's face matching result to management server 8 of the external system.

[0061] The operation status monitoring unit 26 monitors the operation status of its own device, and notifies the face management server 5 of the operation status of its own device.

[0062] Next, a description will be given of the management terminal 2. FIG.

[0063] The management terminal 2 includes a communication unit 31, a display 32 (display unit), an input device 33 (operation unit), a storage unit 34, and a control unit 35.

[0064] The communication unit 31 communicates with the face management server 5 via the network. In this embodiment, the communication unit 31 receives screen information and the like from the face management server 5, and transmits to the face management server 5 operation information and the like of the administrator in response to the screen information and the like.

[0065] The display 32 displays various screens. The input device 33 is a mouse, a keyboard, etc., and is used to operate the screen displayed on the display 32.

[0066] The storage unit 34 stores a program (management application) executed by a processor constituting the control unit 35 and the like.

[0067] The control unit 35 includes a GUI control unit 41. The control unit 35 is configured with a processor, and each unit of the control unit 35 is realized by executing a program (management application) stored in the storage unit 34 by the processor.

[0068] The GUI control unit 41 displays various operation screens delivered from the face management server 5 on the display 32. In addition, in response to an input operation by an administrator using the input device 33, it acquires input information and controls the screen. In this embodiment, it performs display input control for a screen related to login, specifically, a login screen. It also performs display input control for a screen related to user management, specifically, a screen related to registration (individual registration, batch registration), reference, update, and deletion related to a user. It also performs display input control for a screen related to comparison group management, specifically, a screen related to registration (individual registration, batch registration), reference, update, and deletion related to a comparison group. It also performs display input control for a screen related to authentication device management, specifically, a screen related to registration, reference, update, and deletion related to the association between an authentication device and a comparison group. It also performs display input control for a screen related to referring to an authentication log.

[0069] Next, we will explain the schematic configuration of the face management server 5. Fig. 7 is a block diagram showing the schematic configuration of the face management server 5.

[0070] The face management server 5 includes a communication unit 51, a storage unit 52, and a control unit 53.

[0071] The communication unit 51 communicates with the management terminal 2 via the network. The communication unit 51 also communicates with the face authentication device 1 via the network. The communication unit 51 also communicates with the face matching server 6.

[0072] The memory unit 52 stores a face information database, a database regarding association information between the face authentication device 1 and a matching group, a database regarding association information between the face matching server 6 and a matching group, an administrator access log, a control program executed by the processor constituting the control unit 53, and the like.

[0073] In the face information database, information about a registered user, such as the user's first and last name, face image, matching group, etc. Note that the face image of the user may be stored in an encrypted state to protect privacy.

[0074] The control unit 53 includes an administrator access management unit 61, a user management unit 62, a matching group management unit 63, a face authentication device management unit 64, a face matching server management unit 65, a face information management unit 66, a database management unit 67, an operation status monitoring unit 68, and an authentication log presentation unit 69. The control unit 53 is configured with a processor, and each unit of the control unit 53 is realized by the processor executing a program stored in the storage unit 52. Each unit of the control unit 53 is configured as a WebAPI (Web Application Programming Interface).

[0075] The administrator access management unit 61 permits or denies access (login) of an administrator depending on the access permission group of the administrator who accesses the management terminal 2 from the own device.

[0076] Furthermore, the administrator access management unit 61 manages access to its own device from the management terminal 2, and when the administrator operates the management terminal 2 to access its own device, the administrator access management unit 61 accumulates the information at that time as an administrator access log (history information) in the storage unit 52. Furthermore, in response to a reference request from the management terminal 2, the administrator access log is presented to the management terminal 2.

[0077] The user management unit 62 manages information related to users, and performs necessary processing related to users in response to requests from the management terminal 2. In this embodiment, requests for registration, reference, update, and deletion related to users are made from the management terminal 2, and the necessary processing is performed in response to the requests.

[0078] The matching group management unit 63 manages information related to matching groups, and performs necessary processing related to matching groups in response to requests from the management terminal 2. In this embodiment, requests for registration, reference, update, and deletion related to matching groups are made from the management terminal 2, and necessary processing is performed in response to the requests. In addition, matching group information, i.e., information required for the face authentication device 1 to request face authentication from the face matching server 6 that matches its own matching group, is generated for each face authentication device 1, and the matching group information is provided to the face authentication device 1.

[0079] The face authentication device management unit 64 manages information related to the face authentication device 1, and performs necessary processing related to the face authentication device 1 in response to a request from the management terminal 2. In this embodiment, a request is made from the management terminal 2 to register, refer to, update, and delete associations between the face authentication device 1 and a matching group, and the necessary processing is performed in response to the request.

[0080] The face matching server management unit 65 manages information relating to the face matching server 6, and performs necessary processing relating to the face matching server 6 in response to a request from the management terminal 2. In this embodiment, a request is made from the management terminal 2 for registration, reference, and deletion of associations between the face matching server 6 and matching groups, and the necessary processing is performed in response to the request.

[0081] The face information management unit 66 manages the face information of the user stored in the device itself and the face matching server 6. In this embodiment, the face information is synchronized so that the face information of the user stored in the device itself (such as the user's first and last name and face image) and the face information of the user stored in the face matching server 6 (the user's face feature amount data) are maintained in a consistent state. The face information management unit 66 also copies the face information (the user's face feature amount data).

[0082] The database management unit 67 manages the database provided in its own device, and backs up and restores the database.

[0083] The operation status monitoring unit 68 monitors the operation status of its own device, and also receives notifications of operation status from the face authentication device 1 and the face matching server 6. In response to operations by the administrator at the management terminal 2, the operation status of the face authentication device 1, its own device (face management server 5), and the face matching server 6 is displayed on the screen of the management terminal 2.

[0084] The authentication log presenting unit 69 acquires the authentication log from the face matching server 6 in response to a reference request from the management terminal 2, and presents the authentication log to the management terminal 2.

[0085] Next, a description will be given of a schematic configuration of face matching server 6. Fig. 8 is a block diagram showing a schematic configuration of face matching server 6.

[0086] The face matching server 6 includes a communication unit 71, a storage unit 72, and a control unit 73.

[0087] The communication unit 71 (face image receiving unit, authentication result transmitting unit) communicates with the face authentication device 1 via the network. In this embodiment, it receives face image data and the like from the face authentication device 1. It also transmits the user's authentication results and the like to the face authentication device 1. The communication unit 71 also communicates with the face management server 5 via the network. In this embodiment, it receives various processing requests and the like from the face management server 5, and transmits corresponding responses and the like to the face management server 5.

[0088] The storage unit 72 stores a face matching database, association information between users and feature amounts, information about the matching group of the own device, an authentication log, a control program executed by a processor constituting the control unit 73, and the like.

[0089] In the face matching database, facial feature data of the registered users is registered as information about the registered users. A face matching database is provided for each matching group, and the facial feature data of the users is stored on a group basis. Here, the facial feature data of the registered users may be backed up and stored in a non-volatile memory such as an HDD or SSD provided in the face matching server 6 in case of unexpected loss from the face matching database.

[0090] The control unit 73 includes a matching group management unit 81, an image quality check unit 82, a facial image extraction unit 83, a facial feature amount generation unit 84, a facial feature amount management unit 85, a facial feature amount matching unit 86, an authentication log management unit 87, a database management unit 88, and an operation status monitoring unit 89. The control unit 73 is configured with a processor, and each unit of the control unit 73 is realized by the processor executing a program stored in the storage unit 72. Each unit of the control unit 73 is configured as a WebAPI.

[0091] The match group management unit 81 manages the match group to which the device itself belongs, and performs registration and deletion processes related to the match group in response to a request from the face management server 5.

[0092] The image quality check unit 82 judges whether the image of the face area in the captured image satisfies a predetermined quality. Specifically, it detects whether a mask and sunglasses are worn from the target image, and calculates the face recognition suitability (evaluation value based on the face direction and expression).

[0093] The face image extraction unit 83 extracts a face image from the captured image of the user acquired from the management terminal 2 during user registration. At this time, it detects the face of the person from the captured image (face detection), determines whether the size of the detected face is appropriate (face size check), cuts out the face area from the captured image (face cutout), and acquires the face image of the person. Note that, during face authentication, a face image may be extracted from the captured image of the subject acquired from the face authentication device 1.

[0094] The facial feature amount generating unit 84 detects facial feature points from facial image data and generates facial feature amount data at the time of user registration and face authentication.

[0095] When a user is registered, the facial feature amount management unit 85 registers the facial feature amount data of the user generated by the facial feature amount generation unit 84 in a face matching database corresponding to the user's matching group. When updating or deleting, the facial feature amount management unit 85 deletes the facial feature amount data registered in the face matching database in response to a request from the face management server 5. When a program related to the facial feature amount generation or face matching algorithm is upgraded, the facial feature amount management unit 85 updates the facial feature amount data registered in the face matching database in response to a request from the face management server 5 so as to correspond to the new program.

[0096] During face authentication, the facial feature amount matching unit 86 compares the facial feature amount data of the subject generated from the facial image data acquired from the face authentication device 1 with the facial feature amount data of a registered person (registered user) stored in the device itself to determine whether the subject is a registered person or not. This facial feature amount matching unit 86 corresponds to a matching group associated with the device itself, and when multiple matching groups are associated with the device itself, the device has multiple facial feature amount matching units 86 for each matching group. This facial feature amount matching unit 86 (face matching process) constitutes a face matching unit in combination with a face matching database.

[0097] The facial feature amount matching unit 86 calculates the similarity (matching score) between the subject and the registrant. By comparing this similarity with a predetermined threshold, it is possible to determine whether the facial authentication has been successful. In addition to notifying the face authentication device 1 of the result of the success or failure as the matching result, the face authentication device 1 may also be notified of the person ID and similarity of the registrant with a high similarity as the matching result.

[0098] When the face authentication device 1 requests the authentication log management unit 87 to perform face authentication, the authentication log management unit 87 accumulates the information at that time, i.e., the device ID of the face authentication device 1 that requested the face authentication, the face matching result, and other information, as an authentication log, in the storage unit 72. In addition, in response to a request from the face management server 5, the authentication log is provided to the face management server 5.

[0099] The database management unit 88 manages the database provided in its own device, and backs up and restores the database.

[0100] The operation status monitoring unit 89 monitors the operation status of its own device, and notifies the face management server 5 of the operation status of its own device.

[0101] In this embodiment, the face matching server 6 has the functions of face image extraction, face feature amount generation, and face feature amount matching as its main functions, but these functions can be configured in separate information processing devices independent of each other. For example, the face image extraction function can be configured in a separate information processing device independent of other functions such as face feature amount generation and face feature amount matching.

[0102] Next, a description will be given of the processing carried out by the face image extraction unit 22 of the face authentication device 1 during face authentication. Fig. 9 is an explanatory diagram showing an overview of the processing carried out by the face image extraction unit 22 of the face authentication device 1.

[0103] The face authentication device 1 can be provided with multiple cameras 11. In this case, a camera image capture process that acquires captured images from the camera 11 is provided for each camera 11. This camera image capture process starts when the face authentication device 1 is started, and passes the captured images acquired from the camera 11 to the control process. The control process performs face detection, face size check, and face cutout processing on the captured images from each camera 11.

[0104] In addition, in the face authentication device 1, the control process reads the setting file and executes the camera image capture process based on the setting information described in the setting file for each camera 11. At this time, the camera image capture process receives each piece of information, such as the camera type, camera connection information, image acquisition rate, and acquired image size, as parameters from the control process during execution.

[0105] In face detection, a face area included in an image captured by the camera 11 is detected. Based on the detection result, a face frame (a rectangular frame surrounding the face area) is set.

[0106] Here, there are cases where the image captured by the camera 11 includes multiple face areas. In this case, a face area to be the subject of face recognition is selected from the multiple detected face areas (detected face selection process). At this time, only one face area to be the subject of face recognition may be selected, or multiple face areas to be the subject of face recognition may be selected. In this case, the number of face areas to be selected may be specified by the administrator.

[0107] In this detected face selection process, a face region to be a target of face recognition may be selected based on the area of ​​the face frame. That is, the area of ​​the face frame is calculated from the width and height of the face frame, and the areas of the face frames are compared to select a predetermined number of face regions in descending order of area. When only one face region is to be selected, the face region with the largest area may be selected. When the face frames have the same area, the face regions may be selected in the order in which they were detected.

[0108] Also, the face area to be the subject of face recognition may be selected based on the width of the face frame. That is, the width of each face frame is compared, and a predetermined number of face areas are selected in descending order of width. When only one face area is selected, the face area with the largest width may be selected. When the widths of the face frames are the same, the face areas may be selected in the order in which they were detected.

[0109] As pre-processing for face detection, for example, grayscale conversion is performed.

[0110] In the face size check process, it is determined whether the size of the face area (face frame) detected from the captured image meets a predetermined standard (appropriate range). Specifically, if the width and height of the face frame acquired by face detection are within a predetermined range, it is determined that the size of the face area is appropriate. Specifically, threshold values ​​(minimum and maximum values) are set for each of the width and height of the face frame, and the width and height of the face frame are compared with the respective minimum and maximum values, and if the width of the face frame is equal to or greater than the minimum value and equal to or less than the maximum value, and at the same time, the height of the face frame is equal to or greater than the minimum value and equal to or less than the maximum value, it is determined that the size of the face area is appropriate.

[0111] 10 and 11 are explanatory diagrams showing an overview of the face cut-out processing performed in the face authentication device 1. FIG.

[0112] In face cutout processing, a cutout area is set for the image captured by the camera based on the face area (face frame) acquired by face detection, and the image of the cutout area (face cutout frame) is cut out as a face image. The cutout area is set with the face area at the center and expanded vertically and horizontally.

[0113] In the example shown in Fig. 10(A), the expansion coefficient k is set to 3, and an area three times the size of the face area detected by face detection is set as the cut-out area. In the example shown in Fig. 10(B), the expansion coefficient k is set to 4, and an area four times the size of the face area is set as the cut-out area.

[0114] Here, as shown in FIG. 10(B), the coordinates (x', y') of the center B of the face frame can be calculated from the coordinates (x, y) of the upper left vertex A of the face frame and the width w and height h of the face frame using the following formula. (x',y')=((x+w) / 2,(y+h) / 2) Furthermore, the width W and height H of the cut-out region are calculated from the width w and height h of the face frame and the expansion coefficient k as follows: W = w × k H = h × k Then, from the coordinates (x', y') of the center B of the face frame and the width W and height H of the cut-out region, the coordinates (X, Y) of the upper left vertex C of the cut-out region are calculated as follows: (X,Y)=(x'-(W / 2),y'-(H / 2)) The cut-out region is defined by the coordinates (X, Y) of the upper left vertex C of this cut-out region, and the width W and height H of the cut-out region.

[0115] Furthermore, there is no problem if the entire cut-out area is included in the captured image, but as shown in Figure 11(A), there are cases where the entire cut-out area is not included in the captured image, i.e., part of the cut-out area extends beyond the captured image.

[0116] In this case, as shown in Fig. 11(B), a background image of the same size as the cut-out area is generated. This background image is used to fill in the blank areas that extend beyond the captured image, and is filled in with a predetermined color (e.g., black). Also, as shown in Fig. 11(C), an image of the cut-out area is cut out from the captured image to obtain a cut-out image. Then, as shown in Fig. 11(D), the cut-out image is superimposed on the background image to generate a facial image.

[0117] Next, the facial image extraction process will be described with reference to Fig. 12, which is an explanatory diagram showing an overview of the facial image extraction process.

[0118] As described above, the facial image extraction unit 22 of the facial authentication device 1 performs a facial image extraction process, i.e., a process of generating facial image data from the captured image data of the user by performing each of the processes of face detection, face size check, and face cut-out, and this facial image extraction process is also performed in the facial image extraction unit 83 of the face matching server 6.

[0119] When a user is registered, photographed image data of the user is sent from the management terminal 2 via the face management server 5 to the face matching server 6, whereupon the face image extraction unit 83 of the face matching server 6 performs face image extraction processing.

[0120] On the other hand, during face authentication, face image extraction processing is performed only in the face authentication device 1, and face image extraction processing is not performed in the face matching server 6. In this case, the face authentication device 1 is given a highly accurate face detection function. Also, the face authentication device 1 constantly captures images of the shooting area with the camera 11, and sends face image information (captured image data and face frame information) to the face matching server 6 when a face is detected. This distributes the load of face image extraction processing to multiple face authentication devices 1, and reduces the load on the face matching server 6. In addition, the amount of communication can be reduced, and therefore the load on the network can be reduced. And, since the face authentication response can be performed at high speed, face authentication of subjects who appear one after another can be performed efficiently.

[0121] That is, since the face authentication process of the user is shared between the face authentication device 1 and the face matching server 6, there is no need to provide a large number of expensive authentication devices that centralize all of the face authentication process of the user in the face authentication device 1 as in the past. Furthermore, even if the face feature amount data needs to be updated, no large-scale maintenance work of the authentication devices is required, and the work can be done in the face matching server 6. Therefore, according to this embodiment, a face authentication system with an inexpensive configuration and excellent operability can be constructed.

[0122] The management terminal 2 may be configured not to have the function of extracting a face image, that is, the management terminal 2 may not be provided with the face image extraction unit 22.

[0123] FIG. 13 is an explanatory diagram showing an outline of a face authentication inquiry based on matching group information.

[0124] The face management server 5 holds information about the matching group to which the user belongs, association information between the face authentication device 1 and the matching group, and association information between the face matching server 6 and the matching group. In addition, the matching group management unit 63 of the face management server 5 generates matching group information for each face authentication device 1 based on the association information between the face authentication device 1 and the matching group and the association information between the face matching server 6 and the matching group.

[0125] This matching group information is information required for the face authentication device 1 to request face authentication from the face authentication server 6 that matches its own matching group. This matching group information includes identification information (group number) of the matching group to which the face authentication device 1 belongs, and the face authentication server 6 to which the face authentication device 1 requests face authentication, i.e., destination information of the face authentication server 6 corresponding to the matching group with the face authentication device 1. Here, the destination information is specifically the network address (e.g. IP address) of the face authentication server 6, and this destination information specifies the face authentication server 6 to which the face authentication request is to be made, and the face authentication device 1 and the face authentication server 6 to which the face authentication request is to be made are associated with each other. Note that, if the face authentication device 1 belongs to multiple matching groups, the address (IP address) of each of the corresponding multiple face authentication servers 6 is included in the matching group information.

[0126] In the face authentication device 1, at the time of startup, the startup processing unit 21 acquires matching group information from the face management server 5 as operation setting information and stores it in the device itself. When the face authentication device 1 detects a person's face, it sends a face matching request to the face matching server 6 that matches the matching group to which the device itself belongs. This face matching request includes information on the matching group to which the device itself belongs.

[0127] In addition, in the face authentication device 1, the timing for obtaining setting information regarding matching group information, etc. from the face management server 5 may be at startup, or may be periodically at a specified timing or at specified intervals, or a configuration in which matching group information is distributed from the face management server 5 may be adopted.

[0128] The face matching server 6 holds association information between the face matching process of its own device and the matching group. When the face matching server 6 receives a face recognition request from the face authentication device 1, it identifies the face authentication process corresponding to the face authentication device 1 based on the association information between the matching group and the face authentication process and the matching group acquired from the face authentication device 1, and causes the face authentication process to perform face matching. As a result, the face matching process is performed in the face authentication process corresponding to the specified matching group.

[0129] In the present embodiment, the face matching server 6 provides a face matching database for each matching group, and stores face feature data for each matching group. Therefore, when matching faces, face matching may be performed using face feature data registered in the face matching database corresponding to the matching group of the face authentication device 1. On the other hand, the face matching server 6 may perform matching regardless of the matching group of the face authentication device 1, and then filter the matching results. That is, face matching may be performed using the face feature data of all users stored in the face matching server 6, and only the matching results with users who belong to the matching group of the face authentication device 1 may be extracted. In this case, it is not necessary to provide a separate face matching database for each matching group.

[0130] Next, administrator access management will be described.

[0131] In this system, access from the management terminal 2 to the face management server 5 and face matching server 6 is managed. The face management server 5 includes an administrator access management unit 61. This access management unit monitors the status of access (login) from the management terminal 2 to the face management server 5 and face matching server 6, and upon detecting access from the management terminal 2, records information about the access (such as the administrator who accessed and the date and time) as an administrator access log (history information). In addition, in response to a request from the management terminal 2 to view the administrator access log, the face management server 5 provides the administrator access log to the management terminal 2, allowing the administrator to view the administrator access log.

[0132] In this embodiment, access rights to user information are set for administrators to restrict access to user information (such as registration, reference, update, and deletion of user information). Administrators are divided into multiple administrator groups according to the differences in access rights given to the administrators. Here, administrators are divided into system operation administrators, service operation administrators, and general administrators.

[0133] In this embodiment, users are divided into a plurality of access permission groups according to the administrator group that can access the user's information. An access permission group is an administrator group that can access the data of the user. Here, users are divided into groups of system operation administrators, service operation administrators, and general administrators.

[0134] When the face management server 5 accepts the login of an administrator, it acquires the administrator group (administrator type) of the logged-in administrator, and determines whether or not to permit access (reference, update, and deletion) to the user's information based on the administrator group and the access permission group to which the target user belongs, and permits access to the user's information. In other words, the administrator can only perform reference, update, and deletion processes on the user's information that are permitted by the administrator group to which the administrator belongs.

[0135] The system operations administrator is a person who manages the operation (maintenance, etc.) of this system, and can access the information of all users, that is, users whose access permission groups are system users, administrative users, or general users. The service operations administrator is a person who manages the operation of the face recognition service by this system (user registration / deletion, etc.), and can access the information of users whose access permission groups are administrative users or general users. The general administrator is a person who has been granted authority by the service operations administrator, and can only access the information of users whose access permission groups are general users.

[0136] FIG. 14 is an explanatory diagram showing a login screen displayed on the management terminal 2. As shown in FIG.

[0137] When the management terminal 2 starts up the management application and accesses the face management server 5, a login screen is displayed. On this login screen, the administrator can enter his or her own user ID and password. When the administrator enters a user ID and password on this login screen and operates the login button, user authentication is performed on the face management server 5. If login is successful, the top screen is displayed. If login is unsuccessful, an error is displayed.

[0138] Next, registration, reference, update and deletion regarding a comparison group will be described. Figures 15 and 16 are explanatory diagrams showing screens displayed on the management terminal 2 when registration, reference, update and deletion regarding a comparison group are performed.

[0139] On the management terminal 2, when registering, referencing, updating, or deleting a comparison group, the reference screen shown in Figure 15(A), the registration screen shown in Figure 15(B), the update screen shown in Figure 15(C), and the deletion screen shown in Figure 16 are displayed.

[0140] 15(A) displays a list of registered comparison groups. By viewing this reference screen, the administrator can check the registered details of the comparison groups.

[0141] This reference screen is provided with a list display section 101 and an update button 102. In the list display section 101, the comparison group ID (number) and a description (installation location) of the comparison group are displayed side by side for each registered comparison group. This allows the administrator to check the registered comparison groups. In this reference screen, operating the update button 102 updates the list display section 101 to the latest registered content. In addition, selecting a comparison group, specifically operating the comparison group ID, transitions to an update screen (see FIG. 15(C)).

[0142] Also, on this reference screen, sorting can be performed by specifying items (matching group, description). Note that it may be possible to specify the number of items to be displayed on the list display unit 101 (the number of face matching servers 6) and the display range of the face matching servers 6 and matching groups. It may also be possible to specify search conditions for each item (device ID, face matching server 6, description) for searching.

[0143] The registration screen (first screen) shown in FIG. 15(B) allows the administrator to register a comparison group.

[0144] This registration screen is provided with an explanation input section 103 and a registration button 104. An explanation regarding the matching group can be input in the explanation input section 103. When the registration button 104 is operated after inputting the explanation, a process of registering the matching group information in the database based on the input contents is performed on the face management server 5.

[0145] The update screen shown in FIG. 15(C) allows the administrator to update the registered contents of the comparison group.

[0146] This update screen is provided with an explanation input section 105 and an update button 106. An explanation regarding the matching group can be input in the explanation input section 105. When the update button 106 is operated after inputting the explanation, a process of updating the matching group information registered in the database with the input contents is performed in the face management server 5.

[0147] The delete screen shown in FIG. 16 allows an administrator to delete a match group.

[0148] This deletion screen is provided with a list display section 107 and a delete button 108. In the list display section 107, a list of registered match groups (numbers) and explanations related to the match groups are displayed. In addition, in this list display section 107, a check box is provided for each match group. A match group can be selected using the check box. When the match group to be deleted is selected using the check box and the delete button 108 is operated, a process of deleting the selected match group is performed on the face management server 5.

[0149] Here, by selecting "reference" on the menu screen (not shown), the screen transitions to the reference screen shown in Fig. 15(A). By selecting "registration" on the menu screen, the screen transitions to the registration screen shown in Fig. 15(B). By selecting "update" on the menu screen, the screen transitions to the update screen shown in Fig. 15(C). By selecting "delete" on the menu screen, the screen transitions to the delete screen shown in Fig. 16.

[0150] Next, the processes of registration, reference, update and deletion related to a comparison group will be described. Fig. 17 is a sequence diagram showing the procedure of the processes of registration, reference and update related to a comparison group. Fig. 18 is an explanatory diagram showing an overview of the processes of registration, reference and update related to a comparison group.

[0151] First, the registration process for the collation group in this system will be described.

[0152] As shown in Fig. 17(A), the administrator performs an operation to instruct registration of a comparison group on the management terminal 2. In this embodiment, the administrator performs an operation to register a comparison group on a registration screen for a comparison group (see Fig. 15(B)).

[0153] 18, in the face management server 5, when an operation to instruct registration of a match group is performed in the management terminal 2, a registration process for the match group is performed in the match group management unit 63. In this registration process, the match group information input by the administrator is registered in the match group database.

[0154] Then, when the registration process for the matching group is completed in the face management server 5, a processing result indicating whether the registration process was completed normally or not is displayed on the screen of the management terminal 2. Also, in the face management server 5, the administrator access management unit 61 records an administrator access log.

[0155] Next, the reference process for the collation group will be described.

[0156] 17(B), the administrator performs an operation to instruct a reference to a comparison group on the management terminal 2. In this embodiment, the administrator performs an operation to open a reference screen for comparison groups (see FIG. 15(A)) on a menu screen or the like.

[0157] In the face management server 5, when an operation is performed in the management terminal 2 to instruct the face management server 5 to refer to the comparison group, the face management server 5 performs a reference process related to the comparison group. In this reference process, a database related to the comparison group is referenced, list information of the comparison group for all the comparison groups is generated, and a reference screen for all the comparison groups is displayed on the management terminal 2 based on the list information. In addition, when the administrator selects an item on the reference screen (see FIG. 15(A)) displayed on the management terminal 2 and instructs sorting, list information in the sorted state is generated, and a reference screen reflecting the sorting is displayed on the management terminal 2.

[0158] Next, the update process for the collation group will be described.

[0159] 17(C), the administrator performs an operation to instruct an update of the comparison group on the management terminal 2. In this embodiment, the administrator inputs the update contents on the update screen for the comparison group (see FIG. 15(C)) and operates the update button 106.

[0160] When an operation to instruct an update of the comparison group is performed on the management terminal 2, the face management server 5 performs an update process for the comparison group. In this update process, the database for the comparison group is updated with the contents specified by the administrator. Then, when the update process ends, the face management server 5 displays on the screen of the management terminal 2 a result indicating whether the update process ended normally or not.

[0161] Next, the deletion process for a comparison group will be described. Fig. 19 is a sequence diagram showing the procedure of the deletion process for a comparison group. Fig. 20 is an explanatory diagram showing an overview of the deletion process for a comparison group.

[0162] 19, the administrator performs an operation to instruct the deletion of a comparison group on the management terminal 2. In this embodiment, on the deletion screen for comparison groups (see FIG. 16), the administrator selects the comparison group to be deleted and operates the delete button 108.

[0163] As shown in FIG. 20, in the face management server 5, when an operation is performed in the management terminal 2 to instruct the deletion of the matching group, a deletion process for the matching group is performed. In this deletion process, the registration information of the matching group designated by the administrator is deleted from the database for the matching group. Also, the association information between the target matching group and the face matching server 6 is deleted. At this time, the association information between the target matching group and the face matching server 6 is deleted, and then the registration information of the matching group is deleted. Also, information about the target matching group is deleted from the database of associations between the matching group and the user. Also, the face information (facial feature amount data) of the user belonging to the target matching group is deleted from the face information database. Also, the association information between the target matching group and the face authentication device 1 is deleted. Also, in the face management server 5, the administrator access management unit 61 records the administrator access log.

[0164] Furthermore, the face management server 5 sends a request for deleting a matching group to the face matching server 6. This request for deleting a matching group includes information on the matching group to be deleted.

[0165] When the face matching server 6 receives a request to delete a matching group from the face management server 5, it performs a deletion process for the matching group. In this deletion process, it deletes the association information between the specified matching group and the face matching unit (face matching process). At this time, it stops the face matching unit for the matching group to be deleted, and deletes the face matching database corresponding to the matching group.

[0166] Then, when the deletion process is completed, the face matching server 6 sends a matching group deletion response to the face management server 5. This response includes a processing result indicating whether or not the deletion process has been completed normally.

[0167] When the face management server 5 receives the response to the matching group deletion from the face matching server 6 and completes its own deletion process, it displays on the screen of the management terminal 2 a processing result indicating whether the deletion process has been completed normally or not.

[0168] In addition, by restarting the face authentication device 1, this process is reflected and the matching group information is deleted.

[0169] Next, the batch registration process for the comparison group will be described with reference to Fig. 21. The batch registration process for the comparison group is shown in Fig. 21 as a sequence diagram.

[0170] In this system, matching groups can be registered in bulk in the face management server 5 by using a file of list information that stores information about the matching groups to be targeted.

[0171] Specifically, first, an operation to instruct batch registration of comparison groups is performed on the management terminal 2. In this embodiment, on a registration screen (not shown), the administrator selects a file that stores list information related to a target comparison group, and operates the registration button.

[0172] At this time, the selected list information file is read and the list information is displayed on the confirmation screen of the management terminal 2. The administrator confirms the registration contents on the confirmation screen and then performs an operation to instruct batch registration of the matching groups. Also, on the confirmation screen, the registration contents can be modified as necessary.

[0173] In the face management server 5, when an operation to instruct batch registration of the comparison groups is performed in the management terminal 2, a batch registration process for the comparison groups is performed. In this batch registration process, the registration process for each target comparison group is repeated based on the list information of the comparison groups acquired from the management terminal 2.

[0174] When the face management server 5 completes the batch registration process, the registration result indicating whether the batch registration process was completed normally or not is displayed on the screen of the management terminal 2. At this time, the matching groups that were unsuccessfully registered are reported to the administrator, and the administrator is prompted to re-register the matching groups. The registration result is output in a log file.

[0175] Although the bulk registration process for the comparison group has been described here, bulk update and bulk deletion processes for the comparison group are also possible.

[0176] Next, registration, reference, update, and deletion of the association between the face authentication device 1 and the matching group in this system will be described. Figures 22 and 23 are explanatory diagrams showing screens displayed on the management terminal 2 when registering, referencing, updating, and deleting the association between the face authentication device 1 and the matching group.

[0177] On the management terminal 2, when registering, referencing, updating, or deleting the association between the face recognition device 1 and the matching group, the reference screen shown in Figure 22(A), the registration screen shown in Figure 22(B), and the deletion screen shown in Figure 23 are displayed.

[0178] The reference screen shown in Fig. 22(A) displays a list of association settings between the registered face authentication device 1 and the matching group. By viewing this reference screen, the administrator can check the registered contents of the matching group.

[0179] This reference screen is provided with a list display section 111 and a face authentication machine designation section 112. In the list display section 111, the device ID of each authentication machine, the matching group (number) associated with each authentication machine, and an explanation of the matching group are displayed. In the face authentication machine designation section 112, the target face authentication machine 1 (device ID) can be selected using a pull-down menu. This allows the display content of the list display section 111 to be narrowed down to a specific face authentication machine 1, and only the matching group associated with the specific face authentication machine 1 is displayed in the list display section 111. Note that if a face authentication machine 1 is not designated in the face authentication machine designation section 112, information on all face authentication machines 1 is displayed in the list display section 111.

[0180] Also, on this reference screen, sorting can be performed by specifying items (device ID, face authentication device 1, description). Note that it may be possible to specify the number of items to be displayed in the list display section 111 (the number of face authentication devices 1) and the display range of the face matching server 6 and matching group. It may also be possible to specify search conditions related to each item (device ID, face authentication device 1, description) for searching.

[0181] In the example shown in FIG. 22(A), the face recognition devices 1 are grouped according to the locations where the face recognition devices 1 are installed (for example, the locations of business establishments).

[0182] The registration screen (second screen) shown in Fig. 22(B) is used to set up an association between the face authentication device 1 and a matching group. Information that associates the authentication device with a matching group can be registered.

[0183] This registration screen is provided with an authentication device designation section 113, a matching group designation section 114, and a registration button 115. In the authentication device designation section 113, a target face authentication device 1 (device ID) can be selected using a pull-down menu. In the matching group designation section 114, a matching group of the target face authentication device 1 can be selected using a pull-down menu. This pull-down menu displays the matching groups registered using the registration screen for matching groups (see FIG. 15(B)). When the face authentication device 1 and matching group are entered on this registration screen and then the registration button 115 is operated, the face management server 5 performs a process of registering the matching group information using the entered information.

[0184] In some cases, multiple matching groups may be associated with one face authentication device 1. In this case, it is sufficient to repeat the registration operation on the registration screen to associate one matching group with one face authentication device 1. Also, multiple matching group designation units 114 may be provided on the registration screen.

[0185] The deletion screen shown in FIG. 23 is for deleting the association setting between the face recognition device 1 and the matching group.

[0186] This deletion screen is provided with a list display section 116, a face authentication device designation section 117, and a delete button 118. In the list display section 116, the device ID of each authentication device, the matching group (number) associated with each face authentication device 1, and an explanation of the matching group are displayed. In addition, this list display section 116 is provided with a check box for each association setting. The association setting can be selected using the check box. When the registration item to be deleted is selected using the check box and the delete button 118 is operated, a process of deleting the selected registration item is performed in the face management server 5. In the face authentication device designation section 117, the face authentication device 1 can be selected using a pull-down menu. As a result, the display contents of the list display section 116 are updated to be narrowed down to a specific face authentication device 1.

[0187] Here, by selecting "reference" on the menu screen (not shown), the screen transitions to a reference screen shown in FIG. 22(A). By selecting "registration" on the menu screen, the screen transitions to a registration screen shown in FIG. 22(B). By selecting "delete" on the menu screen, the screen transitions to a deletion screen shown in FIG. 23. By selecting "update" on the menu screen, the screen transitions to an update screen (not shown). By selecting "associating" the face authentication device 1 with a matching group on the reference screen, the screen transitions to an update screen (not shown).

[0188] In the example shown in Fig. 23, the deleted screen displays a list of registered associations, but deletion may be performed on an individual edit screen. Also, the update screen (not shown) is similar to the registration screen (Fig. 22(B)).

[0189] Next, the registration process for the association between the face authentication device 1 and the matching group will be described. Fig. 24 is a sequence diagram showing the procedure of the registration process for the association between the face authentication device 1 and the matching group. Fig. 25 is an explanatory diagram showing an overview of the registration process for the association between the face authentication device 1 and the matching group.

[0190] The administrator performs an operation to open a registration screen (see the figure) on the management terminal 2. In this embodiment, the administrator performs an operation to select registration on a menu screen (not shown).

[0191] 25, when an operation to open a registration screen is performed on the management terminal 2, the face management server 5 references a database related to the registration devices (face authentication device 1 and face matching server 6) to generate list information for the face authentication device 1. It also references a database related to matching groups to generate list information for the matching groups. Then, based on the list information for the face authentication device 1 and the list information for the matching groups, it generates screen information for the registration screen and displays the registration screen on the management terminal 2.

[0192] Next, in the management terminal 2, the administrator performs an operation on the registration screen to instruct registration of the association between the face authentication device 1 and the matching group. In this embodiment, on the registration screen (see FIG. 22(B)), the administrator inputs a combination of the face authentication device 1 and the matching group and operates the registration button.

[0193] In the face management server 5, when an operation is performed in the management terminal 2 to instruct registration of an association between the face authentication device 1 and a matching group, a registration process is performed regarding the association between the face authentication device 1 and the matching group. In this registration process, association information is registered in a database for a combination of the face authentication device 1 and the matching group specified by the administrator. Note that the association between the face authentication device 1 and the matching group is reflected by restarting the face authentication device 1.

[0194] Then, when the registration process is completed, the face management server 5 displays on the screen of the management terminal 2 a registration result indicating whether the registration process has been completed normally or not.

[0195] Next, a description will be given of a reference process relating to the association between the face authentication device 1 and a matching group. Fig. 26 is a sequence diagram showing the procedure of the reference process relating to the association between the face authentication device 1 and a matching group.

[0196] In the management terminal 2, the administrator performs an operation to instruct to refer to the association between the face matching server 6 and the matching group. In this embodiment, the operation is performed to open a reference screen (see FIG. 22(A)) on a menu screen or the like.

[0197] In the face management server 5, when an operation is performed in the management terminal 2 to instruct a reference to the association between the face authentication device 1 and the matching group, a reference process for the matching group is performed. In this reference process, a database related to the association between the face authentication device 1 and the matching group is referenced, a list of association information for all face matching servers 6 is generated, and a reference screen is displayed on the management terminal 2. Also, when the administrator performs an operation to narrow down the target face authentication device 1 on the reference screen, a list of associations for the selected face authentication device 1 is generated, and a reference screen reflecting the narrowing down is displayed on the management terminal 2. Also, when the administrator performs an operation to select an item on the reference screen and instruct sorting, list information in a sorted state is generated, and a reference screen reflecting the sorting is displayed on the management terminal 2.

[0198] In addition, the face management server 5 records an administrator access log.

[0199] Next, a description will be given of the update process for the association between the face authentication device 1 and the matching group. Fig. 27 is a sequence diagram showing the procedure of the update process for the association between the face authentication device 1 and the matching group.

[0200] The administrator performs an operation to open an update screen (not shown) on the management terminal 2. In this embodiment, the administrator performs an operation to select update on a menu screen (not shown).

[0201] When an operation to open the update screen is performed on the management terminal 2, the face management server 5 references the database related to the registered devices to generate list information of the face authentication device 1. Also, references the database related to the matching group to generate list information of the matching group. Then, based on the list information of the face authentication device 1 and the list information of the matching group, screen information of the registration screen is generated and the update screen is displayed on the management terminal 2. On this update screen, the currently registered combination is displayed in the initial state based on the registration information related to the association between the face authentication device 1 and the matching group.

[0202] Next, in the management terminal 2, the administrator performs an operation on the update screen to instruct updating of the association between the face authentication device 1 and the matching group. In this embodiment, the administrator inputs the update contents on the update screen (not shown) and operates the update button. The update contents are to change the combination of the authentication device and the matching group.

[0203] When an operation is performed in the management terminal 2 to instruct an update of the association between the face authentication device 1 and the matching group, the face management server 5 performs an update process regarding the association between the face authentication device 1 and the matching group. In this update process, the database regarding the association between the face authentication device 1 and the matching group is updated with the content specified by the administrator. A process is performed to update the association information in the database with the combination of the authentication device and the matching group entered by the administrator. Then, when the update process is completed, the face management server 5 displays a result on the screen of the management terminal 2 indicating whether the update process was completed successfully or not.

[0204] Next, a description will be given of a deletion process relating to an association between the face authentication device 1 and a matching group. Fig. 28 is a sequence diagram showing the procedure of a deletion process relating to an association between the face authentication device 1 and a matching group.

[0205] In the management terminal 2, the administrator performs an operation to instruct deletion of the association between the authentication device and the comparison group. In this embodiment, on the deletion screen (see FIG. 23), the administrator selects the registration item (association) and operates the delete button.

[0206] In the face management server 5, when an operation is performed in the management terminal 2 to instruct deletion of the association between the face authentication device 1 and the matching group, a deletion process is performed regarding the association between the face authentication device 1 and the matching group. In this deletion process, the association information between the face authentication device 1 and the matching group specified by the administrator is deleted.

[0207] Then, when the deletion process is completed in the face management server 5, a result indicating whether the deletion process was completed normally or not is displayed on the screen of the management terminal 2. In addition, the face management server 5 records an administrator access log.

[0208] Next, registration, reference and deletion of associations between face matching server 6 and matching groups in this system will be described. Figures 29 and 30 are explanatory diagrams showing screens displayed on management terminal 2 when registration, reference and deletion of associations between face matching server 6 and matching groups.

[0209] On the management terminal 2, when registering, referencing, updating or deleting the association between the face matching server 6 and the matching group, a reference screen shown in Figure 29(A), a registration screen shown in Figure 29(B), and a deletion screen shown in Figure 30 are displayed.

[0210] 29(A) displays a list of association settings between registered face matching servers 6 and matching groups. By viewing this reference screen, the administrator can check the registered contents of the matching groups.

[0211] This reference screen is provided with a list display section 121 and a face matching server designation section 122. In the list display section 121, the device ID of each face matching server 6, the matching group (number) associated with each authentication device, and an explanation of the matching group are displayed. In the face matching server designation section 122, the target face matching server 6 (device ID) can be selected using a pull-down menu. This allows the display contents of the list display section 121 to be narrowed down to a specific face matching server 6, and only the matching groups associated with the specific face matching server 6 are displayed in the list display section 121. Note that the face matching server Designation part 122 If no face matching server 6 is specified in step 2, information on all face matching servers 6 is displayed in list display section 121.

[0212] Also, on this reference screen, sorting can be performed by specifying items (device ID, face matching server 6, description). Note that it may be possible to specify the number of items to be displayed in list display section 121 (the number of face matching servers 6) and the display range of face matching servers 6 and matching groups. It may also be possible to search by specifying search conditions related to each item (device ID, face matching server 6, description).

[0213] The registration screen (second screen) shown in FIG. 29(B) is used to specify the association between the matching group and the face matching server 6. In the registration screen (second screen) shown in FIG.

[0214] This registration screen is provided with a face matching server designation section 123, a matching group designation section 124, and a registration button 125. Designation part 122 In the comparison group selection section 122, a pull-down menu can be used to select the device ID of face matching server 6. In the comparison group designation section 124, a pull-down menu can be used to select a comparison group. The pull-down menu displays the comparison groups registered on the registration screen for comparison groups (see FIG. 15(B)).

[0215] When a face matching server 6 and a matching group are specified and a registration button 125 is operated, association information between the matching group and face matching server 6 is registered for the combination of the specified face matching server 6 and matching group.

[0216] Incidentally, there may be cases where multiple matching groups are assigned to one face matching server 6. In this case, it is sufficient to repeat the registration operation on the registration screen to associate one matching group with one face matching server 6. Also, multiple matching group designation units 124 may be provided on the registration screen.

[0217] The deletion screen shown in FIG. 30 is used to delete the association setting between the face matching server 6 and the matching group.

[0218] This deletion screen is provided with a list display section 126, a face matching server designation section 127, and a delete button 128. In the list display section 126, the device ID of each face matching server 6, the matching group (number) associated with each face matching server 6, and an explanation of the matching group are displayed. In addition, in this list display section 126, a check box is provided for each association setting. The association setting can be selected using the check box. When the registration item to be deleted is selected using the check box and the delete button 128 is operated, a process of deleting the selected registration item is performed in the face management server 5. In the face matching server designation section 127, the face matching server 6 can be selected using a pull-down menu. As a result, the display contents of the list display section 126 are updated in a state where it is narrowed down to a specific face matching server 6.

[0219] Here, the reference screen shown in Fig. 29(A) is displayed by selecting reference on a menu screen (not shown). The registration screen shown in Fig. 29(B) is displayed by selecting registration on a menu screen (not shown). The deletion screen shown in Fig. 30 is displayed by selecting deletion on a menu screen (not shown). Furthermore, when face matching server 6 is selected in list display section 121 of the reference screen shown in Fig. 29(A), the screen transitions to the registration screen shown in Fig. 29(B), where the association between face matching server 6 and a matching group can be changed.

[0220] In the example shown in FIG. 30, the deleted associations are displayed in a list on the deletion screen, but the associations may be deleted on an individual edit screen (not shown).

[0221] Next, a registration process for the association between face matching server 6 and a matching group will be described. Fig. 31 is a sequence diagram showing the procedure of the registration process for the association between face matching server 6 and a matching group.

[0222] The administrator performs an operation to open a registration screen (see FIG. 29(B)) on the management terminal 2. In this embodiment, the administrator performs an operation to select registration on a menu screen (not shown).

[0223] When an operation to open a registration screen is performed on the management terminal 2, the face management server 5 references a database related to registration devices to generate list information for the face matching server 6. Also, references a database related to matching groups to generate list information for matching groups. Then, based on the list information of the face matching server 6 and the list information of the matching groups, screen information for the registration screen is generated and the registration screen is displayed on the management terminal 2.

[0224] Next, in management terminal 2, the administrator performs an operation on the registration screen to instruct registration of the association between face matching server 6 and the matching group. In this embodiment, on the registration screen (see FIG. 29(B)), the administrator inputs a combination of face matching server 6 and matching group and operates registration button 125.

[0225] When an operation is performed in management terminal 2 to instruct registration of an association between face matching server 6 and a matching group, face management server 5 performs registration processing regarding the association between face matching server 6 and a matching group. In this registration processing, association information is registered in a database for a combination of face matching server 6 and a matching group specified by the administrator.

[0226] Furthermore, the face management server 5 sends a request for matching group registration to the face matching server 6 that is the target.

[0227] When the face matching server 6 receives a request for matching group registration from the face management server 5, it performs registration processing for the matching group. In this registration processing, the matching group is registered as setting information of the face matching server 6. Then, when the registration processing is completed, the face matching server 6 sends a registration response to the face management server 5.

[0228] When the registration process is completed in the face management server 5, a result indicating whether the registration process was completed normally or not is displayed on the screen of the management terminal 2. In addition, the face management server 5 records an administrator access log.

[0229] , Next, a reference process for the association between face matching server 6 and a matching group will be described. Fig. 32 is a sequence diagram showing the procedure of the reference process for the association between face matching server 6 and a matching group.

[0230] In the management terminal 2, the administrator performs an operation to instruct to refer to the association between the face matching server 6 and the matching group. In this embodiment, the operation is performed to open a reference screen (see FIG. 29(A)) on a menu screen or the like.

[0231] In the face management server 5, when an operation is performed in the management terminal 2 to instruct reference to the association between the face matching server 6 and the matching group, a reference process is performed regarding the association between the face matching server 6 and the matching group. In this reference process, a database regarding the association between the face matching server 6 and the matching group is referenced, a list of association information for all the face matching servers 6 is generated, and a reference screen is displayed on the management terminal 2. Also, when the administrator performs an operation to narrow down the target face matching servers 6 on the reference screen, a list of associations for the selected face matching server 6 is generated, and a reference screen reflecting the narrowing down is displayed on the management terminal 2. Also, when the administrator performs an operation to select an item on the reference screen and execute sorting, a sorted list is generated, and a reference screen reflecting the sorting is displayed on the management terminal 2.

[0232] In addition, the face management server 5 records an administrator access log.

[0233] Next, a deletion process for an association between the face matching server 6 and a matching group will be described. Fig. 33 is a sequence diagram showing the procedure of the deletion process for an association between the face matching server 6 and a matching group. Fig. 34 is an explanatory diagram showing an overview of the deletion process for an association between the face matching server 6 and a matching group.

[0234] In the management terminal 2, the administrator performs an operation to instruct deletion of the association between the face matching server 6 and the matching group. In this embodiment, on the deletion screen (see FIG. 30), the administrator selects the target association and operates the delete button.

[0235] 34, when an operation is performed in the management terminal 2 to instruct deletion of the association between the face matching server 6 and the matching group, the face management server 5 performs a deletion process regarding the association between the face matching server 6 and the matching group. In this deletion process, the association information between the target face matching server 6 and the matching group is deleted. Also, the face registration ID related to the target face matching server 6 is deleted from the face information database.

[0236] Furthermore, the face management server 5 sends a request for deleting the matching group to the face matching server 6 that is the target.

[0237] When the face matching server 6 receives a request to delete a matching group from the face management server 5, the face matching server 6 performs a deletion process for the matching group. In this deletion process, the setting information for the matching group of the face matching server 6 is deleted. Then, when the deletion process is completed, the face matching server 6 sends a response to the face management server 5.

[0238] When the deletion process is completed, the face management server 5 displays a result indicating whether the deletion process was completed normally or not on the screen of the management terminal 2. In addition, the face management server 5 records an administrator access log.

[0239] Next, registration, reference, update and deletion related to a user will be described. Figures 35 and 36 are explanatory diagrams showing screens displayed on the management terminal 2 when registering, referring, updating and deleting related to a user.

[0240] In the management terminal 2, when registering, referring to, updating, or deleting a user, an individual editing screen shown in FIG. 35(A), a search screen shown in FIG. 35(B), and a list display screen shown in FIG. 36 are displayed.

[0241] The individual edit screens (second screen, third screen) shown in FIG. 35(A) are used to perform operations for registering, referencing, updating, and deleting information about a user.

[0242] This registration screen has a name input section 131, a user code input section 132, a matching group input section 133, an access permission group input section 134, an activation date input section 135, an invalidation date input section 136, an additional information input section 137, a facial image input section 138, and an execute button 139.

[0243] In the name input section 131, the user's name can be input. In the user code input section 132, the user code and code number (such as an employee number) can be input. In the comparison group input section 133, the comparison group can be input. In the comparison group input section 133, the comparison group registered by the registration screen (see FIG. 15(B)) related to the comparison group is displayed. The comparison group can be selected by a check box. In the access permission group input section 134, the access permission group can be input. The access permission group can be selected by a pull-down menu. In the activation date input section 135, the activation date, that is, the date on which the face matching of the person can be enabled, can be input. In the invalidation date input section 136, the invalidation date, that is, the date on which the face matching can be disabled, can be input. By setting the activation date and invalidation date, it is possible to start or end services for multiple users at the same time. In the additional information input section 137, additional information can be input.

[0244] The face image input unit 138 is provided with a face image designation unit 141, a selection button 142, and a face image display unit 143. In this face image designation unit 141, a file of a photographed image of the user can be designated. When a file of a photographed image of the user stored in advance in the management terminal 2 is designated and the selection button 142 is operated, the photographed image of the user is sent to the face matching server 6 via the face management server 5, and a process of generating a face image from the photographed image of the user (face detection, face cut-out, etc.) is performed in the face matching server 6, and the face image of the user is displayed on the face image display unit 143.

[0245] When information for each item is entered on this registration screen and an execute button 139 is operated, a process of registering the user information is carried out based on the entered information.

[0246] In this way, on the registration screen, a matching group can be assigned to the user, and as a result, association information between the user and the matching group is registered in the database. Also, on this registration screen, an activation date and an invalidation date can be registered. The activation date and invalidation date specify the period during which the face information (facial feature amount data) of the user is stored in the face matching server 6, and on the activation date, the face image data of the user is registered in the database of the face matching server 6, and on the invalidation date, the face image data of the user is deleted from the database of the face matching server 6.

[0247] Furthermore, on this registration screen, by specifying a photographed image of the user, the face image data of the user is registered in the database of the face matching server 6.

[0248] The search screen shown in Fig. 35(B) is used to input search conditions related to registered user information. Users that meet the search conditions can be narrowed down and displayed on a list display screen. This search screen is provided with a name input section 131, a user code input section 132, a matching group input section 133, an access permission group input section 134, an activation date input section 135, an invalidation date input section 136, an update date and time input section 144, an acquired number input section 145, and a search button 146.

[0249] The name input section 131, user code input section 132, comparison group input section 133, access permission group input section 134, activation date input section 135, and invalidation date input section 136 are the same as those in the registration screen. In the update date and time input section 144, the update date and time (period) can be specified. In the acquisition number input section 145, the number of acquisitions can be specified.

[0250] On this search screen, when search conditions are input in the necessary items and a search button 146 is operated, the screen transitions to a list display screen (see FIG. 36) reflecting the search results.

[0251] The list display screen shown in FIG. 36 is used to display a list of registered users, and to perform bulk updating and bulk deletion.

[0252] By viewing this reference screen, the administrator can confirm the registered details of the user. This list display screen is provided with a list display section 147 and an execute button 148. The list display section 147 displays the user code, first name, collation group, etc. Furthermore, the list display section 147 is provided with a check box for each user. A user can be selected using the check box. When the user to be deleted is selected using the check box and the execute button 148 (delete button) is operated, a process of deleting the selected user is performed on the face management server 5.

[0253] In addition, on the list display screen, sorting (rearrangement) can be performed by specifying an item (user code, first name, matching group, etc.). Note that it may be possible to specify the number of items (number of users) to be displayed in the list display section 147, or the display range of user codes and matching groups. It may also be possible to search by specifying search conditions for each item.

[0254] Here, the individual edit screen (see FIG. 35(A)) is used to perform operations for registering, referencing, updating, and deleting users. Specifically, when registration is selected on the menu screen (not shown), a blank individual edit screen is displayed, where a new user can be registered by entering information about the user. Also, when a user is selected on the list display screen (see FIG. 36), the screen transitions to an individual edit screen, where information about a registered user can be individually confirmed (individual reference), and the registered contents of required items can be re-entered (individual update), and the registered contents of required items can be deleted (individual deletion).

[0255] Furthermore, when updating individually, if a captured image of the user is separately specified in the facial image specification section 141 of the individual editing screen (see FIG. 35(A)), a new facial image of the user will be generated from that captured image, and the facial image at the time of the user's registration can be replaced with the new facial image.

[0256] The list display screen (see FIG. 36) allows for collective viewing, collective updating, and collective deletion of registered users. Specifically, when list display is selected on the menu screen (not shown), the list display screen (see FIG. 36) is displayed, where information on registered users can be confirmed collectively (collective viewing). Furthermore, by selecting a check box, desired users can be deleted collectively (collective deletion). Also, check boxes may be provided separately to allow selection of information on a page-by-page basis in the list.

[0257] It is also possible to update information for multiple users collectively (bulk update). In this case, on the search screen (see FIG. 35(B)), conditions such as the target user or comparison group are specified, and search button 146 is operated, and then on the list display screen (see FIG. 36) displayed accordingly, execute button 148 (bulk update button) is operated. Note that in a bulk update, items other than user-specific information such as user name and user code are subject to update.

[0258] Furthermore, when search is selected on the menu screen (not shown), a search screen (see FIG. 35(B)) is displayed, and when search conditions are entered on this search screen and search button 146 is operated, the screen transitions to a list display screen (see FIG. 36) reflecting the search results.

[0259] Next, the user registration process will be described. Fig. 37 is a sequence diagram showing the procedure of the user registration process. Fig. 38 is an explanatory diagram showing an overview of the user registration process.

[0260] In the management terminal 2, the administrator performs an operation to instruct registration of a user. In this embodiment, on the individual edit screen in the registration mode (see FIG. 35(A)), the administrator inputs user information, designates a file of a photographed image of the user, and then operates the registration button.

[0261] As shown in Fig. 38, in the face management server 5, when an operation is performed in the management terminal 2 to instruct registration of a user, the face management server 5 starts registration processing for the user. In this registration processing, first, a request for facial image extraction is sent to a face matching server 6 whose matching group matches the target user. This request includes a photographed image of the user acquired from the management terminal 2. At this time, if there are multiple face matching servers 6 whose matching group matches the target user, one face matching server 6 is selected, and the request for facial image extraction is sent to that one face matching server 6.

[0262] When the face matching server 6 receives a request for face image extraction from the face management server 5, it performs face image extraction processing. In this face image extraction processing, it performs processing such as face detection and face cut-out on the photographed image of the user acquired from the face management server 5 to extract the face image of the user. Then, it sends a face image extraction response to the face management server 5. This response includes the face image of the user.

[0263] When the face management server 5 receives the face image extraction response from the face matching server 6, it sends a face registration request to the face matching server 6 whose matching group matches the target user. This request can include the face image of the user acquired from a specific face matching server 6. In this case, if there are multiple face matching servers 6 whose matching group matches the target user, the face registration request is sent to all face matching servers 6, including the face image of the user acquired from the specific face matching server 6.

[0264] When the face matching server 6 receives a face registration request from the face management server 5, it performs face registration processing. In this face registration processing, face features are generated from the face image of the user extracted earlier, and the face features are registered in a database. At this time, a face registration ID is assigned in association with the face features of the user. Then, a face registration response is sent to the face management server 5. This response includes a result indicating whether the face registration has been completed normally or not, and the face registration ID. Note that, upon completion of registration of the face features in the database, the extracted face image of the user is deleted. In addition, the face feature data may be backed up and stored in a non-volatile memory such as an HDD or SSD managed separately from the face matching DB of the face matching server 6.

[0265] The face management server 5 receives the face registration response from the face matching server 6, and if the process has been completed normally, performs registration processing for the user. In this registration processing, the user information acquired from the management terminal 2 is registered in the face information database. The face management server 5 also registers the user's face image acquired from the face matching server 6 in the face information database. The face management server 5 also registers the face registration ID issued by the face matching server 6 in the face information database as user information.

[0266] Next, the face management server 5 displays a result indicating whether the update regarding the user has been completed normally or not on the screen of the management terminal 2. In addition, the face management server 5 records a log in the administrator access log.

[0267] Next, the bulk registration process for users will be described with reference to Fig. 39, which is a sequence diagram showing the procedure of the bulk registration process for users.

[0268] In this embodiment, the user information can be registered collectively by using a file that stores list information about a plurality of target users.

[0269] Specifically, first, the administrator performs an operation to instruct batch registration of users on the management terminal 2. In this embodiment, on a registration screen (not shown), the administrator selects a file in which list information related to the target users is stored and operates a registration button.

[0270] At this time, the file of the selected list information is read, and the list information is displayed on the confirmation screen of the management terminal 2. The administrator confirms the registration contents on the confirmation screen and then performs an operation to instruct the bulk registration of users. Also, on the confirmation screen, the registration contents can be modified as necessary.

[0271] In the face management server 5, when an operation to instruct collective registration of users is performed in the management terminal 2, collective registration processing of users is started. In this collective registration processing, the registration processing is repeated for each of the target users. The registration processing for each user is the same as the registration processing for the user (individual registration processing) shown in FIG. 37.

[0272] Then, the face management server 5 receives the face registration responses for all target users from the face matching server 6, and when the batch registration process is completed, displays the registration result indicating whether the batch registration process was completed normally or not on the screen of the management terminal 2. At this time, the users who failed to register are reported to the administrator, and the administrator is prompted to re-register the users. The registration result is also output in a log file.

[0273] Next, the user-related reference process will be described with reference to Fig. 40, which is a sequence diagram showing the procedure of the user-related reference process.

[0274] In the management terminal 2, the administrator performs an operation to instruct display of a list related to users. In this embodiment, the administrator performs an operation to open a list display screen (see FIG. 36) related to comparison groups on a menu screen or the like.

[0275] In the face management server 5, when an operation is performed on the management terminal 2 to instruct displaying a list of users, the face management server 5 refers to the database to generate list information including all users, and displays a list display screen (see Figure 36) on the management terminal 2 based on the list information.

[0276] Furthermore, the administrator performs an operation to instruct a search related to a user on the management terminal 2. Specifically, the administrator displays a search screen (see FIG. 35(B)), inputs search conditions on the search screen, and operates the search button.

[0277] In the face management server 5, when an operation is performed on the management terminal 2 to instruct a search for users, the face management server 5 refers to the database, generates list information for users who meet the search criteria, and displays a list display screen (see Figure 36) as the search results on the management terminal 2 based on the list information.

[0278] Furthermore, the administrator performs an operation to instruct sorting regarding the list display of users on the management terminal 2. Specifically, the administrator performs an operation to instruct sorting by specifying an item on the list display screen.

[0279] In the face management server 5, when an operation to instruct sorting is performed in the management terminal 2, user list information sorted by the specified item is generated, and a list display screen reflecting the sorting is displayed on the management terminal 2 based on the user list information.

[0280] When the administrator selects a user on the list display screen (see FIG. 36) and instructs individual display, the face management server 5 refers to the database, acquires information about the selected user, and displays an individual editing screen (see FIG. 35(A)) on the management terminal 2 based on the information about that user.

[0281] Next, the update process related to a user will be described. Fig. 41 is a sequence diagram showing the procedure of the update process related to a user. Fig. 42 is an explanatory diagram showing an overview of the update process related to a user.

[0282] First, a case where a change is made to a photographed image of a user will be described. In this case, the administrator performs an operation to instruct an update regarding the user on the management terminal 2. In this embodiment, the administrator re-enters the user's information on an individual edit screen (not shown) in update mode displayed by specifying the user to be updated, and also replaces the file of the photographed image of the user, and then operates the update button.

[0283] As shown in Fig. 42, when an operation to instruct an update regarding a user is performed in the management terminal 2, the face management server 5 starts an update process regarding the user. In this update process, first, a request for facial image extraction is sent to a face matching server 6 whose matching group matches the target user. This request includes a photographed image of the user acquired from the management terminal 2. At this time, if there are multiple face matching servers 6 whose matching group matches the target user, one face matching server 6 is selected, and a request for facial image extraction is sent to that one face matching server 6.

[0284] When the face matching server 6 receives a request for face image extraction from the face management server 5, the face image extraction unit 83 performs face image extraction processing. In this face image extraction processing, face detection, face cut-out, and other processing are performed on the photographed image of the user acquired from the face management server 5 to generate a face image of the user. Then, a face image extraction response is sent to the face management server 5. This response includes the face image of the user.

[0285] When the face management server 5 receives the face image extraction response from the face matching server 6, it sends a face registration request to the face matching server 6 whose matching group matches the target user. This request can include the face image of the user acquired from a specific face matching server 6. In this case, if there are multiple face matching servers 6 whose matching group matches the target user, the face registration request is sent to all face matching servers 6, including the face image of the user acquired from the specific face matching server 6.

[0286] When the face matching server 6 receives a face registration request from the face management server 5, it performs face registration processing. In this face registration processing, face features are generated from the face image of the user extracted earlier, and the face features are registered in a database. At this time, a face registration ID is assigned in association with the face features of the user. Then, a face registration response is sent to the face management server 5. This response includes a result indicating whether or not the face registration has been completed normally, and the face registration ID.

[0287] When the face management server 5 receives the face registration response from the face matching server 6, if the face registration is successful, it sends a request for deleting facial features to the same face matching server 6 that sent the face registration request. However, if the face registration is unsuccessful, the request for deleting facial features is not sent, and face matching is performed with the old facial features unchanged.

[0288] When the face matching server 6 receives a facial feature deletion request from the face management server 5, it performs a facial feature deletion process. In this facial feature deletion process, old facial features of the subject registered in the database are deleted. Then, a facial feature deletion response is sent to the face management server 5. This response includes a result indicating whether the facial feature deletion has been completed normally.

[0289] The face management server 5 receives a response to the facial feature deletion from the face matching server 6, and if the process has been completed normally, performs an update process related to the user. In this update process, the registration contents of the face information database are updated with the user information acquired from the management terminal 2. Also, the face image of the user acquired from the face matching server 6 is registered in the face information database. Also, the registration contents of the face information database are updated with the new face registration ID issued by the face matching server 6. In other words, the old face registration ID is deleted, and the new face registration ID is registered in its place.

[0290] Next, the face management server 5 displays the processing result, which indicates whether the update regarding the user has been completed normally, on the screen of the management terminal 2. In addition, the face management server 5 records the administrator access log.

[0291] Next, a case where there is no change in the photographed image of the user, that is, where there is a change in items other than the photographed image, will be described. In this case, the administrator performs an operation to instruct an update regarding the user on the management terminal 2, and at this time, the administrator re-enters the user information on the individual edit screen (not shown) in the update mode and then operates the update button.

[0292] When an operation to instruct an update regarding a user is performed on the management terminal 2, the face management server 5 performs an update process regarding the user. In this update process, the registered contents of the face information database are updated with the user information entered by the administrator. Then, a result indicating whether the update regarding the user has been completed normally is displayed on the screen of the management terminal 2. In addition, the face management server 5 records an administrator access log.

[0293] Next, the deletion process for a user will be described. Fig. 43 is a sequence diagram showing the procedure of the deletion process for a user. Fig. 44 is an explanatory diagram showing an overview of the deletion process for a user.

[0294] In the management terminal 2, the administrator performs an operation to instruct deletion of a user. In this embodiment, the administrator operates the delete button on an individual edit screen (not shown) in the deletion mode that specifies and displays the user to be deleted. Also, on a list display screen (not shown) in the deletion mode, the administrator selects the target user and operates the bulk delete button.

[0295] 44, when an operation to instruct deletion of a user is performed on the management terminal 2, the face management server 5 starts a deletion process for the user. In this deletion process, first, a face deletion request is sent to a face matching server 6 whose matching group matches the target user. At this time, if there are multiple face matching servers 6 whose matching group matches the user, the face deletion request is sent to all face matching servers 6.

[0296] When the face matching server 6 receives a face deletion request from the face management server 5, it performs a deletion process on the face information of the target user. In this deletion process, the information of the corresponding user (facial feature amount data) is deleted from the database.

[0297] Furthermore, when the deletion process is completed, the face matching server 6 sends a face deletion response to the face management server 5. This response includes a result indicating whether or not the deletion process has been completed normally.

[0298] Furthermore, the face management server 5 receives a face deletion response from the face matching server 6, and if the process has been completed successfully, performs a deletion process for the user. In this deletion process, the information of the target user is deleted from the face information database. Then, when the deletion process is completed, the processing result indicating whether the deletion process has been completed successfully or not is displayed on the screen of the management terminal 2. Furthermore, the face management server 5 records an administrator access log.

[0299] Next, a description will be given of the face image extraction processing carried out by the face matching server 6. Fig. 45 is an explanatory diagram showing an overview of the face image extraction processing.

[0300] The face management server 5 sends a request for face image extraction to the face matching server 6. This request includes the captured image data of the user and face cut-out conditions (face image extraction conditions). The face cut-out conditions are parameters used in the face image extraction process (face detection, face size check, face cut-out). Specifically, the face cut-out conditions include the number of faces to be adopted when multiple faces are detected (number of detected faces), the face adoption method for determining whether the area or height of the face frame is used as the basis for detecting a face, and the minimum and maximum values ​​for the width and height of the face frame that is used as the basis for adopting a face (face size check threshold).

[0301] When the face matching server 6 receives a request for face image extraction from the face management server 5, the face image extraction unit 83 performs face image extraction processing. This face image extraction processing includes face detection, face size check, and face cut-out processing. Then, a face image extraction response is sent to the face management server 5. This response includes face image data (there may be multiple pieces of face image data) and position information of the face frame (the coordinates of the upper left vertex of the face frame, and the width and height of the face frame).

[0302] The process performed by the face image extraction unit 83 of the face matching server 6 is the same as that performed by the face image extraction unit 22 of the face authentication device 1. That is, first, a face area included in the captured image acquired from the face authentication device 1 is detected (face detection). Here, if the captured image includes multiple face areas, the face areas are selected based on the area and width of the face areas. Next, it is determined whether or not the size of the face area (face frame) detected from the captured image meets a predetermined standard (appropriate range) (face size check). Next, a cutout area is set for the captured image of the camera 11 based on the face area (face frame) acquired by face detection, and the image of the cutout area is cut out as a face image (face cutout), and the face image is acquired.

[0303] Next, a description will be given of the image quality check performed by face matching server 6. Fig. 46 is an explanatory diagram showing an overview of the image quality check process.

[0304] The image quality check unit 82 of the face matching server 6 judges whether the image of the face area (face frame) detected from the captured image data satisfies a predetermined quality. Specifically, it detects from the image whether a mask and sunglasses are worn. It also calculates the face authentication suitability from the image. The face authentication suitability is an evaluation value according to the face orientation and facial expression.

[0305] When checking the image quality, first, the face management server 5 sends a request for checking the image quality to the face matching server 6. This request includes the captured image data and face frame information.

[0306] When the face matching server 6 receives a request for checking image quality from the face management server 5, the image quality check unit 82 determines whether the image of the face area (face frame) in the captured image satisfies a predetermined quality and calculates a quality evaluation value. Then, the face matching server 6 sends a response to the image quality check to the face management server 5. This response includes the quality evaluation value.

[0307] Next, a facial feature amount generation process performed during face registration will be described. Fig. 47 is an explanatory diagram showing an overview of the facial feature amount generation process.

[0308] The facial feature amount generating unit 84 of the face matching server 6 generates facial feature amount data from the facial image data of the user previously extracted, and stores the facial feature amount data in the face matching DB.

[0309] When generating this facial feature amount, first, the face management server 5 or the face authentication device 1 sends a request for generating facial feature amount to the face matching server 6.

[0310] When the face matching server 6 receives a request for generating facial features from the face management server 5, the facial feature generating unit 84 generates facial feature data from the image of the face area previously extracted. Then, the face matching server 6 transmits a response to the facial feature generation to the face management server 5.

[0311] Next, a description will be given of a facial feature amount registration process performed during face registration. Fig. 48 is an explanatory diagram showing an overview of the facial feature amount registration process (feature amount designation). Fig. 49 is an explanatory diagram showing an overview of the facial feature amount registration process (image designation).

[0312] The face matching server 6 has a function of extracting a face image and a function of generating face features, and performs the processes of extracting a face image and generating face features in response to a request from the face management server 5. On the other hand, when there are multiple face matching servers 6 in the same matching group, it is necessary to register the face features of the user in duplicate in the face matching database of each face matching server 6.

[0313] At this time, one face matching server 6 performs the processes of facial image extraction and facial feature generation, and provides the facial feature acquired here to the remaining face matching servers 6, thereby enabling registration of the facial feature in all face matching servers 6. In this case, the face management server 5 receives the facial feature data generated by one face matching server 6 in a response to the facial feature generation, and issues an instruction to register the facial feature by adding the facial feature data to a request for facial feature registration to the remaining face matching servers 6 (feature designation).

[0314] Furthermore, one face matching server 6 performs face image extraction processing, and the acquired face image data is provided to the remaining face matching servers 6, which then perform face feature generation processing, thereby making it possible to register face feature data in all face matching servers 6. In this case, the face management server 5 receives face image data, etc. acquired from one face matching server 6 in a response to the face image extraction, and issues an instruction to register face feature data to the remaining face matching servers 6 by adding the face image data, etc. to a request to register face feature data (image designation).

[0315] 48, in the case of feature designation, first, the face management server 5 sends a facial feature registration request to the face matching server 6. This request includes the user's matching group, facial feature data, and user code.

[0316] Face matching server 6 receives a request for face feature registration. Face Management Server 5 When the request is received from the user, the facial feature amount management unit 85 registers the facial feature amount data of the user added to the request in the face matching database. Matching The face is registered in the face matching database of the unit. In addition, the face matching server 6 assigns a registrant code to the user who has completed face registration. The registrant code is registered in the face matching database in association with the facial feature amount.

[0317] Then, when the face feature registration process is completed, the face matching server 6 sends a response to the face management server 5. This response includes a processing result indicating whether the process was completed normally or not, and a face registration ID (registrant code).

[0318] 49, in the case of image designation, first, the face management server 5 sends a request for facial feature registration to the face matching server 6. This request includes the user's matching group, the user's photographed image data, face frame information, and the user code.

[0319] Face matching server 6 receives a request for face feature registration. Face Management Server 5 When the request is received from the server 6, the facial feature amount generating unit 84 generates facial feature amount data from the facial image data of the user (captured image data and face frame information) added to the request, and the facial feature amount managing unit 85 registers the facial feature amount data of the user in the face matching database. At this time, the data is registered in the face matching database of the face authentication process corresponding to the matching group to which the user belongs. Furthermore, the face matching server 6 assigns a registrant code to the user who has completed face registration. Furthermore, the registrant code is registered in the face matching database in association with the facial feature amount.

[0320] Then, the face matching server 6 transmits a response to the face feature amount registration to the face management server 5. This response includes a processing result indicating whether or not the processing has been completed normally, and a face registration ID.

[0321] Next, a facial feature amount deletion process performed at the time of updating and deleting will be described below. Fig. 50 is an explanatory diagram showing an overview of the facial feature amount deletion process.

[0322] When deleting a user, the facial feature data of the target user is deleted from the facial matching database of face matching server 6. When updating a user, old facial feature data of the target user is deleted from the facial matching database of face matching server 6. When deleting a facial feature, other information of the target user is also deleted, and information regarding the correspondence between the facial feature and the user is also deleted.

[0323] Specifically, first, the face management server 5 sends a request for deleting a facial feature to the face matching server 6. This request includes a matching group and a face registration ID.

[0324] When the face matching server 6 receives a deletion request from the face management server 5, the face feature amount management unit 85 identifies the user based on the face registration ID added to the request, and deletes the facial feature amount of the target user from the face matching database. At this time, if the target user belongs to multiple matching groups, the deletion process is performed for each matching group. In other words, the facial feature amount of the target user is deleted from the face matching database for each matching group.

[0325] Furthermore, when the deletion process is completed, the face matching server 6 sends a facial feature deletion response to the face management server 5. This deletion response includes a processing result indicating whether the deletion process has been completed normally or not.

[0326] Next, the face registration performed during the registration process for a user will be described below. Fig. 51 is an explanatory diagram showing an overview of the face registration.

[0327] In the face management server 5, when performing registration processing for a user, first, the face image extraction processing is performed by the face matching server 6, and the face image data acquired from the face matching server 6 is registered in the face information database, and then the face image data is provided to the face matching server 6, and the face registration processing is performed by the face matching server 6. In this face registration processing, a face feature amount is generated from the face image acquired from the face management server 5, and the face feature amount is registered in the face matching database.

[0328] Specifically, first, the face management server 5 sends a face registration request to the face matching server 6. This face registration request includes a user code, a matching group, a user registration ID, and face image data.

[0329] When the face matching server 6 receives a face registration request from the face management server 5, the image quality check unit 82 performs an image quality check on the user's face image added to the request. Next, the face feature amount generation unit 84 generates face feature amount data from the user's face image data, and the face feature amount management unit 85 registers the user's face feature amount data in a face matching database corresponding to the user's matching group. At this time, association information between the user registration ID used in the face information database of the face management server 5 and the registration ID used in the face matching database of the face matching server 6 at the time of face feature registration is saved. Also, if the user registration ID has already been registered, the original face feature amount is deleted and replaced. Also, a user code is registered in the face matching database as user information.

[0330] When the face matching server 6 has completed the registration process in the face matching database corresponding to all matching groups to which the user belongs, it sends a face registration response to the face management server 5. This response includes a processing result indicating whether the registration process has been completed normally or not.

[0331] Next, a description will be given of the face information synchronization process performed by the face management server 5. Fig. 52 is an explanatory diagram showing an overview of the face information synchronization process.

[0332] The face information management unit 66 of the face management server 5 executes a synchronization process with the face matching server 6. In this synchronization process, it is assumed that the registration contents of the face information database in the face management server 5 match the registration contents of the face matching database in the face matching server 6. Specifically, it is assumed that the user information (face image) registered in the face information database in the face management server 5 and the facial feature amount data of the user registered in the face matching database in the face matching server 6 match each other. It is also assumed that the registration contents related to the matching group in the face matching server 6 match between the face management server 5 and the face matching server 6.

[0333] Regarding the timing of the synchronization process, the synchronization process is executed periodically at a predetermined cycle. The synchronization process is also executed when user information is registered, updated, or deleted. The synchronization process is also executed when a matching group is deleted or when the face matching server 6 is started.

[0334] In addition, regarding users who are the targets of the synchronization process, synchronization is performed for all users, for specified users, and for users who have been updated during a specified period.

[0335] In addition, in this embodiment, an activation date and an invalidation date are registered for each user as user information. The activation date is the date on which facial matching for the target user can be enabled, and during the period in which the user is in an enabled state after this activation date, facial authentication is performed for the user and the user can receive facial authentication. The invalidation date is the date on which facial matching can be disabled, and during the period in which the user is in an disabled state after this invalidation date, facial authentication is not performed for the user and the user cannot receive facial authentication.

[0336] In addition, the face information management unit 66 of the face management server 5 determines whether to activate or deactivate all users as a synchronization timing determination. When the face matching of a user is activated and the activation period begins, the face information management unit 66 instructs the face matching server 6 to perform face registration processing, i.e., processing to register the face feature amount data of the target user. When the face matching of a user is deactivated and the invalidation period begins, the face information management unit 66 instructs the face matching server 6 to perform face deletion processing, i.e., processing to delete the face feature amount data of the target user. Therefore, the face information (facial feature amount data) of the user is not stored in the face matching server 6 during the period before the activation date or the period after the invalidation date.

[0337] In addition, the face management server 5 determines whether a predetermined process has been normally completed in the face matching server 6, and if it determines that the process has failed and needs to be re-executed, it instructs the face matching server 6 to re-execute the process. The processes to be executed here are face registration, face deletion, matching group registration, and matching group deletion.

[0338] The face management server 5 records a synchronization log (history information of the synchronization process).

[0339] Next, the copy processing of the facial feature amount data will be described with reference to Fig. 53, which is an explanatory diagram showing an overview of the copy processing of the facial feature amount data.

[0340] In this embodiment, a face matching database is provided for each matching group in the face matching server 6, and the face feature amount data of users is managed on a matching group basis, and multiple face matching databases for the same matching group are provided for load balancing and life and death correspondence. Therefore, the face feature amount data of the same user needs to be registered in multiple face matching databases. On the other hand, the process of generating face feature amount data from the face image data of a user is a heavy load, and it is not efficient to repeatedly perform this process on multiple face matching servers 6.

[0341] Therefore, in this embodiment, when facial feature amount data of a user has already been generated and registered in a face matching database, the facial feature amount data is copied between two face matching databases. In other words, the facial feature amount data registered in the original face matching database can be used to register the facial feature amount data in another face matching database.

[0342] The face information management unit 66 of the face management server 5 manages the registration status of the facial feature data for each user in the face matching database based on the face information database, and when any change occurs to the facial image data of a user, instructs copying of the facial feature data between the two face matching databases (transfer instruction).

[0343] At this time, if there is a change on an individual basis, the facial feature amount data on an individual basis is copied (individual copy) as shown in Fig. 53(A). For example, when a new user is registered or updated, the facial feature amount data on an individual basis is copied. In addition, in order to improve the accuracy of face matching for a specific user, facial feature amount data may be additionally generated using different facial image data, and multiple pieces of facial feature amount data may be registered for the same user. In this case as well, the facial feature amount data on an individual basis is copied only for the specific user.

[0344] Furthermore, when there is a change in the matching group unit, the facial feature amount data is copied in matching group units (group copy) as shown in Fig. 53(B). For example, when a new matching group is registered, the facial feature amount data of the face matching database in which the facial feature amount data was first registered is copied collectively to the second and subsequent face matching databases with the same registered contents. Also, when multiple face matching databases with the same registered contents are provided for load distribution or redundancy, the facial feature amount data of the user is copied in matching group units.

[0345] Next, the process of face authentication in this system will be described. Fig. 54 is an explanatory diagram showing an overview of the process of face authentication.

[0346] When the face authentication device 1 detects a person's face from an image captured by the camera 11 and acquires a face image, it transmits a face matching request to the face matching server 6. This face matching request includes the device ID of the requesting face authentication device 1, the matching group of the face authentication device 1, the captured image data of the person to be face authenticated, face frame information, and matching conditions.

[0347] In the face matching server 6, first, the face feature amount generating unit 84 generates the face feature amount of the subject from the face image of the subject acquired from the face authentication device 1. Next, the face feature amount corresponding to the matching group of the face authentication device 1 is generated. Matching The facial feature matching unit 86 of the unit matches the facial feature of the subject with the facial feature of the user registered in the facial recognition database. At this time, face matching is performed in a face recognition process in which the matching group matches the face recognition device 1 that made the request. When face matching is completed, a face matching response is sent to the authentication device that made the request. This face matching response includes the matching result (success or failure), matching score, and user code.

[0348] The facial feature amount matching unit 86 calculates a matching score that indicates the degree of similarity between the subject of facial authentication and the registered user, and if this matching score is equal to or greater than a predetermined reference value, the subject of facial authentication is deemed to be the registered user, and a matching result is generated indicating that facial matching has been successful.On the other hand, if the matching scores for all users are not equal to or greater than the reference value, it is determined that the subject of facial authentication is not the registered user, and a matching result is generated indicating that facial matching has failed.

[0349] In addition, the processing conditions (parameters for face matching) performed by the face matching server 6 may be added to the matching request. This allows the face authentication device 1 to instruct the processing contents of face matching performed by the face matching server 6. For example, a threshold value for the matching score may be specified as the processing condition for face matching, and matching results with matching scores equal to or greater than the predetermined threshold may be included in the response. Also, the number of matching results may be specified as the processing condition for face matching, and a predetermined number of matching results starting from the highest matching scores may be included in the response.

[0350] In addition, in the face matching server 6, when face matching is completed, the authentication log management unit 87 stores information such as the matching result acquired in the face matching in a database as an authentication log (face recognition history information). At this time, in addition to storing only the face recognition result (success or failure) as the authentication log, the matching score may also be included in the authentication log. Also, the facial image of the subject acquired from the face recognition device 1 may be included in the authentication log. In this case, it is preferable to encrypt and store the facial image of the subject.

[0351] In addition, when many valid matching results whose matching scores exceed the threshold are obtained, a predetermined number of matching results may be narrowed down in order from the highest matching scores and saved as an authentication log. In addition, when face matching does not end normally or when no valid matching results whose matching scores exceed the threshold are obtained, only information included in the request from the face authentication device 1 may be saved as an authentication log.

[0352] FIG. 55 is an explanatory diagram showing an overview of the reference process relating to the authentication log.

[0353] In response to a request from management terminal 2 via face management server 5, face matching server 6 sends the authentication log (face recognition history information) accumulated in its own device to management terminal 2, and an administrator can view the authentication log information on management terminal 2. At this time, the administrator can specify conditions, and the administrator can view authentication logs that meet the specified conditions.

[0354] Specifically, the management terminal 2 displays a screen for inputting search conditions for searching the authentication log, and on this screen the administrator can specify as search conditions the face authentication device 1 that requested the face authentication, the user, the number of extractions, the matching date and time when the face matching was performed (extraction period), and the matching result (success, failure), etc.

[0355] When the administrator inputs search conditions in the management terminal 2 to instruct a search, the authentication log presenting unit 69 of the face management server 5 sends a request to refer to the authentication log to the face matching server 6. This request to refer to the authentication log includes the search conditions specified by the administrator. At this time, the authentication log presenting unit 69 identifies the face matching server 6 whose matching group matches the target face authentication device 1 or user based on information regarding the association between the face authentication device 1 and the matching group, information regarding the association between the face matching server 6 and the matching group, and information regarding the user, and sends a request to refer to the authentication log to that face matching server 6. In addition, the face management server 5 records the administrator access log.

[0356] When the face matching server 6 receives a request to refer to the authentication log from the face management server 5, it searches for the authentication log according to the search conditions included in the request, generates a list of authentication logs that match the search conditions as the search results, and attaches this list of authentication logs to a response to the authentication log reference and sends it to the face management server 5. Note that a facial image of the subject can be sent to the face management server 5 as the authentication log, but in this case, since the facial image of the subject is encrypted, it is decrypted and then added to the response.

[0357] When the face management server 5 receives a response to refer to the authentication logs from the face matching server 6, it causes the management terminal 2 to display a search result screen displaying a list of authentication logs that meet the search criteria specified by the administrator, based on the list of authentication logs included in the response. At this time, for search criteria that cannot be narrowed down by the face matching server 6, authentication logs that meet the search criteria are extracted from the list of authentication logs, and a list of narrowed down authentication logs is generated.

[0358] In addition, in the management terminal 2, an item can be specified to instruct sorting, and when this operation is performed, the face management server 5 sorts the list of authentication logs by the specified item and generates a list of sorted authentication logs. In addition, in the management terminal 2, the number of authentication logs to be acquired can be specified, and when this operation is performed, the face matching server 6 extracts only that number of authentication logs and generates a list of authentication logs. In addition, in the management terminal 2, a period can be specified, and when this operation is performed, the face matching server 6 extracts authentication logs included in the period and generates a list of authentication logs.

[0359] FIG. 56 is an explanatory diagram showing an authentication log reference screen displayed on the management terminal 2. As shown in FIG.

[0360] An authentication log reference screen is displayed on the management terminal 2. This authentication log reference screen is provided with a search condition designation section 151 and a search result display section 152.

[0361] In the search condition specification unit 151, the device ID of the target face authentication device 1, the user code, the number of acquisitions, the matching date and time, and the matching result can be input as search conditions. In the matching date and time field, the period (start point and end point) can be specified. In the matching result field, success, failure, or both success and failure can be selected from a pull-down menu. After inputting the search conditions for each field in this search condition specification unit 151, operating the search button 153 executes a search to search for authentication logs that meet the search conditions, and the search results are displayed in the search result display unit 152.

[0362] The search result display unit 152 displays a list of data for each item of matching date and time, matching result (success, failure), and face image as the search result. In the face image item, if the matching result is successful, the face image of the person to be face-authenticated obtained from the face authentication device 1 and the face image of the registered user are displayed. If the matching result is unsuccessful, only the face image of the person to be face-authenticated is displayed. This allows the administrator to check the authentication log related to the specified face authentication device 1 or user, and also allows the administrator to narrow down and check the authentication log by matching date and time or matching result.

[0363] In this search result display section 152, items can be specified and sorted (rearranged). In addition, when a save button 154 is operated, the authentication log information displayed in the search result display section 152 is saved in a predetermined format. This allows the administrator to check the authentication logs related to the specified face authentication device 1 and user in chronological order.

[0364] Next, a description will be given of the external application linking function of the face authentication device 1. Fig. 57 is an explanatory diagram showing an overview of the external application linking function of the face authentication device 1.

[0365] The face authentication device 1 has a function of linking with an external application, and notifies the authentication result acquired from the face matching server 6 to an external application, i.e., a management server 8 (management device) of an external system. The external system uses the authentication result of this system to provide various services to users. For example, the external system is an entrance / exit management system that performs control to unlock the door if the authentication result is successful. Also, the external system is a payment system that performs control to execute payment if the authentication result is successful.

[0366] The face authentication device 1 also includes an authentication result adjustment unit 25 (connection application), and notifies the authentication result to the management server 8 of the external system via this authentication result adjustment unit 25. The face authentication device 1 notifies the authentication result (success, failure) to the authentication result adjustment unit 25 via a named pipe (inter-process communication). The authentication result adjustment unit 25 notifies the authentication result to the management server 8 of the external system via the network using a predetermined protocol (WebSocket). The authentication result adjustment unit 25 may perform necessary processing such as processing or adding information depending on how the authentication result is used in the external system.

[0367] The face matching request unit 23 of the face authentication device 1 sends a face matching request to the face matching server 6, and when it receives a face matching response from the face matching server 6 in response, it writes the authentication result (authentication successful, authentication failed) contained in the response into a named pipe and notifies the connection application. At this time, the face authentication device 1 further obtains identification information of the authenticator and camera 11 as the authentication result and adds it to the notification of the authentication result. Specifically, if the authentication result is successful, it writes the user code and the device ID of the face authentication device 1 (camera 11) into the named pipe. On the other hand, if the authentication result is unsuccessful, it writes only the authentication result (authentication failed) into the named pipe.

[0368] In addition, the authentication result adjustment unit 25 of the face authentication device 1 suppresses the notification of the authentication result to the external system so as not to repeatedly notify the same authentication result. In this embodiment, when the authentication result is notified to the connected application, a notification suppression time is set, and notification of the same authentication result is temporarily stopped during the notification suppression time.

[0369] When a notification of authentication success is made for a certain person, other notifications of authentication success for that person are suppressed for a predetermined time (authentication success notification suppression time) after the notification of authentication success. Also, when a notification of authentication failure is made for a certain person, other notifications of authentication failure for that person are suppressed for a predetermined time (authentication failure notification suppression time) after the notification of authentication failure.

[0370] This notification suppression time is set for each authentication result (authentication success, authentication failure) and person, and if the authentication result is different for the same person, the notification suppression time is set differently, and if the authentication result is the same for different people, the notification suppression time is set differently. Therefore, during the notification suppression time for authentication failure for a certain person, a notification of authentication success for that person can be made, and during the notification suppression time for authentication success for a certain person, a notification of authentication failure for that person can be made. Also, during the notification suppression time for authentication failure for a certain person, a notification of authentication failure for another person can be made, and during the notification suppression time for authentication success for a certain person, a notification of authentication success for another person can be made.

[0371] Next, the operation status monitoring function in this system will be explained. Figure 58 is an explanatory diagram showing an overview of the operation status monitoring function in this system.

[0372] The face authentication device 1 includes an operation status monitoring unit 26. This operation status monitoring unit 26 monitors the operation status of its own device, and notifies the face management server 5 (monitoring server) of the operation status (alive or dead) of its own device periodically at a predetermined cycle. In addition, the face matching server 6 includes an operation status monitoring unit 89. This operation status monitoring unit 89 monitors the operation status of its own device, and notifies the face management server 5 (monitoring server) of the operation status (alive or dead) of its own device periodically at a predetermined cycle.

[0373] The face management server 5 includes an operation status monitoring unit 68. This operation status monitoring unit 68 monitors the operation status of its own device, and periodically records the operation status (alive or dead status) of its own device in a predetermined cycle. In addition, the face management server 5 records the operation status notified from the face authentication device 1 and the face matching server 6 in its own device.

[0374] The information related to the operation status notified here includes the device ID, acquisition time, alive status, program version information, and the like.

[0375] Furthermore, in the management terminal 2, by making a request to the face management server 5 to refer to the operating status, the manager can check the operating status.

[0376] Next, the update of the face matching database accompanying the version upgrade of the program in face matching server 6 will be described.

[0377] In order to improve the accuracy of face matching, the face matching server 6 appropriately updates the face feature generation program and face matching program. On the other hand, the face feature data depends on the algorithms of face feature generation and face matching, and if the algorithms are changed, the face feature data also changes, so that it is necessary to regenerate the face feature data.

[0378] Therefore, in this embodiment, when a program related to the algorithm of facial feature generation or face matching is upgraded, the facial feature data registered in the face matching database is updated to correspond to the new program. That is, the facial feature data already stored is deleted, and the facial feature data is regenerated based on the facial image data at the time of the user's registration, and the new facial feature data is stored.

[0379] Specifically, when face management server 5 detects an upgrade of the face matching program by the operation status notification function, it sends a request for updating the face feature amount data to face matching server 6. This request includes the face image of the user. When face feature amount management unit 85 in face matching server 6 receives the request for updating the face feature amount data, it causes face feature amount generation unit 84 to perform processing for generating face feature amount data from the face image of the user, and registers the new face feature amount data in the face matching database.

[0380] In this case, if the face matching process and face matching database are the same in multiple face matching servers 6, that is, the face feature data registered in the face matching database is the same between multiple face matching servers 6, and the face matching process performs face matching processing on face authentication devices 1 in the same matching group, then when one face matching server 6 regenerates the face feature data and registers the new face feature data in the face matching database, the face feature data registered in that face matching database is copied to another face matching server 6, making the registration contents of the face matching database identical.

[0381] At this time, the face management server 5 instructs the destination face matching server 6 to make a copy, and the facial feature data stored in the source face matching server 6 is transferred to the destination face matching server 6 via the face management server 5.

[0382] Next, backup and restore will be described.

[0383] In this embodiment, the databases and files stored in the face management server 5 and the face matching server 6 can be backed up and restored.

[0384] The information stored in the face management server 5 that is to be backed up includes a database of user information (face information), facial image files, a database of matching group information, a database of authentication device and matching group association information, a database of matching server and matching group association information, and an administrator access log file.

[0385] The information stored in the face matching server 6 that is to be backed up includes the authentication log database and face image files.

[0386] 59 is an explanatory diagram showing the backup screen and restore screen displayed on the management terminal 2. The backup screen and restore screen can be switched using tabs within the screen.

[0387] As shown in FIG. 59(A), the backup screen is provided with a backup destination designation section 161, a backup target designation section 162, a progress status display section 163, and an execute button 164. In the backup destination designation section 161, the backup destination can be designated. In the backup target designation section 162, the face management server 5 and the face matching server 6 can be selected as the devices to be backed up. The backup is executed when the execute button 164 is operated. In the progress status display section 163, a list of data items to be backed up is displayed. In these data items, a check mark is displayed upon completion of the backup, allowing the administrator to recognize the progress status of the backup.

[0388] The data to be backed up here includes a database of user information (facial information), files of facial images of registered users, a database of matching group information, a database of association information between the facial recognition device 1 and a matching group, a database of association information between the facial matching server 6 and a matching group, a file of administrator access logs, a database of authentication logs, and files of facial images of persons subject to facial recognition.

[0389] As shown in Figure 59(B), the restore screen displays the following information in the same way as the backup screen: Restoration Destination designation unit 165 and Restoration An object designation section 166, a progress status display section 167, and an execution button 168 are provided. Restoration In the destination designation section 165, the restore destination can be designated. Restoration In the target designation section 166, the face management server 5 and the face matching server 6 can be selected as the devices to be restored. When the execute button 168 is operated, the restoration is executed. In the progress display section 167, the data items to be restored are displayed in a list. For these data items, a check mark is displayed as the restoration is completed, allowing the administrator to recognize the progress of the restoration.

[0390] Next, a modified example of this embodiment will be described below. Fig. 60 is an explanatory diagram showing an overview of face image extraction processing according to the modified example of this embodiment.

[0391] In the above embodiment (see FIG. 12), the face image extraction process for generating face image data from the captured image data of the user is performed by the face authentication device 1 and the face matching server 6, but in this modified example, the face image extraction process is performed by the face authentication device 1 and the face management server 5, and is not performed by the face matching server 6. Specifically, the face image extraction unit 83 provided in the face matching server 6 in the above embodiment is omitted, and a face image extraction unit 201 is provided in the face management server 5.

[0392] During user registration, photographed image data of the user is sent from the management terminal 2 to the face management server 5, and face image extraction processing is performed by the face image extraction unit 201 of the face management server 5. On the other hand, during face authentication, face image extraction processing is performed by the face image extraction unit 22 of the face authentication device 1, which is the same as in the above embodiment.

[0393] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0394] In recent years, in the world of IoT (Internet of Things), attention has been focused on CPS (Cyber ​​Physical Systems), a new concept that creates new added value by information cooperation between physical space and cyberspace. In this context, the present embodiment can also adopt the CPS concept. That is, as a basic configuration of CPS, for example, an edge device arranged in a physical space and a cloud server arranged in a cyberspace are connected via a network, and face authentication processing can be distributed and processed by processors mounted on the edge device and the cloud server. In this face authentication processing, the edge device acquires face image data of a user photographed by a camera of an authentication machine. Next, the cloud server executes a process of generating feature data from the face image data received from the edge device via the network, compares the feature data with the feature data of the user registered in advance, executes authentication processing of the user, and takes over the authentication result related to the authentication processing by face authentication application software, and displays it on a display or the like in an output format defined on the application. Here, each data generated in the edge device or cloud server is preferably generated by a Web application or the like mounted on a standardized platform, and by using such a standardized platform, it is possible to improve the efficiency of building a system including a group of various sensors and IoT application software. [Industrial Applicability]

[0395] The facial recognition server device, facial recognition method, and facial recognition program of the present invention have the effect of enabling the facial recognition function to be operated stably and efficiently for each facial recognition service provider, and are useful as a facial recognition server device, facial recognition method, and facial recognition program that executes facial recognition processing based on image data photographing a subject of facial recognition. [Explanation of symbols]

[0396] 1. Face recognition machine 2 Management terminal (terminal device, face recognition management device) 3. Face recognition server (server device) 5. Face management server (face image management section) 6 Face matching server (face image matching section)

Claims

1. A face authentication system that registers face image data of a user and performs face authentication to determine whether the subject is the user or not based on face image data obtained by photographing the subject, A plurality of face recognition devices each having a camera for photographing the subject; a face recognition server device connected to the plurality of face recognition devices via a network; Equipped with The face authentication server device a face image management unit that accumulates face image data of the user at the time of the registration; a plurality of face image matching units that perform a matching process between face feature amount data of the subject generated from face image data of the subject acquired from the face authentication device and face feature amount data of the user generated from face image data of the user during the face authentication; an authentication result transmission unit that transmits an authentication result obtained by the face image matching unit, the authentication result indicating whether the subject is the user or not, to the face authentication device; having A facial recognition system characterized in that, during the facial recognition, the facial recognition device, in a facial image extraction process in which an excision area is set in the image data of the subject and facial image data of the subject is excised, detects the facial area included in the image data of the subject and sets a rectangular face frame surrounding the facial area, and if the size of the set face frame is within a predetermined appropriate range, sets the excision area expanded vertically and horizontally by an integer multiple centered on the face frame, and if the entire excision area is not included in the image data of the subject, superimposes the excision image of the image data of the subject on a background image of the same size as the excision area to generate the facial image data of the subject.

2. A face authentication method for registering face image data of a user, and performing face authentication to determine whether or not the subject is the user based on face image data of the subject, using a server device, comprising: At the time of said registration, a face image management unit included in said server device stores face image data of said user; During the face authentication, a camera of a plurality of face authentication devices connected to the server device via a network photographs the subject, and a plurality of face image matching units included in the server device execute a matching process between face feature amount data of the subject generated from face image data of the subject acquired from the face authentication device and face feature amount data of the user, and transmit an authentication result indicating whether the subject is the user or not to the face authentication device; A facial recognition method characterized in that, during the facial recognition, the facial recognition device, in a facial image extraction process in which an excision area is set in image data of the subject and facial image data of the subject is excised, detects a facial area included in the image data of the subject and sets a rectangular face frame surrounding the facial area, and if the size of the set face frame is within a predetermined appropriate range, sets the excision area expanded vertically and horizontally by an integer multiple centered on the face frame, and if the entire excision area is not included in the image data of the subject, superimposes the excised image of the image data of the subject on a background image of the same size as the excision area to generate facial image data of the subject.

3. A face recognition program for causing a computer to execute the face recognition method according to claim 2.

Images