Facial recognition terminal, facial recognition system, control method, and control program

Abstract

This prevents anyone other than the person being authenticated from seeing their facial image during facial recognition. [Solution] The facial recognition terminal 3 includes an imaging device 34 for capturing images of a person to be authenticated, a display device 35 for displaying information on the screen for the person to be authenticated, and a processor 31 that controls the screen display of the display device based on the captured image acquired by the imaging device. The processor detects a person's face from the captured image acquired by the imaging device and displays the region of the captured image containing the detected face on the display device in place of a substitute image.

Description

Technical Field

[0001] The present disclosure relates to a face recognition terminal and a face recognition system that perform guidance display during face recognition, and a control method and a control program for the guidance display related to face recognition.

Background Art

[0002] In recent years, services using a face recognition device (hereinafter referred to as a face recognition service) have become widespread in order to manage entry and exit to facilities and to perform payment procedures for goods and services. The face recognition device performs so-called face matching in which an imaging image of an authentication target person is acquired and the face image extracted from the imaging image is compared with a pre-registered face image. Thereby, it can be determined whether the user is a legitimate registered person.

[0003] There is known a face recognition device in which wall members are provided on both sides of a display that displays a face image and an authentication result in order to ensure the privacy of an authentication target person (authenticated person) (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the face recognition device of Patent Document 1, if a person other than the authentication target person peeks over the wall member and looks into the display, there is a risk that the face image will be seen.

[0006] Therefore, a main object of the present disclosure is to provide a face recognition terminal and a face recognition system that can prevent a face image from being seen by a person other than the authentication target person during face recognition, and a control method and a control program for the guidance display related to face recognition. [Means for solving the problem]

[0007] The facial recognition terminal of this disclosure comprises an imaging device for capturing images of a person to be authenticated, a display device for displaying information on the screen for the person to be authenticated, and a processor for controlling the screen display of the display device based on the captured image acquired by the imaging device. The processor is configured to detect a person's face from the captured image acquired by the imaging device and to display the region of the captured image containing the detected face on the display device in place of a substitute image.

[0008] The facial recognition system of this disclosure comprises an imaging device for capturing images of a person to be authenticated, a display device for providing guidance to the person to be authenticated, and a processor that causes the display device to display an image to guide the person to the imaging range of the imaging device, wherein the processor detects a person's face from the captured image acquired by the imaging device and displays the region of the captured image containing the detected face on the display device in place of a substitute image.

[0009] The method for controlling the guidance display related to facial recognition according to this disclosure is executed by a facial recognition terminal having an imaging device for capturing images of a person to be authenticated, a display device for displaying information on the screen to the person to be authenticated, and a processor that controls the screen display of the display device based on the captured image acquired by the imaging device, wherein the processor detects a person's face from the captured image acquired by the imaging device and displays the region of the captured image containing the detected face on the display device in place of a substitute image.

[0010] The control program for the guidance display related to facial recognition of this disclosure is executed by a facial recognition terminal having an imaging device for capturing images of a person to be authenticated, a display device for displaying information on the screen to the person to be authenticated, and a processor that controls the screen display of the display device based on the captured image acquired by the imaging device, wherein the processor detects a person's face from the captured image acquired by the imaging device and displays the region of the captured image containing the detected face on the display device in place of a substitute image. [Effects of the Invention]

[0011] According to this disclosure, it is possible to prevent anyone other than the person being authenticated from seeing the facial image during facial recognition. [Brief explanation of the drawing]

[0012] [Figure 1] Overall configuration diagram of the facial recognition system [Figure 2] Figure 1 shows a functional block diagram of the facial recognition server and facial recognition terminal. [Figure 3] An explanatory diagram showing an example of an image-captured facial information table. [Figure 4] This is an explanatory diagram showing the image after the face region of the captured image has been replaced with a substitute image (the substitute images are (A) a blurred circle, (B) a blurred circle with a figure indicating the positions of the eyes and nose superimposed, (C) an image with an illustration superimposed on the face region, and (D) a blurred image of the face region). [Figure 5] Examples of display screens including guidance information (guidance display screens): ((A) When the face area is away from the guide frame, (B) When the face area is small, (C) When the face corresponding to the face area is facing left) [Figure 6] (A) First example and (B) Second example of the display screen during authentication execution (authentication execution screen) [Figure 7] Examples of display screens showing authentication results (A) to C: when authentication is successful, D: when authentication fails) [Figure 8] Flowchart showing the base image generation process in a facial recognition terminal. [Figure 9] Flowchart showing the process flow of authentication execution in a face recognition terminal [Figure 10] Flowchart of a determination process for determining whether or not authentication start conditions are satisfied [Figure 11] Sequence diagram showing an example of the operations of a face recognition server and a face recognition terminal after the start of face recognition until the result is displayed [Figure 12] First example of the screen display of a face recognition terminal until the result of face recognition is displayed [Figure 13] Second example of the screen display of a face recognition terminal until the result of face recognition is displayed

Mode for Carrying Out the Invention

[0013] A first invention made to solve the above problems is a face recognition terminal, comprising an imaging device for imaging an authentication target person, a display device for performing screen display on the authentication target person, and a processor for controlling the screen display of the display device based on the captured image acquired by the imaging device, wherein the processor detects a human face from the captured image acquired by the imaging device, and replaces the area of the captured image containing the detected face with a substitute image and causes the display device to display the substitute image.

[0014] According to this, the display device displays the area including the face in the captured image by replacing it with a substitute image. Therefore, it is possible to prevent the face image of the authentication target person from being displayed on the display device during face authentication and to prevent the face image from being seen by persons other than the authentication target person.

[0015] Further, in a second invention, the processor is configured to detect the position of a human face from the captured image, arrange the substitute image at the detected position of the face, and cause the display device to display the substitute image.

[0016] According to this, since the substitute image is arranged at the position of the face in the captured image, it is possible to satisfactorily prevent the face image of the authentication target person from being displayed on the display device.

[0017] Furthermore, the third invention is configured such that the processor superimposes a guide frame for guiding the person to be authenticated to the appropriate imaging position onto a base image obtained by replacing the region containing the detected face in the captured image with the substitute image, and displays this on the display device.

[0018] According to this, a simple configuration can guide the person being authenticated to move to a position where they can be properly imaged.

[0019] Furthermore, the fourth invention is configured such that the processor acquires the result of matching the image of a face located inside the guide frame with pre-stored face information.

[0020] According to this, the facial image located within the guide frame will be the target of facial recognition. Therefore, by placing the guide frame in a position where an image suitable for matching can be obtained, the person to be authenticated can be guided to a position where an image of their face suitable for matching can be obtained.

[0021] Furthermore, the fifth invention is configured such that when the processor compares a face image extracted from the captured image with pre-stored face information, it displays a progress image on the display device indicating the progress of the comparison.

[0022] According to this system, a progress image showing the matching progress is displayed on the display device. Therefore, the person being authenticated can understand the matching progress from the progress image.

[0023] Furthermore, the sixth invention is configured such that the processor acquires the orientation of a face detected from the captured image and generates the alternative image according to the acquired face orientation.

[0024] According to this, the person being authenticated can determine the detected facial orientation by reviewing alternative images. This allows the person being authenticated to, for example, change their facial orientation to one that is more likely to be authenticated.

[0025] Furthermore, the seventh invention is configured such that the processor displays on the display device a notification superimposed on a base image obtained by replacing the region containing the detected face in the captured image with the substitute image, based on at least one of the position, size, posture, change in position, and change in size of the face detected in the captured image.

[0026] According to this, by having the person being authenticated check the notification, they can obtain information related to one of the following: the position, size, or change in position of the face detected in the captured image.

[0027] Furthermore, the eighth invention is configured such that, when the processor detects multiple faces of people in the captured image acquired by the imaging device, it displays the regions of the captured image containing the faces of people on the display device in place of the corresponding substitute image.

[0028] According to this, when the display device detects the faces of multiple people, it replaces the areas of the captured image containing the faces with alternative images. Therefore, during face recognition, the display device does not show the areas containing the faces, preventing anyone other than the person being authenticated from seeing the face image.

[0029] Furthermore, the ninth invention is configured such that, when the processor detects multiple faces from the captured image acquired by the imaging device, it superimposes the substitute images so that the substitute image of the larger face overlaps the substitute image of the smaller face, and displays them on the display device.

[0030] According to this, the display device can superimpose and display the corresponding alternative images in the order in which they should be authenticated.

[0031] Furthermore, the tenth invention is configured such that, when the processor detects multiple faces of people from the captured image acquired by the imaging device, it superimposes the substitute images on the display device such that the substitute image of the face with a large change in size overlaps the substitute image of the face with a small change in size.

[0032] According to this, the display device can superimpose and display the corresponding alternative images in the order in which they should be authenticated.

[0033] Furthermore, the eleventh invention is configured such that when the processor fails to match the face image of a face extracted from the captured image with pre-stored face information, it displays a notification indicating the cause of the failure on the display device.

[0034] According to this, the person being authenticated can understand the reason for the verification failure through a notification displayed on the device.

[0035] The twelfth invention made to solve the aforementioned problems is a facial recognition system comprising: an imaging device for capturing images of a person to be authenticated; a display device for displaying information on the screen for the person to be authenticated; and a processor for controlling the screen display of the display device based on the captured image acquired by the imaging device, wherein the processor detects a person's face from the captured image acquired by the imaging device and displays the region of the captured image containing the detected face on the display device in place of a substitute image.

[0036] According to this system, the display device replaces the area containing the face in the captured image with a substitute image. Therefore, during facial recognition, the face image of the person being authenticated is not displayed on the display device, preventing anyone other than the person being authenticated from seeing the face image.

[0037] The thirteenth invention, made to solve the aforementioned problems, is a method for controlling a guidance display related to facial recognition, which is executed by a facial recognition terminal having an imaging device for capturing images of a person to be authenticated, a display device for displaying an image on the screen for the person to be authenticated, and a processor for controlling the screen display of the display device based on the image captured by the imaging device, wherein the processor detects a person's face from the image captured by the imaging device and displays the region of the image captured that contains the detected face on the display device in place of a substitute image.

[0038] According to this system, the display device replaces the area containing the face in the captured image with a substitute image. Therefore, during facial recognition, the face image of the person being authenticated is not displayed on the display device, preventing anyone other than the person being authenticated from seeing the face image.

[0039] The fourteenth invention, made to solve the aforementioned problems, is a control program for a guidance display related to facial recognition, which is executed by a facial recognition terminal having an imaging device for capturing images of a person to be authenticated, a display device for displaying images on the screen for the person to be authenticated, and a processor for controlling the screen display of the display device based on the captured image acquired by the imaging device, wherein the processor detects a person's face from the captured image acquired by the imaging device and displays the region of the captured image containing the detected face on the display device in place of a substitute image.

[0040] According to this system, the display device replaces the area containing the face in the captured image with a substitute image. Therefore, during facial recognition, the face image of the person being authenticated is not displayed on the display device, preventing anyone other than the person being authenticated from seeing the face image.

[0041] The facial recognition system disclosed herein is used for purposes such as access control to facilities and facial payment at cash registers. Examples of such facilities include corporate facilities such as offices, entertainment facilities such as concert venues, movie theaters, and event venues, educational institutions such as universities, and transportation facilities such as airports and train stations. Embodiments of this disclosure will be described below with reference to the drawings.

[0042] The facial recognition system 1 of this disclosure is used for access control to corporate facilities such as offices. Figure 1 is an overall configuration diagram of the facial recognition system 1 of this disclosure. Figure 2 is a functional block diagram of the facial recognition server 2 and facial recognition terminal 3 shown in Figure 1. Figure 1 is an explanatory diagram showing an example of the captured facial information table 4 shown in Figure 2. Figure 2 is an explanatory diagram showing an image after the face region of the captured image has been replaced with a substitute image 5. Figures 5(A) to (C) are examples of display screens (guidance display screen 72) that include guidance information. Figure 6 is an example of a display screen (authentication execution screen 80) when authentication is performed, and Figures 7(A) to (D) are examples of display screens (result display screen 84) that show the authentication result.

[0043] As shown in Figure 1, the facial recognition system 1 comprises a facial recognition server 2 and at least one facial recognition terminal 3 (also called an authentication device). The facial recognition system 1 may also include an administrator terminal (not shown) used by a business that provides facial recognition services, and a business server (not shown) for providing services to those whose faces have been recognized.

[0044] The facial recognition server 2 and the facial recognition terminal 3 can communicate with each other via a publicly known communication network 7 such as the Internet. However, the facial recognition server 2 and the facial recognition terminal 3 may also send and receive necessary information and data to and from each other via other information processing devices (i.e., indirectly) not shown.

[0045] The facial recognition server 2 consists of computers that perform the processing necessary to provide the facial recognition service. The facial recognition server 2 may be operated by a company using the office, or it may be operated by a business that provides facial recognition services via a communication network 7 such as the Internet.

[0046] The facial recognition server 2 stores the facial information (hereinafter referred to as registered facial information) of multiple individuals who have been registered in advance (i.e., multiple users whose facial information has already been registered). Based on instructions from the facial recognition terminal 3, the facial recognition server 2 can execute a predetermined program (hereinafter referred to as the facial recognition program) using the facial image acquired by the facial recognition terminal 3 and the registered facial information, and perform information processing (hereinafter referred to as facial recognition processing) to authenticate the facial image of the user to be authenticated.

[0047] As shown in Figure 1, the facial recognition server 2 comprises a processor 11 consisting of a CPU or MPU, memory 12 such as RAM or ROM, storage 13 such as an HDD or SSD, and communication devices 14 such as a network interface or wireless communication module. The facial recognition server 2 may also include other known hardware such as a display (not shown) and input devices (not shown).

[0048] As shown in Figure 2, the facial recognition server 2 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0049] The communication unit 21 may mainly consist of communication devices 14. The communication unit 21 communicates wirelessly or via wired connection with other devices (in this case, a facial recognition terminal 3, etc.) via a network in accordance with known communication protocols.

[0050] The storage unit 22 may be composed of storage devices such as memory 12 and storage 13. The storage unit 22 stores programs for facial recognition processing and various information necessary for executing facial recognition processing. For example, the storage unit 22 stores a database (registered facial information database 25) in which registered facial information is recorded.

[0051] The control unit 23 may be mainly composed of the processor 11. The control unit 23 includes an image acquisition unit 26, a feature detection unit 27, and a face recognition unit 28 as functional units for performing face recognition processing. The image acquisition unit 26, the feature detection unit 27, and the face recognition unit 28 may each be configured by the processor 11 executing a face recognition program.

[0052] The image acquisition unit 26 acquires the face image of the person to be authenticated (the person to be authenticated) acquired by the face authentication terminal 3 during the face authentication process. The image acquisition unit 26 may acquire the face image of the person to be authenticated from the face authentication terminal 3, or it may acquire the face image of the person to be authenticated by extracting the face portion from the captured image acquired by the face authentication terminal 3.

[0053] The feature detection unit 27 obtains the facial features of the person to be authenticated by detecting facial feature points from the facial image of the person to be authenticated acquired by the image acquisition unit 26.

[0054] In the face recognition process, the face recognition unit 28 determines whether the user is a legitimate registrant by comparing the facial features of the person to be authenticated, acquired by the feature acquisition unit, with the registered face information stored in the storage unit 22. The face recognition unit 28 transmits the determination result (also called the authentication result or matching result) to the face recognition terminal 3.

[0055] If the facial recognition unit 28 determines that the user is not a legitimate registrant and obtains the reason why the matching could not be done, i.e., the cause of failure, it may include the information related to the cause of failure in the determination result and transmit it to the facial recognition terminal 3.

[0056] The facial recognition terminal 3 captures an image of the person to be authenticated and transmits it to the facial recognition server 2, and the facial recognition server 2 notifies the person of the result of the authentication.

[0057] As shown in Figure 1, the facial recognition terminal 3 comprises a processor 31 consisting of a CPU and an MPU, memory 32 such as RAM and ROM, and storage 33 such as an HDD and an SSD.

[0058] The facial recognition terminal 3 includes a processor 31, memory 32, and storage 33, as well as an imaging device 34, a display device 35, and a communication device 36 for communicating with the facial recognition server 2. The display device 35 includes a display 35A (for example, a liquid crystal display) and displays information on the screen to provide various instructions related to facial recognition to the person to be authenticated. The imaging device 34 is a device for imaging the person to be authenticated, and it images the area around the facial recognition terminal 3 and acquires the captured image. The imaging device 34 may be composed of a camera 34A.

[0059] The facial recognition terminal 3 may also be equipped with other known hardware, such as an audio output device (not shown) like a speaker, and an input device (not shown) like a touch panel. The display device 35 may be composed of a touch panel display in which a display 35A and a touch panel are integrated.

[0060] The facial recognition terminal 3 executes a control program to control the guidance display on the display device 35 for facial recognition, and executes a control process for the guidance display related to facial recognition (hereinafter referred to as the authentication terminal process) to implement a control method for the guidance display related to facial recognition. By executing the control program, the facial recognition terminal 3 controls the guidance display on the display device 35 and guides the person to be authenticated to a position where a facial image suitable for facial recognition can be acquired. The facial recognition terminal 3 also controls the guidance display on the display device 35 and informs (notifies) the person to be authenticated of the progress of facial recognition and the result of facial recognition.

[0061] As shown in Figure 2, the facial recognition terminal 3 has a communication unit 41, a storage unit 42, an imaging unit 43, a display unit 44, and a control unit 45 as functional units for performing authentication terminal processing.

[0062] The communication unit 41 may mainly consist of communication devices 36. The communication unit 41 communicates wirelessly or via wired connection with other devices (in this case, a facial recognition server 2, for example) via a network, according to known communication protocols.

[0063] The memory unit 42 may be composed of storage devices such as memory 32 and storage 33. The memory unit 42 stores a control program for controlling the guidance display by the display device 35 for facial recognition, as well as various information necessary for the execution of the control program.

[0064] The imaging unit 43 is comprised of an imaging device 34. The imaging unit 43 acquires an image and outputs it to the control unit 45.

[0065] The display unit 44 is composed of a display device 35. Based on instructions from the control unit 45, the display unit 44 displays guidance information on the screen.

[0066] The control unit 45 can be realized by having one or more processors 31 execute a control program for controlling the guidance display and perform authentication terminal processing. The control unit 45 comprehensively controls the operation of the face recognition terminal 3. The control unit 45 has the following functional units for performing authentication terminal processing: a face detection unit 51, a face information acquisition unit 52, an alternative image generation unit 53, a superposition order setting unit 54, a base image generation unit 55, a guidance display unit 56, a matching processing unit 57, and an authentication result display unit 58.

[0067] The face detection unit 51 causes the imaging device 34 to acquire captured images at predetermined intervals (hereinafter referred to as interval time). When the imaging device 34 acquires captured images, the face detection unit 51 detects human faces within those captured images. In order to detect faces from the captured images, the face detection unit 51 analyzes the entire captured image and identifies areas that may contain faces. The acquisition of captured images by the face detection unit 51 may be configured to occur between the start of the authentication terminal processing and the time when an instruction to perform face authentication is given to the face authentication server 2.

[0068] Next, the face detection unit 51 applies a pre-trained, known learning model to the identified region to determine whether it is a face. If it is determined to be a face, the face detection unit 51 sets a region containing the detected face (hereinafter referred to as the face region) for each detected face. Furthermore, the face detection unit 51 acquires face region information 61 (see Figure 3) for each face region, such as the position (center position), width (left-right width), height (up-down width), and size (area, also called size), as information to identify each face region. The face region information 61 is also called face frame information because it corresponds to information for setting a frame surrounding the face.

[0069] If multiple regions are determined to be faces, the face detection unit 51 acquires face region information 61 for each of those regions.

[0070] The face detection unit 51 outputs all identified face region information 61, along with the captured image, to the face information acquisition unit 52 and the matching processing unit 57.

[0071] The face information acquisition unit 52 updates the table (hereinafter referred to as the image capture face information table 4) stored in the storage unit 42 so that all face-related information detected by the face detection unit 51 is consistent with the table. In other words, the face information acquisition unit 52 updates the image capture face information table 4 so that all face-related information detected by the face detection unit 51 is recorded in the image capture face information table 4, and no information other than faces detected by the face detection unit 51 is recorded in the image capture face information table 4. In this way, the face information acquisition unit 52 tracks people captured in the image capture and acquires and updates information about those people captured in the image capture.

[0072] Figure 3 shows an example of an image capture face information table 4. The image capture face information table 4 stores identification numbers (hereinafter referred to as face IDs), face images, and face information 62 in association with each other. Face information 62 includes face region information 61. Face information 62 may also include the amount of change in the position of the face region (i.e., the amount of movement of the face region) and the amount of change in the size of the face region after an interval of time has elapsed. Face information 62 represents amounts that change according to the face, such as the face region information 61 of the face corresponding to the face image and the orientation of the face. Note that the image capture face information table 4 may be stored in the storage 33 of the face authentication terminal 3, or it may be temporarily stored in the volatile memory (RAM) of the face authentication terminal 3. Specifically, for example, the information recorded in the image capture face information table 4 shown in Figure 3 may be temporarily stored in the volatile memory of the face authentication terminal 3 as face IDs and their corresponding values. In addition, the image capture face information table 4 may include information on the size of the face (width and height), and the amount of change in position, the amount of change in size, and the overlap width may not be stored in the image capture face information table 4 but may be calculated for each frame.

[0073] When the face information acquisition unit 52 acquires face region information 61 and captured images from the face detection unit 51, it acquires a face image for each of the detected faces. Subsequently, the face information acquisition unit 52 updates the captured face information table 4 based on the acquired face images.

[0074] Specifically, when the face information acquisition unit 52 acquires a face image, it determines whether the acquired face image is recorded in the captured face information table 4. The determination of whether it is recorded in the captured face information table 4 may be based on a known method (for example, based on position, size, etc.). If it is not recorded, the face information acquisition unit 52 assigns a new face ID, associates the detected face image with the face region information 61, and adds it to the captured face information table 4.

[0075] Furthermore, when acquiring a face image, the face information acquisition unit 52 should acquire the corresponding left-right rotation angle (yaw angle), up-down rotation angle (pitch angle), and tilt angle (roll angle) of the face and add them to the imaged face information table 4.

[0076] If recorded, the face information acquisition unit 52 updates the face information 62 corresponding to the face image with the face information 62 acquired by the face detection unit 51. Furthermore, when acquiring a face image, the face information acquisition unit 52 should acquire the corresponding left-right rotation angle (yaw angle), up-down rotation angle (pitch angle), and tilt angle (roll angle) of the face and update the imaged face information table 4.

[0077] Furthermore, when updating, the face information acquisition unit 52 should calculate the amount of change (movement) in the position of the face region and update the captured face information table 4. Specifically, first, the face information acquisition unit 52 obtains the position of the face region before the update from the captured face information table 4. Then, based on the position of the face region before the update and the position of the face region newly acquired by the face detection unit 51, the face information acquisition unit 52 should calculate the amount of change (movement) in the position of the face region and update the captured face information table 4.

[0078] Furthermore, during the update, the face information acquisition unit 52 should calculate the amount of change (movement) in the size of the face region and update the captured face information table 4. Specifically, first, the face information acquisition unit 52 obtains the size of the face region before the update from the captured face information table 4. Then, based on the position of the face region before the update and the size of the face region newly acquired by the face detection unit 51, the face information acquisition unit 52 should calculate the amount of change in the size of the face region and update the captured face information table 4.

[0079] Furthermore, the captured face information table 4 may also record the elapsed time (hereinafter referred to as the "still time") when a face region remains within the region of the captured image from which a face image suitable for face recognition is obtained. The region of the captured image from which a face image suitable for face recognition is obtained (also called the "imaging position") is represented by a guide frame 65 displayed near the center of the display screen (for example, see the dashed circle in Figure 4(A), also called the "guide frame").

[0080] If the image capture face information table 4 includes a static time, when updating, the face information acquisition unit 52 should use the position of the face region corresponding to the face image to determine whether or not that face region is within the guide frame 65. If it is present, it should update the static time by adding the elapsed time since the last image was acquired, i.e., the interval time. If it is not present, the face information acquisition unit 52 should reset the static time to 0.

[0081] Furthermore, if a face image other than the acquired face image is recorded in the captured face information table 4, the face information acquisition unit 52 deletes that face image, the face ID corresponding to that face image, and the face information 62 associated with that face ID from the captured face information table 4.

[0082] The alternative image generation unit 53 performs a "face concealment process" in the privacy protection process, which hides the face area of ​​the captured image, making it impossible to identify personal information in the captured image. The face concealment process is performed by replacing the face area detected in the captured image with an alternative image 5.

[0083] The alternative image generation unit 53 uses the image capture face information table 4 to obtain the position, width, and height of each face detected by the face detection unit 51, and identifies the respective face regions. Subsequently, the alternative image generation unit 53 sets an alternative image 5 for each face region. The alternative image 5 is used to replace each face region (i.e., the region of the image capture that contains the detected face). The alternative image generation unit 53 may generate alternative images 5 with different colors and shapes for each face region (i.e., for each face). The alternative image generation unit 53 may also obtain the orientation (posture) of the face corresponding to each face region and generate the alternative image 5 according to the obtained face orientation.

[0084] Figures 4(A) to 4(D) show examples in which the face region of the captured image has been replaced with alternative image 5. Figure 4(A) shows an example in which a blurred circle (a circle with a smoothly blurred outline) was used as alternative image 5.

[0085] Figure 4(B) uses a blurred circle with superimposed shapes representing eyes and a nose as the alternative image 5. In this case, the position and shape of the shapes representing the eyes and nose in alternative image 5 are set according to the orientation of the face corresponding to the face region. This allows alternative image 5 to be configured so that the orientation of the face recognized by the person being authenticated can be understood, enabling the person being authenticated to orient their face in an orientation that makes authentication easier.

[0086] Figure 4(C) shows an example where an image with a shape (stamp) superimposed on the face region of the captured image was used as alternative image 5. Figure 4(D) shows an example where an image (blurred image) with Gaussian blur, mosaic processing, or mask processing applied to the face region of the captured image was used as alternative image 5.

[0087] The superposition order setting unit 54 sets the superposition order of the alternative images 5 generated by the alternative image generation unit 53. The superposition order here refers to the order in which the alternative images 5 are superimposed (the order when they are superimposed from bottom to top), with higher-order images being positioned higher and lower-order images superimposed on lower-order images. For example, the superposition order setting unit 54 sets the superposition order based on the size of the face images recorded in the captured face information table 4 (specifically, in ascending order of size). Alternatively, the superposition order setting unit 54 may set the superposition order based on the amount of change in face size recorded in the captured face information table 4. The superposition order setting unit 54 may also record the set superposition order in the captured face information table 4 (see Figure 3).

[0088] The base image generation unit 55 generates a base image 70 by superimposing the face regions of the captured images in the superposition order set by the superposition order setting unit 54, and replacing them with alternative images 5 generated by the alternative image generation unit 53.

[0089] This substitution places the alternative image 5 at the location of the detected face in the captured image. Furthermore, when multiple faces are detected in the captured image, the base image 70 is generated by replacing each face region with the corresponding alternative image 5.

[0090] In the base image 70, the alternative images 5 are superimposed in the order of superposition. For example, if the superposition order is based on the size of the face region, the base image generation unit 55 should generate the base image 70 by superimposing the alternative images 5 in order of increasing face region size (i.e., the alternative images 5 of larger face regions are superimposed on the alternative images 5 of smaller face regions in order). Also, if the superposition order is based on the amount of change in the face region, the base image generation unit 55 should generate the base image 70 by superimposing the alternative images 5 in order of increasing face region size change (i.e., the alternative images 5 of larger face regions are superimposed on the alternative images 5 of smaller face regions in order).

[0091] The guidance display unit 56 displays a guidance display screen 72 on the display unit 44 to guide the person to be authenticated to a position where facial recognition is easily performed.

[0092] The guidance display unit 56 generates guidance information to display the guidance display screen 72. The guidance information includes a guide frame 65 (guidance frame) that is displayed near the center of the display screen of the face recognition terminal 3. The guide frame 65 guides the person to be authenticated to an appropriate imaging position where a face image suitable for face recognition can be obtained.

[0093] In addition, the guidance information includes a guidance notification 74 that directs the person to be authenticated so that their face area moves within the guide frame 65. The person corresponding to the largest face area in the superimposition order (the person on whom the corresponding alternative image 5 is superimposed last) is the person to be authenticated. The guidance display unit 56 generates the guidance notification 74 based on at least one of the position, size, change in position, and change in size of the largest face area in the superimposition order.

[0094] The guidance display unit 56 generates text 74A such as "Get closer to the center" as a guidance notification 74 when, for example, the position of the largest face region in the superimposed order is far from the guide frame 65. When the size of the largest face region in the superimposed order is smaller than the corresponding threshold, it generates text 74A such as "Get closer" as a guidance notification 74. When the amount of change in the position or the amount of change in the size of the largest face region in the superimposed order is greater than the corresponding threshold, it generates text 74A such as "Stop" as a guidance notification 74.

[0095] In addition, the guidance information may include notifications (text 74A or illustrations) to guide the person being authenticated to assume a posture that facilitates facial recognition. The guidance display unit 56 may generate guidance notifications 74 based on the left-right rotation angle, up-down rotation angle, and tilt angle (i.e., facial posture) of the face corresponding to the largest face area in the superimposed order.

[0096] The guidance display unit 56 displays a guidance display screen 72 on the display unit 44, which is created by superimposing guidance information (guide frames 65 and various notifications) onto the base image 70. As a result, the display device 35 replaces the face area of ​​the captured image with the substitute image 5 and displays the guidance information superimposed on it.

[0097] Figures 5(A) to 5(C) show examples of the guidance display screen 72, respectively.

[0098] Figure 5(A) shows an example of a guidance display screen 72 that includes guidance information (text 74A: "Center") when the face area is far from the guide frame 65. Figure 5(B) shows an example of a guidance display screen 72 that includes guidance notification 74 (text 74A: "Get closer") when the face area is too small. Figure 5(C) shows an example of a guidance display screen 72 that includes guidance notification 74 (text 74A: "Face forward") when the face of the person being authenticated is not facing forward. In Figure 5(C), a blurred circle with superimposed shapes indicating the positions of the eyes and nose is used as the alternative image 5, and an example of a guidance display screen 72 is shown when the face corresponding to the largest face area in the superimposed order is facing left. In the guidance display screen 72 shown in Figure 5(C), the person being authenticated can understand the orientation of their face as recognized by the face authentication terminal 3 by the alternative image 5.

[0099] When the matching processing unit 57 detects that the face image in the captured image meets the predetermined authentication initiation conditions, it transmits the face image to the face authentication server 2 and instructs it to perform authentication. At this time, the matching processing unit 57 may also transmit the captured image and the position and size of the face image to be authenticated to the face authentication server 2 instead of the face image.

[0100] At this time, the matching processing unit 57 generates a base image 70 by superimposing the face regions of the captured image in the superposition order set by the superposition order setting unit 54, and replacing them with alternative images 5 generated by the alternative image generation unit 53, when the authentication process begins. Subsequently, the matching processing unit 57 displays an authentication execution screen 80 on the display unit 44, which superimposes text 76 indicating that face authentication is in progress onto the base image 70, until it obtains an authentication result from the face authentication server 2. Figure 6(A) shows an example (first example) of the authentication execution screen 80 displayed on the display unit 44 in this case.

[0101] In addition, the matching processing unit 57 may obtain the progress status of face matching from the face recognition server 2 and display an authentication execution screen 80 on the display unit 44, which superimposes text 77 indicating that face recognition is in progress and an image showing the progress status of face recognition (progress image 78) onto the base image 70. Figure 6(B) shows an example (second example) of the authentication execution screen 80 displayed on the display unit 44 in this case. In this case, as shown in Figure 6(B), the text 77 indicating that face recognition is in progress may include a numerical value or the like indicating the progress status (degree of progress).

[0102] The authentication initiation conditions may include at least one of three conditions: a condition relating to the position of the face in the captured image (hereinafter referred to as the position condition), a condition relating to the size of the face in the captured image (hereinafter referred to as the size condition), and a condition relating to the orientation of the face in the captured image (hereinafter referred to as the orientation condition).

[0103] Positional conditions may include various conditions related to the position of the face region, such as the face region being located within the guide frame 65 in the captured image, the face region remaining within the guide frame 65 for a predetermined time or longer, and the amount of change in the position of the face region (e.g., the amount of movement within the interval time) being below a predetermined threshold. At a minimum, the positional conditions include the face region being located within the guide frame 65 in the captured image. Other positional conditions may include the face region being maintained within the guide frame 65 for a predetermined time or longer, and the amount of change in its position during that predetermined time being below a predetermined threshold.

[0104] Size requirements may include the size of faces in the captured image being greater than or equal to a threshold.

[0105] The orientation condition may include that at least one of the face's left-right rotation angle (yaw angle), up-down rotation angle (pitch angle), and tilt angle (roll angle) is within a predetermined range. For example, the orientation condition may be that the face's left-right rotation angle is less than or equal to a predetermined first threshold, and the orientation angle is less than or equal to a predetermined second threshold.

[0106] If the authentication initiation conditions include three conditions: position, size, and orientation, the matching processing unit 57, when it determines that all three conditions are met, sends an image of the face region that satisfies those conditions to the face authentication server 2 and instructs it to perform face matching.

[0107] Upon receiving a command to perform face matching, the face recognition server 2 obtains facial features from the received face image, compares the obtained facial features with registered face information, and performs face recognition. For example, if the positional condition includes that the face region is located within the guide frame 65 in the captured image, an image of the face region located within the guide frame 65 is sent to the face recognition server 2. The face recognition server 2 obtains facial features from the received image of the face region (the image of the face region located inside the guide frame 65) and performs face recognition by comparing it with registered face information.

[0108] If the registered face information can be matched (i.e., the registered face image corresponding to the image of the face region can be extracted), the face authentication server 2 (specifically the processor 11 of the face authentication server 2) determines that face authentication was successful. If the registered face information cannot be matched (i.e., the registered face image corresponding to the image of the face region cannot be extracted), the face authentication server 2 (specifically the processor 11 of the face authentication server 2) determines that face authentication was unsuccessful. Once the face authentication server 2 obtains the authentication result (matching result), it transmits the authentication result to the face authentication terminal 3 (specifically the matching processing unit 57).

[0109] When the authentication result display unit 58 obtains an authentication result from the face authentication server 2, it displays a result display screen 84 on the display unit 44, which includes a notification indicating the authentication result (hereinafter referred to as the result notification 82). Specifically, when the image of the face region and the registered face information can be matched (i.e., authentication is successful), the authentication result display unit 58 displays a result display screen 84 on the display unit 44, which includes a result notification 82 indicating that the match was successful. When the image of the face region and the registered face information cannot be matched (i.e., authentication is unsuccessful), the authentication result display unit 58 displays a result display screen 84 on the display unit 44, which includes a result notification 82 indicating that the match could not be made.

[0110] Figures 7(A) to 7(D) show examples of the result display screen 84. In the examples shown in Figures 7(A) to 7(D), the authentication result display unit 58 displays the result display screen 84 on the display unit 44 by superimposing the result notification 82 onto the base image 70 generated at the start of the authentication process. Figures 7(A) to 7(C) show examples of the result display screen 84 when authentication is successful, and Figure 7(D) shows an example of the result display screen 84 when authentication fails.

[0111] Figure 7(A) shows an example in which a check mark 82A and text 82B ("Matching successful") which includes the name of the person being authenticated and indicates that facial recognition was successful are used as the result notification 82 indicating that facial recognition was successful. Figure 7(B) shows an example in which an illustration 82C that the person being authenticated can understand and text 82B ("Matching successful") which includes the name of the person being authenticated and indicates that facial recognition was successful are used as the result notification 82 indicating that facial recognition was successful. Figure 7(C) shows an example in which a symbol 82D (graphic) that the person being authenticated can understand and text 82B ("Matching successful") which includes the name of the person being authenticated and indicates that facial recognition was successful are used as the result notification 82 indicating that facial recognition was successful.

[0112] However, the name included in text 82B is not mandatory, and text 82B may only contain the success or failure of facial recognition. In that case, illustration 82C (symbol 82D) may be displayed instead of the name. Illustration 82C (symbol 82D) and the name may be those registered by the person being authenticated when registering with facial recognition system 1. Furthermore, from a privacy perspective, it is preferable that the name included in text 82B be a nickname or something similar rather than the real name of the person being authenticated.

[0113] Figure 7(D) shows an example where the result notification 82, indicating that face recognition failed, uses the text 82E ("Matching failed"). Figure 7(D) also shows an example of a result notification 82 issued when the face recognition terminal 3 obtains both the authentication result and the cause of failure. Figure 7(D) shows an example where the result notification 82 includes the text 82F and illustration 82G indicating the cause of face recognition failure (blurry image).

[0114] Next, the authentication terminal processing performed by the facial recognition terminal 3 of the facial recognition system 1 will be described. The authentication terminal processing includes two processes: a process for generating a base image 70 (hereinafter referred to as the base image generation process (ST100)) and a process related to the execution of facial recognition (hereinafter referred to as the authentication execution process (ST200)).

[0115] Below, we will first explain the details of the base image generation process (ST100). Figure 8 is a flowchart of the base image generation process (ST100). Note that the base image generation process (ST100) is performed at time intervals.

[0116] As shown in Figure 8, in the base image generation process (ST100), the processor 31 of the face recognition terminal 3 first acquires the captured image obtained by the imaging device 34, detects the faces in the captured image, and obtains the corresponding face region information 61 (ST101).

[0117] Once face detection is complete, the processor 31 of the face recognition terminal 3 acquires the face image of the detected face and updates the captured face information table 4 to match the captured image using the face region information 61 (ST102).

[0118] Specifically, the processor 31 of the face recognition terminal 3 determines whether the acquired face image is recorded in the captured face information table 4. If the acquired face image is not recorded, the face information acquisition unit 52 assigns a new face ID, associates the detected face image with the face region information 61, and adds it to the captured face information table 4.

[0119] If an acquired face image is recorded, the processor 31 of the face recognition terminal 3 updates the image capture face information table 4 using the newly acquired face region information 61. At this time, the processor 31 of the face recognition terminal 3 may also update various information recorded in the image capture face information table 4 (specifically, the amount of change in position and the amount of change in size) using information recorded in the image capture face information table 4 before the update (for example, the position of the face region and the size of the face region).

[0120] Furthermore, if the facial recognition terminal 3's processor 31 finds that the captured facial information table 4 contains a facial image that was not detected, it removes the information associated with the facial ID corresponding to that facial image from the captured facial information table 4.

[0121] Next, the processor 31 of the face recognition terminal 3 sets the superposition order using the captured face information table 4 (ST103). In this embodiment, the processor 31 of the face recognition terminal 3 sets the superposition order in ascending order of the size of the face regions recorded in the captured face information table 4. Alternatively, the processor 31 of the face recognition terminal 3 may set the superposition order in ascending order of the amount of change in the size of the face regions recorded in the captured face information table 4.

[0122] Once the superposition order is set, the processor 31 of the face recognition terminal 3 generates an alternative image 5 for each face region of the captured image to replace that face region (ST104). An alternative image 5 is set for each face region of the captured image (i.e., for each face ID corresponding to a face region). The alternative image 5 may be generated by a blurred circle containing the face region, as shown in Figure 4(A), or by an illustration containing the face region, as shown in Figure 4(B). Alternatively, as shown in Figure 4(C), the alternative image 5 may be generated by cutting out the corresponding face region and superimposing a stamp. In addition, as shown in Figure 4(D), the alternative image 5 may be generated by cutting out the corresponding face region and applying Gaussian blur, mosaic processing, or mask processing to that face region.

[0123] Once the generation of the alternative image 5 is complete, the processor 31 of the face recognition terminal 3 generates the base image 70 by superimposing the alternative image 5 according to the superposition order and replacing each face region of the captured image with the alternative image 5 (ST105). Once the generation of the base image 70 is complete, the processor 31 of the face recognition terminal 3 finishes the base image generation process (ST100).

[0124] Next, we will explain the details of the authentication execution process (ST200). Figure 9 is a flowchart showing the flow of the authentication execution process (ST200) in the facial recognition terminal 3. Figure 10 is a flowchart of the determination process that determines whether or not the authentication start conditions are met.

[0125] As shown in Figure 9, in the authentication execution process (ST200), the processor 31 of the face authentication terminal 3 first performs a determination process on each of the face regions recorded in the captured face information table 4 to determine whether or not the authentication start conditions are met (ST201).

[0126] As shown in Figure 10, in the determination process, the processor 31 of the face recognition terminal 3 refers to the captured face information table 4 and determines whether the size of the face region is greater than or equal to the corresponding threshold (ST301). If the size of the face region is less than the corresponding threshold (No in ST301), the processor 31 of the face recognition terminal 3 determines that the authentication start condition is not met.

[0127] If the size of the face region is greater than or equal to the corresponding threshold (Yes in ST301), the processor 31 of the face authentication terminal 3 refers to the captured face information table 4 and determines whether the left-right rotation angle of the face corresponding to that face region is less than or equal to the corresponding threshold (ST302). If the left-right rotation angle is greater than the corresponding threshold (No in ST302), the processor 31 of the face authentication terminal 3 determines that the authentication start condition is not met.

[0128] If the left-right rotation angle is less than or equal to the corresponding threshold (Yes in ST302), the processor 31 of the face recognition terminal 3 refers to the imaged face information table 4 and determines whether the up-down rotation angle of the face corresponding to that face region is less than or equal to the corresponding threshold (ST303). If the up-down rotation angle is greater than the corresponding threshold (No in ST303), the processor 31 of the face recognition terminal 3 determines that the authentication start conditions are not met.

[0129] If the vertical rotation angle is less than or equal to the corresponding threshold (Yes in ST303), the processor 31 of the face recognition terminal 3 refers to the imaged face information table 4 and determines whether the tilt angle of the face corresponding to that face region is less than or equal to the corresponding threshold (ST304). If the tilt angle is greater than the corresponding threshold (No in ST304), the processor 31 of the face recognition terminal 3 determines that the authentication start condition is not met.

[0130] If the tilt angle is less than or equal to the corresponding threshold (Yes in ST304), the processor 31 of the face recognition terminal 3 refers to the imaged face information table 4 and determines whether the amount of change in the position of the face region, i.e., the amount of movement, is less than or equal to the corresponding threshold (ST305). If the amount of movement is greater than the corresponding threshold (No in ST305), the processor 31 of the face recognition terminal 3 determines that the authentication start condition is not met.

[0131] When the amount of movement is less than or equal to the corresponding threshold (Yes in ST305), the processor 31 of the face recognition terminal 3 refers to the captured face information table 4, obtains the static time of the face region, and determines whether the static time is longer than the corresponding threshold (ST306). If the static time is greater than (longer than) the corresponding threshold (Yes in ST306), the processor 31 of the face recognition terminal 3 determines that the authentication start condition is met. On the other hand, if the static time is less than or equal to the corresponding threshold (No in ST306), the processor 31 of the face recognition terminal 3 determines that the authentication start condition is not met.

[0132] As shown in Figure 9, the processor 31 of the facial recognition terminal 3 performs a determination process on each of the facial regions recorded in the captured facial information table 4 and obtains a determination result (ST201), and then determines whether or not there is a facial region that satisfies the authentication start conditions (ST202).

[0133] If no facial region satisfies the authentication initiation conditions, that is, if it is determined that the authentication initiation conditions are not met for any facial region (No in S202), the processor 31 of the facial recognition terminal 3 uses the captured facial information table 4 to generate guidance information (guide frame 65 and text). Subsequently, the processor 31 of the facial recognition terminal 3 displays a guidance display screen 72 (see Figures 5(A) to (D)) on the display device 35, with the guidance information superimposed on the base image 70 (ST203).

[0134] Subsequently, the processor 31 of the facial recognition terminal 3 performs a determination process for each of the facial regions recorded in the captured facial information table 4, for example, after the interval time has elapsed (or after a time equal to or greater than the interval time has elapsed) (ST201).

[0135] If at least one facial region satisfies the authentication initiation conditions (Yes in S202), the processor 31 of the facial authentication terminal 3 acquires an image of the facial region that satisfies the authentication initiation conditions (facial image) from the captured image, sends the acquired image to the facial authentication server 2, and instructs the facial authentication server 2 to perform facial authentication (ST204).

[0136] Once the facial recognition execution command is completed, the processor 31 of the facial recognition terminal 3 displays an authentication execution screen 80 (see Figure 6(A)) on the display device 35, which is an authentication execution screen 80 (see Figure 6(A)) in which text 76 indicating that facial recognition is in progress is superimposed on the base image 70 at the time of the execution command (when authentication is performed) (ST205).

[0137] Once the display of the authentication execution screen 80 (ST205) is complete, the processor 31 of the facial recognition terminal 3 determines whether or not it has obtained an authentication result from the facial recognition server 2 (ST206). If it has not obtained a result (No in ST206), the processor 31 of the facial recognition terminal 3 waits until it obtains an authentication result from the facial recognition server 2 (ST206).

[0138] The processor 31 of the facial recognition terminal 3 may, after the facial recognition execution instruction is completed and until the authentication result is obtained, obtain the matching progress from the facial recognition server 2, generate a progress image 78 corresponding to the progress information, and display an authentication execution screen 80 (see Figure 6(B)) on the display device 35 by superimposing the generated progress image 78 on the base image 70 at the time of execution instruction (when authentication is performed).

[0139] If the authentication result has been obtained from the facial recognition server 2 (Yes in ST206), the processor 31 of the facial recognition terminal 3 displays the result display screen 84 (see Figures 7(A) to (D)) on the display device 35 (ST207). Once the display of the result display screen 84 is complete, the processor 31 of the facial recognition terminal 3 finishes the authentication execution process (ST200).

[0140] Although not shown in Figure 9, after the display of the results screen 84 is complete, if facial recognition is successful at the facial recognition server 2, the processor 31 of the facial recognition terminal 3 may execute processing for successful facial recognition (for example, processing to unlock the door for entering the corporate facility for a predetermined period of time).

[0141] Next, we will explain the processing flow in the facial recognition system 1 until the facial recognition result is displayed, and the screen display on the display device 35 of the facial recognition terminal 3 until the facial recognition result is displayed. Figure 11 is a sequence diagram showing an example of the operation of the facial recognition server 2 and the facial recognition terminal 3 from the start of facial recognition until the result is displayed. Figures 12 and 13 show the first and second examples of the screen display on the display device 35 until the facial recognition result is displayed, respectively.

[0142] As shown in Figure 11, the processor 31 of the face recognition terminal 3 performs base image generation processing (ST100) at intervals, including face detection and acquisition of face region information 61 (ST101), updating of the captured face information table 4 (ST102), setting of the superposition order (ST103), generation of alternative image 5 (ST104), and generation of base image 70 (ST105).

[0143] Furthermore, the processor 31 of the face recognition terminal 3 performs authentication execution processing (ST200) at intervals. In the authentication execution processing (ST200), the processor 31 of the face recognition terminal 3 determines whether each face region in the captured image (i.e., the face region recorded in the captured face image table) satisfies the authentication start conditions (ST202). If none of the face regions satisfy the authentication start conditions, the processor 31 displays a guidance display screen 72 on the display device 35, which has a guidance notification 74 superimposed on the base image 70.

[0144] Figures 12(A) to (C) show examples of guidance screens 72 displayed on the display device 35 of the face recognition terminal 3 until the authentication start conditions are met. In Figures 12(A) and (B), since the detected face area is far from the guide frame 65, the text 74A "Center" is generated as guidance information and displayed on the screen of the display device 35. In Figure 12(C), since the detected face area is smaller than the threshold, the text 74A "Get closer" is generated as guidance information and displayed on the screen of the display device 35.

[0145] As shown in Figure 11, when at least one facial region satisfies the authentication initiation conditions, an image of the facial region satisfying the authentication initiation conditions is sent to the facial authentication server 2, and the server is instructed to perform facial authentication (ST204). Once the instruction to perform facial authentication is complete, the processor 31 of the facial authentication terminal 3 displays the authentication execution screen 80 on the display device 35 (ST205).

[0146] Figure 12(D) shows an example of the authentication execution screen 80. As shown in Figure 12(D), after the instruction to perform facial recognition is completed, and until the facial recognition terminal 3 obtains the authentication result, the display device 35 of the facial recognition terminal 3 displays the authentication execution screen 80, in which text 76 indicating that facial recognition is in progress is superimposed on the base image 70 at the start of the authentication execution.

[0147] As shown in Figure 11, when the face recognition server 2 receives a command to perform face recognition, it obtains face features from the image of the face region transmitted from the face recognition terminal 3 (ST401). Subsequently, the face recognition server 2 compares the obtained face features with the registered face information (ST402) and transmits the authentication result (matching result) to the face recognition terminal 3 (ST403).

[0148] When the facial recognition terminal 3 receives the authentication result, it displays the result display screen 84 (ST207). Figure 12(E) shows the result display screen 84. The result display screen 84 shown in Figure 12(E) includes a check mark 82A indicating that facial recognition was successful, and text 82B containing the name of the person being authenticated.

[0149] As shown in Figures 12(A) to (E), the display device 35 of the facial recognition terminal 3 displays various information superimposed on the base image 70. Since the base image 70 is generated by replacing the face region in the captured image with a substitute image 5, the face image is not displayed on the display device 35 until facial recognition is completed. Therefore, even if someone other than the person being authenticated looks at the screen of the display device 35 during facial recognition, they will not be able to see the face image. Thus, it is possible to prevent people other than the person being authenticated from seeing the face image during facial recognition.

[0150] Figures 13(A) to 13(E) show the screen display of the display device 35 of the face recognition terminal 3 when multiple faces are detected in the captured image. In Figures 13(A) to 13(E), as in Figures 12(A) to 12(E), various information is superimposed on the base image 70 and displayed on the display device 35 of the face recognition terminal 3.

[0151] However, in the base image generation process (ST100), the superposition order is set based on the captured face information table 4, and the base image 70 is generated by superimposing the alternative images 5 based on that superposition order. Therefore, as shown in Figure 13(C), the alternative images 5 of larger face regions are superimposed on the alternative images 5 corresponding to smaller face regions and displayed on the display device 35. This allows the corresponding alternative images 5 to be superimposed and displayed on the display device 35 in the order in which they should be authenticated, so that users of the face authentication system 1 can easily identify the person to be authenticated by the display device 35. If the superposition order is based on the amount of change in position, the alternative images 5 of face regions with a large change in size are superimposed on the alternative images 5 of face regions with a small change in size and displayed on the display device 35. This allows the alternative image 5 of a person approaching the imaging device 34 and attempting to perform face authentication to be superimposed on the alternative image 5 of another person.

[0152] In the above embodiment, an example was described in which the facial recognition system 1 is configured by a facial recognition server 2 and a facial recognition terminal 3. However, the functions of the facial recognition server 2 and the functions of the facial recognition terminal 3 may be realized by a single device (facial recognition device).

[0153] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0154] The facial recognition terminal, facial recognition system, control method, and control program related to this disclosure are useful as facial recognition terminals, facial recognition systems, control methods, and control programs for preventing persons other than the person being authenticated from viewing facial images during facial recognition. [Explanation of Symbols]

[0155] 1: Facial recognition system 2: Face recognition server 3: Facial recognition terminal 4: Image-captured facial information table 5: Alternative image 7: Communication Network 11: Processor 12: Memory 13: Storage 14: Communication equipment 21: Communications Department 22: Storage section 23: Control Unit 25: Registered facial information database 26: Image acquisition unit 27: Feature detection unit 28: Facial Recognition Department 31: Processor 32: Memory 33: Storage 34: Imaging device 34A: Camera 35:Display device 35A: Display 36: Communication equipment 41: Communications Department 42: Storage section 43: Imaging Unit 44: Display section 45: Control Unit 51: Face detection unit 52: Facial Information Acquisition Unit 53: Alternative image generation section 54: Superposition Order Setting Section 55: Base image generation unit 56: Information display section 57: Verification Processing Unit 58: Authentication Result Display Section 61: Facial area information 62: Facial Information 65: Guide frame 70: Base image 72: Information display screen 74: Notice 74A: Text 76: Text 77: Text 78: Progress image 80: Authentication execution screen 82:Result notification 82A: Checkmark 82B: Text 82C: Illustration 82E: Text 82F: Text 82G: Illustration 84:Result display screen

Claims

1. It is a facial recognition terminal, An imaging device for capturing images of the person to be authenticated, A display device that shows a screen to the person to be authenticated, The system includes a processor that controls the screen display of the display device based on the captured image acquired by the imaging device, The aforementioned processor, A person's face is detected from the captured image acquired by the aforementioned imaging device. A face recognition terminal that displays the region of the captured image containing the detected face on the display device in place of a substitute image.

2. The face recognition terminal according to claim 1, wherein the processor detects the position of a person's face from the captured image, and places the substitute image at the detected face position and displays it on the display device.

3. The face authentication terminal according to claim 1 or 2, wherein the processor superimposes a guide frame for guiding the person to be authenticated to the appropriate imaging position onto a base image obtained by replacing the region containing the detected face in the captured image with the alternative image, and displays it on the display device.

4. The facial recognition terminal according to claim 3, wherein the processor obtains the result of matching an image of a face located inside the guide frame with pre-stored facial information.

5. The face authentication terminal according to claim 1, wherein the processor displays a progress image on the display device indicating the progress of the matching when matching a face image of a face extracted from the captured image with pre-stored face information.

6. The face recognition terminal according to claim 1, wherein the processor acquires the orientation of a face detected from the captured image and generates the alternative image according to the acquired face orientation.

7. The face authentication terminal according to claim 1, wherein the processor causes the base image obtained by replacing the region of the captured image containing the detected face with the substitute image, to superimpose a notification on the display device based on at least one of the position, size, posture, change in position, and change in size of the face detected in the captured image.

8. The face recognition terminal according to claim 1, wherein, when the processor detects multiple faces of persons from the captured image acquired by the imaging device, it causes the region of the captured image containing the faces of persons to be displayed on the display device in place of the corresponding substitute image.

9. The face recognition terminal according to claim 1, wherein, when the processor detects multiple faces from the captured images acquired by the imaging device, the processor superimposes the substitute images such that the substitute image of the larger face overlaps the substitute image of the smaller face, and displays them on the display device.

10. The face recognition terminal according to claim 1, wherein, when the processor detects multiple faces from the captured image acquired by the imaging device, the processor superimposes the substitute images on the display device such that the substitute images of faces with a large change in size overlap the substitute images of faces with a small change in size.

11. The face authentication terminal according to claim 1, wherein the processor, when it fails to match a face image of a face extracted from the captured image with pre-stored face information, displays a notification on the display device indicating the cause of the failure.

12. It is a facial recognition system, An imaging device for capturing images of the person to be authenticated, A display device that shows a screen to the person to be authenticated, The system includes a processor that controls the screen display of the display device based on the captured image acquired by the imaging device, The aforementioned processor, A person's face is detected from the captured image acquired by the aforementioned imaging device. A face recognition system that displays the region of the captured image containing the detected face on the display device in place of a substitute image.

13. A method for controlling guidance displays related to facial recognition, An imaging device for capturing images of the person to be authenticated, A display device that shows a screen to the person to be authenticated, A facial recognition terminal having a processor that controls the screen display of the display device based on the captured image acquired by the imaging device, is executed. The aforementioned processor, A person's face is detected from the captured image acquired by the aforementioned imaging device. A control method for displaying on the display device the region of the captured image that contains the detected face, in place of a substitute image.

14. A control program for displaying guidance related to facial recognition, An imaging device for capturing images of the person to be authenticated, A display device that shows a screen to the person to be authenticated, A facial recognition terminal having a processor that controls the screen display of the display device based on the captured image acquired by the imaging device, is executed. The aforementioned processor, A person's face is detected from the captured image acquired by the aforementioned imaging device. A control program that causes the region of the captured image containing the detected face to be displayed on the display device in place of a substitute image.

Images