Facial recognition device, facial recognition method, and computer program

Abstract

To provide a facial recognition device that can efficiently detect impersonation. [Solution] A facial recognition device comprising: an image acquisition means for acquiring an image of a subject; a face detection means for detecting a face from the image; a facial recognition means for identifying a person by matching the face; a reference image holding means for holding a reference face image that serves as a comparison standard for the face; and a deviation determination means for determining that a face is a fake when it deviates from a predetermined determination standard obtained from the reference face image.

Description

Technical Field

[0001] Relates to a face authentication device, a face authentication method, a computer program, etc.

Background Art

[0002] In an entrance and exit system that unlocks a door using face authentication technology, for example, a camera such as a tablet PC captures the face of the person entering the room, the authentication result is displayed on the display, and the door lock is unlocked.

[0003] When using face authentication technology in this way, there is a problem of fraudulently impersonating others by presenting a photo of another person's face to the camera. As solutions to these problems, for example, in Patent Document 1, when the biometric information input at the time of authentication exactly matches the biometric information included in the system, the authentication is made to fail, thereby determining repeated impersonation acts using photos or the like.

Prior Art Documents

Patent Documents

[0004] [[ID=二十五]] [[ID=二十六]] [[ID=二十七]]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] [[ID=三十八]] [[ID=三十九]]However, in the prior art disclosed in the above-mentioned patent document, in the prior art method, it is impossible to determine impersonation without continuously having a large amount of authentication history. Also, it can only be determined if it is an impersonation act using the same image as the image once used. [[ID=四十]] [[ID=四十一]]

[0006] [[ID=四十二]] [[ID=四十三]]Therefore, one of the objectives of the present invention is to provide a face authentication device capable of efficiently determining impersonation in order to solve the problems of the prior art. [[ID=四十四]] [[ID=四十五]]

Means for Solving the Problems

[0007] A facial recognition device according to an embodiment of the present invention, An image acquisition means for acquiring an image of the subject, A face detection means for detecting faces from the aforementioned image, A facial recognition means that identifies a person by matching the aforementioned face, A reference image holding means for holding a reference face image that serves as a comparison standard for the aforementioned face, A deviation determination means that determines that the face is an impersonation if it deviates from predetermined determination criteria obtained from the reference face image, It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a facial recognition device that can efficiently detect impersonation. [Brief explanation of the drawing]

[0009] [Figure 1] (A) is a hardware block diagram showing an example of the hardware configuration of the facial recognition device 100 according to Embodiment 1 of the present invention, and (B) is a functional block diagram showing an example of the functional configuration of the facial recognition device 100 according to Embodiment 1. [Figure 2] This flowchart shows an example of a processing method for facial recognition according to Embodiment 1 of the present invention. [Figure 3] (A) and (B) are diagrams illustrating an example of the display of Embodiment 1 of the present invention. [Figure 4] This flowchart shows an example of the process for determining deviations in a facial recognition method according to Embodiment 1 of the present invention. [Figure 5] This figure illustrates an example of an AI model for a score learned from past history change trends in the facial recognition method of Embodiment 1 of the present invention. [Figure 6] This flowchart shows an example of the processing of the facial recognition method according to Embodiment 2 of the present invention. [Figure 7] This flowchart shows an example of the deviation detection process for the facial recognition method of Embodiment 2 of the present invention. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0011] <Embodiment 1> Figure 1(A) is a hardware block diagram showing an example of the hardware configuration of a facial recognition device 100 according to Embodiment 1 of the present invention. 101 is a CPU (Central Processing Unit) as a computer, and functions as a control means that controls the operation of each part of the entire device based on a computer program stored in memory as a storage medium.

[0012] The facial recognition device 100 is equipped with memory, which includes program memory and data memory. The ROM (Read-Only Memory) 102 is the program memory and stores a computer program for CPU control, which includes various processing procedures described later.

[0013] The RAM (Random Access Memory) 103 is a data memory and includes a work area for the CPU 101's program, a data storage area for error handling, and a load area for the control computer program.

[0014] Alternatively, program memory may be realized by loading a computer program into RAM 103 from an external storage device connected to the facial recognition device 100. HDD 104 is a hard disk for storing multiple electronic data and computer programs according to this embodiment.

[0015] Furthermore, an external storage device may be used to perform the same function as the HDD104. Here, the external storage device can be implemented, for example, by a media (recording medium) and an external storage drive to enable access to that media.

[0016] As such media, for example, CD-ROM, DVD, USB memory, MO, flash memory, etc. are used. Further, the external storage device may be a server device connected via a network or the like.

[0017] The input device 105 is a device for capturing the user's operation information into the face authentication device 100, and includes, for example, pointing devices such as a mouse and a touch panel, a joystick, a keyboard, and the like.

[0018] The output device 106 is a device for performing display output. The communication interface (I / F) 107 performs bidirectional communication, wired or wireless, with other information processing devices, communication devices, external storage devices, etc. by a known communication technique.

[0019] In the present embodiment, the communication I / F 107 is connected to an imaging device 108 which is a network camera via a LAN, and receives the captured video in real time. The imaging device 108 is, for example, a network camera installed in a state connected to a network, and captures a video with the user's face within the angle of view.

[0020] Note that the imaging device 108 is not limited to a network camera, and may be a web camera incorporated in this face authentication device. Alternatively, the imaging device 108 may be a camera connected by USB, a smartphone with a camera, a single-lens reflex camera, a video camera, an in-vehicle camera, or the like.

[0021] The external system 109 is an external system such as an entrance / exit management system that receives an instruction according to the result of the processing performed by the face authentication device 100, and executes various processes related to unlocking and locking an electric lock, for example. Note that 110 is a system bus.

[0022] Figure 1(B) is a functional block diagram showing an example of the functional configuration of the facial recognition device 100 according to Embodiment 1. The facial recognition device 100 includes an image acquisition unit 111, a face detection unit 112, a deviation determination unit 113, a storage unit 114, a display unit 115, a communication unit 116, a facial recognition unit 117, a shooting status acquisition unit 118, an expression estimation unit 119, an age estimation unit 120, a biometric determination unit 121, and a misrecognition response unit 122.

[0023] These functional blocks are realized by the CPU 101 loading the programs stored in ROM 102 into RAM 103 and executing processes according to the flowcharts described later. The memory unit 114 is a functional block of RAM 103.

[0024] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, each functional block shown in Figure 1(B) does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0025] The image acquisition unit 111 functions as an image acquisition means for acquiring images of subjects captured by the imaging device 108. The face detection unit 112 functions as a face detection means for detecting the user's face from the image acquired by the image acquisition unit 111. The result of face detection shall include at least face position information.

[0026] The deviation determination unit 113 determines whether a face deviates from a predetermined determination criterion, using the trends of various information obtained from reference face images (registered face images and authentication history images) stored in the memory unit 114 as predetermined determination criteria. The information used for the determination is obtained by the face recognition unit 117, the shooting situation acquisition unit 118, the facial expression estimation unit 119, the age estimation unit 120, the biometric determination unit 121, etc.

[0027] Furthermore, the deviation detection unit 113 functions as a deviation detection means for determining impersonation by determining whether the face deviates from predetermined criteria obtained from reference face images (registered face images or authentication history images).

[0028] The memory unit 114 functions as a reference image holding means that performs a reference image holding step when the face recognition unit 117 performs face matching, by holding a reference face image (registered face image or authentication history) that serves as the basis for comparing faces. The authentication history includes images.

[0029] The display unit 115 displays various information, such as whether authentication was successful or not. The communication unit 116 transmits information to the external system 109.

[0030] The facial recognition unit 117 functions as a facial recognition means that identifies a person by matching the face detected by the facial detection unit 112, and extracts facial features from the face detected by the facial detection unit 112. It also compares the extracted facial features with the facial features of registered images stored in the storage unit 114 to determine whether or not they are the same person.

[0031] The matching result is output as a similarity score, and if it exceeds a predetermined threshold, it is determined to be the same person. The output similarity score is also used by the deviation determination unit 113 to determine deviations. The reference image holding means holds the authentication history by the face recognition means. The reference face image includes registered face images used by the face recognition means and history images included in the authentication history.

[0032] The shooting status acquisition unit 118 acquires information about the shooting status of the face image used by the deviation determination unit 113. The shooting status acquired includes lighting conditions, exposure settings, etc., which affect the brightness of the face, blur or blur on the face, clothing, etc., and a score value is obtained for each of these.

[0033] The facial expression estimation unit 119 acquires facial expression information from the face image used by the deviation determination unit 113. The acquired facial expression is, for example, a smile, and a score of the degree of smile is obtained.

[0034] The age estimation unit 120 estimates the age of the face image used in the deviation determination unit 113. A numerical value representing the age is obtained as a score.

[0035] The biometric determination unit 121 performs biometric determination on the facial image used in the deviation determination unit 113. The biometric determination result is output as a biometric determination score.

[0036] The misauthentication response unit 122 detects misjudgments based on the frequency with which the deviation determination unit 113 has determined a deviation, and takes predetermined action.

[0037] Figure 2 is a flowchart showing an example of the processing of a facial recognition method according to Embodiment 1 of the present invention, illustrating the processing procedure when facial recognition is performed. The CPU 101 and other components within the facial recognition device 100 execute a computer program stored in memory, thereby sequentially performing each step of the flowchart in Figure 2.

[0038] One possible method of committing fraud by impersonating someone else is to hold an ID card with a facial image or a facial image displayed on a smartphone over the facial recognition device 100 and have it photographed by the imaging device 108.

[0039] In step S200, the image acquisition unit 111 acquires an image of the user captured by the imaging device 108. Here, step S200 functions as an image acquisition step to acquire an image of the subject.

[0040] In step S201, the face detection unit 112 performs face detection on the image acquired in step S200. The face detection result acquired in this embodiment is assumed to include at least the position, width, and height of the center of the face within the image. Step S201 functions as a face detection step to detect a face from an image.

[0041] In this embodiment, we describe an example where only one person is detected, but the result is not limited to one person. Also, if two or more people are detected, the authentication may be made to fail and the process terminated, so that each person is authenticated one by one.

[0042] In step S202, the facial recognition unit 117 performs facial matching processing to determine the identity of the user. First, it extracts features from the face acquired by the face detection unit 112. The extracted facial features are compared with the registered user images stored in the storage unit 114, and a similarity score is output for each registered user image.

[0043] A predetermined threshold is set for the similarity score, and if the score exceeds the threshold, the user is identified as a registered person. If the similarity scores of multiple people exceed the threshold, the person with the highest score is identified as the user. Here, step S202 functions as a facial recognition step that matches faces to identify individuals.

[0044] In step S203, it is determined whether or not the person was identified through facial recognition in step S202. If the person was identified, the process proceeds to step S204; otherwise, the process proceeds to step S210.

[0045] In step S204, the system retrieves registered images from the storage unit 114 for use in the deviation detection unit 113, and in step S205, it retrieves authentication history from the storage unit 114 for use in the deviation detection unit 113. If multiple imaging devices 108 are connected and there is a sufficiently large amount of authentication history to retrieve, the system may narrow it down to the same authentication history as the imaging device 108 used in step S200.

[0046] In step S206, the deviation determination unit 113 performs a deviation determination process on the user's face detected in step S201. Here, step S206 functions as a deviation determination step in which it is determined that an impersonation has occurred if the face deviates from predetermined determination criteria obtained from the reference face image.

[0047] The details of this step will be described later using the flowchart in Figure 4. If the result of processing in this step is determined to be impersonation, proceed to step S208; if it is determined not to be impersonation, proceed to step S207.

[0048] In step S207, processing is performed upon successful authentication. Specifically, for example, the display unit 115 displays a message indicating that authentication was successful, and the face image acquired in step S201 is registered in the authentication history as a face that was successfully authenticated.

[0049] Figures 3(A) and 3(B) illustrate an example of the display of Embodiment 1 of the present invention. As shown in the example in Figure 3(A), in step S207, the name of the person corresponding to the registered image is displayed along with the message "Authentication OK".

[0050] Furthermore, in step S207, in order to operate in cooperation with the external system 109, the communication unit 116 sends an instruction to the external system 109 if authentication is successful in this step. For example, in the case of an access control system, an instruction to unlock is sent in step S207.

[0051] In step S208, the misauthentication response unit 122 performs a predetermined misauthentication response process in case the deviation determination unit 113 determines that there is a deviation. That is, if the misauthentication response unit 122 determines in step S206 that it is an impersonation attempt, it accepts authentication by another method (second authentication means) such as an ID card, vein authentication, or passcode input.

[0052] Furthermore, as part of the false authentication handling process in step S208, if the frequency of authentication failures reaches a predetermined condition, the administrator may be notified that there may be a problem with the imaging environment, such as the imaging device 108, or that there is a possibility of impersonation.

[0053] In other words, the system may have a notification means that provides notification when the frequency with which the deviation determination means determines a deviation exceeds a predetermined level. In this case, in addition to providing notification, the processing of steps S204, S205, and S206 may be temporarily suspended.

[0054] In step S209, the misauthentication response unit 122 determines that if authentication by another method (second authentication means) is successful in the process of step S209, it proceeds to step S207; otherwise, it proceeds to step S210.

[0055] Thus, the facial recognition device of this embodiment includes a second authentication means, and when the deviation determination means determines that a deviation has occurred, the second authentication means determines whether or not a deviation has occurred in a manner different from that of the facial recognition means. If authentication is performed by the second authentication means, it is determined that there has been no deviation.

[0056] Furthermore, by proceeding to step S207, authentication is considered successful, and the authentication history (including the image taken at that time) is stored in the authentication history. This history information will then be used for authentication in subsequent attempts. As a result, even if you are mistakenly identified as an impersonator due to temporary changes in appearance such as hairstyle, makeup, or tanning, the likelihood of being identified as not an impersonator in subsequent attempts increases.

[0057] In step S210, as a process for handling authentication failure, the display unit 115 displays a message indicating that authentication failed. At that time, as shown in the example display in Figure 3(B), it is also possible to display a message indicating that although facial recognition with the person corresponding to the registered image was successful, authentication failed because it was determined to be a spoof by the judgment in step S206. In the example in Figure 3(B), the message indicating that authentication failed due to being determined to be a spoof is displayed as "Spoofing Judgment Result: NG".

[0058] Furthermore, in order to operate in cooperation with the external system 109 even in the event of authentication failure, the communication unit 116 should send an instruction to the external system 109 in this step indicating what to do if authentication fails. This would, for example, maintain the lock on a door or gate.

[0059] Figure 4 is a flowchart showing an example of the deviation detection process of the facial recognition method according to Embodiment 1 of the present invention, and shows a detailed example of the deviation detection process shown in step S206. Note that the operation of each step in the flowchart of Figure 4 is performed sequentially by the CPU 101 and other components of the facial recognition device 100, which act as a computer, executing a computer program stored in memory.

[0060] Furthermore, the registered image acquired in step S204 does not necessarily have to be used as the reference image for determining deviations; the deviation determination shown in Figure 4 can be performed using only the image included in the authentication history acquired in step S205.

[0061] In step S400, the shooting status acquisition unit 118 calculates a score related to the shooting status of the face detected in step S201. When fraudulent activity such as impersonation occurs, the shooting status of the image fraudulently acquired in step S200 may differ significantly from the shooting status of the registered image acquired in step S204.

[0062] Therefore, this information is used to determine deviations. The shooting conditions calculated by the shooting conditions acquisition unit 118 include the brightness due to lighting conditions and exposure settings of the face area included in the image, the degree of blur or out-of-focus, etc., and each of these is expressed numerically.

[0063] For example, for brightness scores, the lowest brightness is set to 0, and the higher the score, the higher the brightness, with a maximum of 100. Similarly, for the degree of blur, the lowest degree of blur is set to 0, and the higher the score, the greater the degree of blur, with a maximum of 100.

[0064] The brightness and degree of blur or out-of-focus of the face detected in step S201 are added together to form the image shooting condition score Ss calculated in step S201. In addition to the conditions mentioned above, other conditions such as clothing may also be added to the shooting condition score.

[0065] When dealing with clothing, the clothing is classified using image analysis, and a predetermined numerical score is assigned to each item. For example, collarless clothing is 0, collared clothing is 25, clothing with a tie is 50, and clothing with a tie and jacket is 75.

[0066] Alternatively, a numerical value representing the clothing can be calculated from the average color of the chest area of ​​the image. Note that the judgment based on the shooting conditions in step S400 can be omitted. In that case, the judgment based on the shooting conditions is also omitted when determining deviation in step S406.

[0067] In step S401, the age estimation unit 120 estimates and scores the age of the face detected in step S201. The estimated age of the face detected in step S201 should change gradually. If there is a sudden decrease in age, or if the estimated age is younger than past history or registered images, it may indicate the theft of images from the web or social media.

[0068] Therefore, this information is used to determine deviations. The age estimation unit 120 calculates the estimated age for each face included in the image. The score for the estimated age can be simply the estimated age itself, which is then used as the score Sa.

[0069] The score Sa is set to 0 for age 0, with a maximum of 100, and can be rounded to 100 if the age exceeds 100. Step S401 can be omitted. In that case, the age estimation judgment is also omitted when determining the deviation in step S406.

[0070] In step S402, the facial expression estimation unit 119 estimates and scores the facial expression of the face detected in step S201. If a person who normally performs facial recognition with a neutral expression is recognized with a smile, it raises suspicion of image theft from the web or social media.

[0071] Therefore, this information is used to determine deviations. The facial expression estimation unit 119 outputs a score indicating the degree of smiling in the faces included in the image, for example, with a neutral expression set to 0 and an upper limit of 100, where a larger smile results in a higher score.

[0072] The output from facial expression estimation is denoted as the facial expression estimation score Sf. Furthermore, the facial expressions handled are not limited to smiles. Other expressions such as anger, sadness, and surprise may also be scored and handled. Note that step S402 can be omitted. In that case, the judgment based on facial expression estimation is also omitted when determining deviation in step S406.

[0073] In step S403, the biometric detection unit 121 performs biometric detection on the face detected in step S201 and assigns a score. It is known that, given the same conditions, the biometric detection score is usually a certain value for each individual, and if the same person undergoes biometric detection in the same location as usual, it is highly likely that the results will be very similar each time.

[0074] Generally, rather than simply determining whether or not something is a living being using biometric detection, which can vary greatly from person to person, standardizing the shooting environment and the conditions of the person makes it easier to detect subtle differences in trends, thus enabling the detection of suspected image theft on the web and social media.

[0075] Therefore, this information is used to determine deviations. The output of the biological determination unit 121 is a determination score that represents biological characteristics, for example, with a minimum value of 0 and a maximum value of 100, so that the higher the biological characteristics, the higher the score.

[0076] For the face detected in step S201, the system determines its bio-likeness based on comparisons with the color and texture of the face on the ID card or smartphone used for impersonation, and outputs a judgment score. The output for bio-detection is denoted as Sl.

[0077] The method of biometric detection is not limited to this method. For example, biometric responses such as blinking, changes in facial angle, or pulse rate in the image may be detected and scored. Step S403 can also be omitted. In that case, the judgment based on biometric detection will also be omitted when determining deviation in step S406.

[0078] In step S404, the facial recognition unit 117 performs facial recognition on the face detected in step S201 using the registered facial image and assigns a score. If the same person performs facial recognition in the same location as usual, there is a high probability that the facial recognition results will be very similar each time.

[0079] Even if the faces are those of the same person, there is a high probability that subtle differences in characteristics between the actual face and the face on the ID card or smartphone used for impersonation can be detected, making it possible to detect suspected theft of images from the web or social media.

[0080] The facial recognition unit 117 outputs a list of similarity scores between faces in an image and registered facial images. The similarity score with the highest score among the registered images is designated as Ssim.

[0081] The similarity score Ssim is set with a minimum value of 0 and a maximum value of 100, so that a higher similarity results in a higher score. Step S404 can be omitted. In that case, the facial recognition judgment is also omitted when determining the deviation in step S406.

[0082] In step S405, the registered images obtained in step S204 and the authentication history images obtained in step S205 are filtered by user. Specifically, only images related to the person identified by facial recognition in step S202 are filtered. Note that this step may be omitted, and all information obtained in steps S204 and S205 may be used for the determination.

[0083] In step S406, the deviation determination unit 113 uses the scores obtained in steps S400 to S404 to determine if the image acquired in step S201 has deviated from the given criteria. Specifically, it determines whether the image deviates from a predetermined reference score range (determination criteria) based on the difference between these score values ​​and the scores obtained from the images narrowed down in step S405.

[0084] For the images narrowed down in step S405, the same processing as in steps S400 to S404 is performed on each, and the shooting conditions, age estimation result, facial expression estimation result, biometric determination result, and face recognition result are scored and used. Thus, in this embodiment, the deviation determination means narrows down to the reference face image of the person identified by the face recognition means and then determines whether it deviates from a predetermined determination criterion.

[0085] Furthermore, instead of scoring all of these factors as predetermined criteria, predetermined criteria may be generated based on at least one of the following: shooting conditions, age estimation results, facial expression estimation results, biometric determination results, and facial recognition results.

[0086] Furthermore, in the case of registered images, these processes may be performed at the time of registration and stored in the storage unit 114 along with the image. In the case of historical images, the results of the processes in steps S400 to S404 may be stored in the storage unit 114 along with the authentication result. The upper and lower limits of the reference score range for each judgment are determined from the distribution of these scores.

[0087] Regarding the shooting condition score Ss, if the lower limit of the reference score range for determining the shooting condition is Ls and the upper limit is Hs, then it is determined that the score deviates from the reference score range unless the following equation 1 is satisfied. Ls≦Ss≦Hs (Formula 1)

[0088] For the age score Sa, if the lower limit of the reference score range for age determination is La and the upper limit is Ha, then it is determined that the score is outside the reference score range unless the following equation 2 is satisfied. La≦Sa≦Ha...(Formula 2)

[0089] Furthermore, regarding age determination, instead of relying on the determination in Equation 2, if the highest estimated age of the person identified in step S405 exceeds the age of the face detected in step S201, it may be determined that the score deviates from the reference range, and the process may proceed to step S407.

[0090] Regarding the facial expression score Sf, if the lower limit of the reference score range for facial expression judgment is Lf and the upper limit is Hf, then it is determined that the score deviates from the reference score range unless the following equation 3 is satisfied. Lf≦Sf≦Hf (Formula 3)

[0091] For the biological assessment score Sl, if the lower limit of the reference score range in biological assessment is Ll and the upper limit is Hl, then it is determined that the score deviates from the reference score range unless the following equation 4 is satisfied. Ll≦Sl≦Hl (Formula 4)

[0092] For the facial recognition similarity score Ssim, if the lower limit of the reference score range for facial recognition determination is Lsim and the upper limit is Hsim, then it is determined that the score deviates from the reference score range unless the following equation 5 is satisfied. Lsim≦Ssim≦Hsim (Formula 5)

[0093] In step S406, if any of the conditions in equations 1 to 5 are not met, it is determined that the score has deviated from the standard score range, and the process proceeds to step S407. On the other hand, if all of equations 1 to 5 are met, it is determined that the score has not deviated from the standard score range, and the process proceeds to step S408.

[0094] Furthermore, it is also possible to determine a deviation only when several predetermined conditions among Equations 1 to 5 are not met. Alternatively, the above predetermined conditions may be changed depending on the shooting conditions and the application being applied. It is desirable that the respective lower and upper limits above be determined for each user based on the authentication history, etc., and that deviations be determined using the respective upper and lower limits set for each user.

[0095] Furthermore, the method for determining deviation is not limited to the method described above. For example, the above multiple scores may be weighted and added together, with a predetermined lower limit L and upper limit H, and if the result does not satisfy the following equation 6, it may be determined that a deviation has occurred. Note that a, b, c, d, and e are arbitrary coefficients and should be adjusted in advance according to the shooting environment, etc. L≦aSs+bSa+cSf+dSl+eSsim≦H (Formula 6)

[0096] Alternatively, an AI model may be created that learns the trend of changes in past history from various scores such as normal shooting conditions (without fraudulent activity), age, facial expression, biometric detection results, and facial recognition results, and the determination may be made based on the deviation from the trend of changes in past history. In other words, a learning model generation means may be provided that learns the time-series change trend of predetermined judgment criteria and generates a learning model for predetermined judgment criteria.

[0097] When creating an AI model through learning, after installing the facial recognition device 100 of this embodiment, data is collected by having users use the system several times before actual operation, under conditions where it is guaranteed that no fraudulent activity will occur.

[0098] This acquired data is used as ground truth data, and a model is created that learns various scores for each user. Using this model, if the various judgment scores obtained from the image acquired in step S201 deviate from the change trend of the model that learned past history, it may be determined that there is a deviation. In other words, the deviation determination means may determine the deviation based on a learning model of predetermined judgment criteria.

[0099] Figure 5 illustrates an example of an AI model for a score learned from past history change trends in the face recognition method of Embodiment 1 of the present invention. The judgment score obtained from the face in the image acquired in step S200 is represented by the formula Sx, the change trend of the upper limit learned from past change trends is Hx, and the change trend of the lower limit is Lx.

[0100] Sx is one of the judgment scores Ss, Sa, Sf, Sl, or Ssim, with the vertical axis of the graph representing the score and the horizontal axis representing time. When represented in this way, if the time of capture of the image acquired in step S200 is time t, then it can be determined that a deviation has occurred if the following equation 7 is not satisfied. Lx(t)≦Sx(t)≦Hx(t) (Formula 7)

[0101] Furthermore, the AI ​​models used to indicate changing trends, as described above, may be created as separate models for each score, or a single model may be created and used to analyze the combined score.

[0102] In step S407, the deviation determination unit 113 determines that the face acquired in step S201 is a fake and terminates the processing flow shown in Figure 4.

[0103] In step S408, the deviation detection unit 113 determines that the face acquired in step S201 is not an impersonation and terminates the processing flow shown in Figure 4. With this processing flow, fraudulent activity due to impersonation can be efficiently detected.

[0104] <Embodiment 2> In the example described in Embodiment 1, an example was described in which at least the images included in the authentication history are used in the various determination processes executed by the deviation determination unit 113. In Embodiment 2, only age estimation is used in the determination process executed by the deviation determination unit 113, and deviation determination is performed using only registered images without using the images included in the authentication history.

[0105] Figure 6 is a flowchart showing an example of the processing of the facial recognition method according to Embodiment 2 of the present invention. The CPU 101 and other components within the facial recognition device 100 execute a computer program stored in memory, which sequentially performs the operations of each step in the flowchart of Figure 6. The steps with the same reference numerals as those in the flowchart of Figure 2 described above perform the same processing, so their explanation is omitted.

[0106] In step S600, all registered images to be used by the deviation detection unit 113 are obtained from the storage unit 114. Note that the authentication history acquisition process performed in step S205 is not performed, and the process proceeds to step S601.

[0107] In step S601, the deviation determination unit 113 makes a deviation determination on the user's face detected in step S201. Details of step S601 will be described later using the flowchart in Figure 7. If the result of the processing in step S601 is determined to be impersonation, the process proceeds to step S208; if it is determined not to be impersonation, the process proceeds to step S207.

[0108] Figure 7 is a flowchart showing an example of the deviation detection process of the facial recognition method of Embodiment 2 of the present invention, and a detailed example of the deviation detection process in step S601 will be explained using the flowchart in Figure 7.

[0109] Furthermore, the CPU 101 and other components within the facial recognition device 100 execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart of Figure 7. The steps with the same symbols as those in the flowchart of Figure 4 described above perform the same processing, so their explanation is omitted.

[0110] In step S700, the deviation detection unit 113 estimates and scores the age of the face acquired in step S201. If no fraudulent activity using previously taken images has occurred, the face acquired in step S201 will have the highest estimated age at that time. Therefore, this information is used to determine deviation.

[0111] In step S701, the registered images obtained in step S600 are narrowed down to only those related to the person identified by facial recognition in step S202. In other words, the images used are limited to only the registered images of the person themselves.

[0112] In step S702, it is determined whether there is a contradiction in the estimated age. That is, if the estimated age of the face detected in step S200 is the highest, i.e., higher than the age of the face in the registered image, then there is no contradiction in the estimated age, and the result in step S702 is determined to be No. Thus, in this embodiment, the deviation determination means determines whether there is a contradiction between the age of the person in the registered face image and the estimated age of the face.

[0113] In step S702, the age of the face in the registered image was estimated from the face in the registered image. However, for example, the date the registered image was taken or the date of birth of the person in the registered image may be linked to the registered image and saved. Then, the age of the person in the registered image's face can be calculated based on the date the registered image was taken or the date of birth of the person in the registered image, and this calculated age can be compared with the estimated age of the face detected in step S200.

[0114] If the result in step S702 is No, proceed to step S408; if the result in step S702 is Yes, proceed to step S407. After that, the processing flow shown in Figure 7 is terminated.

[0115] As described above, in Embodiment 2, fraudulent activities due to impersonation can be efficiently detected using only registered images.

[0116] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0117] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0118] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the facial recognition device, etc., via a network or various storage media.

[0119] Furthermore, the computer (or CPU, MPU, etc.) in the facial recognition device may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0120] (Configuration 1) A face recognition device comprising: an image acquisition means for acquiring an image of a subject; a face detection means for detecting a face from the image; a face recognition means for identifying a person by matching the face; a reference image holding means for holding a reference face image that serves as a comparison standard for the face; and a deviation determination means for determining that a face is an imposter if it deviates from a predetermined determination standard obtained from the reference face image.

[0121] (Configuration 2) The face recognition device according to Configuration 1, wherein the reference image holding means holds the authentication history by the face recognition means, and the reference face image includes the history image included in the authentication history.

[0122] (Configuration 3) The facial recognition device according to Configuration 1 or 2, characterized in that the reference facial image includes a registered facial image used in the facial recognition means.

[0123] (Configuration 4) The facial recognition device according to Configuration 3, characterized in that the deviation determination means determines whether there is a discrepancy between the age of the person in the registered facial image and the estimated age of the face.

[0124] (Configuration 5) The facial recognition device according to any one of Configurations 1 to 4, characterized in that the deviation determination means narrows down the reference facial image of the person identified by the facial recognition means and then determines whether it deviates from the predetermined determination criteria.

[0125] (Configuration 6) The facial recognition device according to any one of Configurations 1 to 5, characterized in that the predetermined judgment criteria are generated based on at least one of the following: shooting conditions, age estimation result, facial expression estimation result, biometric determination result, and facial recognition result.

[0126] (Configuration 7) A facial recognition device according to any one of Configurations 1 to 6, characterized in that it has a second authentication means that determines whether or not a deviation has occurred in a manner different from that of the facial recognition means when the deviation determination means determines that a deviation has occurred.

[0127] (Configuration 8) A facial recognition device according to any one of Configurations 1 to 7, characterized in that it has a notification means that gives a notification when the frequency at which the deviation determination means determines a deviation is greater than a predetermined value.

[0128] (Configuration 9) A facial recognition device according to any one of Configurations 1 to 8, comprising a learning model generation means that learns the time-series change trend of the predetermined judgment criteria and generates a learning model of the predetermined judgment criteria, wherein the deviation determination means determines the deviation based on the learning model of the predetermined judgment criteria.

[0129] (Method) A face recognition method characterized by comprising: an image acquisition step of acquiring an image of a subject; a face detection step of detecting a face from the image; a face recognition step of identifying a person by matching the face; a reference image holding step of holding a reference face image that serves as a comparison standard for the face; and a deviation determination step of determining that the face is an impersonation if it deviates from a predetermined determination standard obtained from the reference face image.

[0130] (Program) A computer program for controlling each means of a facial recognition device described in any one of configurations 1 to 9 by a computer. [Explanation of Symbols]

[0131] 100: Facial recognition device 108: Imaging device 109: External Systems

Claims

1. An image acquisition means for acquiring an image of the subject, A face detection means for detecting faces from the aforementioned image, A facial recognition means that identifies a person by matching the aforementioned face, A reference image holding means for holding a reference face image that serves as a comparison standard for the aforementioned face, A deviation determination means that determines that the face is an impersonation if it deviates from predetermined determination criteria obtained from the reference face image, A facial recognition device characterized by having the following features.

2. The aforementioned reference image holding means holds the authentication history by the facial recognition means, The facial recognition device according to claim 1, characterized in that the reference facial image includes a history image included in the authentication history.

3. The facial recognition device according to claim 1, characterized in that the reference facial image includes a registered facial image used in the facial recognition means.

4. The facial recognition device according to claim 3, characterized in that the deviation determination means determines whether there is a discrepancy between the age of the person in the registered facial image and the estimated age of the face.

5. The facial recognition device according to claim 1, characterized in that the deviation determination means narrows down the reference facial image of the person identified by the facial recognition means and then determines whether it deviates from the predetermined determination criteria.

6. The facial recognition device according to claim 1, characterized in that the predetermined determination criteria are generated based on at least one of the following: shooting conditions, age estimation result, facial expression estimation result, biometric determination result, and facial recognition result.

7. The facial recognition device according to claim 1, further comprising a second authentication means that determines whether or not a deviation has occurred in a manner different from that of the facial recognition means when the deviation determination means determines that a deviation has occurred.

8. The facial recognition device according to claim 1, further comprising a notification means that provides notification when the frequency at which the deviation determination means determines a deviation exceeds a predetermined value.

9. The system includes a learning model generation means that learns the time-series change trend of the predetermined judgment criteria and generates a learning model of the predetermined judgment criteria, The facial recognition device according to claim 1, characterized in that the deviation determination means determines the deviation based on the learning model of the predetermined determination criteria.

10. Image acquisition step to obtain an image of the subject, A face detection step for detecting a face from the aforementioned image, A facial recognition step that matches the aforementioned face to identify the person, A reference image holding step, which holds a reference face image that serves as a comparison standard for the aforementioned face, A deviation determination step in which, if the face deviates from predetermined determination criteria obtained from the reference face image, it is determined to be an impersonation; A facial recognition method characterized by comprising the following features.

11. A computer program for controlling each means of a facial recognition device according to any one of claims 1 to 9 by computer.

Images