Facial recognition methods, programs, and computer systems

Abstract

To provide a method, program and computer system for face authentication, which do not use the step of instructing a user to make a facial expression, and offer high accuracy, less burden on authentication targets and highly real-time performance.SOLUTION: A computer-implemented face authentication method is provided, comprising a first authentication step of computing a degree of similarity based on feature point extraction, a second authentication step of computing a degree of similarity based on expression parameter extraction, and a step of making an authentication determination based on both the degree of similarity computed in the first authentication step and the degree of similarity computed in the second authentication step.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present application broadly relates to a method, program, and apparatus for authentication based on human facial images, and more particularly relates to a face authentication method, program, and computer system that achieves high accuracy and reduces the burden on the person to be authenticated. relates to.

Background Art

[0002] Authentication methods that identify a person based on a human facial image have become widespread. For example, they are widely used for user authentication during sign-in of portable information terminals, security at building entrances, and the like. Face authentication has the advantages of low burden on users as it eliminates the need to remember passwords or carry dedicated devices such as tokens, relatively low difficulty in spoofing, and relatively low psychological resistance from users.

[0003] In general, in face authentication technology, a feature quantity is extracted from an image of a user's face, and the feature quantity is compared with a pre-stored feature quantity to determine whether the user is the genuine person, or to determine which of a plurality of persons the user corresponds to. Here, how to set the threshold for determining face matching becomes an issue. If the threshold is set too high, the probability of false positive, which is misrecognizing a stranger as the genuine person, increases, and if the threshold is set too low, there is a problem that the probability of another misrecognition, false negative, in which the genuine person is not recognized as the genuine person, increases.

[0004] One way to address these challenges is through facial recognition technology that uses the user's facial expressions. For example, a technology has been disclosed that improves the accuracy of facial recognition by applying a technology that can quantify the degree of a person's facial expression (for example, Patent Document 1) and having the user make a smile (for example, Patent Document 2). However, such methods have drawbacks, such as being burdensome to the person being authenticated (especially in a public environment such as authentication at the entrance of a building, people may be reluctant to make facial expressions) and requiring time for authentication (especially difficult to apply to cases where a person passing through needs to be authenticated in real time).

[0005] The present invention has been made in view of these problems, and aims to provide a facial recognition method, program, and computer system that are highly accurate (high recognition rate and low misrecognition rate), have minimal burden on the user, and can be applied to real-time authentication. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Published Patent Application JP 2014-206903 [Patent Document 2] Published Patent Application JP 2016-009453 [Overview of the project] [Problems that the invention aims to solve]

[0007] To provide a highly accurate, low-burden, and real-time facial recognition method, program, and computer system. [Means for solving the problem]

[0008] The present invention solves the above problem by providing a computer-based facial recognition method that includes a first authentication step of calculating similarity with registered facial information based on feature point extraction, a second authentication step of calculating similarity with registered facial parameters based on facial parameter extraction, and a step of making an authentication determination based on both the first similarity calculated in the first authentication step and the second similarity calculated in the second authentication step, and does not include a step of instructing the user to make an expression.

[0009] Furthermore, the present invention solves the above problem by providing a facial recognition method described in paragraph 0008, wherein in the step of determining the authentication, authentication is determined to be successful only when the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold.

[0010] Furthermore, the present invention solves the above problem by providing a face recognition method described in paragraph 0008, in which, in the second authentication step, a similarity determination is performed using representative values ​​of facial expression parameters extracted from a video image of the user's face within a predetermined time.

[0011] Furthermore, the present invention solves the above problem by providing a facial recognition method described in paragraph 0008, in which the second authentication step uses facial expression parameters for anger, sadness, surprise, and joy, but does not use facial expression parameters for disgust, fear, and neutral expression.

[0012] Furthermore, the present invention solves the above problem by providing a face recognition program that causes a computer to execute a first set of commands for calculating similarity based on feature point extraction, a second set of authentication commands for calculating similarity based on facial expression parameter extraction, and a set of commands for making an authentication determination based on both the first similarity calculated by the first set of authentication commands and the second similarity calculated by the second set of authentication commands, without including a set of commands that instructs the user to make a facial expression.

[0013] Furthermore, the present invention solves the above problem by providing a facial recognition program as described in paragraph 0012, wherein the set of commands for determining authentication includes a set of commands that determines that authentication is successful only when the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold.

[0014] Furthermore, the present invention solves the above problem by providing a face recognition program as described in paragraph 0012, wherein the second set of authentication commands includes a set of commands that perform similarity determination using representative values ​​of facial expression parameters extracted from a video image of the user's face within a predetermined time.

[0015] Furthermore, the present invention solves the above problem by providing a facial recognition program as described in paragraph 0012, in which the second set of authentication instructions uses facial expression parameters for anger, sadness, surprise, and joy, but does not use facial expression parameters for disgust, fear, and neutral expression.

[0016] Furthermore, the present invention solves the above problems by providing a computer system configured to execute a program described in any one of paragraphs 0012 to 0015. [Effects of the Invention]

[0017] A highly accurate, low-burden, and real-time facial recognition method, program, and computer system are provided. [Brief explanation of the drawing]

[0018] [Figure 1] This is an embodiment of an information system in which the method according to the present invention is implemented. [Figure 2] This is the functional structure of an embodiment of the authentication device according to the present invention. [Figure 3] This is a flowchart of the process performed in the first authentication unit of the embodiment of the present invention. [Figure 4] This is a conceptual diagram representing the degree of facial expression of users in their normal state. [Figure 5]This is a flowchart of processing performed by the second authentication unit of the embodiment according to the present invention. [Figure 6] This is a diagram showing another embodiment according to the present invention. [Modes for Carrying Out the Invention]

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Figure 1 shows an embodiment of an information system in which the method according to the present invention is executed. Although the present embodiment assumes security for a gate of a building, the embodiments of the present invention are not limited thereto, and can also be applied to, for example, user authentication of mobile information terminals, and identification of suspicious individuals from an unspecified large number of passersby on the street.

[0020] The camera (100) is a means for capturing an image of a user's face, and is preferably capable of acquiring moving images at a predetermined frame rate. The authentication device (101) is a computer device for processing the user's face image acquired by the camera (100) and performing face authentication. It is not limited to a computer device installed in the facility managed by an organization such as an enterprise, and at least a part of its functions may be a computer device installed outside the organization and managed by a cloud service or the like. The user (102) is a user who is the subject of face authentication. The output device (103) is a device for outputting the authentication result, and may be a display device (display, monitor) for visual confirmation by a human, a device that generates a signal such as an audio alarm, an interface device for operating the gate opening and closing mechanism, or a relay program for transmitting the authentication result to another program.

[0021] Figure 2 shows the functional structure of an authentication device (101) according to one embodiment of the present invention. The face image acquisition unit (201) is a means for acquiring an image to be used for face recognition from the image acquired by the camera (100). In the present invention, since it is necessary to read facial expression parameters from the image, it is desirable that the image be a moving image of a few seconds in length rather than a still image. It is desirable that the device be configured to automatically select an image that includes the entire face of the user (103) and is in focus, determined by a program. The program may also be configured to give instructions to the user (103) via messages or voice so that the user's face can be properly captured.

[0022] The first authentication unit (202) is a means for calculating the similarity between the user's (103) face and the registered face information based on the feature points of the user's face image. The details of its processing will be described later. The second authentication unit (203) is a means for calculating the similarity between the user's (103) face and the registered face information based on facial expression parameters. The details of its processing will be described later. The authentication determination unit (204) is a means for returning the result of the final determination (whether authentication was successful or not, whether the user's face matches the registered one, which of the multiple registered face information is closest to the user's face, etc.) based on the similarity calculated by the first authentication unit (202) and the similarity calculated by the second authentication unit (203). The face information database (205) is a means for storing data of comparison subjects for the face images of one or more users (102), and it is desirable that the face information data be set up in advance by taking and processing the user's (102) face image. It is desirable that the facial information database (205) pre-stores the facial image feature quantities used by the first authentication unit (202) and the facial expression parameters used by the second authentication unit (203), obtained using the same acquisition method as during authentication. It is desirable that each record in the facial information database (205) includes an identifier (key) that uniquely identifies each person being compared. In addition, each record (or another storage means accessed by the record's identifier) ​​may include attribute information of the person being compared (identification number, name, age, gender, etc.). When applying the present invention to facial authentication processing when signing in to a smartphone or personal computer, the facial information database (205) only needs to contain facial information for one person or a very small number of people, rather than multiple people.

[0023] Each of the above functions may be implemented through the coordinated processing of computer hardware and a dedicated program (software), but it is not necessary for each of the above functions to correspond to a single program. For example, the first authentication unit (202) and the second authentication unit (203) may be implemented by a single program. As mentioned above, these functions are not limited to computers managed by the company; for example, a part of them may be implemented by computers that operate or are managed outside the organization via cloud services. They may also be implemented by hardware installed in cameras, smartphones, etc.

[0024] Figure 3 shows a flowchart of the processing performed by the first authentication unit (202) in an embodiment of the present invention. This embodiment assumes a case where it is determined whether there is facial information in the facial information database (205) that is sufficiently similar to the face of the user (103) (for example, security at a building gate), but even in a case where the target facial information has already been determined and it is determined whether the user (103) is the person in question (for example, security when signing in to a smartphone), the same principle applies in that it returns a similarity score based on facial features as the result.

[0025] First, the first authentication unit (202) acquires a facial image of the user (102) to be authenticated from the facial image acquisition unit (201) (S301). This image may be a still image or a short (a few seconds) moving image. It is desirable that the facial image acquisition unit (201) selects an image in which the user's (102) face is clearly visible and that the image is provided after appropriate preprocessing such as brightness normalization and edge enhancement. Next, the first authentication unit (202) extracts features from the facial image (S302). Known features such as Haar-like features may be used. Next, the first authentication unit (202) reads the features of the facial information registered in the facial information database (205) and compares them with the features of the user's (102) facial image to calculate the similarity (matching score) (S303). Known methods in the field of facial authentication technology may be used to calculate the similarity. Alternatively, machine learning software or a neural network may be trained using training data to output similarity scores. In addition, the feature point detection step (S302) may also be performed by a neural network (typically a convolutional layer of a CNN (Convolutional Neural Network)). In this case, attribute information such as age and gender included in the face information database (205) may be used as input parameters for similarity determination (experiments by the inventors have shown that this can further improve the accuracy of face recognition). Finally, the first authentication unit (202) returns the identifier and similarity score of the face information in the face information database (205) with the highest similarity score (S304).

[0026] As shown in Figure 4, the inventor's experiments have revealed that even when a user is not consciously making an expression, the facial expression parameters in their facial image exhibit values ​​unique to that user. Even when definitive similarity determination is difficult using facial feature point information alone, using differences in facial expression parameters in addition enables highly accurate person identification. Conventional facial recognition has a trade-off: lowering the similarity threshold in similarity determination increases the possibility of misidentifying the target person as someone else (false negative), and raising the similarity threshold increases the possibility of misidentifying another person as the target person (false positive). However, by adding authentication based on facial expression parameters that unconsciously appear on the user's face even when they are not making an expression, both the possibility of false negatives and false positives can be reduced.

[0027] Figure 5 shows a flowchart of the processing performed by the second authentication unit (203) in an embodiment of the present invention. The second authentication unit (203) calculates similarity based on the user's facial expression parameters. Here, facial expression refers to the facial expression in the user's normal (unconscious) state, not by intentionally creating an expression (for example, by being instructed to "smile"). First, the second authentication unit (203) acquires a facial image of the user (102) to be the target of facial authentication from the facial image acquisition unit (201) (S501). This image is preferably a short (a few seconds) video. Next, the second authentication unit (203) extracts facial expression parameters from the facial image (S502). The extraction of facial expression parameters may be performed by a method such as that disclosed in Patent Document 1, but it is preferable to calculate a representative value such as the average value from multiple values ​​of facial expression parameters in multiple frames of the video and use it for authentication processing. Next, the second authentication unit (203) reads the facial expression parameters from the record in the face information database (205) that corresponds to the identifier output by the first authentication unit (202) (S503). Finally, the second authentication unit (203) compares the facial expression parameters obtained from the user's (102) face image with the facial expression parameters of the record in the face information database (205) and calculates the similarity (S504).

[0028] The facial expression parameters may consist of anger, sadness, surprise, happiness, disgust, fear, and neutral, and each may be expressed as a normalized numerical value. The similarity between the facial expression parameters extracted from the user's face image and the facial expression parameters in the facial information database (204) may be calculated by calculating the Euclidean distance between the facial expression parameters and normalizing it, but it may also be calculated based on a different formula. The similarity may also be calculated by training a machine learning or deep learning model. In this case, attribute information such as age and gender included in the facial information database (205) may be used as input parameters for similarity determination (experiments by the inventors have shown that this can further improve the accuracy of facial recognition).

[0029] Experiments by the inventor have shown that the expression parameters for anger, sadness, surprise, and happiness show relatively large individual differences and tend to take values ​​unique to each person, while the expression parameters for disgust, fear, and neutral show little interpersonal variation. Therefore, the second authentication unit (203) may calculate similarity using only the expression parameters for anger, sadness, surprise, and happiness, or it may calculate similarity by relatively increasing the weighting of the expression parameters for anger, sadness, surprise, and happiness. The processing of the first authentication processing unit (202) and the processing of the second authentication processing unit (203) may be executed sequentially, but they may be executed concurrently to speed up the authentication process.

[0030] The authentication determination unit (204) makes a final authentication determination (typically, whether the user's (102) face image belongs to the same person as the face information registered in the face information database (204)) based on both the similarity calculated by the first authentication processing unit (202) and the similarity calculated by the second authentication processing unit (203). For example, threshold A (a threshold for stricter determination) and threshold B (a threshold for more lenient determination) may be set, and if the similarity calculated by the first authentication processing unit (202) is between threshold A and threshold B (i.e., there is a high probability that they are the same person, but it is not certain), the second authentication processing unit (203) may determine whether they are the same person based on whether the similarity calculated exceeds threshold C (a predetermined value different from thresholds A and B). Alternatively, authentication may be deemed successful (the user matches a person registered in the face information database (204)) only if both the first authentication processing unit (202) and the second authentication processing unit (203) have confirmed successful authentication (the similarity between the user's face and the item in the face information database (204) is above a predetermined threshold). As an alternative, the final similarity may be calculated from the similarity determined by the first authentication processing unit (202) and the similarity determined by the second authentication processing unit (203), and the final authentication decision may be made based on this. The final similarity may be calculated based on a weighted average of the similarity determined by the first authentication processing unit (202) and the similarity determined by the second authentication processing unit (203), but a non-linear calculation formula may also be used. The final authentication decision may also be made by training a machine learning or deep learning model that takes the similarity calculated by the first authentication processing unit (202) and the similarity calculated by the second authentication processing unit (203) as input.

[0031] Figure 6 shows another embodiment of the present invention. Figure 6a) is an embodiment in which the facial recognition method of the present invention is applied to the authentication of a user (602) when signing in (logging in) to a portable information device such as a smartphone (601) or a personal computer. In this embodiment, the authentication device (101) is the smartphone (601) itself (some processing may be performed by another computer device via a network), and the output device (103) may be a program that transmits the authentication result to the program that performs the login processing on the smartphone. In this case, the element corresponding to the facial information database (205) may be a file stored on the portable information device, and may be configured to store only the facial information of one user (the legitimate user of that portable information device). In this case, the result of the first authentication unit (202) may be only the similarity between the user (602) who is signing in and the facial information (feature points) in the file, and the second authentication unit (203) may compare it with the facial expression parameters in the facial information database (205) (file) without obtaining an identifier from the first authentication unit (202).

[0032] Figure 6b) shows an example of applying the facial recognition method according to the present invention to multiple moving subjects (603). The camera (604) may be fixed in place or mounted on a mobile device such as a drone. In this embodiment, the authentication device (101) and output device (103) may be the same as those described in Figure 1. In this embodiment, it is desirable that the image acquisition unit (201) is configured to execute software for facial recognition, extract the human face portion from the captured video image, perform necessary preprocessing on it, and then pass it to the first authentication processing unit (202) and the second authentication processing unit (203). Furthermore, the output of the output device (103) may be attribute information (identification number, name, etc.) included in records of the facial information database (205) whose similarity to the multiple subjects (603) is above a predetermined threshold. In any of the embodiments in Figure 6, the hardware used (camera, computer, etc.) may be the same as that used in conventional facial recognition, and the present invention can be implemented by simply changing the software.

[0033] (Technically remarkable effects of the present invention) Facial recognition methods have inherent advantages such as low burden on users, relatively difficult impersonation, and relatively low psychological resistance from users. However, the present invention provides further advantages by improving authentication accuracy without increasing psychological resistance from users or sacrificing real-time capabilities. Furthermore, hardware equipment such as cameras can be reused from those used in conventional facial recognition systems, and authentication accuracy can be improved simply by changing the software, making it highly cost-effective.

Claims

1. A first authentication step of calculating a first similarity with registered facial information based on feature point extraction from a user's facial image, A second authentication step involves extracting facial expression parameters from the aforementioned facial image, which numerically represent the degree of each of several types of facial expressions, and comparing the extracted facial expression parameters with registered facial expression parameters to calculate a second similarity score. The process includes the step of making an authentication determination based on both the first similarity and the second similarity, It does not include a step in which the user is instructed to make facial expressions. A computer-based facial recognition method.

2. In the step of determining whether authentication is successful, authentication is determined to be successful only if the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold. The facial recognition method according to claim 1.

3. In the second authentication step, similarity determination is performed using representative values ​​of facial expression parameters extracted from the user's facial video image within a predetermined time period. The facial recognition method according to claim 1.

4. In the aforementioned second authentication step, the facial expression parameters for anger, sadness, surprise, and joy are used, but the facial expression parameters for disgust, fear, and neutral expression are not used. The facial recognition method according to claim 1.

5. A first set of authentication commands that calculates a first similarity with registered facial information based on feature point extraction from a user's facial image, A second set of authentication commands extracts facial expression parameters from the aforementioned facial image, representing the degree of each of several types of facial expressions as numerical values, and calculates a second similarity score by comparing the extracted facial expression parameters with registered facial expression parameters. The computer is instructed to execute a set of instructions that perform authentication based on both the first similarity and the second similarity. It does not include a set of commands that instruct the user to make facial expressions. Facial recognition program.

6. The set of instructions for determining the authentication includes a set of instructions that determines that authentication is successful only when the first similarity is equal to or greater than a predetermined first threshold and the second similarity is equal to or greater than a predetermined second threshold. The facial recognition program according to claim 5.

7. The second set of authentication commands includes a set of commands that perform similarity determination using representative values ​​of facial expression parameters extracted from video images of the user's face within a predetermined time period. The facial recognition program according to claim 5.

8. The second set of authentication commands described above uses the facial expression parameters for anger, sadness, surprise, and joy, but does not use the facial expression parameters for disgust, fear, and neutral expression. The facial recognition program according to claim 5.

9. A computer system configured to execute the program described in any one of claims 5 to 8.

Images