Information processing device, information processing method, and program

Abstract

To enable highly accurate biometric determination in a short time in biometric determination accompanying face recognition.SOLUTION: An information processing device has: storage means which stores biometric determination parameters used for determining whether a person appearing in an image is a living body; face recognition means which compares face feature quantities obtained from a face image of a recognition target person with face feature quantities of a registered user to perform face recognition of whether the recognition target person is the registered user; and determination means which determines, using the biometric determination parameters, whether the recognition target person recognized as the registered user by the face recognition is a living body. The storage means stores biometric determination parameters determined on the basis of detection results of eye opening / closing motions of the registered user. The determination means determines whether the recognition target person is a living body by comparing detection results of motions including eye opening and closing of the recognition target person recognized as the registered user by the face recognition with the biometric determination parameters.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to information processing technology associated with face recognition. [Background technology]

[0002] In recent years, facial recognition has been widely used as a means of identity verification and recognition. Furthermore, facial recognition systems may be configured to perform a biometrics determination to determine whether a person appearing in an image (hereinafter referred to as a recognition target) is a living person, as a countermeasure against attempts to impersonate a registered user using a photograph or the like. The biometrics determination to determine whether a recognition target is a living person is performed, for example, by requesting the recognition target to perform a specific action, such as blinking or turning their head, and then confirming whether the specific action was performed based on the image. When confirming whether a blink was performed as a specific action, a method is used in which a score representing the degree of eye opening or the probability that the eyes are open is obtained from the image, and the score is compared with a predetermined threshold to determine whether a blink was performed. Note that when obtaining a score representing the degree of eye opening or the probability that the eyes are open from an image, a configuration using an algorithm such as template matching or machine learning is often used.

[0003] Furthermore, as a method for determining whether the eyes are open or closed, Patent Document 1 discloses a method for learning images of the eyes both open and closed, and determining thresholds for determining whether the eyes are open and closed based on the images. Furthermore, Patent Document 2 discloses a method for determining thresholds based on the frequency distribution of scores over a certain period of time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-134490 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-151798 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in biometric determination associated with face recognition, it is not desirable to apply a method of learning or determining a threshold value for determining whether the eyes are open or closed based on the image input during recognition, because learning or determining a threshold value takes a certain amount of time, which increases the time required for recognition and causes the person being recognized to have to wait.

[0006] Therefore, an object of the present invention is to make it possible to perform accurate biometric determination in a short time in biometric determination associated with face recognition. [Means for solving the problem]

[0007] The information processing device of the present invention comprises a memory means for storing biometric determination parameters used to determine whether a person appearing in an image is a living body; a face recognition means for comparing facial features acquired from a facial image of the person to be recognized with facial features of a registered user to perform facial recognition to determine whether the person to be recognized is the registered user; and a determination means for determining whether the person to be recognized, who has been recognized as the registered user by the face recognition, is a living body using the biometric determination parameters, wherein the memory means stores the biometric determination parameters determined based on detection results of the eye opening and closing movements of the registered user, and the determination means determines whether the person to be recognized is a living body by comparing detection results of movements including eye opening and closing of the person to be recognized, who has been recognized as the registered user by the face recognition, with the biometric determination parameters stored in the memory means. [Effects of the Invention]

[0008] According to the present invention, in biometric determination associated with face recognition, it is possible to achieve highly accurate biometric determination in a short time. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration applicable to an information processing apparatus. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 3] FIG. 4 is a diagram illustrating an example of registration information stored in a storage unit. [Figure 4] 1 is a conceptual diagram of an example of a system to which an information processing device is applied. [Figure 5] FIG. 10 is a diagram showing an example of a face for each individual and an example of transition of eye opening / closing scores. [Figure 6] 10 is a flowchart of an information registration process. [Figure 7] 10 is a flowchart of a face recognition process. [Figure 8] 10 is a flowchart of a biometric determination parameter determination process according to the first embodiment. [Figure 9] 4 is a flowchart of a living body determination process according to the first embodiment. [Figure 10] 10 is a flowchart of a biometric determination parameter update process according to the first embodiment. [Figure 11] 10 is a flowchart of an eye open / close score calculation process. [Figure 12] 10A and 10B are diagrams used to explain the eye open / close score calculation process. [Figure 13] 10A and 10B are diagrams illustrating examples of individual faces and examples of transitions in eye opening and closing scores before and after correction. [Figure 14] 10 is a flowchart of a biometric determination parameter determination process according to the second embodiment. [Figure 15] 10 is a flowchart of a living body determination process according to the second embodiment. [Figure 16] 10 is a flowchart of a biometric determination parameter update process according to the second embodiment. [Figure 17] 10 is a flowchart of a biometric determination parameter determination process according to the third embodiment. [Figure 18] 13 is a flowchart of a biometric determination parameter update process according to the third embodiment. [Figure 19] 13 is a flowchart of a biometric determination parameter determination process according to the fourth embodiment. [Figure 20]10 is a flowchart of a living body determination process according to a fourth embodiment. [Figure 21] 13 is a flowchart of a biometric determination parameter update process according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, the present invention may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, redundant descriptions of the same or similar configurations and processing steps will be omitted.

[0011] First Embodiment 1 is a diagram showing an example of a hardware configuration applicable to an information processing device 100 of this embodiment. The information processing device 100 includes, for example, a CPU 101, a ROM 102, a RAM 103, a mass storage device 104, a communication unit 105, an input device 106, and an output device 107. The communication unit 105 is connected to a network 108.

[0012] The CPU 101 reads out a control program recorded in the ROM 102 and an information processing program according to this embodiment and executes various processes. The RAM 103 is used as a temporary storage area such as a main memory or a work area. The mass storage device 104 is a hard disk drive (HDD), a solid-state drive (SSD), or the like, and is used for long-term data storage. The communication unit 105 is a circuit for communication via a network 108. The input device 106 is a device for inputting instructions and data to the information processing device 100 from the outside. Specifically, the input device 106 includes a camera for acquiring images, a keyboard, a mouse, a touch panel, and the like for accepting user input. The output device 107 is a device for outputting instructions and data from the information processing device 100 to the outside. Specifically, the output device 107 includes a display device such as a display for displaying recognition results and information to the user, and an interface for outputting recognition results, an unlocking signal when recognition is successful, and the like to an external device.

[0013] It should be noted that the information processing device 100 does not necessarily have to include all of the components shown in Fig. 1. For example, if all input and output with the outside world is performed using other devices interconnected via a network 108, the input device 106 and the output device 107 are not required. Furthermore, the information processing device 100 may include components not shown in Fig. 1. For example, the information processing device 100 may include a GPU (graphical processing unit) or FPGA (field programmable gate array) that executes image processing.

[0014] As described above, the hardware configuration of the information processing device 100 has the same hardware components as those installed in a personal computer (PC), a tablet terminal, a smartphone, etc. Therefore, various functions realized by the information processing device 100 can be implemented as software that runs on a PC, etc. The information processing device 100 can realize various functional units of the information processing device 100 and processes shown in each flowchart described below by the CPU 101 executing an information processing program according to this embodiment.

[0015] Fig. 2 is a functional block diagram showing each functional unit realized in the information processing device 100 of this embodiment. Each functional unit shown in Fig. 2 can be realized in terms of hardware by elements and mechanical devices such as a computer CPU, and in terms of software by a computer program, etc., but here, functional blocks realized by cooperation between them are depicted. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combining hardware and software.

[0016] 2, the information processing device 100 has functional units such as an image acquisition unit 202, a determination unit 203, a face recognition unit 204, and a storage unit 205. The face recognition unit 204 has functional units such as a feature acquisition unit 206 and a comparison unit 207. The information processing device 100 is connected to an image capture device 201 via a network 108. Note that the information processing device 100 and the image capture device 201 may be connected via input / output interfaces included in the input device 106 and the output device 107, rather than via the network 108.

[0017] Each functional unit will be described below. The imaging device 201 is a camera that captures still images and moving images. The imaging device 201 captures an image of a person (recognition target) whose face is to be recognized, and sends the image to the information processing device 100 via the network 108. The image can be a moving image in any format, such as Motion JPEG or H.264, or a still image in any format, such as JPEG. The image acquisition unit 202 acquires an image from the image capture device 201 via the network 108 .

[0018] The determination unit 203 performs a biometric determination based on the image acquired by the image acquisition unit 202 to determine whether the person shown in the image is a living organism, rather than an image of a person shown in a photograph, etc. Details of the biometric determination process performed by the determination unit 203 will be described later using a flowchart. The face recognition unit 204 performs face recognition based on the image acquired by the image acquisition unit 202. The face recognition unit 204 includes a feature acquisition unit 206 and a comparison unit 207. The feature acquisition unit 206 calculates face features from the face image captured in the image acquired by the image acquisition unit 202 using a pre-trained Deep Learning model or the like. The comparison unit 207 compares two face features to obtain a matching score that correlates with the degree of similarity. In this embodiment, the comparison unit 207 compares face features registered in the storage unit 205 with face features calculated from the image acquired by the image acquisition unit 202 to obtain a matching score. Details of the face recognition process performed by the face recognition unit 204 will be described later using a flowchart.

[0019] The storage unit 205 is a database that stores registration information for face recognition. The registration information includes at least facial feature amounts for face recognition and biometric determination parameters used for biometric determination. In addition, the registration information may include any information such as a facial image, name, and ID (identification information) of a registered user (hereinafter referred to as a registered user). 3 is a diagram showing an example of registration information stored in the storage unit 205. Registration information 301 and registration information 302 each include a registered user's ID, name, registered face image 303, registered face feature amounts, and biometric determination parameters.

[0020] In this embodiment, the storage unit 205 is embodied as a database configured on the mass storage device 104, but is not limited to this. For example, a mobile terminal or an ID card with an IC chip carried by a registered user may store registration information for face recognition as the storage unit 205, and the registration information may be read by the information processing device 100 via a reading device connected to the information processing device 100.

[0021] FIG. 4 is a conceptual diagram showing an example of a system to which the information processing device 100 is applied. The imaging device 201 captures an image of the face of a person to be recognized 401. The person to be recognized 401 may be a registered user or an unregistered person. The information processing device 100 acquires an image of the face of the person to be recognized 401 from the imaging device 201 via the network 108 and performs face recognition processing.

[0022] In addition to face recognition processing, the information processing device 100 of this embodiment also performs a biometrics determination process to determine whether the person to be recognized 401 is a living person, as a countermeasure against impersonation of registered users using photographs or the like. The biometrics determination process involves requesting the person to be recognized 401 to perform a specific action, such as blinking or changing the direction of their face, and then confirming whether the specific action was performed based on an image. In this embodiment, blinking is used as an example of the specific action required of the person to be recognized during biometrics determination. The information processing device 100 acquires an eye opening / closing score representing the detection result of the degree of eye opening and the probability of the eyes being open from an image of the person to be recognized 401 captured by the imaging device 201, and determines whether the person blinked by comparing the eye opening / closing score with a threshold. If the person to be recognized blinked, it can be determined that the person to be recognized 401 is a living person, not a person image captured in a photograph or the like. The eye opening / closing score representing the degree of eye opening and the probability of the eyes being open from an image can be acquired using an algorithm such as template matching or machine learning, for example.

[0023] In the first embodiment, an example will be described in which a threshold value serving as a criterion for determining whether the eyes are open or closed is registered as a biometric determination parameter in an information processing device 100 that performs biometric determination by detecting blinking of a recognition target person. FIG. 5(a) is a diagram showing a pair of a face of a person 501 and a graph 502 showing a time-series transition of an opening / closing score representing the detection result of the opening / closing movement of the person 501, calculated from each frame of a video. FIG. 5(b) is a diagram showing a pair of a face of another person 503 and a graph 504 showing a time-series transition of the person 503's eye opening / closing score. The transition of the opening / closing score shown in graph 502 represents a time-series transition of the opening / closing score calculated from images of each frame of a video capturing the person 501 by the eye opening / closing score calculation process shown in the flowchart of FIG. 11 (to be described later). Similarly, graph 504 represents a time-series transition of the opening / closing score calculated from images of each frame of a video capturing the person 503 by the eye opening / closing score calculation process of FIG. 11. Graph 502 shows that the opening / closing score decreased because the person 501 blinked when the image of frame number 14 was captured. Similarly, graph 504 shows that the person 503 blinked when the images of frame numbers 9 and 18 were captured, resulting in a low open / close score.

[0024] Comparing graphs 502 and 502, it can be seen that person 503 has a lower level of open / close score compared to person 501. This is thought to be due to the fact that person 503 has narrower eyes than person 501, among other physical characteristics. Because the level of the open / close score differs depending on an individual's physical characteristics, determining whether the eyes are open or closed by comparing a general-purpose threshold with the open / close score may not yield accurate results. Therefore, to accurately determine whether the eyes are open or closed based on the open / close score, it is necessary to apply a different threshold to each individual, taking into account the difference in the level of the open / close score due to each individual's physical characteristics. For example, in the case of graph 502 for person 501, it is thought that if the threshold is set to around 0.50 of the open / close score value, it will be possible to determine whether the eyes are open or closed from the open / close score. On the other hand, in the case of graph 504 for person 503, it is not appropriate to set the threshold to around 0.50 of the open / close score value; for example, it is thought that a threshold of around 0.45 would be more appropriate.

[0025] Therefore, the information processing device 100 of the first embodiment sets the threshold value used to determine whether the eyes are open or closed to an appropriate value for each individual, and includes the threshold value set for each individual in the registration information as a biometric determination parameter of the registered user, and registers it in the memory unit 205.

[0026] 6 to 11 are flowcharts showing the flow of each information process executed by the information processing device 100 of this embodiment. Hereinafter, the flow of each information process in the information processing device 100 of this embodiment will be described with reference to these flowcharts. In the following description of each flowchart, the description of each process (step) will be omitted by adding the symbol S to the beginning of each process (step). However, the information processing device 100 does not necessarily have to perform all of the steps described in these flowcharts.

[0027] Fig. 6 is a flowchart of information registration processing in which the information processing device 100 acquires facial feature amounts from a facial image of a registered user, further acquires biometric determination parameters and information such as the name of the registered user, and stores the information in a database as registration information. The information processing device 100 starts the processing of the flowchart in Fig. 6 in response to a call of the information registration processing function by, for example, a user who wishes to register themselves (hereinafter referred to as a user to be registered). Note that the call of the information registration processing function of the information processing device 100 may be made by a specific operator or the like.

[0028] When the information registration process of FIG. 6 is started, first, in the process of S601, the image acquisition unit 202 acquires a facial image of the face of the user to be registered from the imaging device 201. Next, in the process of S602, the feature amount acquisition unit 206 calculates the feature amount of the acquired face image. Next, in the process of S603, the determination unit 203 determines biometric determination parameters. Details of the biometric determination parameter determination process of S603 will be explained later using the flowchart of FIG. Furthermore, in the process of S604, the information processing device 100 prompts the user to be registered to input a name via the output device 107. The user to be registered uses the input device 106 to input the name to be registered.

[0029] Thereafter, in the process of S605, the information processing device 100 registers each piece of information, such as the ID, the face image obtained in S601 to S604, the feature amounts of the face image, the biometric determination parameters, and the name, in the storage unit 205 (database) as registration information of the user to be registered. The ID may be input by the user to be registered in the same way as the name, or the information processing device 100 may automatically determine an ID that is unique for each registered user in the database. Once the registration information of the user to be registered is registered in the database by this information registration process, the information processing device 100 thereafter treats the user to be registered as a registered user.

[0030] 7 is a flowchart of face recognition processing performed by the face recognition unit 204 of the information processing device 100. The information processing device 100 starts the processing of the flowchart in FIG. 7, for example, in response to a call by a person to be recognized of a face recognition function. Note that, in a case where the information processing device 100 is used as part of an entrance / exit gate of an entrance / exit system, for example, the information processing device 100 may start the processing of the flowchart in FIG. 7 when it detects that a person to be recognized is approaching the entrance / exit gate. The person to be recognized may be a registered user or an unregistered person.

[0031] When the face recognition process of the flowchart in FIG. 7 is started, first, in the process of S701, the image acquisition unit 202 acquires a face image of the face of the person to be recognized from the imaging device 201. Next, in the process of S702, the feature amount acquisition unit 206 calculates the feature amount of the face image of the recognition target person. Next, in the processing of S703, the comparison unit 207 compares the facial features of the person to be recognized calculated in S702 with the facial features of each piece of registration information registered in the memory unit 205, and calculates a matching score that correlates with the degree of similarity.

[0032] Then, in the process of S704, the face recognition unit 204 determines whether the highest matching score among the matching scores calculated in S703 is equal to or greater than a preset threshold value for the matching score. If the highest matching score is equal to or greater than the threshold value, the face recognition unit 204 determines that face recognition has been successful, and proceeds to the next step of S705, assuming that the registered information having the facial feature value from which the combined score was calculated is the registered information of the person to be recognized. On the other hand, if the highest matching score is less than the threshold value, the face recognition unit 204 proceeds to the process of S710.

[0033] When the process proceeds to S705, the information processing device 100 refers to the biometric determination parameters from the registration information stored in the storage unit 205, and also refers to information such as the name as necessary. Next, in the process of S706, the determination unit 203 performs a liveness determination process using the liveness determination parameters referenced from the storage unit 205. Details of the liveness determination process of S706 will be explained later using the flowchart of FIG. Next, in the process of S707, the determination unit 203 checks whether the biometrics determination process of S706 was successful. If the biometrics determination was successful, the determination unit 203 proceeds to the process of S708, whereas if the biometrics determination was unsuccessful, the determination unit 203 proceeds to the process of S710.

[0034] If the process proceeds to S708, the determination unit 203 performs processing to update the biometric determination parameters included in the registration information of the registered user. Details of the biometric determination parameter update processing in S708 will be described later using the flowchart in Fig. 10. Then, after S708, the information processing device 100 proceeds to processing in S709.

[0035] When the process proceeds to S709, the information processing device 100 performs processing when the facial recognition and biometrics determination of the recognition target person are successful. Note that the processing when the facial recognition and biometrics determination are successful is optional. For example, if the information processing device 100 is used as part of an entrance / exit gate of an entrance / exit system, the information processing device 100 can notify the recognition target person via the output device 107 that the facial recognition has been successful, and can send an unlock signal to the associated entrance / exit gate, for example. On the other hand, if the process proceeds to S710, the information processing device 100 performs processing for when face recognition or biometric determination fails. The processing for when face recognition or biometric determination fails is also optional. For example, if the information processing device 100 is part of an entrance / exit gate of an entrance / exit system, the information processing device 100 can notify the person to be recognized that face recognition has failed via the output device 107, and can, for example, send a signal to not unlock the entrance / exit gate.

[0036] FIG. 8 is a flowchart showing details of the biometric determination parameter determination process performed by the determination unit 203 in S603. First, in the process of S801, the determination unit 203 calculates an eye opening / closing score for the face image of the user to be registered acquired in S601. Details of the eye opening / closing score calculation process of S801 will be described later with reference to the flowchart of FIG. Next, in the process of S802, the determination unit 203 calculates a threshold value used to determine whether the eyes are open or closed based on the eye open / closed score calculated in S801, and determines the threshold value as a biometric determination parameter.

[0037] Here, it is assumed that the registered face image of the registered user stored in the storage unit 205 as registration information is registered as a face image with eyes well-opened. Therefore, the eye open / closed score calculated in S801 is the eye open / closed score when the user to be registered has their eyes open. The determination unit 203 calculates an appropriate threshold for the eye open / closed score when the user to be registered has their eyes open, based on the relationship between the eye open / closed score when the eyes are open and an appropriate threshold for the eye open / closed score. The relationship between the eye open / closed score when the eyes are open and an appropriate threshold for the eye open / closed score is experimentally determined in advance. For example, it is assumed that it is possible to appropriately determine whether the eyes are open or closed by acquiring a time-series transition of the eye open / closed scores for multiple people, such as graphs 502 and 504, and using a value obtained by subtracting 0.05 from the eye open / closed score based on the transition as a threshold. Therefore, in this case, the determination unit 203 subtracts 0.05 from the eye open / closed score calculated in S801 as the threshold to be used when determining whether the user to be registered has their eyes open or closed, and determines the threshold as the biometric determination parameter.

[0038] Note that a registered user may register multiple facial images of themselves as registered facial images, as in the registered facial image 303 in Fig. 3. In such a case, the determination unit 203 may calculate an open / close score for each of the facial images in S801, obtain a threshold for each of the facial images in S802, and determine the liveness determination parameter by using the average of the multiple thresholds as a final threshold. Alternatively, the determination unit 203 may check conditions such as the facial orientation for multiple facial images of the registered user, and perform the process of the flowchart in Fig. 8 based on only the facial image with the best conditions to determine the liveness determination parameter as a threshold.

[0039] FIG. 9 is a flowchart showing details of the biometric determination process performed by the determination unit 203 in S706. First, in the process of S901, the determination unit 203 requests the recognition target person to blink via the output device 107. The request to the recognition target person to blink is made, for example, by displaying a message such as "Please blink" on a display included in the output device 107. Then, in the process of S902, the image acquisition unit 202 acquires a face image obtained by capturing an image of the face of the person to be recognized by the imaging device 201.

[0040] Next, in the process of S903, the determination unit 203 calculates an eye opening / closing score based on the face image acquired in S902, similar to S801. Details of the eye opening / closing score calculation process of S903 will be described later with reference to the flowchart of FIG. Furthermore, in the process of S904, the determination unit 203 compares the open / closed score calculated in S903 with the threshold value referred to as the biometric determination parameter in S705. If the open / closed score is equal to or greater than the threshold value, the determination unit 203 proceeds to the process of S905 and determines that the eyes of the recognition target person are open. On the other hand, if the open / closed score is less than the threshold value, the determination unit 203 proceeds to the process of S906 and determines that the eyes of the recognition target person are closed.

[0041] Next, in the process of S907, the determination unit 203 determines whether the person to be recognized has successfully blinked. For example, if the determination unit 203 determines that the eyes of the person to be recognized are closed in S906, the determination unit 203 may determine that the person to be recognized has successfully blinked. Alternatively, the determination unit 203 may store the state of the eyes of the person to be recognized in chronological order, and determine that the person to be recognized has successfully blinked when the state transitions from open state to closed state to open state. If the determination unit 203 determines that the person to be recognized has successfully blinked, the process proceeds to S909. On the other hand, if the determination unit 203 does not determine that the person to be recognized has successfully blinked, the process proceeds to S908.

[0042] If the process proceeds to S908, the determination unit 203 determines whether the biometric determination process has timed out. For example, the determination unit 203 may determine that the process has timed out if the person to be recognized has not successfully blinked within 10 seconds since the request to blink was issued in S901. If the determination unit 203 determines that the process has timed out, it proceeds to S910. On the other hand, if the process has not timed out, it returns to S902, where the next image is acquired in S902 and the same processes as those from S903 onwards are performed.

[0043] If the process proceeds to S909, the determination unit 203 determines that the recognition target person has been successfully determined to be a living person, and determines that the recognition target person who is the subject is a living person, that is, a registered user in living body form, not a photograph or the like. On the other hand, if the process proceeds to S910, the determination unit 203 determines that the biometrics of the person to be recognized are not successful. That is, in this case, it is determined that the facial image of the person to be recognized is not the facial image of a biometric registered user. This makes it possible to prevent impersonation of a registered user by using a facial photo of the registered user or the like to pass face recognition.

[0044] Fig. 10 is a flowchart showing details of the process of updating the biometric determination parameters in the registration information of the registered user, which is performed by the determination unit 203 in S708. The biometric determination parameter update process in Fig. 10 is a process that is performed when the determination unit 203 determines that the recognition target is a biometric registered user in the process of the flowchart in Fig. 7 described above.

[0045] First, in the process of S1001, the determination unit 203 refers to the value of the open / close score calculated from the facial image of the recognition target in S903 during the liveness determination process of S706, that is, the value of the eye open / close score of the registered user who is the recognition target and whose liveness determination was successful in S909 of Fig. 9. At the same time, the determination unit 203 also refers to whether the eye state was determined to be open or closed based on the open / close score. Then, the determination unit 203 acquires the level of the open / close score when the eyes of the registered user who was successful in the liveness determination are in the open state (for example, the average value of the open / close scores determined to be open) and the level of the open / close score when the eyes are in the closed state (similarly, the average value of the open / close scores determined to be closed).

[0046] Next, in the process of S1002, the determination unit 203 determines an appropriate threshold value for each registered user acquired in S1001. The method for determining the threshold value may be the same as that of S802, or a different method. For example, similar to the example of the process of S802, the determination unit 203 may set the threshold value to a value obtained by subtracting 0.05 from the level of the open / close score of the registered user when their eyes are open, acquired in S1001. Alternatively, unlike the example of the process of S802, the determination unit 203 may set the threshold value to a value that is exactly midway between the level of the open / close score of the registered user when their eyes are open and the level of the open / close score when their eyes are closed, acquired in S1001. Then, in the next process of S1003, the determination unit 203 registers the threshold determined in S1002 as the biometric determination parameter of the registered user in the storage unit 205 (database). As a result, the already registered threshold is overwritten, thereby performing the biometric determination parameter update process.

[0047] Fig. 11 is a flowchart showing details of the eye opening / closing score calculation process performed by the determination unit 203 based on a face image in S801 and S903. Fig. 12 is a diagram used to explain the concept of the eye opening / closing score calculation process. First, in the process of S1101, the determination unit 203 creates an input image by cutting out the eye region from the face image. Input image 1201 in Fig. 12 shows an example of the input image generated by the determination unit 203 by cutting out the eye region from the face image in S1101. Note that although the input image 1201 shows an example of an image in which the eyes are open, if the eyes of the face image are closed, the eye region is cut out from the face image in which the eyes are closed to create the input image.

[0048] Next, in the process of S1102, the determination unit 203 calculates HOG (Histgrams of Oriented Gradients) features for the input image 1201. HOG features 1204 in FIG. 12 show an example of features calculated by the determination unit 203 from the input image 1201.

[0049] Next, in the process of S1103, the determination unit 203 acquires HOG features of the template images of open and closed eyes. The template image of open eyes is a template image of a human eye in an open state that is stored in advance in the information processing device 100. The template image of open eyes may be created by any method. For example, the template image of open eyes may be a template image of a typical human eye in an open state created using computer graphics, or may be created based on an image of an arbitrary person with their eyes in an open state. Similarly, the template image of closed eyes is a template image of a human eye in a closed state that is stored in advance in the information processing device 100. The template image of closed eyes may be created by any method. For example, the template image of closed eyes may be a template image of a typical human eye in a closed state created using computer graphics, or may be created based on an image of an arbitrary person with their eyes in an closed state. The open template image 1203 in FIG. 12 is an example of an open eye template image, and the closed template image 1202 in FIG. 12 is an example of a closed eye template image.

[0050] The determination unit 203 acquires HOG features for both the open eye template image and the closed eye template image. Feature 1205 in FIG. 12 shows an example of an HOG feature calculated from the closed eye template image 1202, and feature 1206 shows an example of an HOG feature calculated from the open eye template image 1203. The information processing device 100 may store only the HOG features calculated from the open and closed eye template images, without storing them. In this case, in S1103, the determination unit 203 acquires the stored HOG features instead of calculating HOG features from the template images.

[0051] Next, in the process of S1104, the determination unit 203 calculates the Euclidean distance D1 between the HOG feature amount 1204 calculated from the input image 1201 and the HOG feature amount 1205 calculated from the closed-eye template image 1202. Similarly, in the process of S1105, the determination unit 203 calculates the Euclidean distance D2 between the HOG feature amount 1204 of the input image 1201 and the HOG feature amount 1206 of the open-eye template image 1203.

[0052] Next, in the process of S1106, the determination unit 203 calculates the opening / closing score SC by the following opening / closing score calculation formula (1).

[0053] SC=D1 / (D1+D2) Equation (1)

[0054] Here, the input image 1201 with the eyes open and the eye-open template image 1203 have similar image shapes, so the Euclidean distance D2 is small. Conversely, the input image 1201 with the eyes open and the eye-closed template image 1202 have different image shapes, so the Euclidean distance D1 is large. Therefore, when the input image 1201 with the eyes open is acquired in S1101, the open / close score SC calculated using the open / close score calculation formula (1) is large. In contrast, for example, when an input image with the eyes closed is acquired, the Euclidean distance D2 between the input image and the eye-open template image 1203 is large, and conversely, the Euclidean distance D1 between the input image and the eye-closed template image 1202 is small. Therefore, when the input image with the eyes closed is acquired in S1101, the open / close score SC calculated using the open / close score calculation formula (1) is small. From these facts, the larger the value of the open / close score SC, the higher the likelihood that the eyes in the input image are open.

[0055] In S1106, the determination unit 203 may prepare multiple pairs of template images of open and closed eyes, calculate an open / close score for each pair, and then calculate the average value as the final open / close score SC. Furthermore, the determination unit 203 may calculate the open / close score SC for either the right eye or the left eye, or may calculate it for both eyes and average it.

[0056] The process of the flowchart shown in FIG. 11 is an example of a process for calculating the eye open / closed score, and the determination unit 203 may calculate the eye open / closed score using other known methods. For example, the determination unit 203 may detect the upper and lower eyelids using a Deep Learning model capable of detecting facial organ points or an algorithm based on the brightness gradient of an image, and calculate the eye open / closed score based on the distance between the upper and lower eyelids. Alternatively, the determination unit 203 may be equipped with a learning device that learns multiple images of open and closed eyes and outputs the likelihood that an input image is an image of open eyes as the open / closed score. In this case, the learning device may use an algorithm such as Deep Learning or SVM (Support Vector Machine). Furthermore, the images used in learning may be created by any method, such as the closed template image 1202 and the open template image 1203 described above.

[0057] As described above, according to the information processing device 100 of the first embodiment, in biometric determination associated with face recognition, it is possible to apply appropriate biometric determination parameters tailored to the individual user more quickly and in a manner that does not impose a burden on the user.

[0058] <Second embodiment> In the second embodiment, an example will be described in which coefficients of a correction formula for normalizing or correcting an open / close score are registered as biometric determination parameters. Note that the hardware configuration applicable to the information processing device 100 of the second embodiment is the same as that shown in FIG. 1, the functional blocks are the same as those shown in FIG. 2, and an application example of the information processing device 100 is the same as that shown in FIG. 4, so illustration and description thereof will be omitted. In addition, in the information processing device 100 of the second embodiment, the registration information is generally the same as that shown in FIG. 3, and the flowchart of the information registration process is generally the same as that shown in FIG. 6, and the flowchart of the face recognition process is generally the same as that shown in FIG. 7, so illustration thereof will also be omitted.

[0059] Fig. 13 is a diagram used to explain an example of an open / close score corrected for each individual in the information processing device 100 of the second embodiment. Person 501 and graph 502 in Fig. 13(a) are the same as the example in Fig. 5(a), and person 503 and graph 504 in Fig. 13(b) are the same as the example in Fig. 5(b), so their explanation will be omitted. As explained in Fig. 5, the open / close score levels of person 501 and person 503 are different, and it is difficult to accurately determine whether the eyes are open or closed using a predetermined threshold value.

[0060] In the first embodiment described above, a threshold value was determined, but in the second embodiment, the open / close scores are corrected (normalized) so that the level of the open / close score for each person when the eyes are open is about 0.55, and the level of the open / close score for each person when the eyes are closed is about 0.45. That is, in the information processing device 100 of the second embodiment, by correcting (normalizing) the open / close scores, it is possible to accurately determine whether the eyes are open or closed using a predetermined threshold value (for example, 0.50).

[0061] Graph 1301 in Fig. 13(a) shows an example of the transition of the open / close score after correcting the level of the open / close score in the eyes open state shown in graph 502 to about 0.55. Similarly, graph 1303 in Fig. 13(b) shows an example of the transition of the open / close score after correcting the level of the open / close score in the eyes open state shown in graph 504 to about 0.55. Graph 1302 in Fig. 13(a) shows an example of the transition of the open / close score after correcting the level of the open / close score when the eyes are open shown in graph 502 so that the level of the open / close score when the eyes are open is about 0.55 and the level of the open / close score when the eyes are closed is about 0.45. Similarly, graph 1304 in Fig. 13(b) shows an example of the transition of the open / close score after correcting the level of the open / close score when the eyes are open shown in graph 504 so that the level of the open / close score when the eyes are open is about 0.55 and the level of the open / close score when the eyes are closed is about 0.45. Details of the process of correcting the level of the open / close score will be described later. In this way, the information processing apparatus 100 of the second embodiment registers, as parameters, the coefficients of the correction formula for correcting the level of the opening / closing score for each individual.

[0062] The information processing device 100 of the second embodiment, which corrects the level of the opening / closing score in accordance with each individual, will be described below, focusing on the differences from the first embodiment. Fig. 14 is a flowchart showing details of the biometric determination parameter determination process performed by the determination unit 203 in S603 of Fig. 6 in the information processing device 100 of the second embodiment. Note that the process of S801 is the same as that described in the first embodiment, and therefore description thereof will be omitted.

[0063] When the process proceeds to S1401 after S801, the determination unit 203 determines, as the biometric determination parameters, the coefficients of the correction formula for correcting the eye open / close score calculated in S801. Here, as in the previous example, it is assumed that the registered face image of the registered user stored in the storage unit 205 as registration information is registered as a face image with eyes well opened. The eye open / close score calculated in S801 is set to the open / close score SC1 when the eyes are open. Based on these, the determination unit 203 determines the coefficients of the correction formula for correcting the open / close score so that the level of the open / close score when the eyes are open is approximately 0.55. Specifically, the determination unit 203 determines the coefficients a and b of the correction formula for correcting the open / close score SC to the corrected open / close score SCc using the calculation formula shown in the following equation (2). For example, the coefficient a is set to 1, and the coefficient b is calculated using the following equation (3).

[0064] SCc=a·SC+b Formula (2) b=0.55-SC1 Equation (3)

[0065] Note that, as in the registered face information 403 of FIG. 3, a registered user may register multiple face images as registered face images. In such a case, the determination unit 203 may calculate an open / close score for each face in S801, calculate a coefficient of a correction formula for each face in the process of S1401, and further determine the average value of the calculated scores as the final coefficient to be used as the biometric determination parameter. Alternatively, the determination unit 203 may calculate an open / close score for each face in the process of S801, and further determine the average value as the open / close score SC1 for the registered user's eyes in an open state, and then calculate the coefficient in S1401. As another method, the determination unit 203 may check conditions such as the face orientation for multiple face images, and determine the coefficient based on only the face image with the best conditions, by performing the process of the flowchart of FIG. 14.

[0066] After the biometric determination parameter determination process shown in FIG. 14, in S605 of FIG. 6 in the second embodiment, the information processing device 100 registers the coefficients a and b of the correction formula obtained in the process of S1401 in the storage unit 205 (database) as biometric determination parameters.

[0067] 15 is a flowchart showing details of the biometrics determination process performed by the determination unit 203 in S706 of Fig. 7 in the information processing device 100 of the second embodiment. The processes of S901 to S903 and S905 to S910 are the same as those described in the first embodiment, and therefore their description will be omitted. In the case of the second embodiment, after processing S903, the determination unit 203 performs processing S1501, and then further performs processing S1502.

[0068] In the process of S1501, the determination unit 203 corrects the eye open / close score obtained in S903. Specifically, the determination unit 203 calculates a corrected open / close score SCc based on the open / close score SC obtained in S903, the coefficient a and the coefficient b determined as the biometric determination parameters in S705 and stored in the storage unit 205, and the above-mentioned formula (2).

[0069] Next, in the process of S1502, the determination unit 203 compares the corrected opening / closing score SCc calculated in S1501 with a predetermined threshold value (e.g., 0.50). If the corrected opening / closing score SCc is equal to or greater than the threshold value, the determination unit 203 proceeds to the process of S905, and if it is less than the threshold value, the determination unit 203 proceeds to the process of S906. The subsequent processes are the same as those described above.

[0070] Fig. 16 is a detailed flowchart of the biometrics determination parameter update process performed by the determination unit 203 in S708 of Fig. 7 in the information processing device 100 of the second embodiment. The biometrics determination parameter update process of Fig. 16 is a process performed when the determination unit 203 determines that the recognition target is a registered biometric user in the process of the flowchart of Fig. 7 described above.

[0071] First, in the process of S1001 described above, the determination unit 203 obtains the level of the open / close score when the target person's eyes are open (for example, the average value of the open / close scores determined to be in the open state) and the level of the open / close score when the target person's eyes are closed (for example, the average value of the open / close scores determined to be in the closed state). Here, the level of the open / close score when the eyes are open is defined as SC1, and the level of the open / close score when the eyes are closed is defined as SC2.

[0072] Next, in the process of S1601, the determination unit 203 determines the coefficient of the correction formula for correcting the open / close score. The coefficient may be determined using the same method as in S1401, or a different method. For example, similar to the example of the process of S802, the determination unit 203 calculates coefficient b using equation (3) from the level SC1 of the open / close score for the eyes open state obtained in S1001, and sets coefficient a to 1.

[0073] Alternatively, the judgment unit 203 may determine the coefficients a and b of the correction formula using the following equations (4) and (5) from the open / close score levels SC1 and SC2 so that the open / close score level when the eyes are open is approximately 0.55 and the open / close score level when the eyes are closed is approximately 0.45.

[0074] a=0.1 / (SC1-SC2) Equation (4) b=0.55-a·SC1 Equation (5)

[0075] Next, in the process of S1602, the determination unit 203 registers the coefficients of the correction formula determined in S1601 as biometric determination parameters in the storage unit 205 (database). That is, the determination unit 203 updates the biometric determination parameters by overwriting the already registered thresholds.

[0076] <Third embodiment> In the third embodiment, an example will be described in which an open-eye template image and a closed-eye template image appropriate for each individual are registered as biometric determination parameters. Note that the hardware configuration, functional blocks, and application examples applicable to the information processing device 100 of the third embodiment are the same as those in Figures 1, 2, and 4 described above, and therefore illustrations and descriptions thereof will be omitted. Furthermore, in the information processing device 100 of the third embodiment, the registration information, the flowchart of the information registration process, and the flowchart of the face recognition process are generally the same as those in Figures 3, 6, and 7, and therefore illustrations thereof will also be omitted.

[0077] In the third embodiment, closed and open eye images of a registered user are registered and used as open and closed eye template images (for example, closed template image 1202 and open template image 1203 in FIG. 12) used to calculate the eye open / closed score shown in Fig. 11. This makes it possible to calculate an open / closed eye score that can accurately determine whether the eyes are open or closed, regardless of the individual. The information processing device 100 of the third embodiment, which registers and uses closed and open eye images of a registered user as biometric determination parameters, will be described below, focusing on the differences from the first embodiment.

[0078] FIG. 17 is a detailed flowchart of the biometric determination parameter determination process performed by the determination unit 203 in S603 in the information processing apparatus 100 of the third embodiment. In the process of S1701, the determination unit 203 creates an eye-open template image based on the face image acquired in S601 and determines it as the biometric determination parameter to be registered. Specifically, it is assumed that a face image with good eye-open conditions is registered as the registered face image for face recognition of the registered user. As in the process of S1101, the determination unit 203 creates an image by cutting out the eye area from the face image acquired in S601 and sets this as the eye-open template image.

[0079] Note that, as in the registered face image 303 of FIG. 3, a registered user may register multiple face images as registered face images. In this case, the determination unit 203 may perform averaging on the eye-open template images obtained from the multiple face images and use the resulting average image as the final eye-open template image. Furthermore, in this case, the determination unit 203 may perform preprocessing to adjust the position and scale based on the position and scale of the pupils before performing the averaging. Alternatively, as in S801, the determination unit 203 may calculate an open / close score for each of pre-stored general-purpose template images and create an eye-open template image based on the face image with the highest open / close score. Similarly, the determination unit 203 may check conditions such as facial orientation for multiple face images and create an eye-open template image based only on the face image with the best conditions. Furthermore, when a configuration is adopted in which the eye-open / closed score is calculated based on a set of multiple eye-open and eye-closed templates, the determination unit 203 may create multiple eye template images from multiple face images.

[0080] After the biometric determination parameter determination process shown in FIG. 17, as the process of S605 in FIG. 6 in the third embodiment, the information processing device 100 registers the open eye template image obtained in S1701 in the storage unit 205 (database) as a biometric determination parameter.

[0081] Fig. 18 is a detailed flowchart of the biometrics determination parameter update process performed by the determination unit 203 in S708 in the information processing device 100 of the third embodiment. The biometrics determination parameter update process in Fig. 18 is performed when the determination unit 203 determines that the person to be recognized is a registered biometric user in the process of the flowchart in Fig. 7 described above.

[0082] First, in the process of S1801, the determination unit 203 acquires the input image (an image obtained by cutting out the eye area from the face image) created in S1101 in the biometric determination process of S706. The determination unit 203 also simultaneously refers to whether the eye state is determined to be open or closed based on the input image.

[0083] Next, in the process of S1802, the determination unit 203 creates template images of open or closed eyes based on each of the input images for which the eye state was determined to be open or closed, acquired in S1801. Note that if multiple input images are acquired in S1801, the best image may be selected as the template image, as in the process of S1701, or some or all of the multiple images may be used as the template image.

[0084] Next, in the process of S1803, the determination unit 203 registers the template image created in S1802 as a biometric determination parameter in the storage unit 205 (database). At this time, an already registered template image may be overwritten, or a new template image may be added to the already registered one.

[0085] As described above, in the case of the information processing device 100 of the third embodiment, a template image is referred to as a biometric determination parameter for each individual in the process of S705. Furthermore, the information processing device 100 calculates the eye opening / closing score using the referred template image in the eye opening / closing score calculation process shown in FIG.

[0086] <Fourth embodiment> In the first to third embodiments, an information processing device 100 that performs a biometric determination based on the blinking of a recognition target person has been described. In the fourth embodiment, an example of performing a biometric determination using a method different from that of the above-described embodiments will be described. In the fourth embodiment, for example, an example of performing a biometric determination process will be described in which a facial image of the recognition target person is checked for changes in skin color due to pulse waves, and based on the results, the recognition target person is determined to be a biometric person with a pulse wave or a non-bimetric person such as a photograph without a pulse wave. In the fourth embodiment, a method of registering the pulse rate history of each individual as a biometric determination parameter will be described in the information processing device 100 that performs a biometric determination based on the pulse wave of the recognition target person.

[0087] Note that the hardware configuration, functional blocks, and application examples applicable to the information processing device 100 of the fourth embodiment are the same as those in Figures 1, 2, and 4 described above, and therefore illustrations and descriptions thereof will be omitted. Also, illustrations and descriptions of flowcharts and the like similar to those in the previously described embodiments will be omitted for the information processing device 100 of the fourth embodiment. The information processing device 100 of the fourth embodiment will be described below, focusing on the differences from the previously described embodiments.

[0088] FIG. 19 is a flowchart of the biometric determination parameter determination process performed by the determination unit 203 in S603 in the information processing apparatus 100 of the fourth embodiment. First, in the process of S1901, the image acquisition unit 202 continuously acquires face images of the face of the user to be registered, captured by the imaging device 201, at a constant frame rate.

[0089] Next, in the process of S1902, the determination unit 203 extracts a skin region pixel by pixel from the face image acquired in S1901. Any extraction method can be used to extract the skin region. For example, the determination unit 203 converts the color representation of each pixel in the image into an HSV color space format and extracts a set of pixels in which H (hue), S (saturation), and V (lightness) each have values within a specific range determined in advance as the skin region. Alternatively, the determination unit 203 may exclude pixels that are statistically outliers from the HSV values of all pixels in the face region, under the assumption that the skin region occupies the majority of the face region, and extract the set of remaining pixels as the skin region. Alternatively, the determination unit 203 may include an extractor that extracts a person's skin region pixel by pixel, trained by deep learning.

[0090] Next, in the process of S1903, the determination unit 203 acquires an average hue value for the skin region extracted from the face images continuously acquired at a constant frame rate, thereby obtaining time-series data of the average hue value. Next, in the process of S1904, the determination unit 203 performs frequency analysis on the time-series data of the hue average value to obtain a frequency spectrum. For the frequency analysis, for example, FFT (Fast Fourier Transform) or the like can be used. Note that, when performing the frequency analysis, the determination unit 203 may perform preprocessing on the time-series data of the hue average value in advance using a window function or a band-pass filter.

[0091] Thereafter, in the process of S1905, the determination unit 203 estimates the pulse rate of the user to be registered from the frequency spectrum obtained in S1904. For example, the determination unit 203 estimates the frequency of the frequency component with the maximum amplitude among frequency components corresponding to the range of pulse rates of a typical human as the pulse rate of the user to be registered. Specifically, the determination unit 203 estimates the frequency of the frequency component with the maximum amplitude among frequency components in the range of 0.83 Hz to 3.33 Hz, which corresponds to the range of 50 bpm (beats per minute) to 200 bpm, as the pulse rate of the user to be registered.

[0092] After the biometric determination parameter determination process of FIG. 19, the information processing device 100 registers the pulse rate history obtained in S1905 as the biometric determination parameter of the registered user in the storage unit 205 (database) as the process of S605 of FIG.

[0093] 20 is a flowchart of the biometrics determination process performed by the determination unit 203 in S706 in the information processing device 100 of the fourth embodiment. The processes from S2001 to S2004 are the same as the processes from S1901 to S1904 in FIG. 19, and therefore, description thereof will be omitted.

[0094] After S2004, the process proceeds to S2005, where the determination unit 203 calculates the sum of the amplitudes of the specific frequency bands from the frequency spectrum obtained in S2004 as the pulse wave score. First, the determination unit 203 determines the specific frequency band based on the pulse rate history of the registered user referenced as the biometric determination parameter in S705. For example, the determination unit 203 calculates the average pulse rate from the pulse rate history of the registered user and determines a frequency band within a certain range from the frequency of the average pulse rate as the specific frequency band. Alternatively, the determination unit 203 calculates the average pulse rate and a standard deviation from the pulse rate history of the registered user, and further determines a frequency band within a range determined based on the standard deviation from the frequency of the average pulse rate as the specific frequency band. The range determined based on the standard deviation is, for example, within twice the standard deviation for both positive and negative values. This allows the determination unit 203 to more appropriately acquire the amplitude of the vibration component of the skin color due to the pulse wave of the registered user.

[0095] Next, in the process of S2006, the determination unit 203 checks whether the pulse wave score calculated in S2005 is equal to or greater than a predetermined threshold. The threshold is determined by experimentally searching for a threshold that can best classify living and non-living objects based on the pulse wave score based on the amplitude of a specific frequency band (for example, a threshold that maximizes the F value, which is the harmonic mean of the precision and recall). If the pulse wave score is equal to or greater than the threshold, the determination unit 203 proceeds to S907, and if the pulse wave score is less than the threshold, the determination unit 203 proceeds to S908.

[0096] When the process proceeds to S2007, the determination unit 203 determines that the biometrics of the person to be recognized who was determined to be a registered user in the face recognition process has been successful, as in S909, and determines that the person to be recognized is a registered biometric user. On the other hand, when the process proceeds to S2008, the determination unit 203 determines that the biometrics has not been successful, as in S9010.

[0097] Fig. 21 is a flowchart of the biometrics determination parameter update process performed by the determination unit 203 in S708 in the information processing device 100 of the fourth embodiment. The biometrics determination parameter update process in Fig. 21 is a process performed when the determination unit 203 determines that the recognition target is a registered biometric user.

[0098] First, in the process of S2101, the determination unit 203 refers to the frequency spectrum obtained in S2004 in the living body determination process of S706. Next, in the process of S2102, the determination unit 203 estimates the pulse rate based on the frequency spectrum referred to in S2101, in the same manner as in the process of S1905. Thereafter, in the process of S2103, the information processing device 100 registers (that is, updates) the pulse rate history obtained in S2102 as a living body determination parameter in the storage unit 205 (database).

[0099] In the fourth embodiment, an example has been described in which the pulse rate history of a registered user is registered as a biometric determination parameter. However, the information processing device 100 may register other parameters for appropriately performing biometric determination based on a pulse wave according to an individual. For example, the information processing device 100 may register the pulse rate at the time of face image registration as a reference value and use this instead of the average value in S2005. Furthermore, the information processing device 100 may register statistics such as the average value and standard deviation used in S2005 instead of the history. Furthermore, the information processing device 100 may register a threshold value used for determination and parameters used to correct the pulse wave score, as in the first and second embodiments.

[0100] In the fourth embodiment, only the pulse wave is used for the living body determination, but the living body determination may be performed by combining the eye opening / closing in the first to third embodiments and the pulse wave in the fourth embodiment. By combining these, the living body determination can be performed with higher accuracy.

[0101] <Other embodiments> In the above-described embodiments, the biometric determination was performed based on the detection results of the eye opening / closing movement and the pulse wave, but the biometric determination may also be performed based on other detection results. For example, the information processing device may request the recognition target to speak a specific word and perform the biometric determination using the detection results of the lip movement when the recognition target speaks the specific word. In this example, by using biometric determination parameters determined based on the lip movement when the registered user speaks the specific word, the biometric determination can be performed quickly and in a manner that does not burden the recognition target, as in the above-described embodiments.

[0102] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of specific embodiments for implementing the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0103] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a storage means for storing a liveness determination parameter used to determine whether a person shown in an image is a live person; a face recognition means for comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user, and performing face recognition to determine whether the person to be recognized is the registered user; a determination means for determining whether the person to be recognized as the registered user by the face recognition is a living body by using the living body determination parameter; and the storage means stores the biometric determination parameters determined based on the detection result of the eye opening and closing movement of the registered user; The information processing device is characterized in that the determination means determines whether the person to be recognized is a living being by comparing the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the facial recognition with the living being determination parameters stored in the memory means. (Configuration 2) the storage means also stores a facial image or facial feature amount of the registered user; The information processing device described in configuration 1 is characterized in that the face recognition means performs the face recognition by comparing the facial feature amounts of the registered user stored in the storage means or the facial feature amounts acquired from the facial image of the registered user with the facial feature amounts acquired from the facial image of the person to be recognized. (Configuration 3) the biometric determination parameter is a threshold determined based on a detection result of an eye opening / closing movement of the registered user, The information processing device according to configuration 1 or 2, characterized in that the determination means determines that the person to be recognized is a living body when the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the face recognition include detection results that are equal to or greater than the threshold value. (Configuration 4) the determination means includes a parameter determination means for determining the biometric determination parameter as the threshold value based on a time-series transition of a detection result of an eye opening / closing movement detected from a face image of the registered user, 4. The information processing apparatus according to configuration 3, wherein the storage means stores the threshold determined by the parameter determination means as the biometric determination parameter corresponding to the registered user. (Configuration 5) The information processing device according to configuration 4, wherein the parameter determination means determines a plurality of thresholds based on a time series transition of the detection results of eye opening and closing movements detected from a plurality of face images of the registered user, and determines an average value of the plurality of thresholds as the biometric determination parameter. (Configuration 6) The information processing device described in configuration 4, wherein the parameter determination means determines the biometric determination parameter as the threshold value based on a time-series transition of the detection results of eye opening and closing movements detected from a face image that meets predetermined conditions among a plurality of face images of the registered user. (Configuration 7) the biometric determination parameter is a coefficient for normalizing or correcting a detection result of an eye opening / closing movement, The information processing device according to configuration 1 or 2, characterized in that the determination means determines that the person to be recognized is a living body when the detection results, obtained by normalizing or correcting the detection results of movements including opening and closing the eyes of the person to be recognized who has been recognized as the registered user by the face recognition using the coefficient, include a detection result that is equal to or exceeds a predetermined threshold value. (Configuration 8) the determination means includes a parameter determination means for determining the biometric determination parameters as the coefficients of a correction formula for normalizing or correcting the detection results of the eye opening and closing movements based on a time-series transition of the detection results of the eye opening and closing movements detected from the face image of the registered user, 8. The information processing apparatus according to configuration 7, wherein the storage means stores the coefficients determined by the parameter determination means as the biometric determination parameters corresponding to the registered user. (Configuration 9) The information processing device according to configuration 8, wherein the parameter determination means determines a plurality of the coefficients based on a time series transition of the detection results of eye opening and closing movements detected from a plurality of face images of the registered user, and determines an average value of the plurality of coefficients as the biometric determination parameter. (Configuration 10) The information processing device according to configuration 8, wherein the parameter determination means determines the biometric determination parameter as the coefficient based on a time-series transition of the detection results of eye opening and closing movements detected from a face image that meets predetermined conditions among a plurality of face images of the registered user. (Configuration 11) the liveness determination parameter is at least one of a template image of the eye in an open state and a template image of the eye in a closed state, The information processing device described in configuration 1 or 2, characterized in that the determination means compares an image of the eyes acquired as a result of detecting movements including opening and closing of the eyes of the person to be recognized as the registered user by the face recognition with the template image to determine whether the person to be recognized is a living body. (Configuration 12) the determination means includes a parameter determination means for determining, as the biometric determination parameter, at least one of a template image of eyes in an open state and a template image of eyes in a closed state, both of which are detected from the face image of the registered user; 12. The information processing apparatus according to configuration 11, wherein the storage means stores the template image determined by the parameter determination means as the biometric determination parameters corresponding to the registered user. (Configuration 13) The information processing device according to configuration 12, wherein the parameter determination means determines, as the biometric determination parameter, at least one of a template image obtained by averaging a plurality of template images of the eyes with the eyes open, detected from a plurality of face images of the registered user, and a template image obtained by averaging a plurality of template images of the eyes with the eyes closed. (Configuration 14) The information processing device according to configuration 12, characterized in that the parameter determination means determines, as the biometric determination parameter, at least one of a template image of the eyes with the eyes open and a template image of the eyes with the eyes closed, detected from a face image that meets predetermined conditions among a plurality of face images of the registered user. (Configuration 15) The information processing device described in any one of configurations 4 to 6, 8 to 10, and 12 to 14, characterized in that when the determination means determines that the person to be recognized as the registered user by the face recognition is a living body, the determination means updates the living body determination parameter stored in the storage means corresponding to the registered user with the living body determination parameter determined by the parameter determination means based on the facial image of the person to be recognized. (Configuration 16) a storage means for storing a liveness determination parameter used to determine whether a person shown in an image is a live person; a face recognition means for comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user, and performing face recognition to determine whether the person to be recognized is the registered user; a determination means for determining whether the person to be recognized as the registered user by the face recognition is a living body by using the living body determination parameter; and the storage means stores the biometric evaluation parameters determined based on the pulse wave detection results of the registered user; The information processing device is characterized in that the determination means compares the pulse wave detection results of the person to be recognized as the registered user through the face recognition with the biometric determination parameters stored in the storage means to determine whether the person to be recognized is a living being. (Configuration 17) the storage means also stores a facial image or facial feature amount of the registered user; The information processing device described in configuration 16, wherein the face recognition means performs the face recognition by comparing the facial features of the registered user stored in the storage means or the facial features acquired from the facial image of the registered user with the facial features acquired from the facial image of the person to be recognized. (Configuration 18) the biometric determination parameter is a value indicating at least one of a pulse rate history, a pulse rate reference value, and a pulse rate statistic detected from a face image of the registered user, The information processing device described in configuration 16 or 17, characterized in that the determination means determines that the person to be recognized is a living person if the value detected according to the biometric determination parameter from the facial image of the person to be recognized as the registered user by the facial recognition is equal to or greater than a predetermined threshold value. (Configuration 19) 19. The information processing device according to configuration 18, wherein the predetermined threshold is a threshold determined based on the detection result of the pulse wave of the registered user. (Configuration 20) the living body determination parameter is a coefficient for normalizing or correcting the detection result of the pulse wave, The information processing device described in configuration 16 or 17, characterized in that the determination means determines that the person to be recognized is a living body when the detection results of the pulse wave of the person to be recognized who has been recognized as the registered user by the face recognition and normalized or corrected by the coefficient include detection results that are equal to or greater than a predetermined threshold. (Configuration 21) the determining means includes a parameter determining means for determining the living body determination parameters based on a detection result of the pulse wave of the registered user; 21. The information processing device according to any one of configurations 16 to 20, wherein the storage means stores the biometric determination parameters determined by the parameter determination means as the biometric determination parameters corresponding to the registered user. (Configuration 22) The information processing device described in configuration 21 is characterized in that when the determination means determines that the person to be recognized as the registered user by the face recognition is a living body, the determination means updates the living body determination parameter stored in the storage means corresponding to the registered user with the living body determination parameter determined by the parameter determination means based on the detection result of the pulse wave of the person to be recognized. (Method 1) a storage step of storing liveness determination parameters used to determine whether a person shown in an image is a live person; a face recognition step of comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user to perform face recognition to determine whether the person to be recognized is the registered user; a determination step of determining whether the person to be recognized as the registered user by the face recognition is a living body using the living body determination parameter; and The storage step stores the biometric evaluation parameters determined based on the detection result of the eye opening and closing movement of the registered user; An information processing method characterized in that in the judgment process, it is determined whether the person to be recognized is a living being by comparing the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the facial recognition with the living being judgment parameters stored in the storage process. (Method 2) a storage step of storing liveness determination parameters used to determine whether a person shown in an image is a live person; a face recognition step of comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user to perform face recognition to determine whether the person to be recognized is the registered user; a determination step of determining whether the person to be recognized as the registered user by the face recognition is a living body using the living body determination parameter; and The storage step stores the biometric determination parameters determined based on the pulse wave detection results of the registered user, An information processing method characterized in that the judgment process compares the pulse wave detection results of the person to be recognized who has been recognized as the registered user by the face recognition process with the biometric judgment parameters stored in the storage process to determine whether the person to be recognized is a living being. (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 22. [Explanation of symbols]

[0104] 100: Information processing device, 201: Imaging device, 302: Image acquisition unit, 203: Determination unit, 204: Face recognition unit, 205: Storage unit, 206: Feature amount calculation unit, 207: Comparison unit

Claims

1. a storage means for storing a liveness determination parameter used to determine whether a person shown in an image is a live person; a face recognition means for comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user, and performing face recognition to determine whether the person to be recognized is the registered user; a determination means for determining whether the person to be recognized as the registered user by the face recognition is a living body by using the living body determination parameter; and the storage means stores the biometric determination parameters determined based on the detection result of the eye opening and closing movement of the registered user; The information processing device is characterized in that the determination means determines whether the person to be recognized is a living being by comparing the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the facial recognition with the living being determination parameters stored in the memory means.

2. the storage means also stores a facial image or facial feature amount of the registered user; 2. The information processing device according to claim 1, wherein the facial recognition means performs the facial recognition by comparing the facial features of the registered user stored in the storage means or the facial features acquired from the facial image of the registered user with the facial features acquired from the facial image of the person to be recognized.

3. the biometric determination parameter is a threshold determined based on a detection result of an eye opening / closing movement of the registered user, The information processing device according to claim 1, characterized in that the determination means determines that the person to be recognized is a living body when the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the face recognition include detection results that are above the threshold.

4. the determination means includes a parameter determination means for determining the biometric determination parameter as the threshold value based on a time-series transition of a detection result of an eye opening / closing movement detected from a face image of the registered user, 4. The information processing apparatus according to claim 3, wherein the storage means stores the threshold determined by the parameter determination means as the biometric determination parameter corresponding to the registered user.

5. The information processing device according to claim 4, characterized in that the parameter determination means determines a plurality of thresholds based on the time-series progression of the detection results of eye opening and closing movements detected from a plurality of facial images of the registered user, and determines an average value of the plurality of thresholds as the biometric determination parameter.

6. The information processing device according to claim 4, characterized in that the parameter determination means determines the biometric determination parameter as the threshold value based on the time-series progression of the detection results of eye opening and closing movements detected from face images that meet predetermined conditions among multiple face images of the registered user.

7. the biometric determination parameter is a coefficient for normalizing or correcting a detection result of an eye opening / closing movement, The information processing device according to claim 1, characterized in that the determination means determines that the person to be recognized is a living body when the detection results, obtained by normalizing or correcting the detection results of movements including opening and closing the eyes of the person to be recognized who has been recognized as the registered user by the face recognition using the coefficient, include detection results that are equal to or greater than a predetermined threshold value.

8. the determination means includes a parameter determination means for determining the biometric determination parameters as the coefficients of a correction formula for normalizing or correcting the detection results of the eye opening and closing movements based on a time-series transition of the detection results of the eye opening and closing movements detected from the face image of the registered user, 8. The information processing apparatus according to claim 7, wherein the storage means stores the coefficients determined by the parameter determination means as the biometric determination parameters corresponding to the registered user.

9. The information processing device according to claim 8, characterized in that the parameter determination means determines the multiple coefficients based on a time series transition of the detection results of eye opening and closing movements detected from multiple facial images of the registered user, and determines an average value of the multiple coefficients as the biometric determination parameter.

10. The information processing device according to claim 8, characterized in that the parameter determination means determines the biometric determination parameter as the coefficient based on a time series change in the detection results of eye opening and closing movements detected from a face image that meets predetermined conditions among a plurality of face images of the registered user.

11. the liveness determination parameter is at least one of a template image of the eye in an open state and a template image of the eye in a closed state, The information processing device according to claim 1, characterized in that the determination means compares an image of the eyes acquired as a result of detecting movements including opening and closing of the eyes of the person to be recognized as the registered user by the face recognition with the template image to determine whether the person to be recognized is a living being.

12. the determination means includes a parameter determination means for determining, as the biometric determination parameter, at least one of a template image of eyes in an open state and a template image of eyes in a closed state, both of which are detected from the face image of the registered user; 12. The information processing apparatus according to claim 11, wherein the storage means stores the template image determined by the parameter determination means as the biometric determination parameters corresponding to the registered user.

13. 13. The information processing device according to claim 12, wherein the parameter determination means determines, as the biometric determination parameter, at least one of a template image obtained by averaging a plurality of template images of the eyes with the eyes open, detected from a plurality of face images of the registered user, and a template image obtained by averaging a plurality of template images of the eyes with the eyes closed.

14. The information processing device according to claim 12, characterized in that the parameter determination means determines, as the biometric determination parameter, at least one of a template image of the eyes with the eyes open and a template image of the eyes with the eyes closed, detected from a face image that meets predetermined conditions among a plurality of face images of the registered user.

15. An information processing device as described in any one of claims 4 to 6, 8 to 10, and 12 to 14, characterized in that when the determination means determines that the person to be recognized as the registered user by the facial recognition is a living body, the determination means updates the living body determination parameter stored in the storage means corresponding to the registered user with the living body determination parameter determined by the parameter determination means based on the facial image of the person to be recognized.

16. a storage means for storing a liveness determination parameter used to determine whether a person shown in an image is a live person; a face recognition means for comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user, and performing face recognition to determine whether the person to be recognized is the registered user; a determination means for determining whether the person to be recognized as the registered user by the face recognition is a living body by using the living body determination parameter; and the storage means stores the biometric evaluation parameters determined based on the pulse wave detection results of the registered user; The information processing device is characterized in that the determination means compares the pulse wave detection results of the person to be recognized as the registered user through the face recognition with the biometric determination parameters stored in the storage means to determine whether the person to be recognized is a living being.

17. the storage means also stores a facial image or facial feature amount of the registered user; 17. The information processing device according to claim 16, wherein the face recognition means performs the face recognition by comparing the facial features of the registered user stored in the storage means or the facial features acquired from the facial image of the registered user with the facial features acquired from the facial image of the person to be recognized.

18. the biometric determination parameter is a value indicating at least one of a pulse rate history, a pulse rate reference value, and a pulse rate statistic detected from a face image of the registered user, The information processing device according to claim 16, characterized in that the determination means determines that the person to be recognized is a living person when the value detected according to the biometric determination parameter from the facial image of the person to be recognized as the registered user by the facial recognition is equal to or greater than a predetermined threshold value.

19. 19. The information processing apparatus according to claim 18, wherein the predetermined threshold is determined based on a result of detecting a pulse wave of the registered user.

20. the living body determination parameter is a coefficient for normalizing or correcting the detection result of the pulse wave, The information processing device according to claim 16, characterized in that the determination means determines that the person to be recognized is a living body when the detection results obtained by normalizing or correcting the pulse wave detection results of the person to be recognized as the registered user by the face recognition include a detection result that is equal to or greater than a predetermined threshold.

21. the determining means includes a parameter determining means for determining the living body determination parameters based on a detection result of the pulse wave of the registered user; 21. The information processing apparatus according to claim 16, wherein the storage means stores the biometric determination parameters determined by the parameter determination means as the biometric determination parameters corresponding to the registered user.

22. The information processing device according to claim 21, characterized in that when the determination means determines that the person to be recognized as the registered user by the face recognition is a living body, the determination means updates the living body determination parameter stored in the storage means corresponding to the registered user with the living body determination parameter determined by the parameter determination means based on the detection result of the pulse wave of the person to be recognized.

23. a storage step of storing liveness determination parameters used to determine whether a person shown in an image is a live person; a face recognition step of comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user to perform face recognition to determine whether the person to be recognized is the registered user; a determination step of determining whether the person to be recognized as the registered user by the face recognition is a living body using the living body determination parameter; and The storage step stores the biometric evaluation parameters determined based on the detection result of the eye opening and closing movement of the registered user; An information processing method characterized in that in the judgment process, it is determined whether the person to be recognized is a living being by comparing the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the facial recognition with the living being judgment parameters stored in the storage process.

24. a storage step of storing liveness determination parameters used to determine whether a person shown in an image is a live person; a face recognition step of comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user to perform face recognition to determine whether the person to be recognized is the registered user; a determination step of determining whether the person to be recognized as the registered user by the face recognition is a living body using the living body determination parameter; and The storage step stores the biometric determination parameters determined based on the pulse wave detection results of the registered user, An information processing method characterized in that the judgment process compares the pulse wave detection results of the person to be recognized who has been recognized as the registered user by the face recognition process with the biometric judgment parameters stored in the storage process to determine whether the person to be recognized is a living being.

25. Computer, a storage means for storing a liveness determination parameter used to determine whether a person shown in an image is a live person; a face recognition means for comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user, and performing face recognition to determine whether the person to be recognized is the registered user; a determination means for determining whether the person to be recognized as the registered user by the face recognition is a living body by using the living body determination parameter; and the storage means stores the biometric determination parameters determined based on the detection result of the eye opening and closing movement of the registered user; The determination means is a program that functions as an information processing device that determines whether the person to be recognized is a living being by comparing the detection results of the movements, including opening and closing of the eyes, of the person to be recognized who has been recognized as the registered user by the facial recognition with the living being determination parameters stored in the storage means.

26. Computer, a storage means for storing a liveness determination parameter used to determine whether a person shown in an image is a live person; a face recognition means for comparing a facial feature amount acquired from a face image of a person to be recognized with a facial feature amount of a registered user, and performing face recognition to determine whether the person to be recognized is the registered user; a determination means for determining whether the person to be recognized as the registered user by the face recognition is a living body by using the living body determination parameter; and the storage means stores the biometric evaluation parameters determined based on the pulse wave detection results of the registered user; The determination means is a program that functions as an information processing device that compares the pulse wave detection results of the person to be recognized as the registered user through the facial recognition with the biometric determination parameters stored in the storage means to determine whether the person to be recognized is a living being.

Images