Judgment method, judgment device and computer program product

Abstract

A judgment method, comprising: selecting two or more living body judgment processes from a plurality of living body judgment processes for judging whether a person included in an image is alive, determining an order for executing the two or more selected living body judgment processes, executing the two or more selected living body judgment processes respectively in the determined order, and performing in each of the two or more selected living body judgment processes: prompting a user to perform an action required for the living body judgment process; obtaining a facial image including the face of the user when performing the prompted action; judging whether the user is alive based on features of facial parts included in the facial image, judging whether the judgment results obtained from the two or more selected living body judgment processes respectively meet specified conditions, judging that the user is alive if the judgment results meet the specified conditions, and judging that the user is not alive if the judgment results do not meet the specified conditions.

Description

Technical Field

[0001] The present invention relates to a technology for determining whether a person included in an image is a living body. Background Art

[0002] With the recent development of online systems, identity verification, previously conducted in person, is increasingly being conducted online. When conducting identity verification online, fraudulent use of images, such as still or moving images of another person's face, prepared by the user, has become a problem. As a countermeasure to this fraud, there exists liveness determination technology that determines whether the person depicted in an identity verification image is alive (hereinafter referred to as liveness determination). The following patent documents are known as examples of liveness determination related to liveness determination.

[0003] For example, Patent Document 1 discloses a technology for determining liveness based on the degree of a specific expression, facial orientation, eye openness, mouth openness, or gaze direction acquired from a user's facial image. Patent Document 2 discloses a technology for presenting prompt information to a user and performing liveness determination based on temporal changes in gaze relative to the prompt information detected from a sequence of facial images representing the user's face. Patent Document 3 discloses a technology for determining liveness based on movements of the living being, such as blinking, contained in images of the living being.

[0004] Patent Document 4 discloses a technology for determining liveness based on temporal changes in specified color information extracted from a sequence of facial images. Patent Document 5 discloses a technology for determining liveness based on two-dimensional image data of the face of the subject being authenticated, captured by an imaging element, and image plane phase difference information (segments

[0030] -

[0032] ). Patent Document 6 discloses a technology for determining liveness based on the presence of a frame surrounding the face of the subject person in a captured image.

[0005] However, anti-spoofing countermeasures in liveness determination need further improvement.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-009453

[0009] Patent Document 2: International Publication No. 2016 / 059786

[0010] Patent Document 3: Japanese Patent Application Laid-Open No. 2017-049867

[0011] Patent Document 4: Japanese Patent Gazette No. 6544244

[0012] Patent Document 5: Japanese Patent Application Publication No. 2018-173731

[0013] Patent Document 6: Japanese Patent Application Publication No. 2018-169943 Summary of the Invention

[0014] The present invention is made to solve the above-mentioned problems, and its purpose is to further improve the anti-spoofing countermeasures in living body determination.

[0015] A judgment method according to one embodiment of the present invention is a judgment method of a judgment device, wherein a computer of the judgment device executes the following steps: selecting two or more living body judgment processes from a plurality of living body judgment processes for judging whether a person included in an image is a living body; determining an order for executing the two or more selected living body judgment processes; executing the two or more selected living body judgment processes respectively according to the determined order; performing, in each of the two or more selected living body judgment processes: (1) prompting a user for an action required for the living body judgment process; (2) acquiring a facial image including the face of the user when performing the prompted action; (3) judging whether the user is a living body based on features of facial parts included in the facial image; judging whether the judgment results obtained from the two or more selected living body judgment processes meet a specified condition; and, if the judgment results meet the specified condition, judging that the user is a living body, and if the judgment results do not meet the specified condition, judging that the user is not a living body. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an external view of the living body assessment system according to the first embodiment of the present invention.

[0017] Figure 2 This is a block diagram showing an example of the overall configuration of the living body assessment system according to the first embodiment of the present invention.

[0018] Figure 3 This is a flowchart showing an example of processing by the determination device according to the first embodiment of the present invention.

[0019] Figure 4 This is a flowchart showing an example of living body determination processing based on facial orientation.

[0020] Figure 5 This is a flowchart showing an example of living body determination processing based on eye direction.

[0021] Figure 6 This is a flowchart showing an example of living body determination processing based on the eye opening and closing status.

[0022] Figure 7 This is a flowchart showing an example of living body determination processing based on the mouth opening and closing state.

[0023] Figure 8 It is a diagram showing the feature points of a face.

[0024] Figure 9 This is a flowchart showing an example of living body determination processing based on the wearing and removal status of glasses.

[0025] Figure 10 This is a flowchart showing an example of frame determination processing.

[0026] Figure 11 This is a diagram showing an example of a facial region detected from a user image.

[0027] Figure 12 This is a diagram showing an example of a horizontal edge detected in the vertical line detection process.

[0028] Figure 13 This is a diagram showing an example of vertical edges detected in the horizontal line detection process.

[0029] Figure 14 This is a diagram showing an example of a facial region included in a user image.

[0030] Figure 15 This is a diagram showing another example of the face region included in the user image. DETAILED DESCRIPTION

[0031] The process of completing the present invention

[0032] With the recent development of online systems, identity verification that used to be done face-to-face is increasingly being conducted online. For example, in the field of eKYC (electronic Know Your Customer) such as online account opening at financial institutions, identity verification is now provided by submitting identity verification documents and selfies online.

[0033] Furthermore, in recent years, biometric authentication using human body parts such as fingerprints and faces has become increasingly common. Facial recognition technology, in particular, is extremely convenient and easy to implement because it can be performed without physical contact with the user and using images captured by cameras without specialized features. Consequently, its use as an online identity verification method has attracted considerable attention.

[0034] However, the sheer convenience of facial recognition technology has led to the problem of impersonation by using images containing someone else's face. To combat this impersonation, various companies have developed living body detection technologies to determine whether the person in an image is alive. For example, one known technology illuminates an area believed to be the user's face with a dot pattern of light when capturing an image for facial recognition. If the facial contours are detected, the user is determined to be alive. Furthermore, the living body detection technologies described in Patent Documents 1 to 6 are also known.

[0035] However, if one of the above-mentioned living body determination technologies is fixedly used for living body determination, a malicious user can easily prepare an image that is determined to be a living body by the used living body determination technology, thereby facilitating fraud.

[0036] In view of this, the present inventors have devoted themselves to studying a technique for making it difficult for a malicious user to cheat using a pre-prepared image by appropriately using a plurality of living body determination techniques, and have come up with the embodiments of the present invention shown below.

[0037] One aspect of the present invention relates to a judgment method of a judgment device, which allows a computer of the judgment device to execute the following steps: selecting two or more living body judgment processes from a plurality of living body judgment processes for judging whether a person included in an image is a living body; determining the order in which the two or more selected living body judgment processes are executed; executing the two or more selected living body judgment processes respectively according to the determined order; performing in each of the two or more selected living body judgment processes: (1) prompting a user for an action required for the living body judgment process; (2) acquiring a facial image including the face of the user when performing the prompted action; (3) judging whether the user is a living body based on features of facial parts included in the facial image; judging whether the judgment results obtained from the two or more selected living body judgment processes meet a specified condition; and, if the judgment results meet the specified condition, judging that the user is a living body, and if the judgment results do not meet the specified condition, judging that the user is not a living body.

[0038] According to this configuration, two or more liveness determination processes are selected from a plurality of liveness determination processes, and the order in which they are to be executed is determined. These two or more liveness determination processes are then executed in the determined order. Therefore, the number, content, and order of the liveness determination processes executed to determine whether a user is alive exponentially increase the number of patterns in which liveness determination is executed.

[0039] Furthermore, in this configuration, during each living body determination process, the user is prompted with the actions required for the living body determination process, and whether the user is alive is determined based on facial features contained in a facial image, where the facial image includes the user's face when performing the actions prompted. Furthermore, whether the user is alive is determined based on whether the determination result obtained from each living body determination process satisfies predetermined conditions.

[0040] Therefore, this configuration makes it virtually impossible for a malicious user to prepare in advance a facial image containing facial features corresponding to the motions suggested in each living body determination process. Thus, this configuration can make it difficult to cheat using a prepared image.

[0041] In the above-mentioned determination method, when selecting the two or more living body determination processes, the two or more living body determination processes may be randomly selected from the plurality of living body determination processes.

[0042] According to this configuration, two or more living body determination processes are randomly selected from a plurality of living body determination processes. Therefore, this configuration makes it difficult for a malicious user to prepare a facial image that is determined to be living in all living body determination processes.

[0043] In the above determination method, when determining the order in which the two or more selected living body determination processes are to be executed, the order in which the two or more selected living body determination processes are to be executed may be determined randomly.

[0044] According to this configuration, the order in which the selected two or more living body determination processes are executed is randomly determined. Therefore, this configuration makes it difficult for a malicious user to prepare a facial image that is determined to be living in all living body determination processes.

[0045] In the above-mentioned judgment method, the multiple liveness judgment processes may also include: liveness judgment processing using face orientation as the part feature of the face; liveness judgment processing using eye orientation as the part feature of the face; liveness judgment processing using eye opening and closing state as the part feature of the face; liveness judgment processing using mouth opening and closing state as the part feature of the face; and liveness judgment processing using glasses wearing and removing state as the part feature of the face.

[0046] According to this configuration, executing two or more of the liveness determination processings of using face orientation as a face feature, using eye orientation as a face feature, using eye open / closed state as a face feature, using mouth open / closed state as a face feature, and using glasses wearing / removing state as a face feature can make it difficult to commit fraud using pre-prepared images.

[0047] In the above-mentioned determination method, in each of the two or more selected living body determination processes, the processes (1) to (3) may be performed multiple times, and an action to be prompted to the user may be randomly determined in each process (1).

[0048] This configuration exponentially increases the number, content, and order of liveness determination processes performed to determine whether a user is alive, as well as the number and content of actions prompted during each liveness determination process. This makes it more difficult for a malicious user to prepare facial images that could be judged as alive.

[0049] In the above determination method, when selecting the two or more living body determination processes, the state of the user may be detected, and a living body determination process whose action required by the living body determination process is suitable for the detected state of the user may be selected.

[0050] According to this configuration, two or more living body assessment processes requiring actions appropriate to the user's state can be selected.

[0051] In the above-mentioned judgment method, when judging whether the judgment result satisfies the prescribed conditions, the judgment result may be quantified, and when the result of weighted addition of the quantified result using the coefficient prescribed for each living body judgment processing satisfies the prescribed numerical conditions, it may be judged that the judgment result satisfies the prescribed conditions.

[0052] According to this configuration, the numerical values ​​of the determination results of each living body determination process are weighted and added together using coefficients specified for each living body determination process. If the result of this weighted addition satisfies a specified numerical condition, the determination result is determined to have satisfied the specified condition. Therefore, this configuration can improve the accuracy of determining whether the user is alive.

[0053] In the above judgment method, the computer may also further perform the following steps: obtaining a user image containing the user's face, judging whether the user image contains a frame surrounding the person's face, and if it is judged that the user image contains the frame, judging that the user is not a living body; if it is judged that the user image does not contain the frame, selecting the two or more living body judgment processing and subsequent processing.

[0054] According to this configuration, it is determined whether the acquired user image contains a frame surrounding a person's face. Therefore, it is possible to determine whether the user image contains a frame of a photograph or display containing a person's face.

[0055] If the user's image is determined to contain a frame, it is assumed that the user is attempting to use a photograph or display image containing another person's face for the purpose of determining a living being. In this case, this configuration can determine that the user is not a living being. On the other hand, if the user's image is determined to contain no frame, the process following the selection of two or more living being determination processes is performed. Thus, the same effects as the above-described determination method can be achieved.

[0056] In the above-mentioned determination method, when determining whether the user image includes the frame, a facial area including the user's face may be detected from the user image, and vertical line detection processing and horizontal line detection processing may be performed. In the vertical line detection processing, a first edge closest to the left end of the facial area and having a length greater than a first specified length and a second edge closest to the right end of the facial area and having a length greater than the first specified length, among edges extending in the vertical direction included in the user image, are detected. In the horizontal line detection processing, a third edge closest to the upper end of the facial area and a fourth edge closest to the lower end of the facial area, among edges extending in the horizontal direction included in the user image, are detected. In the horizontal line detection processing, both ends of the third edge are closer to the facial area than the first and second edges and have a length greater than a second specified length, and both ends of the fourth edge are closer to the facial area than the first and second edges and have a length greater than the second specified length. If the total number of edges detected by the vertical line detection processing and the horizontal line detection processing is three or more, it is determined that the user image includes the frame.

[0057] In this configuration, vertical line detection processing is performed to detect a facial region containing the user's face from a user image, and to detect, among the vertically extending edges contained in the user image, a first edge and a second edge that are respectively closest to the left and right ends of the facial region and have a length greater than a predetermined length. This allows detection of two vertically extending frame lines that constitute the frame of a photograph or displayed image containing a person's face.

[0058] Furthermore, horizontal line detection processing is performed to detect a third edge and a fourth edge, respectively, of the horizontally extending edges included in the user image, which are closest to the upper and lower ends of the facial region and are closer to the facial region than the detected first and second edges. This allows detection of two horizontally extending frame lines that constitute the frame of a photograph or displayed image containing a person's face.

[0059] Furthermore, according to this configuration, if the total number of edges detected by the vertical line detection processing and the horizontal line detection processing is three or more, it is determined that the user image contains a frame. Therefore, when this configuration acquires a user image containing a photograph or display that includes a person's face, and if four frame lines can be detected from this user image, it can be determined that the user image contains a frame. Furthermore, even if, for example, due to the shooting environment, a user image is acquired that does not contain a single frame line of the frame of the photograph or display that includes a person's face, and only three frame lines can be detected from this user image, it can still be determined that the user image contains a frame.

[0060] In the above-mentioned judgment method, when judging whether the user image contains the border, a facial area containing the user's face can be detected from the user image. When the width of the facial area is greater than a prescribed upper limit width, or when the height of the facial area is greater than a prescribed upper limit height, it is judged that the user image contains the border.

[0061] To prevent a photo or display containing another person's face from being captured within its frame, a malicious user might bring the photo or display close to the camera and capture the captured image as the user's image. In this case, the width of the facial area in the user's image may exceed the upper limit, or the height of the facial area may exceed the upper limit.

[0062] However, in this configuration, if the width of the facial region detected from the user image exceeds the upper width limit or the height of the facial region exceeds the upper height limit, the user image is determined to contain a frame, and the user corresponding to the user image is determined to be non-living. Therefore, this configuration makes it difficult for a malicious user to use pre-prepared images such as those described above to commit fraud.

[0063] In the above-mentioned judgment method, when judging whether the user image contains the frame, a facial area containing the user's face can be detected from the user image. When the upper end of the facial area exists within a first margin specified from the upper end of the user image, or when the lower end of the facial area exists within a second margin specified from the lower end of the user image, and when the first edge and the second edge are detected in the vertical line detection process, it is judged that the user image contains the frame.

[0064] A malicious user may capture a photo or display containing another person's face so that the face is positioned near the top or bottom of the captured image in order to minimize the frame of the photo or display. This captured image may be used as the user's image. In this case, only the left and right frame lines of the photo or display may be included in the user's image.

[0065] However, in this configuration, if the upper end of the facial region detected from the user image lies within a first margin specified from the upper end of the user image, or if the lower end of the facial region lies within a second margin specified from the lower end of the user image, and if first and second edges extending in the vertical direction are detected, the user image is determined to contain a frame. In this case, this configuration determines that the user is not a living being. Therefore, this configuration makes it difficult for malicious users to use pre-prepared images such as those described above to commit fraud.

[0066] In the above-mentioned judgment method, when judging whether the user image includes the border, when the left end of the facial area exists within a third margin specified from the left end of the user image, or when the right end of the facial area exists within a fourth margin specified from the right end of the user image, and when the first edge or the second edge detected in the vertical line detection processing intersects the third edge or the fourth edge detected in the horizontal line detection processing, it is judged that the user image includes the border.

[0067] To minimize the appearance of the frame of a photo or display containing another person's face, a malicious user might capture the photo or display so that the person's face is positioned near the left or right edge of the captured image and then use the captured image as the user's image. In this case, only the right and lower borders of the photo or display's frame, such as the two L-shaped intersecting borders, may be included in the user's image.

[0068] However, in this configuration, if the left end of the facial region detected in the user image lies within a third margin specified from the left end of the user image, or the right end of the facial region lies within a fourth margin specified from the right end of the user image, and the first or second edge extending vertically is detected and intersects the third or fourth edge extending horizontally, the user image is determined to contain a frame. In this case, this configuration determines that the user corresponding to the user image is not a living person. Therefore, this configuration makes it difficult for malicious users to use pre-prepared images such as those described above to commit fraud.

[0069] Another aspect of the present invention relates to a determination device including: a determination method determination unit that selects two or more living body determination processes from a plurality of living body determination processes for determining whether a person included in an image is alive and determines an order in which the two or more selected living body determination processes are executed; an execution unit that executes the two or more selected living body determination processes in the determined order; an output unit that prompts a user to perform an action required for the living body determination process in each of the two or more selected living body determination processes; a facial image acquisition unit that acquires a facial image including the user's face when performing the prompted action; a first determination unit that determines whether the user is alive based on features of facial parts included in the facial image; and a second determination unit that determines whether a determination result obtained from each of the two or more selected living body determination processes satisfies a specified condition, and if the determination result satisfies the specified condition, determines that the user is alive, and if the determination result does not satisfy the specified condition, determines that the user is not alive.

[0070] Yet another aspect of the present invention relates to a judgment program that causes a computer to function as a judgment device, the judgment program causing the computer to function as the following units: a judgment method determination unit that selects two or more living body judgment processes from a plurality of living body judgment processes for determining whether a person included in an image is living, and determines an order in which to execute the selected two or more living body judgment processes; an execution unit that executes each of the selected two or more living body judgment processes in the determined order; an output unit that prompts a user to perform an action required for the living body judgment process in each of the selected two or more living body judgment processes; a facial image acquisition unit that acquires a facial image including the user's face when performing the prompted action; a first judgment unit that determines whether the user is living based on features of facial parts included in the facial image; and a second judgment unit that determines whether a judgment result obtained from each of the selected two or more living body judgment processes satisfies a predetermined condition, and if the judgment result satisfies the predetermined condition, determines that the user is living, and if the judgment result does not satisfy the predetermined condition, determines that the user is not living.

[0071] According to these configurations, the same operational effects as those of the above-mentioned determination method can be obtained.

[0072] The present invention can also be implemented as a judgment system that operates according to the judgment program. In addition, it is needless to say that the computer program can be circulated through a computer-readable non-transitory storage medium such as a CD-ROM or a communication network such as the Internet.

[0073] In addition, the embodiments described below all represent a specific example of the present invention. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not used to limit the present invention. In addition, among the constituent elements of the following embodiments, the constituent elements not recorded in the independent claims representing the most superior concepts are described as arbitrary constituent elements. In addition, in all embodiments, each content can also be combined.

[0074] Implementation Method 1

[0075] Figure 1 This is an external view of a living body assessment system 100 according to Embodiment 1 of the present invention. Living body assessment system 100 is implemented using a mobile terminal device such as a smartphone or tablet. However, this is merely an example; living body assessment system 100 may also be implemented using a combination of a stationary computer or cloud server, a camera, and a display.

[0076] The living body assessment system 100 includes a assessment device 1, an imaging device 2, and a display 3. The assessment device 1 performs so-called living body assessment to determine whether a person U1 included in an image captured by the imaging device 2 is a living body.

[0077] The imaging device 2 is formed by a camera installed in a mobile terminal device and captures a color visible light image at a predetermined frame period.

[0078] The display 3 is formed of a display device such as a liquid crystal display device or an organic EL (ElectroLuminescence) display device installed in a mobile terminal device. The display 3 displays the determination result of the determination device 1 on whether the person U1 included in the image captured by the imaging device 2 is a living body.

[0079] Figure 2 This is a block diagram showing an example of the overall configuration of a living body assessment system 100 according to Embodiment 1 of the present invention. Assessment device 1 includes a processor 10 (computer), memory 20, sensor 30, operating unit 40, and communication unit 50. Processor 10 is, for example, a CPU (Central Processing Unit). Processor 10 includes a reception unit 11, a determination method determination unit 12, an execution unit 13, an output unit 14, a facial image acquisition unit 15, a first assessment unit 16, and a second assessment unit 17. Reception unit 11 through second assessment unit 17 are implemented, for example, by processor 10 executing a living body assessment program (assessment program).

[0080] The memory 20 is formed of a nonvolatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), and stores various information used by the processor 10 for control.

[0081] The sensor 30 is formed of a human body sensor or an image sensor, and detects the presence of a person within a predetermined short distance from the sensor 30. The operation unit 40 is formed of a touch panel, etc., and is used by the user to perform various operations on the determination device 1. The communication unit 50 is formed using a communication circuit compatible with any communication method such as Ethernet (registered trademark), and communicates with external devices.

[0082] The receiving unit 11 receives input of an instruction to execute the liveness determination. Specifically, the receiving unit 11 receives input of an instruction to execute the liveness determination when a detection signal indicating the detection of a human being is input from the sensor 30. Alternatively, the receiving unit 11 receives input of an instruction to execute the liveness determination when a user operates the operating unit 40 and inputs information indicating an instruction to execute the liveness determination. Alternatively, the receiving unit 11 receives input of an instruction to execute the liveness determination when information indicating a request to execute the liveness determination is input from an external device via the communication unit 50 or when information indicating an instruction to execute the liveness determination is input from an application currently being executed by the processor 10. The method for inputting an instruction to execute the liveness determination is not limited to this.

[0083] The determination method determination unit 12 selects two or more living body determination processes from a plurality of living body determination processes for determining whether a person included in an image is a living body, and determines the execution order of the selected two or more living body determination processes.

[0084] Specifically, the determination method determination unit 12 randomly selects two or more living body determination processes from a plurality of living body determination processes. These plurality of living body determination processes include: a living body determination process based on facial orientation; a living body determination process based on eye orientation; a living body determination process based on eye opening and closing; a living body determination process based on mouth opening and closing; and a living body determination process based on whether glasses are worn or removed. A control program implementing these plurality of living body determination processes is stored in the memory 20.

[0085] Furthermore, the multiple living body assessment processes may also include living body assessment processes based on other facial features. Furthermore, the method by which the assessment method determination unit 12 selects two or more living body assessment processes is not limited to the method described above. For example, the assessment method determination unit 12 may regularly select two or more living body assessment processes from the multiple living body assessment processes by selecting a predetermined number of living body assessment processes in the order in which the control programs implementing the multiple living body assessment processes are stored in the memory 20.

[0086] The determination method determination unit 12 randomly determines the order in which the two or more selected living body determination processes are to be executed. However, the determination method determination unit 12 is not limited to this method. For example, the determination method determination unit 12 may regularly determine the order in which the two or more living body determination processes are to be executed, for example, by determining the order in which the control programs implementing the two or more living body determination processes are stored in the memory 20.

[0087] The execution unit 13 executes the two or more living body determination processes selected by the determination method determination unit 12 in the order determined by the determination method determination unit 12 .

[0088] During each living body determination process executed by the execution unit 13, the output unit 14 presents information indicating the actions required for each living body determination process (hereinafter referred to as action information) to the user. Specifically, the output unit 14 presents the action information to the user by displaying the action information on the display 3. However, the output unit 14 is not limited to this embodiment; the output unit 14 may also present the action information to the user by causing a speaker (not shown) provided in the determination device 1 to output a voice indicating the action information.

[0089] The facial image acquisition unit 15 acquires a facial image during each living body determination process executed by the execution unit 13. The facial image includes the face of the user when performing an action according to the action information provided by the output unit 14. Specifically, after the output unit 14 provides the action information to the user, the facial image acquisition unit 15 causes the camera 2 to start capturing an image and transmits the image of the person U1 ( Figure 1 ) is acquired as a facial image. In addition, the facial image acquisition unit 15 sequentially acquires facial images captured at a predetermined frame rate.

[0090] The first determination unit 16 determines whether the user is alive based on the facial features included in the facial image acquired by the facial image acquisition unit 15. Specifically, the multiple liveness determination processes described above include: a liveness determination process that uses facial orientation as a facial feature; a liveness determination process that uses eye orientation as a facial feature; a liveness determination process that uses eye opening and closing as a facial feature; a liveness determination process that uses mouth opening and closing as a facial feature; and a liveness determination process that uses glasses being worn or removed as a facial feature. For example, in the liveness determination process that uses facial orientation as a facial feature, the first determination unit 16 determines whether the user is alive based on the facial orientation included in the facial image acquired by the facial image acquisition unit 15.

[0091] The second determination unit 17 determines whether the determination results obtained from each of the two or more living body determination processes selected by the determination method determination unit 12 satisfy a predetermined condition. If the determination results obtained from each of the two or more living body determination processes satisfy the predetermined condition, the second determination unit 17 determines that the user is living. If the determination results obtained from each of the two or more living body determination processes do not satisfy the predetermined condition, the second determination unit 17 determines that the user is not living.

[0092] For example, the second determination unit 17 determines that the predetermined condition is satisfied when all the determination results obtained from the two or more selected living body determination processes indicate that the user is alive. Alternatively, the second determination unit 17 may determine that the predetermined condition is satisfied when a predetermined number or more (e.g., more than half) of the determination results obtained from the two or more selected living body determination processes indicate that the user is alive.

[0093] Next, explain Figure 2 The processing of the judgment device 1 is shown. Figure 3 This is a flowchart showing an example of processing of the determination device 1 according to the first embodiment of the present invention.

[0094] In step S100 , if the receiving unit 11 receives input of an instruction to execute a living body assessment, then in step S101 , the assessment method determination unit 12 selects two or more living body assessment processes from a plurality of living body assessment processes and determines the order in which the two or more selected living body assessment processes are to be executed.

[0095] In step S102, the execution unit 13 sequentially executes the two or more living body assessment processes selected in step S101 in the determined order. In each living body assessment process, steps S103 to S105 are executed.

[0096] In step S103, the output unit 14 displays on the display 3 the motion information indicating the motion required for the currently executing living organism determination process. This motion information is presented to the user. In step S104, the facial image acquisition unit 15 acquires a facial image from the camera 2. This facial image includes the user's face as they perform the motions in accordance with the motion information presented in step S103. In step S105, the first determination unit 16 determines whether the user is alive based on the facial features corresponding to the currently executing living organism determination process, which are included in the facial image acquired by the facial image acquisition unit 15.

[0097] In step S106, the execution unit 13 determines whether the two or more living body assessment processes selected in step S101 have all been completed. If, in step S106, it is determined that the two or more living body assessment processes selected in step S101 have not all been completed (step S106: No), the process proceeds to step S102 and subsequent steps. Based on this, the next living body assessment process is executed. If, in step S106, it is determined that the two or more living body assessment processes selected in step S101 have all been completed (step S106: Yes), the process proceeds to step S107.

[0098] In step S107 , the second determination unit 17 determines whether the determination results obtained from the two or more living body determination processes selected in step S101 satisfy a predetermined condition.

[0099] If the second determination unit 17 determines in step S107 that the determination result satisfies the predetermined condition (Yes in step S107), the second determination unit 17 determines in step S108 that the user is a living being. On the other hand, if the second determination unit 17 determines in step S107 that the determination result does not satisfy the predetermined condition (No in step S107), the second determination unit 17 determines in step S109 that the user is not a living being.

[0100] After step S108 and step S109 , in step S110 , the output unit 14 outputs information indicating the determination result of step S108 or step S109 .

[0101] Specifically, in step S110, the output unit 14 displays a message indicating the determination result of step S108 or step S109 on the display 3. Alternatively, the output unit 14 may output a voice indicating the determination result of step S108 or step S109 via a speaker (not shown).

[0102] Furthermore, in step S100, it is assumed that the input of the instruction to execute the living body test is accepted by inputting information indicating a request to execute the living body test from an external device via the communication unit 50. In this case, the output unit 14 may transmit information indicating the result of the test in step S108 or step S109 to the external device via the communication unit 50.

[0103] Alternatively, in step S100, it is assumed that the input of the instruction to execute the living body determination is accepted by the application currently being executed by the processor 10. In this case, the output unit 14 may also transmit information indicating the determination result of step S108 or step S109 to the currently being executed application.

[0104] As described above, according to the determination device 1 of Embodiment 1, the number, content, and order of execution of the liveness determination processes executed to determine whether a user is alive exponentially increase the number of modes in which liveness determination is performed. This effectively makes it impossible for a malicious user to prepare facial images of another person that would be determined to be alive in all modes, making it difficult to commit fraud using pre-prepared images.

[0105] The following describes in detail the various living body assessment processes that can be executed by the execution unit 13. In this embodiment, the various living body assessment processes include a living body assessment process based on facial orientation, a living body assessment process based on eye orientation, a living body assessment process based on eye opening and closing, a living body assessment process based on mouth opening and closing, and a living body assessment process based on whether glasses are worn or removed.

[0106] Living body determination processing based on face orientation

[0107] The following describes in detail the living body determination process based on the face orientation. Figure 4 This is a flowchart showing an example of a living body determination process based on face orientation. If the living body determination process based on face orientation is executed, first, the output unit 14 performs the following operations: Figure 3 Step S103 shown corresponds to step S201. In step S201, the output unit 14 displays motion information indicating the direction of the face on the display 3 as the motion required for the living body determination process.

[0108] Specifically, in step S201, the output unit 14 displays motion information indicating that the face is facing in one of the following directions: up, down, left, or right. Furthermore, the output unit 14 randomly determines the direction indicated in the motion information. However, this is not limiting. The output unit 14 may also regularly determine the direction indicated in the motion information according to a predetermined rule, such as determining in the order of up, down, left, or right.

[0109] Next, the facial image acquisition unit 15 performs the Figure 3 Step S104 shown corresponds to step S202. In step S202, the facial image acquisition unit 15 acquires a facial image including the user's face from the camera 2.

[0110] Next, the first judgment unit 16 performs the Figure 3 Step S105 shown corresponds to step S203 and subsequent processing.

[0111] In step S203 , the first determination unit 16 detects the center coordinates of the face from the facial image acquired in step S202 .

[0112] Specifically, in step S203, the first determination unit 16 first detects a facial region representing a person's face from the facial image. A facial region is, for example, a rectangular area large enough to encompass the entire face. Specifically, the first determination unit 16 detects the facial region by inputting the facial image into a classifier pre-created for facial region detection. This classifier is, for example, a Haar-like cascade classifier.

[0113] Next, the first determination unit 16 detects facial feature points from the detected facial region. Facial feature points are one or more points located at characteristic positions in various facial features, such as the outer and inner canthi of the eyes, the facial contour, the bridge of the nose, the lips, and the eyebrows. Feature points are also called landmarks. The first determination unit 16 detects facial feature points from the facial region by executing landmark detection processing using, for example, a model file of a machine learning framework.

[0114] Figure 8 is a diagram showing the facial feature points 9X. For example, Figure 8 As shown, a plurality of feature points 9X are detected by applying the landmark detection process to the face region 80. Figure 8 In the example of , among the feature points 9X detected by the landmark detection process, 5 feature points 9X are detected on the nose bridge, 2 feature points 9X are detected on the lower side of the nose, 17 feature points 9X are detected on the facial contour, and 4 feature points 95 to 98 are detected on the upper, lower, left, and right ends of the lips. Figure 8 In the example of , two feature points 9X located at the left and right inner corners 92 and two feature points 9X located at the left and right outer corners 93 are detected.

[0115] Furthermore, landmark point numbers are assigned to the feature points 9X, and it is predetermined which feature point with each landmark point number represents which facial part. For example, a landmark point 9X with landmark point number "2" represents the left outer canthus 93, and a landmark point with landmark point number "0" represents the left inner canthus 92. Thus, the first determination unit 16 can determine which facial part a feature point 9X represents based on the landmark point number.

[0116] Furthermore, the first determination unit 16 detects the coordinates of the feature point 9X in the vertical direction, among the plurality of feature points 9X indicating the bridge of the nose, detected from the face region 80, as the center coordinates of the face. Figure 8In the example shown in FIG. 1 , the first determination unit 16 detects the coordinates of the third feature point 94 from the top of the five feature points 9X representing the bridge of the nose as the center coordinates of the face. However, this is not limiting, and the first determination unit 16 may also detect the coordinates of the feature point 9X at the upper or lower end of the bridge of the nose as the center coordinates of the face.

[0117] Next, in step S204, the first determination unit 16 calculates the horizontal component of the face orientation. Figure 8 The content of step S204 will be described in detail.

[0118] The first determination unit 16 sets a vertical centerline and a horizontal centerline based on the facial feature points 9X detected from the facial region 80. For example, the first determination unit 16 sets a straight line that passes through the feature point 94 indicating the center of the face detected in step S203 and is parallel to the vertical side of the facial region 80 as the vertical centerline. Furthermore, the first determination unit 16 sets a straight line that passes through the feature point 94 and is parallel to the horizontal side of the facial region 80 as the horizontal centerline.

[0119] Next, the first determination unit 16 divides the horizontal centerline using the feature points 94, calculates the lengths of the right and left sections, and calculates the ratio of the right section to the left section, assuming the length of the horizontal centerline is 100%. Based on this ratio, the first determination unit 16 calculates the horizontal component of the face's orientation. For example, let the ratio of the right section be α1, the ratio of the left section be α2, and the right side be positive. That is, when the face is facing left, the ratio α2 is smaller than the ratio α1. When the face is facing right, the ratio α1 is smaller than the ratio α2. When the face is facing frontally, the ratio α2 and the ratio α1 are approximately equal. "Approximately equal" means that some difference between the ratios α2 and α1 is permitted.

[0120] Next, the first determination unit 16 subtracts the smaller ratio, α1 or α2, from 50. For example, if α2 is smaller than α1, the first determination unit 16 calculates 50 - α2; if α1 is smaller than α2, the first determination unit 16 calculates 50 - α1. If the face is facing right, since right is positive, the first determination unit 16 calculates 50 - α1 as the horizontal component of the face orientation. On the other hand, if the face is facing left, since left is negative, the first determination unit 16 calculates -(50 - α2) as the horizontal component of the face orientation.

[0121] Thus, as the value of the horizontal component of the face orientation increases in the positive direction, it indicates that the face is facing further to the right, and as the value of the horizontal component of the face orientation increases in the negative direction, it indicates that the face is facing further to the left. Furthermore, when the horizontal component of the face orientation is 0, it indicates that the face is facing forward.

[0122] Next, in step S205, the first determination unit 16 calculates the vertical component of the face orientation. Figure 8 The content of step S205 will be described in detail.

[0123] First determination unit 16 sets a horizontal center line in facial region 80. This horizontal center line is a straight line that passes through feature point 94 representing the center of the face and is parallel to the horizontal side of facial region 80. Alternatively, the horizontal center line may be set at feature point 9X of the nose bridge, other than feature point 94. The horizontal center line setting result in step S204 may also be used for this setting.

[0124] Next, the first determination unit 16 divides the vertical side of the facial region 80 by the intersection of the vertical side and the horizontal centerline, and calculates the length of the upper section and the length of the lower section. Next, the first determination unit 16 calculates the ratio of the upper section to the lower section when the length of the vertical side is set to 100%, and calculates the vertical component of the facial orientation based on this ratio. For example, let the ratio of the upper section be α3, the ratio of the lower section be α4, and upward be positive. That is, when the face is facing upward, the ratio α3 is smaller than the ratio α4. When the face is facing downward, the ratio α4 is smaller than the ratio α3. When the face is facing forward, the ratio α3 and the ratio α4 are approximately equal. The fact that the ratios α3 and α4 are approximately equal has the same meaning as the fact that the ratios α1 and α2 are approximately equal.

[0125] Next, the first determination unit 16 subtracts the smaller ratio, α3 or α4, from 50. For example, if α3 is smaller than α4, the first determination unit 16 calculates 50 - α3. If α4 is smaller than α3, the first determination unit 16 calculates 50 - α4. If the face is facing upward, since upward is positive, the first determination unit 16 calculates 50 - α3 as the vertical component of the face orientation. On the other hand, if the face is facing downward, since upward is positive, the first determination unit 16 calculates -(50 - α4) as the vertical component of the face orientation.

[0126] Thus, as the value of the vertical component of the face orientation increases in the positive direction, it indicates that the face is facing upward, and as the value of the vertical component of the face orientation increases in the negative direction, it indicates that the face is facing downward. In addition, when the vertical component of the face orientation is 0, it indicates that the face is facing forward.

[0127] Next, in step S206 , the first determination unit 16 determines whether the facial orientation indicated by the horizontal component and the vertical component of the facial orientation calculated in steps S204 and S205 is within a predetermined range from the facial orientation indicated by the action information presented in step S201 .

[0128] For example, assume that in step S201, action information indicating that the face is facing right is displayed on display 3. In this case, if the value of the horizontal component of the face orientation calculated in step S204 is within a positive predetermined range (+Z1 to +Z2 (Z2 > Z1)), first determination unit 16 determines that the face orientation is within the predetermined range from the action information presented in step S201.

[0129] Assume that in step S201, action information indicating that the face is facing left is displayed on display 3. At this time, if the value of the horizontal component of the face orientation calculated in step S204 is within a predetermined negative range (-Z1 to -Z2 (Z2 > Z1)), first determination unit 16 determines that the face orientation is within the predetermined range from the action information presented in step S201.

[0130] Assume that in step S201, action information indicating that the face is facing forward is displayed on display 3. At this time, if the value of the horizontal component of the face orientation calculated in step S204 is within a predetermined positive and negative range (-Z3 to +Z4 (Z3, Z4>0)) relative to zero, the first determination unit 16 determines that the face orientation is within the predetermined range from the action information presented in step S201.

[0131] Similarly, assume that in step S201, action information indicating that the face is facing upward is displayed on the display 3. In this case, if the value of the vertical component of the face orientation calculated in step S205 is within a positive predetermined range (+Z5 to +Z6 (Z6 > Z5)), the first determination unit 16 determines that the face orientation is within the predetermined range from the action information presented in step S201.

[0132] Assume that in step S201, action information indicating that the face is facing downward is displayed on display 3. At this time, if the value of the vertical component of the face orientation calculated in step S205 is within a predetermined negative range (-Z5 to -Z6 (Z6 > Z5)), first determination unit 16 determines that the face orientation is within the predetermined range from the action information presented in step S201.

[0133] Assume that in step S201, action information indicating that the face is facing forward is displayed on display 3. At this time, if the value of the vertical component of the face orientation calculated in step S205 is within a predetermined positive and negative range (-Z7 to +Z8 (Z7, Z8>0)) relative to zero, first determination unit 16 determines that the face orientation is within the predetermined range from the action information presented in step S201.

[0134] The method used by the first determination unit 16 to calculate the horizontal component of the facial orientation in step S204 is not limited to the method described above. For example, the horizontal component of the facial orientation may be calculated by subtracting 50 from the larger of the ratios α1 and α2. Furthermore, the horizontal component of the facial orientation may be set as positive when facing left. Furthermore, the method used by the first determination unit 16 to calculate the vertical component of the facial orientation in step S205 is not limited to the method described above. For example, the vertical component of the facial orientation may be calculated by subtracting 50 from the larger of the ratios α3 and α4. Furthermore, the vertical component of the facial orientation may be set as positive when facing downward. Accordingly, in step S206, the method for determining whether the facial orientation is within a predetermined range from the facial orientation indicated by the action information presented in step S201, based on the values ​​of the horizontal and vertical components of the facial orientation calculated in steps S204 and S205, may be modified.

[0135] Assume that the first determination unit 16 determines in step S206 that the facial orientation is within the specified range from the facial orientation indicated in the action information presented in step S201. In this case (YES in step S206), in step S207, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on facial orientation.

[0136] On the other hand, if the first judgment unit 16 determines in step S206 that the facial orientation is not within the specified range of the facial orientation indicated by the action information prompted in step S201 (step S206 is no), in step S208, the first judgment unit 16 determines that the user is not a living body, and outputs information indicating the judgment result to the second judgment unit 17, thereby ending the living body judgment processing based on facial orientation.

[0137] Liveness determination based on eye orientation

[0138] The following describes in detail the living body determination process based on eye direction. Figure 5 This is a flowchart showing an example of a living body determination process based on eye direction. If the living body determination process based on eye direction is executed, first, the output unit 14 performs Figure 3 Step S103 shown corresponds to step S301. In step S301, the output unit 14 displays motion information indicating eye orientation on the display 3 as an action required for the living body determination process.

[0139] Specifically, in step S301, the output unit 14 displays on the display 3 action information indicating that the eyes should be directed to either the left or right direction while the face is facing forward. Furthermore, the output unit 14 randomly determines which left or right direction is indicated in the action information. However, this is not limiting, and the output unit 14 may also regularly determine which left or right direction is indicated in the action information according to a predetermined rule, such as determining in order of left and right.

[0140] Next, the facial image acquisition unit 15 performs the Figure 3 Step S104 shown corresponds to step S302. In step S302, the facial image acquisition unit 15 acquires a facial image including the user's face from the imaging device 2.

[0141] Next, the first judgment unit 16 performs the Figure 3 Step S105 shown corresponds to step S303 and subsequent processing.

[0142] In step S303 , the first determination unit 16 detects the center coordinates of the iris from the facial image acquired in step S302 .

[0143] Specifically, in step S303, the first determination unit 16 and step S203 ( Figure 4 ) Similarly, the facial image is input to the face region detection classifier to detect the face region 80. The first judgment unit 16 inputs the detected face region 80 to the eye region detection classifier to detect the eye region. The eye region is a rectangular area that includes the entire eye and has a slight margin for the size of the eye.

[0144] Next, the first determination unit 16 converts the detected eye region into a grayscale image. For example, the grayscale image conversion process can be performed by calculating the average grayscale values ​​of the red, green, and blue components of each pixel constituting the eye region. However, this is merely an example, and other grayscale conversion processes may also be used.

[0145] Next, the first judgment unit 16 performs binarization processing on the eye region of the grayscale image to generate a binary image of the eye region. The first judgment unit 16 separates the generated binary image of the eye region into a plurality of local regions by predetermined pixel separation in the X direction. The X direction refers to the horizontal (lateral) direction of the image captured by the camera 2. For example, the first judgment unit 16 divides the binary image into 10 equal sections in the horizontal direction. Thus, the binary image is divided into 10 local regions, each of which is in the form of strips with its length in the Y direction. Here, the first judgment unit 16 divides the binary image into 10 local regions, but this is only an example. The number of divisions can be an integer greater than 2 and less than 9, or greater than 11. The Y direction refers to the vertical (longitudinal) direction of the image captured by the camera 2.

[0146] Next, the first determination unit 16 calculates the average luminance value of each of the ten local areas.

[0147] Next, the first determination unit 16 calculates the X coordinate of the estimated iris center position. The estimated iris center position is an estimate of the iris center position and differs from the final calculated iris center position. Due to factors such as double eyelids, thick eyelashes, and false eyelashes, these areas may appear larger as white areas. In such cases, the white of the eye may be filled in. To avoid this, the estimated iris center position is calculated.

[0148] The first determination unit 16 calculates the coordinates of the midpoint in the X direction of the local area with the highest average brightness value among the multiple local areas as the X coordinate of the estimated iris center position. Depending on the width of the local area in the X direction, the midpoint in the X direction of the local area may not be suitable as the X coordinate of the estimated iris center position. In such cases, the left or right end of the local area in the X direction may be calculated as the X coordinate of the estimated iris center position.

[0149] Next, the first determination unit 16 calculates the Y coordinate of the estimated iris center position. The first determination unit 16 detects the uppermost and lowermost white pixel endpoints in the local area where the X coordinate of the estimated iris center position exists, and calculates the midpoint between the uppermost and lowermost endpoints as the Y coordinate of the estimated iris center position.

[0150] Furthermore, due to the influence of eyelashes and makeup, the uppermost and lowermost endpoints may appear in the local area adjacent to the left or right. To address this, the first determination unit 16 may calculate the uppermost and lowermost endpoints in the local area where the X coordinate of the estimated iris center position exists and in the two local areas adjacent to the left and right of the local area, average the three calculated uppermost endpoints to obtain an average uppermost endpoint, and average the three calculated lowermost endpoints to obtain an average lowermost endpoint. The midpoint between the average uppermost and average lowermost endpoints is then calculated as the Y coordinate of the estimated iris center position.

[0151] Next, the first determination unit 16 performs a fill-in process on the binary image. In visible light images, due to factors such as ambient brightness, external light or background light may be reflected into the cornea. When this reflection is significant, bright areas such as white appear within the otherwise dark or brown pupil. In this case, if the eye image is binarized, black islands appear within the pupil area, preventing accurate iris information detection. To address this, the first determination unit 16 performs a fill-in process to fill in the black islands.

[0152] The details of the filling process are as follows. First, the first determination unit 16 sets a vertical line parallel to the Y direction for the X coordinate of the estimated iris center position in the binary image. Next, the first determination unit 16 detects the first white pixel that appears on the vertical line from the top end of the binary image as the top pixel.

[0153] Next, the first determination unit 16 detects the first white pixel that appears on the vertical line from the bottom end of the binary image as the bottom end pixel. Next, the first determination unit 16 determines whether the distance between the top end pixel and the bottom end pixel is greater than a first reference distance. If the first determination unit 16 determines that the distance between the top end pixel and the bottom end pixel is greater than the first reference distance, it determines that the black pixel between the top end pixel and the bottom end pixel on the vertical line satisfies a specified condition and replaces the black pixel with a white pixel.

[0154] On the other hand, if the first determination unit 16 determines that the distance between the upper pixel and the lower pixel is less than or equal to a first reference distance, the vertical line is not replaced. The first reference distance may be a reasonable distance based on, for example, an assumed iris diameter.

[0155] The first determination unit 16 performs this filling process on each vertical line within the range of the left reference distance from the estimated iris center position to the left in the X direction, and also performs this filling process on each vertical line within the range of the right reference distance from the estimated iris center position to the right in the X direction. The sum of the left reference distance range and the right reference distance range is an example of a second reference distance. The left reference distance range and the right reference distance range can be, for example, the same range. The second reference distance can be, for example, a distance slightly larger than the assumed iris diameter. This allows, for example, the filling process to be applied preferentially to vertical lines located in the pupil area.

[0156] Next, the first determination unit 16 detects the left and right end pixels of the pupil area. In the white area of ​​the binary image, the first determination unit 16 examines changes in brightness for each pixel in the left and right directions of the x-direction, starting from the estimated iris center position. Furthermore, the first determination unit 16 detects the black pixel that first appears on the left side of the x-direction as the left end pixel, and the black pixel that first appears on the right side of the x-direction as the right end pixel.

[0157] Next, the first determination unit 16 calculates the middle position between the left end pixel and the right end pixel as the X coordinate of the iris center position.

[0158] Next, the first determination unit 16 detects the upper and lower pixels of the pupil area. In the white area of ​​the binary image, the first determination unit 16 examines changes in brightness for each pixel in the upper and lower directions of the Y direction, starting from the X coordinate of the iris center. Furthermore, the first determination unit 16 detects the black pixel that first appears in the upper Y direction as the upper pixel, and the black pixel that first appears in the lower Y direction as the lower pixel.

[0159] Next, the first determination unit 16 calculates the middle position between the upper pixel and the lower pixel as the Y coordinate of the center position of the iris, thereby calculating the center coordinates of the iris.

[0160] In step S304, the first determination unit 16 detects the positions of the outer and inner corners of the eye from the binary image of the eye region generated in step S303, and calculates the horizontal distance from the iris center coordinates detected in step S303 to the outer corner of the eye and the horizontal distance from the iris center coordinates to the inner corner of the eye.

[0161] Specifically, the frame of the binary image of the eye area is a rectangle circumscribing the white area. Therefore, the first determination unit 16 calculates the X coordinate of the left end of the white area in the binary image as the X coordinate of the left end of the binary image, that is, the X coordinate of the outer corner of the left eye or the inner corner of the right eye. Furthermore, the first determination unit 16 calculates the X coordinate of the right end of the white area as the X coordinate of the right end of the binary image, that is, the X coordinate of the inner corner of the left eye or the outer corner of the right eye.

[0162] The first determination unit 16 then calculates the absolute value of the difference between the X coordinate of the iris center detected in step S303 and the X coordinate of the outer corner of the eye as the horizontal distance from the iris center to the outer corner of the eye. Furthermore, the first determination unit 16 calculates the absolute value of the difference between the X coordinate of the iris center detected in step S303 and the X coordinate of the inner corner of the eye as the horizontal distance from the iris center to the inner corner of the eye. These processes are performed for each binary image corresponding to the left and right eyes.

[0163] In step S305, the first determination unit 16 calculates the horizontal orientation of the eye. Specifically, the first determination unit 16 calculates the ratio β1 of the first distance D1 and the ratio β2 of the second distance D2, assuming that the horizontal distance between the inner corner of the eye and the iris center coordinates calculated in step S304 is the first distance D1, the horizontal distance between the outer corner of the eye and the iris center coordinates is the second distance D2, and the distance between the X coordinates of the inner corner of the eye and the X coordinates of the outer corner of the eye is 100%. This process is performed for each binary image corresponding to the left and right eyes.

[0164] When the eyes are facing left, the left and right eyes are tilted to the left, the second distance D2 of the left eye is shortened, and the first distance D1 of the right eye is shortened. When the eyes are facing right, the left and right eyes are tilted to the right, the second distance D2 of the left eye is lengthened, and the first distance D1 of the right eye is lengthened.

[0165] Based on the above, the rightward deviation of the eye is considered positive. For example, focusing on the right eye, the first determination unit 16 determines that the eye is tilted to the left if the ratio β1 is less than the ratio β2, and that the eye is tilted to the right if the ratio β2 is less than the ratio β1. Focusing on the left eye, the first determination unit 16 determines that the eye is tilted to the left if the ratio β2 is less than the ratio β1, and that the eye is tilted to the right if the ratio β1 is less than the ratio β2.

[0166] If the first determination unit 16 determines that the eye is tilted to the left, it calculates the average of the ratio β2 of the second distance D2 of the left eye and the ratio β1 of the first distance D1 of the right eye, and multiplies the value obtained by subtracting the average from 50 by a negative sign as the horizontal eye heading. The average is calculated because the eye heading does not differ significantly between the left and right eyes. Subtracting the average from 50 increases the horizontal eye heading value as the eye tilts to the left or right from the front of the face. The negative sign is used to multiply the value by the right side to indicate a positive direction.

[0167] When the first determination unit 16 determines that the eye is tilted to the right, it calculates the average value of the ratio β1 of the first distance D1 of the left eye and the ratio β2 of the second distance D2 of the right eye, and calculates the value obtained by subtracting the average value from 50 as the horizontal direction of the eye.

[0168] That is, as the value of the horizontal eye heading increases in the positive direction, it indicates that the eye is facing further to the right relative to the front of the face, and as the value of the horizontal eye heading increases in the negative direction, it indicates that the eye is facing further to the left relative to the front of the face. Furthermore, if the horizontal eye heading is 0, it indicates that the eye is facing the front of the face. The horizontal eye heading is calculated as described above.

[0169] Next, in step S306 , the first determination unit 16 determines whether the horizontal direction of the eye calculated in step S305 is within a predetermined range from the eye direction indicated by the action information presented in step S301 .

[0170] For example, assume that in step S301, action information indicating that the eyes are facing right is displayed on the display 3. In this case, if the value of the horizontal eye orientation calculated in step S305 is within a positive predetermined range (+Z1 to +Z2 (Z2 > Z1)), the first determination unit 16 determines that the eye orientation is within the predetermined range from the action information presented in step S301.

[0171] Assume that in step S301, action information indicating that the eyes are facing left is displayed on display 3. At this time, if the value of the horizontal eye orientation calculated in step S305 is within a predetermined negative range (-Z1 to -Z2 (Z2 > Z1)), the first determination unit 16 determines that the eye orientation is within the predetermined range from the action information presented in step S301.

[0172] Assume that, in step S301, action information indicating that the eyes are facing forward is displayed on the display 3. At this time, if the value of the horizontal eye orientation calculated in step S305 is within a predetermined positive and negative range (-Z3 to +Z4 (Z3, Z4>0)) relative to 0, the first determination unit 16 determines that the eye orientation is within the predetermined range from the action information presented in step S301.

[0173] Alternatively, the left direction may be considered positive for the horizontal eye orientation. Furthermore, the method used by the first determination unit 16 to calculate the horizontal eye orientation in step S305 is not limited to the method described above. Accordingly, in step S306, the determination method may be modified to determine whether the horizontal eye orientation calculated in step S305 is within a predetermined range from the eye orientation indicated in the action information presented in step S301.

[0174] Assume that the first determination unit 16 determines in step S306 that the horizontal eye orientation is within a predetermined range from the eye orientation indicated in the action information presented in step S301. In this case (YES in step S306), in step S307, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on eye orientation.

[0175] On the other hand, if the first judgment unit 16 determines in step S306 that the horizontal direction of the eye is not within the specified range of the eye direction indicated by the action information prompted in step S301 (step S306 is no), in step S308, the first judgment unit 16 determines that the user is not a living body, and outputs information indicating the judgment result to the second judgment unit 17, thereby ending the living body judgment processing based on the eye direction.

[0176] Liveness determination based on eye open / closed status

[0177] The following describes in detail the living body determination process based on the eye opening and closing state. Figure 6 This is a flowchart showing an example of a living body determination process based on the eye opening and closing state. If the living body determination process based on the eye opening and closing state is executed, first, the output unit 14 performs the same Figure 3 Step S103 shown corresponds to step S401. In step S401, the output unit 14 displays on the display 3 motion information indicating whether the eyes are open or closed as motion required for the living body determination process.

[0178] Specifically, in step S401, the output unit 14 displays on the display 3 motion information indicating an instruction to open both eyes, motion information indicating an instruction to close both eyes, or motion information indicating an instruction to blink both eyes a predetermined number of times. Alternatively, in step S401, the output unit 14 displays on the display 3 motion information indicating an instruction to open one of the left and right eyes and close the other, or motion information indicating an instruction to blink only one of the right and left eyes a predetermined number of times.

[0179] The output unit 14 randomly determines the eye opening and closing method indicated in the motion information. However, this is not limiting and the output unit 14 may regularly determine the eye opening and closing method indicated in the motion information according to a predetermined rule, such as determining in the above-mentioned order.

[0180] Next, the facial image acquisition unit 15 performs the Figure 3 Step S104 shown corresponds to step S402. In step S402, the facial image acquisition unit 15 acquires a facial image including the user's face from the imaging device 2.

[0181] Next, the first judgment unit 16 performs the Figure 3Step S105 shown corresponds to step S403 and subsequent processing.

[0182] In step S403 , the first determination unit 16 detects the open or closed state of the eyes from the facial image acquired in step S402 .

[0183] Specifically, in step S403, the first judgment unit 16 first determines Figure 5 Similarly, in step S303 shown, the face region 80 is detected from the face image acquired in step S402, and the eye region is detected from the face region 80. Furthermore, the first determination unit 16 converts the detected eye region into a grayscale image and binarizes the eye region in the grayscale image to generate a binary image of the eye region.

[0184] Based on this, when the eyes are open, a binary image is generated in which the pupil, eyelashes, and darker areas of the eyeball are represented by white pixels, while the rest of the eyeball and lighter areas of the skin are represented by black pixels. On the other hand, when the eyes are closed, a binary image is generated in which the darker areas of the eyelashes and upper eyelid edge are represented by white pixels, while the lighter areas of the skin are represented by black pixels.

[0185] Next, the first determination unit 16 calculates the maximum height of a white area composed of white pixels, where the vertical distance between the upper and lower pixels is the largest. The first determination unit 16 calculates the height of the white area at each X-coordinate on the X-axis by counting the number of white pixels in the white area of ​​the binary image. The height is expressed as the number of white pixels in the vertical direction.

[0186] The first determination unit 16 sequentially shifts the X coordinates by one pixel from the left end to the right end of the white area and calculates the height of each X coordinate. The first determination unit 16 calculates the maximum X coordinate and its height within the white area. This process is performed for each binary image corresponding to the left and right eyes.

[0187] Next, the first determination unit 16 determines whether the maximum height of the white area is greater than a value obtained by multiplying the height of the binary image of the eye area by 0.9.

[0188] Here, the first determination unit 16 determines that the eyes are open if the maximum height of the white area is greater than the value of the height of the binary image of the eye area multiplied by 0.9. That is, when the eyes are open, the ratio of the maximum height of the white area to the height of the binary image of the eye area is close to 1. Therefore, if the maximum height of the white area is greater than the value of the height of the binary image of the eye area multiplied by 0.9, the first determination unit 16 determines that the eyes in the eye area are open.

[0189] On the other hand, if the first determination unit 16 determines that the maximum height of the white area is less than the value of the height of the binary image of the eye area multiplied by 0.9, it then determines whether the maximum height of the white area is less than the value of the height of the binary image of the eye area multiplied by 0.6. Here, if the first determination unit 16 determines that the maximum height of the white area is less than the value of the height of the binary image of the eye area multiplied by 0.6, it determines that the eyes are closed. In other words, when the eyes are closed, the maximum height of the white area is less than 60% of the height of the binary image of the eye area. Therefore, if the maximum height of the white area is less than the value of the height of the binary image of the eye area multiplied by 0.6, the first determination unit 16 determines that the eyes in the eye area are closed.

[0190] On the other hand, when the first determination unit 16 determines that the maximum height of the white area is equal to or greater than the value of the height of the binary image of the eye area multiplied by 0.6, it determines that the eye open / close state cannot be determined.

[0191] The method by which the first determination unit 16 detects the eye open / close state in step S403 is not limited to the method described above. For example, the first determination unit 16 may determine that the eye is closed when the maximum height of the white area is less than the value of the height of the binary image of the eye area multiplied by 0.6, or may determine that the eye is open when the maximum height of the white area is greater than or equal to the value of the height of the binary image of the eye area multiplied by 0.6.

[0192] Next, in step S404 , the first determination unit 16 determines whether the eye opening and closing state detected in step S403 is the same as the eye opening and closing state indicated by the motion information presented in step S401 .

[0193] Assume that the first determination unit 16 determines in step S404 that the eye opening / closing state detected in step S403 is the same as the eye opening / closing state indicated by the motion information presented in step S401. In this case (YES in step S404), in step S405, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the eye opening / closing state.

[0194] On the other hand, suppose that the first determination unit 16 determines in step S404 that the eye opening / closing state is different from the eye opening / closing state indicated by the motion information presented in step S401. In this case (No in step S404), in step S406, the first determination unit 16 determines that the user is not alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the eye opening / closing state.

[0195] In addition, if the first judgment unit 16 determines in step S404 that the eye opening or closing state cannot be determined in step S403, it also determines in step S406 that the user is not a living body, and outputs information indicating the judgment result to the second judgment unit 17, ending the living body judgment processing based on the eye opening or closing state.

[0196] Furthermore, assume that, in step S401, action information indicating blinking a predetermined number of times is displayed on the display 3. In this case, in step S402, the facial image acquisition unit 15 acquires a facial image from the imaging device 2 at a predetermined frame rate at any time. Each time a facial image is acquired, the first determination unit 16 performs step S403 using the acquired facial image, and then proceeds to step S404.

[0197] At this point, if the eye open / closed state detection results in step S403 detect a change from the open state to the closed state a predetermined number of times or more within the predetermined upper limit time, the first determination unit 16 determines in step S404 that the eye open / closed state is the same as the eye open / closed state indicated by the action information presented in step S401. In this case, in step S405, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the eye open / closed state.

[0198] On the other hand, if the eye open / closed state detection results in each step S403 fail to detect a change from the open to closed state more than a predetermined number of times within the predetermined upper limit time, the first determination unit 16 determines in step S404 that the eye open / closed state is different from the eye open / closed state indicated by the action information presented in step S401. In this case, in step S406, the first determination unit 16 determines that the user is not alive, outputs information indicating this determination result to the second determination unit 17, and terminates the living body determination process based on the eye open / closed state.

[0199] Living body determination based on mouth opening and closing status

[0200] The following describes in detail the living body determination process based on the mouth opening and closing state. Figure 7 This is a flowchart showing an example of a living body determination process based on the mouth opening and closing state. If the living body determination process based on the mouth opening and closing state is executed, first, the output unit 14 performs the same Figure 3 Step S103 shown corresponds to step S501. In step S501, the output unit 14 displays on the display 3 motion information indicating the opening and closing of the mouth as the motion required for the living body determination process.

[0201] Specifically, in step S501, the output unit 14 displays on the display 3 action information indicating an instruction to open the mouth, action information indicating an instruction to close the mouth, or action information indicating an instruction to open and close the mouth a predetermined number of times. Furthermore, the output unit 14 randomly determines the mouth opening and closing method indicated in the action information. However, this is not limiting, and the output unit 14 may also regularly determine the mouth opening and closing method indicated in the action information according to a predetermined rule, such as determining in the aforementioned order.

[0202] Next, the facial image acquisition unit 15 performs Figure 3 Step S104 shown corresponds to step S502. In step S502, the facial image acquisition unit 15 acquires a facial image including the user's face from the imaging device 2.

[0203] Next, the first judgment unit 16 performs the Figure 3 Step S105 shown corresponds to step S503 and subsequent processing.

[0204] In step S503 , the first determination unit 16 detects feature points of the lips from the facial image acquired in step S502 .

[0205] Specifically, in step S503, the first judgment unit 16 first determines Figure 4 Similarly, in step S203 shown, the face region 80 is detected from the face image acquired in step S302 , and the landmark detection process is applied to the face region 80 to detect the feature points of the lips. Figure 8 An example is shown in which four feature points 95 to 98 located at the upper, lower, left, and right ends of the lips are detected by applying the landmark detection process to the face region 80 .

[0206] In step S504, the first determination unit 16 calculates the mouth opening amount, which indicates the degree of mouth opening, using the lip feature points detected in step S503. Specifically, the first determination unit 16 divides the distance between the upper end 95 and the lower end 96 of the lips detected in step S503 by the distance between the left end 97 and the right end 98 of the lips detected in step S503, and calculates the result of this division as the mouth opening amount. This allows the mouth opening amount to be appropriately calculated regardless of individual differences in mouth size.

[0207] Next, in step S505 , the first determination unit 16 determines whether the mouth opening and closing state is the same as the mouth opening and closing state indicated by the motion information presented in step S501 based on the mouth opening amount calculated in step S504 .

[0208] Specifically, in step S505, the first determination unit 16 first determines whether the mouth is open or closed based on the amount of mouth opening calculated in step S504. Specifically, the first determination unit 16 determines whether the amount of mouth opening calculated in step S504 is greater than a predetermined threshold (e.g., 0.3). If the amount of mouth opening and closing calculated in step S504 is greater than the predetermined threshold, the first determination unit 16 determines that the mouth is open.

[0209] On the other hand, if the amount of opening and closing calculated in step S504 is less than a predetermined threshold, the first determination unit 16 determines that the mouth is closed. Furthermore, the first determination unit 16 determines whether the mouth opening and closing state determined based on the amount of mouth opening calculated in step S504 is the same as the mouth opening and closing state indicated by the motion information presented in step S501.

[0210] In addition, the method by which the first determination unit 16 determines the mouth opening or closing state in step S505 is not limited to the method using the above-mentioned mouth opening amount, and other methods may be used.

[0211] Assume that the first determination unit 16 determines in step S505 that the mouth opening and closing state determined based on the mouth opening amount is the same as the mouth opening and closing state indicated by the motion information presented in step S501. In this case (YES in step S505), in step S506, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the mouth opening and closing state.

[0212] On the other hand, suppose that the first determination unit 16 determines in step S505 that the mouth opening / closing state is different from the mouth opening / closing state indicated by the motion information presented in step S501. In this case (No in step S505), in step S507, the first determination unit 16 determines that the user is not alive, outputs information indicating this determination result to the second determination unit 17, and terminates the living body determination process based on the mouth opening / closing state.

[0213] Furthermore, if the action information instructing the user to open and close the mouth a predetermined number of times is displayed on the display 3 in step S501, then in step S502, the facial image acquisition unit 15 acquires a facial image from the imaging device 2 at any time at a predetermined frame cycle. Each time a facial image is acquired, the first determination unit 16 performs steps S503 and S504 using the acquired facial image, and then proceeds to step S505.

[0214] In this case, in step S505, the first determination unit 16 determines the mouth opening / closing state based on the mouth opening amount calculated in each step S504 until the predetermined upper limit time has elapsed. Furthermore, if the first determination unit 16 detects a predetermined number of changes from the open mouth state to the closed mouth state, it determines that the mouth opening / closing state is the same as the mouth opening / closing state indicated by the mouth movement information presented in step S501. In this case, in step S506, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the mouth opening / closing state.

[0215] On the other hand, suppose that in step S504, the first determination unit 16 determines the mouth opening / closing state based on the mouth opening amount calculated in each step S504 until the predetermined upper limit time has elapsed, and fails to detect a predetermined number of changes from the open mouth state to the closed mouth state. In this case, the first determination unit 16 determines that the mouth opening / closing state is different from the mouth opening / closing state indicated by the action information presented in step S501. In this case, in step S507, the first determination unit 16 determines that the user is not alive, outputs information indicating this determination result to the second determination unit 17, and terminates the living body determination process based on the mouth opening / closing state.

[0216] Liveness detection based on glasses wearing and removing status

[0217] The following details the living body determination process based on the wearing and removal status of glasses. Figure 9 This is a flowchart showing an example of a living body determination process based on the wearing and removing state of glasses. If the living body determination process based on the wearing and removing state of glasses is executed, first, the output unit 14 performs the same Figure 3 Step S103 shown corresponds to step S601. In step S601, the output unit 14 displays on the display 3 action information for instructing wearing and removing glasses as an action required for the living body determination process.

[0218] Specifically, in step S601, the output unit 14 displays on the display 3 action information indicating a state of wearing glasses (putting on glasses), action information indicating a state of not wearing glasses (taking off glasses), or action information indicating putting on and taking off glasses a specified number of times.

[0219] The output unit 14 randomly determines the wearing and removing method of the glasses indicated in the action information. However, this is not limited to this. The output unit 14 may also regularly determine the wearing and removing method of the glasses indicated in the action information according to a predetermined rule, such as determining in the above-mentioned order.

[0220] Next, the facial image acquisition unit 15 performs Figure 3Step S104 shown corresponds to step S602. In step S602, the facial image acquisition unit 15 acquires a facial image including the user's face from the imaging device 2.

[0221] Next, the first judgment unit 16 performs the Figure 3 Step S105 shown corresponds to step S603 and subsequent processing.

[0222] In step S603 , the first determination unit 16 detects the wearing or removing state of glasses from the facial image acquired in step S602 .

[0223] Specifically, in step S603, the first judgment unit 16 first determines Figure 5 Similarly, in step S303 shown, the face area 80 is detected from the face image acquired in step S302.

[0224] Next, the first determination unit 16 binarizes the detected facial region 80, generating a binary image in which pixels with grayscale values ​​less than a threshold are represented by a first luminance value, and pixels with grayscale values ​​greater than the threshold are represented by a second luminance value. If the facial region 80 is a color image, the first determination unit 16 converts the facial region 80 into a grayscale image having grayscale values ​​ranging from 0 to 255, for example, and then performs the binarization process on the converted grayscale image.

[0225] As a binarization process, for example, Otsu binarization can be used. For example, the first brightness value is white, and the second brightness value is black. In this case, a binary image is generated in which dark areas are represented by white, and bright areas are represented by black. For example, the brightness value of white is 255, and the brightness value of black is 0.

[0226] Furthermore, when converting a color image into a grayscale image, the first determination unit 16 performs a general conversion process. For example, a general conversion process to a grayscale image may be one that calculates the average grayscale value of the red, green, and blue components of each pixel constituting the facial region 80. Furthermore, the first determination unit 16 may calculate the pixel value V of each pixel in the grayscale image using the following equation (1).

[0227] V=0.299·R+0.587·G+0.114·B (1)

[0228] In the above formula (1), R represents the red component value of each pixel in the color image, G represents the green component value of each pixel in the color image, and B represents the blue component value of each pixel in the color image.

[0229] Next, the first determination unit 16 extracts a glasses estimation region, where glasses are estimated to be present, from the generated binary image. Specifically, based on typical facial ratios, the first determination unit 16 extracts the region between the line indicating the position 3 / 10 of the top edge of the facial region 80 and the line indicating the position 6 / 10 of the top edge as the glasses estimation region.

[0230] Next, the first determination unit 16 performs labeling on the extracted glasses estimation region. In the labeling process, the same number is assigned to consecutive white pixels in the binary image. This labeling process detects multiple white regions consisting of consecutive white pixels within the glasses estimation region. As described above, by pre-extracting the binary image of the glasses estimation region from the binary image of the face region 80, the scope of the labeling process is narrowed, thereby shortening the time required for the labeling process.

[0231] Next, the first determination unit 16 calculates the width (horizontal length) of each of the plurality of white areas containing a plurality of consecutive white pixels within the extracted glasses estimation region. The first determination unit 16 then determines whether the width of the longest white area is at least two-thirds of the width of the glasses estimation region.

[0232] The white area representing the eyeglass frame is composed of a plurality of white pixels continuous in the horizontal direction. If the width of the longest white area is at least two-thirds the width of the estimated eyeglass area, it is considered that the user is wearing glasses. Therefore, if the length of the white area is at least two-thirds the width of the estimated eyeglass area, the first determination unit 16 determines that the user is wearing glasses. On the other hand, if the width of the longest white area is less than two-thirds the width of the estimated eyeglass area, the first determination unit 16 determines that the user is not wearing glasses.

[0233] The method used by the first determination unit 16 to detect the wearing or removal of glasses in step S603 is not limited to the method described above. For example, if external light is incident on the glasses frame or the glasses frame is colored, the white area representing the glasses frame may not be detected as an island. Alternatively, the first determination unit 16 may determine that the glasses are being worn if there are at least a predetermined number of white areas with a width greater than a predetermined length. Furthermore, the first determination unit 16 may determine that the glasses are not being worn if there are not at least a predetermined number of white areas with a width greater than a predetermined length.

[0234] Next, in step S604 , the first determination unit 16 determines whether the glasses wearing or removing state detected in step S603 is the same as the glasses wearing or removing state indicated by the action information presented in step S601 .

[0235] Assume that the first determination unit 16 determines in step S604 that the glasses wearing / removing state detected in step S603 is the same as the glasses wearing / removing state indicated by the action information presented in step S601. In this case (YES in step S604), in step S605, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the glasses wearing / removing state.

[0236] On the other hand, suppose that the first determination unit 16 determines in step S604 that the glasses wearing / removing state is different from the glasses wearing / removing state indicated by the action information presented in step S601. In this case (No in step S604), in step S606, the first determination unit 16 determines that the user is not alive, outputs information indicating this determination result to the second determination unit 17, and terminates the living body determination process based on the orientation of the glasses.

[0237] Furthermore, if, in step S601, the action information instructing the user to put on and take off glasses a predetermined number of times is displayed on the display 3, then, in step S602, the facial image acquisition unit 15 acquires a facial image from the imaging device 2 at any time at a predetermined frame cycle. Each time a facial image is acquired, the first determination unit 16 performs step S603 using the acquired facial image and then performs step S604.

[0238] At this point, if the glasses wearing / removing state detection results of each step S603 detect a change from the glasses-not-wearing state to the glasses-wearing state more than a predetermined number of times within the predetermined upper limit time, the first determination unit 16 determines in step S604 that the glasses wearing / removing state is the same as the glasses wearing / removing state indicated by the action information presented in step S601. In this case, in step S605, the first determination unit 16 determines that the user is alive and outputs information indicating this determination result to the second determination unit 17, thereby terminating the living body determination process based on the glasses wearing / removing state.

[0239] On the other hand, if the glasses wearing / removing state detection results in each step S603 fail to detect a change from the glasses-not-wearing state to the glasses-wearing state more than a predetermined number of times within the predetermined upper limit time, the first determination unit 16 determines in step S604 that the glasses wearing / removing state is different from the glasses wearing / removing state indicated by the action information presented in step S601. In this case, in step S606, the first determination unit 16 determines that the user is not alive, outputs information indicating this determination result to the second determination unit 17, and terminates the living body determination process based on the glasses wearing / removing state.

[0240] Implementation Method 2

[0241] When the face included in the facial image is a face included in a photo or a face displayed on a display of a smartphone or tablet, the facial image may include a frame of the photo or display. Figure 3 Before the steps S101 and subsequent processing shown in FIG, a facial image containing the user's face is obtained from the camera device 2, and it is determined whether the acquired facial image contains a frame. If a frame is included, the determination device 1 determines that the user included in the facial image is not a living person. If the facial image does not contain a frame, the determination device 1 performs Figure 3 The processing of step S101 and thereafter is shown.

[0242] According to this, the judgment device 1 of the second embodiment does not perform the judgment when a frame is formed around the face image and it is basically clear that the user is not a living body. Figure 3 On the other hand, when there is no frame around the face image, the judgment device 1 of the second embodiment is the same as the first embodiment, by performing Figure 3 The processing in and after step S101 shown makes impersonation difficult.

[0243] Hereinafter, the determination device 1 according to the second embodiment will be described in detail. Figure 10 In step S100 , if the receiving unit 11 receives an instruction to execute a living body determination, the facial image acquisition unit 15 acquires a facial image (hereinafter referred to as a user image) including the user's face from the imaging device 2 in step S701 .

[0244] In step S702, the second judgment unit 17 and Figure 5 Similarly, in step S303 shown, the face area 80 is detected from the user image acquired in step S701. Figure 11 is a diagram showing an example of the face region 80 detected from the user image 110. For example, Figure 11 An example is shown in which a face region 80 including a face of a person projected on a display of a smartphone is detected.

[0245] Next, in step S703 , the second determination unit 17 performs a vertical line detection process. The vertical line detection process is a process of detecting two vertical lines that may be included in the user image 110 and constitute a frame of a photo or a display.

[0246] Specifically, during vertical line detection, the second determination unit 17 first performs filtering only in the horizontal direction of the user image 110, using, for example, a Sobel filter (a first-order differential) or a Laplacian filter (a second-order differential). A horizontal edge represents a group of pixels whose luminance values ​​differ significantly from those of its horizontally adjacent pixels.

[0247] Figure 12 : is a diagram showing an example of a horizontal edge detected in the vertical line detection process. Figure 12 Represents an image 120 in which only Figure 11 The horizontal edges detected by the above-mentioned filtering process in the horizontal direction of the user image 110 shown are represented by white pixels, and the other pixels in the user image 110 are represented by black. Figure 12 The dotted line shows the Figure 11 The user image 110 is shown with the facial region 80 detected.

[0248] Next, the second determination unit 17 detects a first edge of the detected horizontal edges that extends vertically and is closest to the left end of the facial region 80 and has a length greater than or equal to a first predetermined length, and a second edge that is closest to the right end of the facial region 80 and has a length greater than or equal to the first predetermined length. The first predetermined length can be a reasonable length based on, for example, the vertical length of the user image 110.

[0249] exist Figure 12 In the example shown in FIG. 1 , as horizontal edges extending in the vertical direction, a horizontal edge E1 closest to the left end of the face region 80 and having a length not less than the first predetermined length and a horizontal edge E2 closest to the right end of the face region 80 and extending in the vertical direction are detected. Therefore, the second determination unit 17 detects the horizontal edge E1 as the first edge and the horizontal edge E2 as the second edge.

[0250] Next, in step S704, the second determination unit 17 performs a horizontal line detection process. The horizontal line detection process is a process of detecting two horizontal lines that may be included in the user image 110 and constitute a frame of a photo or a display.

[0251] Specifically, in the horizontal line detection process, the second determination unit 17 first performs the same filtering process as the vertical line detection process only in the vertical direction of the user image 110, thereby detecting vertical edges from the user image 110. A vertical edge represents a group of pixels whose brightness value is significantly different from the brightness value of its vertically adjacent pixels.

[0252] Figure 13 : is a diagram showing an example of a vertical edge detected in the horizontal line detection process. Figure 13 Represents an image 130, which is in Figure 12 The image 120 shown is represented by white pixels. Figure 11 The user image 110 shown is formed by performing the above-mentioned filtering process and detecting vertical edges.

[0253] Next, the second determination unit 17 detects a third edge extending horizontally from among the detected vertical edges, which is closest to the upper end of the facial region 80 and has both ends closer to the facial region 80 than the first and second edges, and has a length of at least a second predetermined length; and a fourth edge closest to the lower end of the facial region 80 and has both ends closer to the facial region 80 than the first and second edges, and has a length of at least the second predetermined length. The second predetermined length can be a reasonable length based on, for example, the horizontal length of the user image 110.

[0254] exist Figure 13 In the example shown in FIG. 1 , a vertical edge E3 having a length not less than the second predetermined length is detected as a vertical edge extending horizontally, the edge being closest to the upper end of the facial region 80 and having both ends closer to the facial region 80 than the horizontal edge E1 detected as the first edge and the horizontal edge E2 detected as the second edge. However, no vertical edge having a length not less than the second predetermined length is detected, the edge being closest to the lower end of the facial region 80 and having both ends closer to the facial region 80 than the first and second edges. Therefore, the second determination unit 17 detects the vertical edge E3 as the third edge and does not detect the fourth edge.

[0255] Next, in step S705 , the second determination unit 17 determines whether the user image 110 includes a frame based on the detection results of the vertical line detection process and the horizontal line detection process.

[0256] Specifically, in step S705, the second determination unit 17 determines that the user image 110 includes a frame when the total number of edges detected by the vertical line detection process and the horizontal line detection process is 3 or more. Figure 13 In the example of , since the first, second, and third edges are detected by the vertical line detection process and the horizontal line detection process, the second determination unit 17 determines that the user image 110 includes a frame.

[0257] If it is determined in step S705 that the user image 110 includes a frame (step S705 is yes), no further processing is performed. Figure 3 Step S101 and subsequent processing are performed as shown. Figure 3 On the other hand, if it is determined in step S705 that the user image 110 does not contain a frame (step S705 is No), the process is performed. Figure 3 The processing of step S101 and thereafter is shown.

[0258] Furthermore, a malicious user who knows that the user is not alive when the user image 110 contains a frame of a photo or display may bring the photo or display closer to the imaging device 2 so that the frame is not included in the user image 110 .

[0259] In this regard, the second judgment unit 17 may also skip steps S703 and S704 and determine in step S705 that the user image 110 contains a border when the height of the facial area 80 detected in step S702 is greater than the specified upper limit height or the width of the facial area 80 is greater than the specified upper limit width.

[0260] The height of the facial region 80 refers to the vertical length of the facial region 80. The upper limit of the height may be set, for example, to be at least 90% of the vertical length of the user image 110. However, the upper limit of the height is not limited to this. The width of the facial region 80 refers to the horizontal length of the facial region 80. The upper limit of the width may be set, for example, to be at least 90% of the horizontal length of the user image 110. However, the upper limit of the width is not limited to this.

[0261] According to this configuration, if a malicious user brings a picture containing another person's face or a display close to the camera device 2 so that the frame of the display is not included in the user image 110, and the face area 80 detected from the user image 110 becomes larger to a degree exceeding the upper limit width or upper limit height, it is determined that the user image 110 contains a frame. Figure 3 In step S109 and subsequent processing, it is determined that the user is not a living person. Therefore, it is possible to prevent a malicious user who attempts to deceive others by using a photo or display containing another person's face from bringing the photo or display close to the camera 2 so that the frame of the photo or display is not included in the user image 110.

[0262] In addition, Figure 10 In the method for determining whether a user image 110 includes a frame, if three or more edges cannot be detected by the vertical line detection process and the horizontal line detection process, the user image 110 is determined to include no frame. Therefore, a malicious user could potentially bring a photo or display containing another person's face close to the imaging device 2 so that two or more frame lines forming the frame of the photo or display are not included in the user image 110.

[0263] Figure 14 1 is a diagram showing an example of a face region 80 included in a user image 110. Specifically, a malicious user may, for example, Figure 14 As shown in FIG, the display is brought close to the camera 2 so that the upper end of the other person's face is located near the upper end of the user image 110 or so that the lower end of the other person's face is located near the lower end of the user image 110. In these cases, for example Figure 14 In this way, sometimes only the left and right frame lines constituting the frame of the display are included in the user image 110 .

[0264] In this regard, in step S705, the second judgment unit 17 may further determine that the user image 110 contains a border when the first edge and the second edge are detected in the vertical line detection processing of step S703, when the upper end of the facial area 80 detected in step S702 exists within a first margin specified from the upper end of the user image 110, or when the lower end of the facial area 80 exists within a second margin specified from the lower end of the user image 110.

[0265] Furthermore, the first margin and the second margin may be set to, for example, 10% or less of the vertical length (height) of the user image 110. With this configuration, even when the user image 110 includes only two frame lines, one on the left and one on the right of the display showing the face of another person, it can be determined that the user image 110 includes a frame.

[0266] In addition, a malicious user may cause the camera device 2 to capture a photo or display containing another person's face so that the other person's face is located near the left or right end of the image captured by the camera device 2 in order to prevent two or more frame lines of the frame of the photo or display containing another person's face from being reflected in the user image 110.

[0267] Figure 15 1 is another example of a face region 80 included in the user image 110. For example, suppose a malicious user causes the camera 2 to capture the display including the face of another person so that the face of the other person is located near the right end of the user image 110. In this case, for example Figure 15 In that case, only the left and lower portions of the L-shaped intersecting frame lines of the display frame may be included in the user image 110. Furthermore, in this case, only the left and upper portions of the L-shaped intersecting frame lines of the display frame may be included in the user image 110. Similarly, if a malicious user causes the camera device 2 to capture the display containing another person's face so that the other person's face is close to the left end of the user image 110, only the right portion and upper or lower portions of the L-shaped intersecting frame lines of the display frame may be included in the user image 110.

[0268] In this regard, in step S705, the second judgment unit 17 may further determine that the user image 110 contains a border when the first edge or the second edge detected in the vertical line detection processing of step S703 intersects the third edge or the fourth edge detected in the horizontal line detection processing of step S704, when the left end of the facial area 80 detected in step S702 exists within the third margin specified from the left end of the user image 110, or when the right end of the facial area 80 exists within the fourth margin specified from the right end of the user image 110.

[0269] The third margin and the fourth margin may be set to, for example, 1 to 5% of the horizontal length (width) of the user image 110. However, the third margin and the fourth margin are not limited thereto.

[0270] That is, the second judgment unit 17 may also determine that the user image 110 contains a border when two edges crossing in an L shape are detected by the vertical line detection processing of step S703 and the horizontal line detection processing of step S704, when the left end of the facial area 80 detected in step S702 is within the third margin specified from the left end of the user image 110, or when the right end of the facial area 80 is within the fourth margin specified from the right end of the user image 110.

[0271] With this configuration, even when the user image 110 includes only two frame lines that intersect in an L shape, such as the left and lower frame lines that constitute the frame of a display displaying another person's face, it can be determined that the user image 110 includes a frame.

[0272] The present invention can adopt the following modified examples.

[0273] (1) In Figure 3 In the case where two or more living body determination processes selected in step S101 are executed in step S102, the processes corresponding to steps S103 to S105 may be performed multiple times in each living body determination process. Furthermore, in each process corresponding to step S103, the output unit 14 may randomly determine the action to be presented to the user.

[0274] For example, in step S102 ( Figure 3 ) is executed based on the eye direction of the living body determination process, it can also be performed multiple times: execute the same as step S103 ( Figure 3 ) corresponds to step S301 ( Figure 5 ), and step S104 ( Figure 3 ) corresponding to step S302 ( Figure 5 ), and step S105 ( Figure 3) corresponding to step S303 ( Figure 5 ) and subsequent processing. In addition, in each step S301 ( Figure 5 ), the output unit 14 can also randomly determine the eye direction indicated in the action information displayed on the display 3.

[0275] This configuration exponentially increases the number, content, and execution order of the liveness determination processes performed to determine whether a user is alive, as well as the number and content of actions presented during each liveness determination process. This makes it more difficult for a malicious user to prepare facial images that would indicate the user is alive.

[0276] (2) In Figure 3 In step S101 shown, the determination method determination unit 12 may detect the state of the user and select a living body determination process whose actions required for the living body determination process are suitable for the detected state of the user.

[0277] This configuration can be implemented, for example, as follows: In step S101 , the determination method determination unit 12 first causes the camera 2 to acquire a facial image containing the user's face. The determination method determination unit 12 then inputs the acquired facial image into a face region detection classifier to detect the face region 80 .

[0278] Next, the judgment method determination unit 12 inputs the detected face region 80 into the eyeglass region detection classifier to detect the eyeglass region included in the face region 80. If the detector does not detect the eyeglass region, it is because the user is in a naked-eye state. Therefore, the judgment method determination unit 12 selects two or more living body determination processes from the plurality of living body determination processes, excluding the living body determination process based on the eyeglass wearing or removing state.

[0279] Furthermore, the judgment method determination unit 12 may input the detected face region 80 into a classifier for detecting sunglasses regions to detect the sunglasses region included in the face region 80. Since the detector cannot detect the eye direction or eye opening / closing status of the user when the sunglasses region is detected, the judgment method determination unit 12 may select two or more living body assessment processes other than the living body assessment process based on eye direction and the living body assessment process based on eye opening / closing status from among the multiple living body assessment processes.

[0280] Furthermore, the judgment method determination unit 12 may input the detected face region 80 into a classifier for detecting mask regions to detect the mask region included in the face region 80. When the detector detects the mask region, it is unable to detect the user's mouth opening or closing state. Therefore, the judgment method determination unit 12 may select two or more living body determination processes other than the living body determination process based on the mouth opening or closing state from among the multiple living body determination processes.

[0281] According to this configuration, two or more living body determination processes requiring actions appropriate to the state of the user can be selected as living body determination processes executed to determine whether the user is alive.

[0282] (3) In Figure 3 In step S107 shown, the second determination unit 17 may quantify the determination results of the first determination unit 16 in each of the two or more living body determination processes selected in step S101 and perform weighted addition of the quantified results using a coefficient specified for each living body determination process. If the result of this weighted addition satisfies a specified numerical condition, the second determination unit 17 may determine that the determination result of the first determination unit 16 satisfies the specified condition.

[0283] This configuration can be implemented, for example, as follows. If the first determination unit 16 determines that the user is alive in each of the two or more living body determination processes selected in step S101, the determination result is numerically converted to 1. On the other hand, if the first determination unit 16 determines that the user is not alive, the determination result is numerically converted to 0. Furthermore, a threshold of 2 / 3 of the number of the two or more living body determination processes selected in step S101 is set, and if the result of the weighted addition of the determination results of each living body determination process is greater than or equal to this threshold, the numerical condition is deemed to be satisfied.

[0284] At this time, the second determination unit 17 sets the coefficient specified for each living body determination process to 1 and performs weighted addition on the determination results of each living body determination process. If the result of this weighted addition is greater than the threshold value, the numerical condition is satisfied, and the determination unit 17 determines that the determination result of the first determination unit 16 satisfies the specified condition. On the other hand, if the result of this weighted addition is less than the threshold value, the numerical condition is not satisfied, and the second determination unit 17 determines that the determination result of the first determination unit 16 does not satisfy the specified condition.

[0285] Or, for example, assume that the living body determination process based on the face direction and the living body determination process based on the eye direction are selected in step S101. Figure 4 Step S201 shown in FIG. 1 displays action information indicating that the face is facing right on the display 3, and Figure 5In step S301 shown, motion information indicating that the eyes are facing right is displayed on the display 3. Furthermore, assuming that the liveness determination process based on facial orientation has a higher accuracy than the liveness determination process based on eye orientation, the coefficient of the liveness determination process based on facial orientation used for weighted addition of determination results is set to 0.7, and the coefficient of the liveness determination process based on eye orientation is set to 0.3.

[0286] At this time, the first judgment unit 16 Figure 4 In step S206 shown in FIG. 1 , the value of the horizontal component of the face orientation calculated in steps S204 and S205 is divided by the value of the horizontal component of the face orientation when the face is facing sideways and to the right, and the result is output as a numerical determination result. Figure 5 In step S306, the result of dividing the horizontal eye orientation value calculated in step S305 by the horizontal eye orientation value when the eyes are facing sideways and rightward is output as a numerical judgment result. Furthermore, a threshold of 2 / 3 of the number of living body assessment processes selected in step S101 is set, and the numerical condition is considered to be satisfied when the result of the weighted addition of the judgment results of each living body assessment process is greater than or equal to this threshold.

[0287] At this time, if the result of weighted addition of the judgment result of the first judgment unit 16, represented by the sum of the product of the judgment result of the living body judgment process based on the face direction and the coefficient 0.7, and the product of the judgment result of the living body judgment process based on the eye direction and the coefficient 0.3, is greater than the threshold value, the numerical condition is satisfied, and the second judgment unit 17 determines that the judgment result of the first judgment unit 16 satisfies the prescribed condition. On the other hand, if the result of the weighted addition is less than the threshold value, the numerical condition is not satisfied, and the second judgment unit 17 determines that the judgment result of the first judgment unit 16 does not satisfy the prescribed condition.

[0288] According to this configuration, as described in the above example, by adjusting the determination accuracy coefficient according to each living body determination process, etc., it is possible to improve the determination accuracy of whether the user is a living body.

[0289] Industrial applicability

[0290] According to the present invention, since the accuracy of living body determination can be improved, it has practical value in the technical field of living body determination.

Claims

1. A judgment method is a judgment method of a judgment device, characterized in that: The computer of the judgment device executes the following steps: selecting two or more living body determination processes from a plurality of living body determination processes for determining whether a person included in an image is a living body; determining an order in which the two or more selected living body determination processes are executed; executing the selected two or more living body determination processes respectively in the determined order; In each of the two or more selected living body determination processes, the following are performed: (1) a process of prompting a user to perform an action required for the living body determination process; (2) a process of acquiring a facial image including the face of the user performing the action prompted; (3) determining whether the user is alive based on the facial features contained in the facial image; determining whether the determination results obtained from each of the two or more selected living body determination processes satisfy a predetermined condition; as well as, If the judgment result satisfies the prescribed condition, the user is judged to be a living body; if the judgment result does not satisfy the prescribed condition, the user is judged to be not a living body. Before selecting the two or more living body assessment processes from the plurality of living body assessment processes, detecting the state of the user, and The two or more living body assessment processes excluding a living body assessment process including a motion inappropriate for determining whether a person included in the image is a living body based on the detected state of the user are selected from the plurality of living body assessment processes.

2. The judgment method according to claim 1, characterized in that: When selecting the two or more living body assessment processes, the two or more living body assessment processes are randomly selected from the plurality of living body assessment processes.

3. The judgment method according to claim 1 or 2, characterized in that: When determining the order of executing the selected two or more living body assessment processes, the order of executing the selected two or more living body assessment processes is randomly determined.

4. The judgment method according to claim 1 or 2, characterized in that: The multiple living body determination processes include: Living body determination processing using facial orientation as a feature of a part of the face; Liveness determination processing using eye direction as a feature of the face; Liveness determination processing using the open / closed state of the eyes as a feature of the face; Living body determination processing using the mouth opening and closing state as the part feature of the face; and Liveness determination processing using the wearing and removing status of glasses as the face feature.

5. The judgment method according to claim 1 or 2, characterized in that: In each of the two or more selected living body determination processes: Repeat the steps (1) to (3) above for several times. In each of the processes (1), an action to be presented to the user is randomly determined.

6. The judgment method according to claim 1 or 2, characterized in that: When judging whether the judgment result satisfies the prescribed condition, the judgment result is quantified, and if the result of weighted addition of the quantified result using the coefficient prescribed for each living body judgment processing satisfies the prescribed numerical condition, it is judged that the judgment result satisfies the prescribed condition.

7. The judgment method according to claim 1 or 2, characterized in that: The computer also performs the following steps: obtaining a user image containing the user's face, Determine whether the user image contains a frame surrounding a person's face. If it is determined that the user image contains the frame, determine that the user is not a living person. If it is determined that the user image does not contain the frame, select the two or more living body determination processes and subsequent processes.

8. The judgment method according to claim 7, characterized in that: When determining whether the user image contains the frame, detecting a face region including the user's face from the user image, Perform vertical line detection processing and horizontal line detection processing, wherein, In the vertical line detection process, a first edge closest to the left end of the face region and having a length greater than a first predetermined length and a second edge closest to the right end of the face region and having a length greater than the first predetermined length are detected among the edges extending in the vertical direction included in the user image. In the horizontal line detection process, a third edge closest to the upper end of the face region and a fourth edge closest to the lower end of the face region are detected among the edges extending in the horizontal direction included in the user image, wherein both ends of the third edge are closer to the face region side than the first edge and the second edge and have a length greater than a second specified length, and both ends of the fourth edge are closer to the face region side than the first edge and the second edge and have a length greater than the second specified length. When the total number of edges detected by the vertical line detection process and the horizontal line detection process is three or more, it is determined that the frame is included in the user image.

9. The judgment method according to claim 7, characterized in that: When determining whether the user image contains the frame, detecting a facial region including the user's face from the user image, When the width of the face region is greater than a predetermined upper limit width, or when the height of the face region is greater than a predetermined upper limit height, it is determined that the frame is included in the user image.

10. The judgment method according to claim 8, characterized in that: When determining whether the user image contains the frame, detecting a facial region including the user's face from the user image, In a case where the upper end of the facial area exists within a first margin specified from the upper end of the user image, or in a case where the lower end of the facial area exists within a second margin specified from the lower end of the user image, when the first edge and the second edge are detected in the vertical line detection processing, it is determined that the user image contains the border.

11. The judgment method according to claim 8 or 10, characterized in that: When determining whether the user image contains the frame, In a case where the left end of the facial area exists within a third margin specified from the left end of the user image, or in a case where the right end of the facial area exists within a fourth margin specified from the right end of the user image, when the first edge or the second edge detected in the vertical line detection processing intersects the third edge or the fourth edge detected in the horizontal line detection processing, it is determined that the user image contains the border.

12. A judgment device, characterized in that include: a determination method determination unit that selects two or more living body determination processes from a plurality of living body determination processes for determining whether a person included in an image is a living body, and determines an order in which the two or more selected living body determination processes are to be executed; an execution unit that executes the selected two or more living body determination processes respectively in the determined order; In each of the two or more selected living body determination processes, an output unit prompts a user to perform an action required for the living body determination process; a facial image acquisition unit acquires a facial image including the face of the user performing the action prompted; and a first determination unit determines whether the user is alive based on facial features included in the facial image. as well as, The second judgment unit judges whether the judgment results obtained from the two or more selected living body judgment processes respectively meet a predetermined condition, and judges that the user is a living body if the judgment result meets the predetermined condition, and judges that the user is not a living body if the judgment result does not meet the predetermined condition. The determination method determination unit detects the state of the user before selecting the two or more living body determination processes from the plurality of living body determination processes, and The determination method determination unit selects the two or more living body assessment processes from the plurality of living body assessment processes excluding a living body assessment process including a motion that is not suitable for determining whether a person included in the image is a living body based on the detected state of the user.

13. A computer program product comprising a judgment program for causing a computer to function as a judgment device, characterized in that: The judgment program causes the computer to function as the following units: a determination method determination unit that selects two or more living body determination processes from a plurality of living body determination processes for determining whether a person included in an image is a living body, and determines an order in which the two or more selected living body determination processes are to be executed; an execution unit that executes the selected two or more living body determination processes respectively in the determined order; In each of the two or more selected living body determination processes, an output unit prompts a user to perform an action required for the living body determination process; a facial image acquisition unit acquires a facial image including the face of the user performing the action prompted; and a first determination unit determines whether the user is alive based on facial features included in the facial image. as well as, The second judgment unit judges whether the judgment results obtained from the two or more selected living body judgment processes respectively meet a predetermined condition, and judges that the user is a living body if the judgment result meets the predetermined condition, and judges that the user is not a living body if the judgment result does not meet the predetermined condition. The determination method determination unit detects the state of the user before selecting the two or more living body determination processes from the plurality of living body determination processes, and The determination method determination unit selects the two or more living body assessment processes from the plurality of living body assessment processes excluding a living body assessment process including a motion that is not suitable for determining whether a person included in the image is a living body based on the detected state of the user.

Images