Determination device, mobile terminal including the determination device, and program for the determination device

The determination device uses light lacking a specific color and convolutional neural networks to securely and efficiently differentiate between real and printed faces, addressing the inefficiencies of existing face authentication methods.

JP7596114B2Active Publication Date: 2024-12-09ELEMENTS INC
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2020173696
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-12-13
Filing Date
2020-10-15
Publication Date
2024-12-09
Estimated Expiration
2040-10-15

AI Technical Summary

Technical Problem

Existing face authentication technologies require multiple imaging sessions, are inconvenient due to method switching, and lack effective prevention of unauthorized use.

Method used

A determination device that captures a person's face using light lacking a specific color spectrum, analyzes the image data by separating color components, and employs convolutional neural networks to distinguish between real faces and printed or displayed images, enhancing security and processing speed.

Benefits of technology

Enables reliable, one-time imaging to determine a real person's face quickly, preventing unauthorized use and improving accuracy against display attacks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596114000001
    Figure 0007596114000001
  • Figure 0007596114000002
    Figure 0007596114000002
  • Figure 0007596114000003
    Figure 0007596114000003
Patent Text Reader

Abstract

To provide a determination device, a portable terminal including the determination device, and a program for the determination device, capable of preventing unauthorized use and determining a person by imaging one time.SOLUTION: A determination device 100 determines a face of a person. The determination device 100 includes an imaging device 200 that images a face of a person only one time, a light-emitting device 300 that performs light-emission with a lack of color when the imaging device 200 performs imaging, an analysis device 400 that analyzes imaging date from the imaging device 200, and a control portion 500 that controls the imaging device 200, the light-emitting device 300, and the analysis device 400. The control portion 500 classifies the image data 200 in color by the analysis device 400, and the face of the person is determined according to a difference in the classified data.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a determination device for determining a human face, a mobile terminal including the determination device, and a program for the determination device. [Background technology]

[0002] Conventionally, research and development has been conducted on a determination device for determining a human face. For example, Patent Document 1 (JP Patent Publication 2007-148968 A) discloses a face recognition technology that can set different security levels according to the purpose of use without forcing a user to perform complicated operations.

[0003] The face recognition device described in Patent Document 1 (JP Patent Publication 2007-148968 A) comprises a storage means for acquiring and storing features from a facial image of a registrant, an authentication means for acquiring facial features from an input image and authenticating whether it matches a registrant stored in the storage means, a security setting means for allowing a user to set a security strength for each purpose of use, and a control means for restricting storage in the storage means if the facial image of the registrant does not satisfy specified shooting conditions based on the security strength, and for changing the content of the authentication process performed by the storage means and authentication means based on the security strength.

[0004] Furthermore, Patent Document 2 (JP Patent Publication 2004-260240A) discloses a mobile phone that can set a security mode that matches the level of awareness of security of each user.

[0005] The mobile phone described in Patent Document 2 (JP Patent Publication No. 2004-260240) is a mobile phone having a plurality of security execution means for executing security functions of the mobile phone, and a selection means for selecting one security execution means from the plurality of security execution means.

[0006] Moreover, Patent Document 3 (JP Patent Publication 2004-80080A) discloses a mobile phone with a personal authentication function that allows the setting of a security level according to the mode of use.

[0007] The mobile phone with personal authentication function described in Patent Document 3 (JP 2004-80080 A) comprises an authentication method selection memory unit that selects at least one personal authentication method from a plurality of personal authentication methods and stores the selected personal authentication method, an information storage unit that stores input information used for the personal authentication method stored in the authentication method selection memory unit, a feature extraction unit that extracts feature information from the input information stored in the information storage unit as authentication information, a feature recording unit that records the user's feature information in advance, and a feature authentication unit that performs personal authentication of the user by comparing the feature information previously recorded in the feature recording unit with the authentication information extracted by the feature extraction unit.

[0008] Furthermore, Patent Document 4 (JP Patent Publication No. 2002-112340) discloses a system and method for authenticating a mobile device, which allows the user to freely change the authentication means depending on the purpose of use, and therefore to set a security level that satisfies the user for each purpose of use.

[0009] The mobile unit authentication system described in Patent Document 4 (JP Patent Publication No. 2002-112340) is an authentication system that authenticates the identity of a mobile unit by communicating with a parent station, and is equipped with a plurality of identity authentication means for authenticating that the user of the mobile unit is the identity of the user, an authentication setting means in which a correspondence between these identity authentication means and the intended use of the mobile unit is set by the user, an authentication information input means in which authentication information for judgment by the identity authentication means is input from the mobile unit, and an identity verification means for comparing the authentication information input by this means with pre-stored authentication information to verify whether the user is the identity of the user, and changes the identity authentication means depending on the intended use of the user based on the correspondence set by the authentication setting means.

[0010] Furthermore, Patent Document 5 (JP Patent Publication 2000-215308 A) discloses that the balance between personal authentication performance and comfort, as well as resistance to environmental changes, can be adjusted according to circumstances, thereby improving convenience.

[0011] The biometric authentication device described in Patent Document 5 (JP Patent Publication 2000-215308 A) is equipped with a biometric information receiving means for acquiring biometric information of a person to be authenticated and extracting the characteristics of the person to be authenticated from the input information to generate biometric feature information, and an identification means for comparing the biometric feature information with registered information prepared in advance for each category and determining the category to which the person to be authenticated belongs, and the biometric authentication device has a mode management means for controlling the biometric information receiving means and the identification means by selecting one from a plurality of authentication logic modes when performing biometric authentication using authentication logic formed by combining at least one of the biometric information receiving means and the identification means. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] JP 2007-148968 A [Patent Document 2] JP 2004-260240 A [Patent Document 3] JP 2004-80080 A [Patent Document 4] JP 2002-112340 A [Patent Document 5] JP 2000-215308 A Summary of the Invention [Problem to be solved by the invention]

[0013] The technology described in Patent Document 1 discloses a technology that allows different security levels to be set depending on the purpose of use, but this technology has the problem that imaging must be performed many times. Furthermore, in the techniques described in Patent Documents 2, 3 and 4, authentication methods must be switched or selected for authentication, resulting in a problem of low convenience. Furthermore, the technology described in Patent Document 5 has the problem that it is difficult to prevent unauthorized use.

[0014] A main object of the present invention is to provide a determination device, a mobile terminal including the determination device, and a program for the determination device, which are capable of preventing unauthorized use and determining a person by capturing an image once. Another object of the present invention is to provide a determination device, a portable terminal including the determination device, and a program for the determination device, which can increase processing speed, prevent fraudulent use by taking an image once, and reliably identify a person. [Means for solving the problem]

[0015] (1) According to one aspect, a determination device for determining a human face includes an imaging device that captures an image of a human face only once, a light-emitting device that emits light lacking a specific color when the imaging device captures an image, an analysis device that analyzes imaging data from the imaging device, and a control device that controls the imaging device, the light-emitting device, and the analysis device, and the control device decomposes the imaging data into color components using the analysis device, and determines whether the imaging data is imaging data of a real person's face based on the difference between component data of a color emitted by the light-emitting device and component data of a color not emitted by the light-emitting device.

[0016] In this case, the face of a person can be reliably identified. In particular, since there is no need to capture images multiple times, it is possible to identify a large number of people in a short period of time. Moreover, being able to reliably determine a person's face means being able to determine whether it is the face of a person who actually exists, or the face of a person that has been photographed and printed. Also, light lacking a particular color is light lacking a particular color spectrum or frequency, and does not mean white light. For example, light lacking a specific color may be light in which, among RGB, G and B, or only G or only B, are dominant over R. Therefore, it includes a state in which no red (R) is contained at all, as well as a state in which a small amount of red (R) is contained.

[0017] (2) In a determination device according to a second aspect of the present invention, the control device may determine whether or not the imaging data is imaging data of a face of a real person at a specific portion of the person's face.

[0018] In this case, a real person's face can be determined using only a portion of the face, that is, a specific feature portion.

[0019] (3) A determination device according to a third aspect of the present invention is the determination device according to the second aspect of the present invention, wherein the specific part is at least one of the eyes, the nose, and the cheeks.

[0020] In this case, a person's face can be identified not by the entire face, but by characteristic parts such as the nose, mouth, eyes, cheeks, and contours.

[0021] (4) In the determination device according to the fourth aspect of the present invention, the control device may instruct the light emitting device to emit light lacking a specific color multiple times, or to emit multiple lights lacking different colors.

[0022] In this case, since the user receives various lights emitted from the light-emitting device, he or she cannot recognize which color of light is missing and the judgment is being made based on that color, thereby improving security and preventing fraudulent use. For example, in an RGB color code, by emitting light in the order of colors #00FF00, #008000, #0000FF, #FFFF00, #00080, #00FFFF, and #800080, and using only the data captured at the moment of #0000FF, the user will not know at what timing the light was emitted, thereby improving security. Note that the above RGB color code indicates the light intensity in the order of RGB from the left, with each color expressed as a two-digit hexadecimal number.

[0023] (5) In the determination device according to the fifth aspect of the present invention, the emission of light lacking a specific color may be any one of light lacking one color in RGB, light lacking one color in the XYZ color system, and light lacking one color in the Munsell color system.

[0024] In this case, the emission of light lacking a specific color consists of one of the following: light lacking one color in RGB, light lacking one color in the XYZ color system, or light lacking one color in the Munsell color system. Therefore, the user cannot recognize the detailed judgment, thereby enhancing security and preventing fraudulent use.

[0025] (6) In the determination device according to the sixth aspect of the present invention, in the determination device according to the sixth aspect of the present invention, the determination device may determine that the imaging data is imaging data of the face of a real person when the brightness of the eye area is greater than the brightness of the surrounding areas, based on data obtained by subtracting component data of colors not emitted by the light-emitting device from component data of colors emitted by the light-emitting device, among data decomposed into each color component.

[0026] In this case, it is possible to reliably determine the image data of the face of a real person.

[0027] (7) A seventh aspect of the present invention provides a determination device, the determination device comprising: 1) subtracting, from among the data separated into color components, component data of colors that the light-emitting device does not emit light from component data of colors that the light-emitting device emits light; 2) determining the peak value of the subtracted data; 3) dividing the subtracted data into first half data and second half data with the peak value as a boundary; 4) reversing the order of the latter half of the data so that the boundary comes last; 5) calculating the Pearson product moment correlation coefficient between the first half data and the second half data whose order has been reversed; 6) determining that the imaging data is imaging data of a face of a real person if the Pearson product moment correlation coefficient is 0.5 or more; and

[0028] In this case, it is possible to easily and quickly determine whether the captured image data is of a real person's face. Note that it is assumed that the first and second halves have the same amount of data. If the peak value is not the center value, the correlation coefficient of a predetermined amount of data is calculated from the peak value.

[0029] (8) The determination device of the eighth aspect of the present invention may calculate a difference between component data of a color emitted by the light-emitting device and component data of a color not emitted by the light-emitting device for a human face part and a background part, and compare the magnitude of the difference for the human face part with the magnitude of the difference for the background part to determine whether or not the imaging data is imaging data of the face of a real person.

[0030] In the case of image data of a real person's face, the color components emitted by the light-emitting device have high brightness in the face area (close to the light-emitting device) due to reflection from the face, and low brightness in the background area (outside the face). In contrast, there is little difference between the face area and the background area in the color components not emitted by the light-emitting device. Therefore, the difference between the color component data emitted by the light-emitting device and the color component data not emitted by the light-emitting device is larger in the face area than in the background area. On the other hand, in the case of image data such as a photograph that does not show the face of a real person, the light from the light-emitting device is also reflected in the background, so there is little difference between the face and the background in terms of the color components emitted by the light-emitting device and the color components not emitted by the light-emitting device. Therefore, the magnitude of the difference between the component data of the color emitted by the light-emitting device and the color components not emitted by the light-emitting device is small between the face and the background. As described above, by comparing the magnitude of the difference between the component data of the color emitted by the light-emitting device and the component data of the color not emitted by the light-emitting device, the magnitude of the difference in the part of the person's face and the magnitude of the difference in the background part, if the ratio of the magnitude of the difference or the difference is equal to or greater than a predetermined value, it can be determined that the imaging data is imaging data of the face of a real person.

[0031] (9) A determination device of the invention according to another aspect is a determination device for determining a human face, and includes an imaging device for imaging the human face, a light-emitting device for emitting light of a specific color when the imaging device captures an image, an analysis device equipped with a convolutional neural network and analyzing imaging data from the imaging device, and a control device for controlling the imaging device, the light-emitting device, and the analysis device, wherein the convolutional neural network has learned to be able to determine whether input data is imaging data of a real person's face using imaging data of a real person's face and imaging data of a photograph of the same person's face while the light-emitting device is emitting light, and the determination device images the human face while the light-emitting device is emitting light, inputs the imaging data with the area around the face extracted into the convolutional neural network, and determines whether the captured human face is a real person's face.

[0032] When the light emitting device is illuminated, there are differences in the luminance of the eyes, the nose, the face and the background between the image data of a real person's face in a particular color and the image data of a photograph of the same person's face when the light emitting device is illuminated. By comparing the differences in luminance of the image data with the image data in a color that is not illuminated, it is possible to determine whether the photographed face is the face of a real person. A convolutional neural network is trained using image data of a real person's face and image data of a photograph of the same person's face, making it possible to comprehensively determine whether or not a face is a real person's face, including differences in the eyes, nose, background, etc. However, to improve the accuracy of the judgment, it is necessary to appropriately select the width (number of neurons) of the convolutional neural network, the number of layers, the number of samples to be learned, etc. For example, it is desirable to have a width of 3x3 filters with 64 channels, the number of layers between 10 and 50, and the number of samples to be 1000 or more. In addition, publicly available trained models such as VGG, ResNet, or denseNet may be used through transfer learning to determine whether a captured human face is a real human face.

[0033] (10) A determination device according to a tenth aspect of the present invention is a determination device according to another aspect, comprising: a first convolutional neural network trained using imaging data captured in a state in which a light-emitting device is emitting light; and a second convolutional neural network trained using imaging data captured in a state in which the light-emitting device is not emitting light; The determination device may input imaging data captured when the light emitting device is emitting light to a first convolutional neural network, and input imaging data captured when the light emitting device is not emitting light to a second convolutional neural network, and determine whether or not the captured human face is the face of a real person based on the determination results of the first convolutional neural network and the determination results of the second convolutional neural network.

[0034] In this case, by combining the determination result with the imaging data obtained when the light emitting device is not emitting light, the following two effects are obtained, enabling more accurate determination. First, when capturing an image of an actual human being, image data captured with the light on contains reflections from the unevenness of the eyes and face, whereas image data captured with the light off contains no reflections from the unevenness of the eyes and face. On the other hand, when a photograph is taken with the light-emitting device on, the image data with the light on reflects not only the unevenness of the eyes and face but also the entire photograph, whereas the image data without the light on shows only reflections from the unevenness of the eyes and face. Due to the above difference, even when a photograph is taken with the light-emitting device on, it is possible to reliably determine whether the face of the person captured is the face of a real person. In addition, when a photograph is taken, there is an unnaturalness caused by taking a photograph in addition to the reflection caused by the unevenness of the eyes and face. The first convolutional neural network trained using image data captured in a light-emitting state naturally recognizes and judges this unnaturalness as a difference, but by further using the second convolutional neural network trained using image data captured in a light-emitting state, it is possible to more reliably judge whether the captured human face is the face of a real person.

[0035] (11) The determination device of an eleventh aspect of the present invention is the determination device of the tenth aspect of the present invention, wherein the analysis device further includes a third convolutional neural network trained using imaging data of a real person's face and imaging data of the same person's face displayed on a screen when the light-emitting device is emitting light, and a fourth convolutional neural network trained using imaging data of the real person's face and imaging data of the same person's face displayed on a screen when the light-emitting device is not emitting light, and the determination device inputs the imaging data when the light-emitting device is emitting light to the first and third convolutional neural networks, and inputs the imaging data when the light-emitting device is not emitting light to the second and fourth convolutional neural networks, and determines whether the captured human face is a real person's face based on the determination results of the first to fourth convolutional neural networks.

[0036] Recently, it has become necessary to distinguish between image data of a face displayed on a liquid crystal display or a mobile display (also called display attack data) and image data of a real person's face, instead of a printed photograph of a face. In this case, the accuracy of the discrimination can be improved by training a convolutional neural network using image data of a real person's face and image data of the same person's face displayed on a display. Then, based on the judgment results of the first and second convolutional neural networks trained using image data of a printed photograph of a face and the judgment results of the third and fourth convolutional neural networks trained using image data of the display, it is possible to comprehensively judge whether or not the face is a real person's face, even if the input data contains display attack data, and thereby a more accurate judgment can be made.

[0037] (12) The determination device of the 12th aspect of the present invention is the determination device of the 11th aspect of the present invention, further including an alarm device, and the control device may cause an imaging device to capture an image of a person's face before performing the determination, and cause the alarm device to notify the person of their standing position so that the size of the person's face is sufficient for making the determination.

[0038] In this case, the position of the person can be adjusted by the notification device. Also, the imaging device may recognize the face and automatically enlarge or reduce the size of the captured face to a predetermined size. The notification device may also be provided with a display device, which displays a captured image and a frame, and when the size and position of the face match the frame, the image may be captured automatically.

[0039] (13) The judgment device of the 13th invention is the judgment device of the 12th invention in one aspect, wherein the control device captures an image of the person's face using an imaging device before performing the judgment, and selects optimal light lacking a specific color from the captured imaging data and emits it from the light-emitting device.

[0040] In this case, the light emitted from the light emitting device can be optimized to lack a specific color. For example, when a person wears contact lenses of a specific color, the light can be emitted in a manner that does not reflect the color of the contact lenses of the specific color. As a result, the judgment can be made appropriately.

[0041] (14) A mobile terminal according to another aspect includes the determination device according to the first to thirteenth aspects of the present invention.

[0042] In this case, since the mobile terminal includes a determination device, it is possible to easily and reliably determine the presence of a person, thereby preventing unauthorized use of the mobile terminal.

[0043] (15) A program for a determination device relating to yet another aspect is a program for a determination device for determining a human face, and includes an imaging process for capturing an image of a human face once, an emission process for emitting light lacking a specific color when capturing an image in the imaging process, an analysis process for analyzing imaging data from the imaging process, and a control process for controlling the imaging process, the emission process, and the analysis process, and the control process decomposes the imaging data into components of each color by the analysis process, and determines whether the imaging data is imaging data of the face of a real person based on the difference between the component data of the color emitted by the emission process and the component data of the color not emitted by the emission process.

[0044] In this case, the face of a person can be reliably identified. In particular, since there is no need to capture images multiple times, it is possible to identify a large number of people in a short period of time. Moreover, being able to reliably determine a person's face means being able to determine whether it is the face of a person who actually exists, or the face of a person that has been photographed and printed. Also, light lacking a particular color is light lacking a particular color spectrum or frequency, and does not mean white light. [Brief description of the drawings]

[0045] [Figure 1] 1 is a schematic structural diagram illustrating an example of a configuration of a mobile terminal including a determination device according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of a configuration of a determination device. [Diagram 3] 10 is a flowchart showing an example of a control flow of a control unit. [Figure 4] 10 is a flowchart showing an example of a control flow of a control unit. [Diagram 5] 4 is a schematic diagram showing an example of face data captured by an imaging device and a light emitting device; FIG. [Figure 6] FIG. 6 is a schematic diagram showing an example of data obtained by analyzing the data in FIG. 5 in an analysis device. [Figure 7] FIG. 6 is a schematic diagram showing an example of data obtained by analyzing the data in FIG. 5 in an analysis device. [Figure 8] FIG. 6 is a schematic diagram showing an example of data obtained by analyzing the data in FIG. 5 in an analysis device. [Figure 9] FIG. 13 is a schematic diagram showing an example of a difference in data analyzed in an analysis device. [Figure 10] FIG. 13 is a schematic diagram showing an example of a difference in data analyzed in an analysis device. [Figure 11] The processes from step S3 to step S5 are carried out using data captured in steps S1 and S2 of a paper on which a facial photograph has been printed. [Figure 12] FIG. 11 is a diagram showing an example of extracting facial features in step S11. [Figure 13] This figure shows an example of data of only R, only G, and only B of facial features when a real person's face is photographed with flash light of RGB code #00FFFF shone on it. [Figure 14] FIG. 11 is a diagram showing an example of a result of the process in step S12. [Figure 15] FIG. 13 is a diagram showing an example of a case where Pearson's product-moment correlation coefficient is calculated. [Figure 16]FIG. 13 is a diagram showing an example of data obtained by capturing an image of a face photograph captured using flashlight, which is printed out (printed) on paper, and then capturing the image again in the processes of steps S1 and S2. [Figure 17] FIG. 13 is a diagram showing an example of data obtained by capturing an image of a face photograph captured using flashlight, which is printed out (printed) on paper, and then capturing the image again in the processes of steps S1 and S2. [Figure 18] FIG. 13 is a diagram showing an example of data obtained by capturing an image of a face photograph captured using flashlight, which is printed out (printed) on paper, and then capturing the image again in the processes of steps S1 and S2. [Figure 19] 10 is a flowchart showing another example of the control flow of the control unit. [Figure 20] 10 is a flowchart showing yet another example of the control flow of the control unit. [Figure 21] 3 is a schematic diagram showing another example of the determination device shown in FIG. 2. [Figure 22] 22 is a flowchart showing an example of a standing position instruction for a person's face when the face is imaged by the imaging device 200 using the determination device 100 shown in FIG. [Figure 23] 11 is a schematic diagram showing an example of a display of the notification device when instructing a standing position of a face. FIG. [Figure 24] 4 is a flowchart showing an example of selecting light lacking a specific color to be emitted from a light emitting device by capturing an image of a real person's face in advance, prior to the processing of step S1 shown in FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] [Embodiment Mode] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[0047] (Overall configuration of the mobile terminal 900 including the determination device 100) FIG. 1 is a schematic structural diagram showing an example of the overall configuration of a mobile terminal 900 including a determination device 100 according to this embodiment, and FIG.

[0048] As shown in FIG. 1, the mobile terminal 900 includes a determination device 100, a display screen 920, and an operation unit 930. The determination device 100 in FIG. 2 includes an imaging device 200, a light emitting device 300, an analysis device 400, and a control unit 500.

[0049] Next, the light emitted from the light emitting device 300 (hereinafter referred to as flash light) has an RGB color code of #00FFFF. That is, it is flash light from which at least the R light has been removed from the three primary colors R, G, and B. The reason for removing the R light in particular will be described later. In this embodiment, the flash light uses the RGB color code #00FFFF, but this is not limited to this, and the flash light may use any one of #FFFF00, #0000FF, and #00FF00, or a combination of two or more. Also, light lacking a specific color may be light in which, among RGB, G and B, or only G or only B, are dominant over R. Therefore, it includes a state in which no red (R) is contained at all, as well as a state in which a small amount of red (R) is contained.

[0050] 3 and 4 are flow charts showing an example of a control flow of the control unit 500. FIG. FIG. 5 is a schematic diagram showing an example of face data captured by the imaging device 200 and the light emitting device 300. As shown in FIG. Next, FIGS. 6, 7, and 8 are schematic diagrams showing an example of data obtained by analyzing the data of FIG. 5 within the analysis device 400, and FIGS. 9 and 10 are schematic diagrams showing an example of the difference between the data analyzed within the analysis device 400.

[0051] First, as shown in FIG. 3, the control unit 500 instructs the imaging device 200 and the light emitting device 300 to capture an image (step S1). At step S1 in this embodiment, the control unit 500 is not sure whether the data captured by the imaging device 200 is the face of an actual person or an image of a face printed on a piece of paper; however, the following explanation will be given assuming that the data is the face of an actual person. Furthermore, the control unit 500 instructs the light emitting device 300 to capture an image with a predetermined flash light during image capture (step S2). In addition, a system may be used in the user interface that automatically presses the shutter when the face is in a predetermined position, or a notification may be provided to encourage the user to move closer to the imaging device 200 when the face is not in the predetermined position. Here, the predetermined flash light according to this embodiment is the light with the color code #00FFFF, as explained above. Incidentally, the face of a real person may be recognized before imaging, and the predetermined flash light may be changed according to the color of the face, the color of the eyes, etc.

[0052] (First analysis) The control unit 500 passes the data captured in step S2 to the analysis device 400, and instructs the analysis device 400 to perform a first analysis (step S3). 5, the data captured by the light emitting device 300 using the color code #00FFFF for the flash light based on an instruction from the control unit 500 is data that is almost free of the color R. Note that since ambient light, such as light from a fluorescent lamp, may enter the image, a small amount of R (red) may be included even if the flash light does not contain R (red).

[0053] Next, the control unit 500 causes the analysis device 400 to divide the face data shown in FIG. 5 received from the imaging device 200 into data of only R, data of only G, and data of only B (step S4). FIG. 6 shows an example of data containing only R, FIG. 7 shows an example of data containing only G, and FIG. 8 shows an example of data containing only B.

[0054] Next, control unit 500 causes analysis device 400 to perform a difference calculation (step S5). Here, analysis device 400 subtracts only the data of R in Fig. 6 from only the data of B in Fig. 8 as shown in Fig. 9, and subtracts only the data of R in Fig. 6 from only the data of G in Fig. 9 as shown in Fig. 10.

[0055] 11 shows the data obtained by imaging a paper on which a face photograph has been printed in steps S1 and S2, and then performing the processes in steps S3 to S5 on the data. This data is called photo attack data.

[0056] Here, the control unit 500 checks the eye reflex and makes a judgment based on the data from the analysis device 400 (step S6). For example, in the case of the face of a real person, that is, when a person is definitely present and the processing of step S2 is performed, the data containing only G and only B will have a high brightness in the eye area, but the data containing only R will not have a high brightness in the eye area, so in the BR and GR data, only the area of ​​the eye where the flash light is present will have a high brightness, as shown in Figures 9 and 10. On the other hand, as shown in FIG. 11, in the case of a paper printout of a face photograph, the bright areas of the eyes disappear.

[0057] That is, in the case of the face of a real person, the reflection from the eyes is particularly strong, but in the case of a paper printout of a face photograph, in the processing of step S2, there is no difference in the light reflection between the eyes and other parts, so the bright parts of the eyes disappear.

[0058] From the above, the control unit 500 can easily determine whether the face data captured by the imaging device 200 is that of a real person or a paper on which a face photograph has been printed.

[0059] (Second analysis) Next, the second judgment will be explained. In the first judgment, papers on which face photographs were printed were excluded, but the inventor found that even among papers on which face photographs were printed, if the same flash light was used as in the original face photograph, there is a possibility that the bright part of the eyes remains. Therefore, a second judgment is performed.

[0060] First, as shown in FIG. 4, control unit 500 instructs analysis device 400 to perform a second analysis using the data used in step S6 (step S10). The control unit 500 extracts facial features from the data (step S11). For example, in this embodiment, the nose is extracted. In addition to the nose, when the cheeks are used as features, it is unlikely that the flash light will hit both cheeks evenly, so unevenness on both cheeks may be used as a feature. In the above embodiment, the facial portion of a person is extracted, but this is not limited to this. It is also possible to extract the outline of the face and then extract the area outside of the outline, that is, the background portion, and then perform the processing from step S12 to step S15 below.

[0061] FIG. 12 is a diagram showing an example of facial feature extraction in step S11, FIG. 13 is a diagram showing an example of R-only data, G-only data, and B-only data of facial features when a real person's face is photographed with flash light of RGB code #00FFFF shone on it, and FIG. 14 is a diagram showing an example of the results of the processing in step S12.

[0062] 12, the control unit 500 extracts the nose tip portion P, the left end portion L, and the right end portion R. The inventors have found that the nose tip portion P has an area that causes specular reflection. As shown in FIG. 13, the data containing only G and data containing only B in this embodiment have a mountain-shaped luminance distribution at the tip P of the nose, the left end L, and the right end R, whereas the data containing only R does not have a mountain-shaped luminance distribution because it is not included in the flash light and is generated by reflecting background light with a wide range of incident angles.

[0063] Next, the control unit 500 causes the analysis device 400 to subtract the minimum value of each luminance distribution from the data containing only R, only G, and only B, and to subtract the data containing only R from the data containing only G, or to subtract the data containing only R from the data containing only B (step S12). Here, the process of step S12 is a process for adjusting the average value of the data to 0, and a method for adjusting the average value of each data to 0 using a similar technique may be used.

[0064] As shown in FIG. 14, when the data of only R is subtracted from the data of only G resulting from the processing in step S12, a peak value appears.

[0065] Next, as shown in FIG. 4, the control unit 500 causes the analysis device 400 to detect a peak value and divide the signal into a first half of the peak value and a second half of the peak value (step S13). Finally, control unit 500 causes analysis device 400 to calculate the Pearson product-moment correlation coefficient between the two series of data, the first half data and the second half data folded back (step S14).

[0066] In this embodiment, the first half and the second half are divided based on the peak value, but this is not limiting, and the first half and the second half may be determined at any position based on the amount of data. For example, each may be set to 50% of the amount of data.

[0067] Figure 15 is a graph in which the average value of the GR in the graph in Figure 14 is calculated, the average value is subtracted from the data for each GR, and then the graph is divided into a first half and a second half with the center point or peak value as the boundary. The graph for the second half of the GR is formed by folding back the part to the right of the center point or peak value of the graph in Figure 14 in the opposite direction from the center position. In the case of Figure 15, there is a positive correlation between the first half and the second half of the GR, and the calculated Pearson product moment correlation coefficient is 0.82.

[0068] 16, 17 and 18 are diagrams showing examples of data obtained by capturing an image of a face photograph captured using flashlight, which is printed out (printed) on paper, and then capturing the image again in the processes of steps S1 and S2. FIG. 16 is a diagram showing an example of data of only R, only G, and only B of the facial feature part of data obtained by re-imaging a paper on which a face photo captured using a flashlight is printed out, and FIG. 17 is a diagram showing an example of the result of processing step S12 of the facial feature part of data obtained by re-imaging a paper on which a face photo captured using a flashlight is printed out. FIG. 18 is a graph obtained by calculating the average value of the GR of the graph in FIG. 17, subtracting the average value from the data of each GR, and dividing the graph into a first half and a second half with the center point as a boundary. The graph of the second half of GR is obtained by folding back the part to the right of the center of the graph in FIG. 17 in the opposite direction from the center position. In the case of FIG. 18, there is little correlation between the first half of GR and the second half of GR, and the Pearson product moment correlation coefficient is calculated to be 0.08.

[0069] As shown in FIG. 4, the control unit 500 causes the analysis device 400 to perform a determination (step S15). Here, when the Pearson product-moment correlation coefficient is 0.5 or more, the control unit 500 determines that the face data is of a real person. For example, the Pearson product-moment correlation coefficient in the case of Fig. 15 is 0.82. On the other hand, if the Pearson product-moment correlation coefficient is less than 0.5, the control unit 500 determines that the data is a re-image of a paper on which a face photograph captured using flashlight has been printed out (printed). For example, the Pearson product-moment correlation coefficient in the case of FIG. 18 is 0.08. Therefore, the Pearson product-moment correlation coefficient between the first half of the data and the second half of the data is calculated, and if the correlation coefficient is 0.5 or greater, it is determined that the captured data is data on the face of a real person, and if the correlation coefficient is less than 0.5, it is determined that the data is a re-image of a piece of paper on which a face photo has been printed.

[0070] [Another embodiment of the determination method] Next, another embodiment for determining whether image data is of the face of an actual person will be described. FIG. 19 is a schematic flow chart of another embodiment of the determination method.

[0071] First, the control unit 500 instructs the imaging device 200 and the light emitting device 300 to capture an image (step S20). Furthermore, the control unit 500 instructs the light emitting device 300 to capture an image with a predetermined flash light when capturing an image (step S21). The control unit 500 extracts an inner region and an outer region of the face outline from the data (step S22). Next, similarly to S12 in FIG. 4, a subtraction value, for example GR or BR, between component data of a color emitted by the light emitting device 300 and component data of a color not emitted by the light emitting device 300 is calculated (step S23). Before subtraction, it is desirable to perform correction so that the minimum value of the component data of the color emitted by the light emitting device 300 and the minimum value (or average value and average value) of the component data of the color not emitted by the light emitting device 300 match. Next, the subtraction values ​​are compared for the inside of the contour, i.e., the face part, and the outside of the contour, i.e., the background part (step S24). Note that, in comparing the face part and the background part, for example, the average brightness values ​​of the respective parts may be compared. Finally, the comparison result between the face part and the background part, for example the ratio or difference, is compared with a predetermined threshold value, and if the ratio or difference is greater than the predetermined threshold value, it is determined that the imaging data is imaging data of the face of a real person (step S25).

[0072] In the case of image data of a real person's face, the color components emitted by the light-emitting device 300 have high brightness in the face area (close to the light-emitting device 300) due to reflection from the face, and low brightness in the background area (outside the face). In contrast, there is little difference between the face area and the background area in the color components not emitted by the light-emitting device 300. Therefore, the difference between the component data of the color emitted by the light-emitting device 300 and the component data of the color not emitted by the light-emitting device 300 is larger in the face area than in the background area. On the other hand, in the case of imaging data such as a photograph that does not show the face of an actual person, the light of the light emitting device 300 is also reflected in the background, so there is little difference between the face and the background in both the color components emitted by the light emitting device 300 and the color components not emitted by the light emitting device 300. Therefore, the magnitude of the difference between the component data of the color emitted by the light emitting device 300 and the color components not emitted by the light emitting device 300 is small between the face and the background. As described above, by comparing the size of the human face part and the size of the background part of the difference between the component data of the color emitted by the light-emitting device 300 and the component data of the color not emitted by the light-emitting device 300, it is possible to determine whether the imaging data is imaging data of the face of a real person.

[0073] [Another embodiment of the determination method] In yet another embodiment of the determination method, machine learning is used to determine whether the image data is image data of a real person's face. Specifically, first, image data of a real person's face and image data of a photograph of the same person's face (also called photo attack data) are input to a convolutional neural network and trained (hereinafter, the trained network is called a photo training network). Thereafter, imaging data of a face captured by the imaging device 200 is input to the photograph learning network, and the network determines whether the input data is imaging data of the face of an actual person or imaging data of a photograph of the face.

[0074] FIG. 20 is a schematic flow chart of yet another embodiment of the determination method. First, a person's face when the light emitting device 300 emits cyan light (hereinafter referred to as cyan data) and a person's face when the light emitting device 300 does not emit light at all (hereinafter referred to as black data) are captured (steps S30, S30'). Next, the facial contour is extracted using the facial contour and characteristic parts such as the eyes and lips (steps S32, S32'). Furthermore, the cyan data and black data around the face are each resized to, for example, 224 × 224. Each pixel value of the resized data may be normalized to a value between 0 and 1 (steps S33, S33'). Note that the same preprocessing such as extraction of the face outline, resizing and normalization of the data around the face must also be performed on the learning data. A convolutional neural network trained using cyan data as input data (hereinafter referred to as the cyan photo learning network) is input with cyan data of a face captured by the imaging device 200 and made to make a judgment, and a convolutional neural network trained using black data (hereinafter referred to as the black photo learning network) is input with black data of a face captured by the imaging device 200 and made to make a judgment (steps S34, S34').

[0075] It is determined whether the imaging data is a real person based on the determination result of the cyan data and the determination result of the black data. When the determination result of the cyan data and the determination result of the black data are both a real person, it may be determined that the imaging data is a real person. Alternatively, when the determination result of either the cyan data or the black data is a real person, it may be determined that the imaging data is a real person. Furthermore, when each determination result is output as an intermediate value rather than 0 or 1, it may be determined by adding up the determination results (step S35).

[0076] In this embodiment, whether the image data is of a real person is determined based on the determination results of the cyan data and the black data, but whether the image data is of a real person may be determined based only on the determination result of the cyan data. Also, the light emission color of the light emitting device 300 may be magenta, yellow, red, blue, or green instead of cyan.

[0077] The purpose of using the judgment result of the black data is as follows. First, when an actual person is photographed, the cyan data shows reflections due to the unevenness of the eyes and face, whereas the black data shows no reflections due to the unevenness of the eyes and face. On the other hand, when a photograph is taken with the light-emitting device 300 on, the cyan data shows that not only the unevenness of the eyes and face but also the entire photograph reflects the light of the light-emitting device 300, whereas the black data shows only reflections due to the unevenness of the eyes and face. From these differences, it is possible to reliably determine whether the face of the photographed person is that of a real person even when the photograph is taken with the light-emitting device 300 on. In addition, when a photograph is taken, there is an unnaturalness caused by taking a photograph in addition to the reflection caused by the unevenness of the eyes and face. The first convolutional neural network trained using cyan data naturally recognizes and judges this unnaturalness as a difference, but by using the second convolutional neural network trained using black data, it is possible to more reliably judge whether the captured human face is the face of a real person.

[0078] Recently, it has also become necessary to distinguish between image data of a face displayed on a liquid crystal display or the like (also called display attack data) and image data of a real person's face, instead of a printed photograph of the face. In this case as well, first, multiple pieces of cyan data of a real person's face and cyan data of the face of the same person displayed on a display are input to the third convolutional neural network to learn (cyan display learning network). Next, multiple pieces of black data of a real person's face and black data of the face of the same person displayed on a display are input to the fourth convolutional neural network to learn (black display learning network). After that, the cyan data of a face captured by the imaging device 200 is input to the cyan display learning network, and the black data of a face captured by the imaging device 200 is input to the black display learning network, so that it can be determined whether the input data is image data of a real person's face.

[0079] The input to the determination device 100 may include image data of real people's faces, image data of printed photographs of faces, and also image data of faces displayed on a liquid crystal display, etc. In order to reliably determine image data of real people's faces from among these inputs, the determination is made according to the following procedure. First, the cyan data and black data captured by the imaging device 200 are input to a cyan photo learning network and a black photo learning network, respectively. Next, the cyan data and black data captured by the imaging device 200 are input to a cyan display learning network and a black display learning network, respectively. Furthermore, the outputs of the above four learning networks are set to 1 when it is determined that the face is that of a real person, and 0 when it is determined that the face is not that of a real person, and if the sum of the outputs of the four learning networks is equal to or greater than a predetermined value, it is determined that the face is that of a real person. Also, if each determination result is output as an intermediate value rather than 0 or 1, the determination may be made by adding up the results of these determinations.

[0080] (Configuration of convolutional neural network) The width (number of neurons) and number of layers of the convolutional neural network can be configured in various ways. For example, if the image data is resized to 224 x 224 and divided into RGB, the number of input signals will be 224 x 224 x 3. There are two output signals: one that is 1 when it is determined to be the face of a real person and 0 when it is determined not to be the face of a real person, and the other that is the opposite. The width and number of hidden layers between them can be selected based on the required determination accuracy and the allowable circuit scale. Recently, a new type of convolutional neural network called ResNet (Deep Residual Network) has been published as a learning model, and in this embodiment, for example, an 18-layer ResNet (ResNet-18) may be used by transfer learning to determine whether or not a captured human face is a real human face. In addition, other published trained models such as VGG and denseNet may be used by transfer learning to determine whether or not a captured human face is a real human face.

[0081] In this embodiment, the control unit 500 performs all judgments, but this is not limited to the above, and the analysis device 400 may have a separate control unit and perform the judgments.

[0082] [Embodiment of the notification device 600] FIG. 21 is a schematic diagram showing another example of the determination device 100 shown in FIG. The determination device 100 shown in Fig. 21 further includes an alarm device 600 in addition to the determination device 100 in Fig. 2. Here, the alarm device 600 may have a sound generating device, may have a display device, may include both of them, or may include any other alarm device 600.

[0083] FIG. 22 is a flow chart showing an example of a standing position instruction for a person's face when the person's face is photographed by the imaging device 200 using the determination device 100 shown in FIG.

[0084] 22, the control unit 500 instructs the imaging device 200 to capture an image (step S41). The imaging device 200 captures an image of a face of a real person in advance. In this case, the imaging device 200 may capture the face of the real person continuously, may capture a moving image, or may capture the face of the real person intermittently multiple times.

[0085] 22, the control unit 500 determines whether the face is within a predetermined frame (step S42). If the face is not within the predetermined frame, the notification device 600 notifies the photographed person to change his / her position (step S43). 22, the control unit 500 judges whether the size of the face captured by the imaging device 200 is an optimal value (step S44). In the present embodiment, it is only judged whether the size of the face is an optimal value, and does not judge whether the face is real or a photo attack. That is, when the determination result indicates that the face size is smaller than the optimum value (No 1 in step S44), the control unit 500 causes the notification device 600 to notify the photographed person to approach the imaging device 200 (step S45). Fig. 23 shows an example of a display on the display device of notification device 600. In Fig. 23, reference numeral 610 denotes the screen of the display device, 620 denotes a frame indicating a desirable face size, and 630 denotes an example of a notification from notification device 600.

[0086] In this step S45, the alarm device 600 may be used to have a voice generating device of the alarm device 600 say, "Please bring your face closer to the imaging device 200," so that the captured face becomes the optimal size, or the display device of the alarm device 600 may say, "Please move your face closer so that it fits into the frame displayed on the display unit," or both.

[0087] On the other hand, if the determination result indicates that the face size is larger than the optimum value (No 2 in step S44), the control unit 500 causes the notification device 600 to notify the photographed person to move away from the imaging device 200 (step S46).

[0088] In this step S46, the alarm device 600 may be used to have a voice generating device of the alarm device 600 say, "Please move your face away from the imaging device 200," or a display device of the alarm device 600 say, "Please move your face away so that it fits within the size of the circle displayed on the display unit," or both, so that the captured face becomes the optimal size.

[0089] Finally, if the control unit 500 determines that the face size is the optimal value (Yes in step S44), the control unit 500 may proceed to the process of step S1 shown in FIG. 3 (step S47). Also, a captured image and a frame 620 may be displayed on a screen 610 of the display device, and when the size and position of the face match the frame 620, an image may be captured automatically.

[0090] [Embodiment of the selection of a specific colorless light of the light emitting device] FIG. 24 is a flow chart showing an example of selecting light lacking a specific color to be emitted from the light emitting device 300 by capturing an image of a real person's face in advance, prior to the processing of step S1 shown in FIG. That is, FIG. 24 shows a process of selecting a specific color-depleted light to be emitted from the light emitting device 300 before one image capture by the imaging device 200.

[0091] 24, the control unit 500 instructs the imaging device 200 to capture an image of a real-life subject (step S51). In accordance with the instruction from the control unit 500, the imaging device 200 captures an image of a real-life face. In this case, the image capturing device 200 may capture the face of a real person continuously, may capture a moving image, or may capture the face intermittently multiple times.

[0092] Next, the imaging device 200 transmits the imaging data to the control unit 500. The control unit 500 receives the imaging data (step S52). The control unit 500 extracts the pupil of the person from the imaging data (step S53).The control unit 500 selects the color of the pupil of the person from the imaging data (step S54).

[0093] Next, the control unit 500 selects a missing light of a specific color to be emitted according to the color of the pupil portion (step S55). Thereafter, the control unit 500 performs the process of step S2 in FIG.

[0094] Here, the reason for selecting the light lacking a specific color to be emitted according to the color of the pupil is to select the light lacking a specific color in consideration of the fact that some users wear colored contact lenses. For example, for a user wearing colored contact lenses of #0000FF, it is easy to determine whether the user is a real person by using the light of #FFFF00, the light of #FF8800, and the light of #88FF00 as the light lacking a specific color. Note that the color of the pupil is not limited to colored contact lenses, and varies depending on race, sex, and place of birth, so it is preferable to carry out the process of FIG.

[0095] As described above, by using the portable terminal 900 and the analysis device 400 and the control unit 500 included in the determination device 100 according to the present invention, the face of a person that actually exists can be determined reliably. In particular, since there is no need to perform multiple image captures, it is possible to perform determination of a large number of people in a short period of time. Furthermore, since data that is not a real human face can be excluded in the first determination, processing speed can be increased.

[0096] Security can be further improved by irradiating the image with various types of flash light to make it unclear to the imaged person which data is being used by the mobile terminal 900 and the determination device 100.

[0097] In this embodiment, the determination device 100 corresponds to the "determination device", the imaging device 200 corresponds to the "imaging device", the flash light corresponds to the "colorless light emission", the light emitting device 300 corresponds to the "light emitting device", the analysis device 400 corresponds to the "analysis device", the control unit 500 corresponds to the "control device", the alarm device 600 corresponds to the "alarm device", the BR data and the GR data correspond to the "difference in data", and the mobile terminal 900 corresponds to the "mobile terminal".

[0098] Although a preferred embodiment of the present invention is as described above, the present invention is not limited thereto. It is understood that various other embodiments can be made without departing from the spirit and scope of the present invention. Furthermore, although the actions and effects of the configuration of the present invention are described in the present embodiment, these actions and effects are merely examples and do not limit the present invention. [Explanation of symbols]

[0099] 100 Judgment device 200 Imaging device 300 Light emitting device 400 Analyzer 500 Control section 600 Notification device 900 Mobile Devices

Claims

1. A determination device for determining a human face, an imaging device that captures an image of the person's face once; a light emitting device that emits light lacking a specific color when the imaging device captures an image; an analysis device for analyzing imaging data from the imaging device; a control device that controls the imaging device, the light emitting device, and the analysis device; The control device is a determination device that uses the analysis device to decompose the imaging data into color components, and determines whether the imaging data is imaging data of the face of a real person based on the difference between the component data of the color emitted by the light-emitting device and the component data of the color not emitted by the light-emitting device.

2. The determination device according to claim 1 , wherein the control device determines whether the imaging data is imaging data of a face of an actual person at a specific portion of the face of the person.

3. The determination device according to claim 2 , wherein the specific part is at least one of the eyes, the nose, and the cheeks.

4. The determination device according to claim 1 , wherein the control device instructs the light emitting device to emit light lacking the specific color a plurality of times, or to emit light lacking a plurality of different colors from each other.

5. 5. The determination device according to claim 1, wherein the light lacking the specific color is any one of light lacking one color of RGB, light lacking one color of the XYZ color system, and light lacking one color of the Munsell color system.

6. 6. The determination device according to claim 1, wherein the determination device determines that the imaging data is imaging data of a face of a real person when the brightness of an eye area is greater than the brightness of a surrounding area based on data obtained by subtracting component data of a color not emitted by the light-emitting device from component data of a color emitted by the light-emitting device, among the data decomposed into each color component.

7. The determination device includes the following steps: 1) subtracting, from the data separated into the color components, component data of a color that is not emitted by the light-emitting device from component data of a color that is emitted by the light-emitting device; 2) determining a peak value of the subtracted data; 3) dividing the subtracted data into first half data and second half data with the peak value as a boundary; 4) reversing the order of the latter half of the data so that the boundary comes last; 5) calculating the Pearson product-moment correlation coefficient between the first half data and the second half data whose order has been reversed; 6) determining that the imaging data is imaging data of a face of a real person if the Pearson's product-moment correlation coefficient is 0.5 or more, thereby determining whether the imaging data is imaging data of a face of a real person.

8. 2. The determination device according to claim 1, wherein the determination device calculates a difference between component data of a color emitted by the light-emitting device and component data of a color not emitted by the light-emitting device for the person's face and a background, and compares a magnitude of the difference for the person's face with a magnitude of the difference for the background to determine whether the imaging data is imaging data of a face of a real person.

9. Further comprising an alarm device, The control device is configured to capture an image of a person's face using the imaging device before performing the judgment, and to have the notification device notify the person of their standing position so that the size of the person's face is such that the judgment can be performed.

10. The control device is configured to capture an image of a person's face using the imaging device before performing the determination, and to select optimal light lacking the specific color from the captured imaging data and emit the light from the light-emitting device.

11. A mobile terminal comprising the determination device according to any one of claims 1 to 10.

12. A program for a determination device for determining a human face, an imaging process for imaging the face of the person once; A light emission process that emits light lacking a specific color when the imaging process captures an image; an analysis process for analyzing imaging data from the imaging process; a control process for controlling the imaging process, the light emission process, and the analysis process; The control process separates the imaging data into components of each color by the analysis process, and controls the imaging data according to a difference between component data of a color for which the light emission process emits light and component data of a color for which the light emission process does not emit light. A program for a determination device that performs processing to determine whether image data is image data of the face of a real person.

Citation Information

Patent Citations

  • Device and method for authenticating biological information

    JP2000215308A

  • Personal authentication system for mobile device and its method

    JP2002112340A

  • Portable telephone with personal authenticating function set

    JP2004080080A

  • Mobile phone

    JP2004260240A

  • Face authentication device and face authentication method

    JP2007011456A