Face authentication device, face authentication method, and program

Abstract

To facilitate miniaturization of a device for face authentication using information on a depth direction.MEANS FOR SOLVING THE PROBLEM: A face authentication device 1 includes: a monocular camera 10 that has an encoding opening, and picks up an image of the face of a user through the encoding opening to acquire a picked-up image showing the face of the user; a depth estimation unit 206 that estimates the depth in at least part of the area of the picked-up image through calculation according to the encoding opening; a depth feature information generation unit 208 that generates depth feature information indicating the feature in a depth direction of the face of the user on the basis of the depth; and a depth face authentication unit 210 that authenticates the user on the basis of the depth feature information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a face recognition device, a face recognition method, and a program. [Background technology]

[0002] Conventionally, a technology is known that improves the reliability of authentication by performing face authentication using depth information in addition to two-dimensional face authentication using two-dimensional information acquired by a monocular camera. For example, Patent Document 1 describes a face authentication technology using depth information acquired by a stereoscopic imaging method. For example, Patent Document 2 describes a face authentication technology using depth information acquired by a distance measurement method using an infrared laser. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-123216 A [Patent Document 2] JP 2013-250856 A [Non-patent literature]

[0004] [Non-Patent Document 1] A. Levin, et al, “Image and depth from a conventional camera with a coded aperture”, ACM Transactions on Graphics, Vol. 26, No. 3, Airticle70, 2007 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technologies of Patent Documents 1 and 2 require the introduction of additional devices such as a stereo camera, an infrared laser emitter, and a light receiver for face recognition using depth information, which takes up space.

[0006] The present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide a face recognition device, a face recognition method, and a program that can facilitate miniaturization of a device for face recognition using depth information. [Means for solving the problem]

[0007] In order to solve the above problem, a face recognition device according to one aspect of the present invention includes a monocular camera having a coded aperture and acquiring an image showing a user's face by capturing an image of the user's face through the coded aperture, a depth estimation unit that estimates a depth within at least a portion of the captured image by calculation according to the coded aperture, a depth feature information creation unit that creates depth feature information indicating depth characteristics of the user's face based on the depth, and a depth face authentication unit that authenticates the user based on the depth feature information.

[0008] A face authentication method according to one aspect of the present invention includes an image acquisition step of acquiring an image showing a user's face by capturing an image of the user's face through a coded aperture using a monocular camera having the coded aperture, a depth estimation step of estimating a depth within at least a portion of the captured image by calculation according to the coded aperture, a depth feature information creation step of creating depth feature information indicating depth characteristics of the user's face based on the depth, and a depth face authentication step of authenticating the user based on the depth feature information.

[0009] A program according to one aspect of the present invention causes a computer to function as an image acquisition means for acquiring an image showing a user's face by imaging the user's face through a coded aperture using a monocular camera, a depth estimation means for estimating a depth within at least a portion of the captured image by calculation according to the coded aperture, a depth feature information creation means for creating depth feature information indicating depth characteristics of the user's face based on the depth, and a depth face authentication means for authenticating the user based on the depth feature information.

[0010] In addition, according to one form of the present invention, the face authentication device further includes a part area recognition unit that recognizes a part area representing a predetermined part of the user's face based on the captured image, and the depth estimation unit estimates the depth within the part area.

[0011] According to one embodiment of the present invention, the part area recognition unit recognizes a plurality of part areas each representing a predetermined part of the user's face based on the captured image, the depth estimation unit estimates the depth within each of the plurality of part areas, and the depth feature information creation unit creates the relative depth of the plurality of parts based on the depth within each of the plurality of part areas as the depth feature information.

[0012] According to an aspect of the present invention, the face recognition device further includes a two-dimensional face recognition unit that performs two-dimensional face recognition based on the captured image.

[0013] In addition, according to one form of the present invention, the face recognition device further includes a deblurring processing unit that performs a deblurring process on the captured image by calculation according to the coded aperture to create a deblurred image, and the part area recognition unit recognizes the part area based on the deblurred image.

[0014] Furthermore, according to one aspect of the present invention, the two-dimensional face authentication unit performs the two-dimensional face authentication based on the image that has been subjected to the blur removal process.

[0015] Furthermore, according to one aspect of the present invention, the depth feature information creation unit calculates an average value of the depth for each of the plurality of part regions, and creates the depth feature information based on the average value. Effect of the Invention

[0016] According to the present invention, it is possible to easily reduce the size of a device for face authentication using information in the depth direction. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of a face authentication device. [Diagram 2] FIG. 11 is a diagram showing an example of a flow of data processing related to face authentication. [Diagram 3] FIG. 11 is a diagram illustrating an example of a flow of creating depth feature information. [Figure 4] FIG. 11 is a flow diagram illustrating an example of a process of face depth authentication. [Diagram 5] FIG. 1 is a diagram illustrating an example of a hardware configuration for implementing a face authentication device. [Figure 6] FIG. 2 is a block diagram showing an example of functions realized by the face recognition device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] [1. Overall configuration of face recognition device] The face recognition device according to this embodiment is a face recognition device that uses information in the depth direction obtained by performing depth estimation using coded imaging technology.

[0020] Fig. 1 is a diagram showing an example of the overall configuration of a face recognition device. As shown in Fig. 1, the face recognition device 1 includes a monocular camera 10 and a control device 20. The monocular camera 10 has a coded aperture 11, a lens 13, and an image sensor 15.

[0021] The coded aperture 11 has an opening region and a light blocking region showing a geometric pattern, and passes a part of the light L incident on the lens 13 while blocking the other part of the light L. The lens 13 collects the light L arriving from the subject and forms an image on the light receiving surface 15a of the image sensor 15. The image sensor 15 is an electronic component that performs photoelectric conversion, and photoelectrically converts the brightness of the image formed on the light receiving surface 15a into an amount of electric charge, and captures the photoelectrically converted electric signal to obtain a captured image.

[0022] The monocular camera 10 captures an image showing the face of the user U by capturing an image of the face of the user U through the coded aperture 11. The image is captured by focusing on a part of the face, but blurring due to defocusing occurs in the captured image in other parts that are not in focus. For example, when capturing an image by focusing on the position of the user U's eyes, no blurring occurs at the focused eye position, but blurring occurs at the position of the nose. The magnitude B of the blurring differs depending on the depth D, which is the distance from the subject to the lens 13.

[0023] The control device 20 is connected to the imaging element 15. The control device 20 performs various processes on the captured image obtained from the imaging element 15. The control device 20 is, for example, a computer.

[0024] [2. Contents of the embodiment] In this embodiment, two-dimensional face recognition and depth face recognition, which is face recognition in the depth direction, are performed from one captured image.

[0025] FIG. 2 is a diagram showing an example of the flow of data processing related to face authentication. A captured image 104, which is acquired by a monocular camera 10 having a coded aperture 11 and shows the face of a user U to be authenticated, is subjected to a blur removal process by calculation according to the coded aperture 11, and a blur removed image 106 is created. Next, two-dimensional feature information 108 showing the features of the face of the user U on a two-dimensional image is created based on the blur removed image 106. Then, two-dimensional face authentication 100 is performed by matching the two-dimensional feature information 108 with two-dimensional user information 110 registered in advance as registration data used in two-dimensional face authentication 100. In addition, a part area showing the parts of the face of the user U is also recognized based on the blur removed image 106, and part area recognition information 112 is created. Next, a depth D is estimated by calculation according to the coded aperture 11 based on the captured image 104 and the part area recognition information 112, and depth estimation information 114 is created. Next, depth feature information 118 indicating the features of the face of the user U in the depth direction is created based on the depth estimation information 114. Then, the depth face authentication 102 is performed by comparing the depth feature information 118 with depth user information 120 registered in advance as registration data used in the depth face authentication 102, and the user U is authenticated.

[0026] As described above, according to this embodiment, it is possible to estimate the depth D from the captured image 104 acquired by the monocular camera 10 having the coded aperture 11, and there is no need to introduce additional devices such as a stereo camera, an infrared laser irradiation device, and a light receiving device, making it easy to miniaturize the device for face recognition using depth information.

[0027] Furthermore, according to this embodiment, both the two-dimensional face authentication 100 and the depth face authentication 102 can be performed from a single captured image 104, making it easy to reduce the time required to acquire data for authenticating the user U.

[0028] The process for the depth face authentication 102 may be performed only when authentication is performed by the two-dimensional face authentication 100. For example, the two-dimensional face authentication 100 is performed first, and when authentication is performed by the two-dimensional face authentication 100, the process proceeds to the process for the depth face authentication 102, such as creating part area recognition information 112 by recognizing the part area, and then creating depth estimation information 114 by estimating the depth, but when authentication is not performed by the two-dimensional face authentication 100, the process for the depth face authentication 102 does not proceed, and authentication fails, and the result is output.

[0029] As described above, according to this embodiment, by performing the process for the depth face authentication 102 only when authentication is performed by the two-dimensional face authentication 100, it is possible to easily reduce the load of computational processing on the computer.

[0030] The captured image 104 is an image showing the face of a user U to be authenticated. The captured image 104 is an image acquired by a monocular camera 10 having a coded aperture 11, capturing an image of the face of the user U through the coded aperture 11. The captured image 104 is composed of a plurality of pixels.

[0031] The deblurred image 106 is an image created by performing deblurring of the captured image 104 through a calculation according to the coded aperture 11. The calculation according to the coded aperture 11 is performed using a well-known technique used in coded imaging technology. For example, it may be performed using the technique described in Non-Patent Document 1. The deblurred image 106 does not need to be an image from which blur has been completely removed, but may be an image from which blur has been reduced.

[0032] The two-dimensional feature information 108 indicates the facial features of the user U on a two-dimensional image, and is information used as matching data for the two-dimensional face authentication 100. The two-dimensional feature information 108 is created, for example, based on the deblurred image 106. For example, the two-dimensional feature information 108 is created based on pixel coordinates that indicate the positions of the user U's facial parts, such as the eyes, nose, mouth, chin, eyebrows, and cheeks, in the image, from the deblurred image 106. Note that the format of the two-dimensional feature information 108 illustrated in FIG. 2 is an example, and is not limited to this, and other formats generally used in two-dimensional face authentication technology may be used.

[0033] The two-dimensional user information 110 indicates the facial features of the user U on a two-dimensional image, and is information that is registered in advance and used as registration data for the two-dimensional face authentication 100. The two-dimensional user information 110 is data in the same format as the two-dimensional feature information 108.

[0034] The two-dimensional face authentication 100 is performed using a known technology as an authentication method that does not use information in the depth direction of an image. For example, the two-dimensional face authentication 100 may be performed using machine learning. In this embodiment, the two-dimensional face authentication 100 is performed based on a captured image 104. More specifically, the two-dimensional face authentication 100 may be performed based on a deblurred image 106 created by performing a deblurring process on the captured image 104. For example, two-dimensional feature information 108 may be created based on the deblurred image 106, and the two-dimensional face authentication 100 may be performed by matching it with two-dimensional user information 110.

[0035] As described above, according to this embodiment, by performing two-dimensional face authentication 100 based on the deblurred image 106 created by performing deblurring processing on the captured image 104, it is possible to easily improve authentication accuracy.

[0036] The part area recognition information 112 is information indicating the result of recognition of a part area expressing a predetermined part of the face of the user U based on the captured image 104. There may be a plurality of predetermined parts of the face of the user U. For example, when a plurality of parts are predetermined as parts of the face, such as the eyes, nose, mouth, and chin, the part area recognition information may be information indicating the result of recognition of the areas expressing the eyes, nose, mouth, and chin. The area is, for example, a rectangle surrounding a part of the face. For example, in the part area recognition information 112 illustrated in FIG. 2, X1, Y1-X2, Y2 of the area expressing the eyes indicate the pixel coordinates of the diagonal corners of the rectangular area. Note that the format of the part area recognition information 112 illustrated in FIG. 2 is an example, and is not limited to this, and other formats generally used in image recognition technology may be used. The part area may be recognized using a well-known technology in image recognition technology. For example, the part area may be recognized using machine learning. Furthermore, part region recognition information 112 may be information indicating the result of recognizing a part region based on deblurred image 106 created by performing deblurring on captured image 104. In this case, the part region is recognized by performing a predetermined calculation on deblurred image 106 using, for example, a trained model.

[0037] As described above, according to this embodiment, by recognizing a part area based on the deblurred image 106 created by performing deblurring processing on the captured image 104, it is possible to easily improve the recognition accuracy.

[0038] The depth estimation information 114 is information indicating the result of estimating the depth D in at least a part of the region of the captured image 104 by calculation according to the coded aperture 11. More specifically, the depth estimation information 114 may be information indicating the result of estimating the depth D in the part region recognized in the part region recognition process in the previous process. When there are multiple part regions recognized in the part region process in the previous process, the depth D in each of the multiple part regions is estimated. The calculation according to the coded aperture 11 is performed using a well-known technique used in coded imaging technology. For example, it may be performed using the technique described in Non-Patent Document 1. The format of the depth estimation information 114 is, for example, the format illustrated in FIG. 2. In the depth estimation information 114 illustrated in FIG. 2, X1, Y1, D1 of the region representing the eye indicates that the estimation result of the depth D in the pixel at the pixel coordinates of X1, Y1 is D1. Note that the format of the depth estimation information 114 illustrated in FIG. 2 is an example and is not limited thereto. Furthermore, the calculation according to the coded aperture 11 may be performed at a predetermined interval, for example, every 20 pixels vertically and every 20 pixels horizontally, etc. Furthermore, the calculation according to the coded aperture 11 may be performed not on the entire captured image 104, but only on a portion corresponding to a plurality of part regions recognized in the part region recognition process in the previous step.

[0039] As described above, according to this embodiment, by performing depth estimation processing only on recognized part regions, it is possible to easily reduce the computational processing load on a computer.

[0040] The depth feature information 118 is information used as matching data for the depth face authentication 102. The depth feature information 118 is information indicating the features in the depth direction of the face of the user U, which is created based on the depth D estimated by the depth estimation in the previous process. More specifically, the depth feature information 118 may be information on the relative depth of a plurality of parts based on the depth D in each of the plurality of part regions estimated by the depth estimation in the previous process, with a predetermined part among the plurality of parts as a reference. Furthermore, the depth feature information 118 may be information created based on an average value calculated for each of the plurality of parts of the depth D in each of the plurality of part regions estimated by the depth estimation in the previous process. As illustrated in FIG. 3, the average value of the depth D is calculated for each of the eyes, nose, mouth, and chin from the depth estimation information 114 indicating the depth D in each of the areas indicating the eyes, nose, mouth, and chin, to create the average depth information 116. Next, for example, the eyes (30.0 cm) are used as a reference and the difference between the eyes and other parts other than the eyes, such as the nose (28.0 cm), mouth (28.7 cm), and chin (29.0 cm), is calculated as a relative depth. In this way, the relative depth of each part, such as the nose (2.0 cm), mouth (1.3 cm), and chin (1.0 cm), using the eyes as a reference is created as depth feature information 118.

[0041] As described above, according to this embodiment, by creating depth feature information 118 based on the average value of the depth D for each part, it is possible to reduce the influence of errors and easily improve the reliability of face authentication.

[0042] The depth user information 120 indicates the characteristics of the face of the user U in the depth direction, and is information that is registered in advance and used as registration data used in the depth face authentication 102. The depth user information 120 is data in the same format as the depth characteristic information 118.

[0043] The depth face authentication 102 authenticates the user U based on the depth feature information 118. For example, the depth face authentication 102 is performed by matching the depth feature information 118 with the depth user information 120 registered in advance in the user terminal. As illustrated in FIG. 4, the depth face authentication 102 may sequentially match and determine each part. For example, first, a nose match (S1) is performed, and it is determined whether the difference between the nose value in the depth feature information 118 and the nose value in the depth user information 120 is within a threshold value (S2). If it is determined in S2 that it is within the threshold value, a determination result that the user U is not the user U is output (S3). If it is determined in S2 that it is within the threshold value, the process proceeds to a mouth match (S4), and similarly, it is determined whether the difference between the mouth value in the depth feature information 118 and the mouth value in the depth user information 120 is within a threshold value (S5). If it is determined in S5 that it is within the threshold value, a determination result that the user U is not the user U is output (S6). If it is determined in S5 that the value is within the threshold, then the chin is matched (S7), and it is similarly determined whether the difference between the chin value in the depth feature information 118 and the chin value in the depth user information 120 is within the threshold (S8). If it is not determined in S8 that the value is within the threshold, a determination result that the user is not the user U is output (S9). If it is determined in S8 that the value is within the threshold, a determination result that the user is the user U is output (S10).

[0044] As described above, according to this embodiment, by sequentially performing the judgment for each part, it is possible to easily reduce the load of the calculation processing on the computer.

[0045] [3. Hardware configuration for implementing face recognition device] A hardware configuration for implementing the face authentication device 1 according to this embodiment will be described with reference to Fig. 5. In this embodiment, a case will be described in which the face authentication device 1 is implemented in a terminal such as a smartphone.

[0046] Fig. 5 is a diagram showing an example of a hardware configuration for realizing the face authentication device 1. As shown in Fig. 5, the control device 20 is equipped with a CPU 21, a memory 22, a display 23, and a touch panel 24.

[0047] The CPU 21 includes at least one processor. The CPU 21 is a type of circuitry. The memory 22 includes storage such as a RAM (Random Access Memory) and a UFS (Universal Flash Storage), and stores programs and data. The CPU 21 executes various processes based on these programs and data. The display 23 is a liquid crystal display, an organic EL display, or the like, and displays an operation screen or the like in response to an instruction from the CPU 21. The touch panel 24 is provided on the display surface of the display 23, and detects touch operations on the surface of the touch panel 24 by a touch sensor of a capacitance type, a resistive film type, or the like.

[0048] The programs and data described as being stored in the memory 22 may be supplied to the control device 20 via a network. The hardware configuration of the control device 20 is not limited to the above example, and various hardware configurations are applicable. For example, the control device 20 may include a reading unit (e.g., an optical disk drive or a memory card slot) that reads a computer-readable information storage medium, or an input / output unit (e.g., a USB terminal) for directly connecting to an external device. In this case, the programs and data stored in the information storage medium may be supplied to the control device 20 via the reading unit or the input / output unit.

[0049] [4. Functions realized by face recognition device] The functional configuration of the control device 20 will be described with reference to FIG.

[0050] Fig. 6 is a block diagram showing an example of functions realized by the face recognition device 1. As shown in Fig. 6, the control device 20 includes a blur removal processing unit 200, a two-dimensional face recognition unit 202, a body part area recognition unit 204, a depth estimation unit 206, a depth feature information creation unit 208, and a depth face recognition unit 210. These functions operate according to a program stored in the memory 22.

[0051] As described in the example of the flow of data processing related to face recognition in this embodiment, for example, the blur removal processing unit 200 performs blur removal processing on the captured image 104 by calculation according to the coded aperture 11 to create a blur removed image 106.

[0052] The two-dimensional face authentication unit 202 performs the two-dimensional face authentication 100 based on the captured image 104. For example, the two-dimensional face authentication unit 202 performs the two-dimensional face authentication 100 based on a deblurred image 106 that is created by performing a deblurring process on the captured image 104.

[0053] The part area recognition unit 204 recognizes part areas representing predetermined parts of the face of the user U based on the captured image 104. For example, the part area recognition unit 204 recognizes a plurality of part areas representing a plurality of predetermined parts of the face of the user U based on the captured image 104. For example, the part area recognition unit 204 recognizes the part areas based on the deblurred image 106 created by performing a deblurring process on the captured image 104.

[0054] Depth estimation unit 206 estimates a depth D within at least a portion of an area of ​​captured image 104 by calculation according to coded aperture 11. For example, depth estimation unit 206 estimates a depth D within a part of a part area recognized by part area recognition unit 204. For example, depth estimation unit 206 estimates a depth D within each of a plurality of part areas recognized by part area recognition unit 204.

[0055] The depth feature information creating unit 208 creates depth feature information indicating the features in the depth direction of the face of the user U based on the depth D estimated by the depth estimation unit 206. For example, the depth feature information creating unit 208 creates, as depth feature information, the relative depths of a plurality of parts based on a predetermined part among the plurality of parts, based on the depth D in each of the plurality of part regions estimated by the depth estimation unit 206. For example, the depth feature information creating unit 208 calculates an average value of the depth D for each of the plurality of parts in each of the plurality of part regions estimated by the depth estimation unit 206, and creates the depth feature information based on the average value.

[0056] The depth face authentication unit 210 authenticates the user U based on the depth feature information created by the depth feature information creation unit 208 .

[0057] [5. Modifications] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.

[0058] For example, the two-dimensional face recognition 100 and the generation of the part area recognition information 112 by recognizing the part area may be performed directly from the captured image 104, not from the deblurred image .

[0059] As described above, according to this modified example, the creation of the two-dimensional face recognition 100 and the part area recognition information 112 by recognizing the part area is performed directly from the captured image 104, which makes it easier to reduce the load of the computational processing on the computer. [Explanation of symbols]

[0060] U user, L light, D depth, B magnitude of blur, 1 face recognition device, 10 monocular camera, 11 coded aperture, 13 lens, 15 imaging element, 15a light receiving surface, 20 control device, 100 two-dimensional face recognition, 102 depth face recognition, 104 captured image, 106 blur-removed image, 108 two-dimensional feature information, 110 two-dimensional user information, 112 body part area recognition information, 114 depth estimation information, 116 average depth information, 118 depth feature information, 120 depth user information, 21 CPU, 22 memory, 23 display, 24 touch panel, 200 blur removal processing unit, 202 two-dimensional face recognition unit, 204 body part area recognition unit, 206 depth estimation unit, 208 depth feature information creation unit, 210 depth face recognition unit.

Claims

1. a monocular camera having a coded aperture and configured to capture an image of a user's face by capturing an image of the user's face through the coded aperture; a depth estimation unit that estimates a depth in at least a part of a region of the captured image by a calculation according to the coded aperture; a depth feature information generating unit that generates depth feature information indicating a feature in a depth direction of the face of the user based on the depth; a depth face authentication unit that authenticates the user based on the depth feature information; Including, Facial recognition device.

2. The face recognition device includes: a part area recognition unit that recognizes a part area representing a predetermined part of the face of the user based on the captured image, The depth estimation unit estimates a depth within the part region. The face recognition device according to claim 1 .

3. the part area recognition unit recognizes a plurality of part areas each representing a predetermined part of the face of the user based on the captured image; the depth estimation unit estimates a depth within each of the plurality of part regions; the depth feature information creation unit creates, as the depth feature information, relative depths of the plurality of parts based on a predetermined part among the plurality of parts as a reference, based on depths in each of the plurality of part regions; The face recognition device according to claim 2 .

4. The face recognition device includes: Further comprising a two-dimensional face authentication unit that performs two-dimensional face authentication based on the captured image. The face recognition device according to claim 3 .

5. The face recognition device includes: a deblurring processor that performs a deblurring process on the captured image by a calculation according to the coded aperture to create a deblurred image, the part area recognition unit recognizes the part area based on the deblurred image. The face recognition device according to claim 4.

6. The two-dimensional face authentication unit performs the two-dimensional face authentication based on the deblurred image. The face recognition device according to claim 5 .

7. the depth feature information creation unit calculates an average value of the depth in each of the plurality of part regions for each of the plurality of part regions, and creates the depth feature information based on the average value. The face recognition device according to any one of claims 3 to 6.

8. a captured image acquisition step of acquiring a captured image showing the user's face by capturing an image of the user's face through the coded aperture by a monocular camera having the coded aperture; a depth estimation step of estimating a depth in at least a part of the captured image by a calculation according to the coded aperture; a depth feature information creating step of creating depth feature information indicating a feature in a depth direction of the face of the user based on the depth; a depth face authentication step of authenticating the user based on the depth feature information; A facial recognition method comprising:

9. an image acquisition means for acquiring an image showing a user's face by capturing an image of the user's face through a coded aperture using a monocular camera having the coded aperture; a depth estimation means for estimating a depth within at least a partial region of the captured image by a calculation according to the coded aperture; a depth feature information generating means for generating depth feature information indicating a feature in a depth direction of the face of the user based on the depth; and a depth face authentication means for authenticating the user based on the depth feature information; A program that makes a computer function as a

Images